This will be a rambling post just to help me get some thoughts down.
With the recent announcement that Chevron is essentially pulling out of all biofuels and renewables, I think it's worth noting some of the unrealistic expectations that Chevron placed on clean technology. Most notably, Chevron placed clean tech processes in direct competition for investment money with its oil exploration projects. Thanks to some of my work in graduate school from some of the best industrial practitioner educators I've ever had, I know that the IRR for a typical conventional oil exploration project is on the order of 20%.
The corresponding number for a bio-renewables project is much lower, on the order of 3-7% for the very best. This is much more typical of what you would see in agriculture. Placing investment money for bio-renewables projects in competition with oil projects, it's easy to see why bio-renewables lost out.
Besides making me profoundly disappointed in Chevron, this kind of news, to me, points to where the future of energy will go. If you accept that the world cannot indefinitely continue to power itself with petroleum-based transportation fuels, then eventually the IRRs of bio-renewable projects (or other types of renewable energy) must approach those of oil. This can happen in one of three different ways.
On the side of petroleum, either the number of opportunities for highly profitable ventures will decrease (a supply-driven slump) or there will not be enough demand to sustain these ventures at a high level of profit (a demand-driven slump).
On the other hand, a second option is for bio-renewables to decrease their costs, increase their IRRs and compete with petroleum on its own terms.
I've staked out my position on demand destruction before, and I still believe that the structural changes that might cause fuels demand to decrease in the same way the IEA and others claim will take decades longer than they predict. That environment means that high prices will continue, and supplier costs will increase as profitable opportunities for exploiting petroleum become depleted - in short, a petroleum supply driven argument. Something like a carbon tax would also help that along, by seriously affecting the economics of bitumen (somewhat less for other forms of unconventional oil, such as tight oil).
As for bio-renewables, after seven years of active work in the area, two of them professionally, realistically I can't see the IRR of any project improving much beyond that 3 to 7% level. Making a low-value product like a fuel just doesn't pay, not when there is the opportunity to avoid the cost of gathering your energy source (i.e. the sun, collected on land) by going with petroleum.
In the end, there will be convergence at some point, when a bio-rewables project and slurping up dead dinosaurs become equally attractive. However, there are too many physical limits to bio-renewable feedstocks to make them more competitive. Instead, the point at which convergence will occur will be when energy as we've known it for the past three quarters of a century is no longer a high yielding investment.