Who does Goldman Sachs buy their futures from?
I take your point that all trades have two sides, and if you define speculation to be the total volume or total net position of trades then yeah, the net speculation of industry and non-industry players must by definition be equal, but I don't find this to be particularly informative. I mean, if one side has a set price/position volume curve, and the other side suddenly comes in and buys and buys and buys, then, yeah, net open interest is the same, but it's pretty clear that it's the side buying as the price rises that's doing the speculation as people understand the word.
It's only when it causes changes in physical holdings that speculation will move the spot price. There are some physical holdings by traders, usually heavily concentrated at Cushing or held offshore in oil tankers contracted to sit around full. They're fairly small, but have large fractional changes.
When I was talking about speculation by industry being small, I was mostly thinking of speculation in the physical market, relative to the side of the market. Except for the products with high seasonality, for the most part the oil industry just doesn't think about speculation as being the industry it's in.
Total profits from holding a volume V at a price P will be the integral of V dP = PV - integral P dV. Over any closed path any constant PV (or PVnormal) has no effect on profits, so the instantaneous profit can be thought of equivalently as V dP or -P dV or (V-Vormal) dP. So long as you keep changes in the volume you hold small, you aren't exposed to losses due to volatility, even if you must hold large volumes, and much of the oil industry runs on this basis.
Where there are large oil industry stock changes, they're mostly either imposed by factors outside of their control, or are hedged against futures markets so that they can view departure from their normal amount of stocks as something which gives a virtually guaranteed profit. This is where futures prices (including industry hedging activity) change the spot price. Of course the oil companies are still exposed to changes in the price basis of what they actually pay or receive relative to reference oil basket, but these are usually comparatively small.
This is a very different price setting mechanism from how I imagine people think oil prices are set. I think people imagine some guy in XOM looking at the markets and saying, "OK, what price can we get away with? Let's set it to that." That's pretty much how prices for cars and many other items are set, but they can only do that for cars because cars are not exactly the same, and because they're willing to shut in car production or increase their car stocks for their particular model until they get the price they asked for.
There's very little shutting in of wells, except for OPEC, and even they're down to UAE, Kuwait and Saudi Arabia being the only ones that ever shut in wells, and in not terribly large volumes any more either.
Mainly, though the price volatility results from the fact that there's so little storage variation that if we produce 1.3% more oil than we consume as we did in the 1Q 2012, that's a huge imbalance. If there were people with a large volume of physical storage who could actually predict the price accurately, we wouldn't have the price volatility we do.
It's only when it causes changes in physical holdings that speculation will move the spot price
What is the mechanism for that effect?
Mainly, though the price volatility results from the fact that there's so little storage variation that if we produce 1.3% more oil than we consume as we did in the 1Q 2012, that's a huge imbalance
Exactly. Demagogues, however, would have us believe that speculation overwhelms normal supply and demand forces. (And monetary policy, too.)