How is this at my expense? It's at the expense of the hedge fund they bought the oil futures from before the price went up. I don't see why I should assume that some hedge fund manager is more on my side than some insider trader.

I think insider trading is robing the market, which affects everyone.