Just have to ask. Will I be spending way more money since my context window is getting so much bigger?

Yes, full context is used to generate each new token.