Isn’t the glaringly obvious problem Garbage Collection?

I’d try digging into this first. Try to completely disable VC and see if there are any slowdowns before you run out of memory. Go from there.