It's actually the remote machine that isn't working, but I get the frustration. What I did to fix this is to override the environment variables in your SSH config like this:

Host * SetEnv TERM=xterm-256color

That’s a neat trick! Didn’t know setting env vars was possible in ssh configs

Not all environment variables will be passed through (depends on the server configuration), but stuff like TERM should