Wouldn't that allow the router to MITM all encrypted data that goes through it?

If it were a CA cert yes. It could instead be a self-signed server (non-CA) cert, that couldn't be used for requests to anything else.