Can't you be ordered to pay the legal fees of the person you sued if you lose badly enough?

Yes, but this only really happens if your lawsuit is bad faith or frivolous. If they have a valid reason for the suit but just lose, this won't happen.