It matters a lot whether the organization is growing. If you get assigned to a toxic manager in a static organization then you're likely to be stuck there indefinitely. In a growing organization there will be opportunities to move up and out to other internal teams.