Launching the rocket from a balloon is not a problem. You can have a platform with a hole underneath the rocket's exhaust and the balloon itself can be a torus through which the rocket flies upwards. There would be minimal impact of a rocket launch on the platform if designed properly.
Heck, you could just drop the rocket and have it ignite (one hopes) with lateral thrust, clearing the launch aerostat.
Or consider the aerostat disposable (at a significant replacement cost, see: <https://news.ycombinator.com/item?id=45333674>).
But either way it's a bit less the firey-burney-explodey problem than the sudden loss of a few thousands tonnes of mass that would displace the equilibrium of the launch platform, should you care to re-use that, or the means by which such platforms (capable of supporting said thousands of tonnes of mass).
Just to put some hard numbers on it, a crewed Falcon9 (Crew Dragon) has a launch mass north of a half-million kilograms, or 500 tonnes.
The Russian Soyuz-FG, also human capable, has a launch mass of slightly over 300 tonnes.
If this spacecraft is only a shuttle to low-Venus orbit with another transfer craft for the flight back to Earth (or other points of interest) that should suffice. If the launch craft is intended for the full return trip of 100--250 days, things could get a bit cozy and interesting depending on the number, disposition, and fragrance of inhabitants.