The analogy isn't letting go of a balloon, it's of dumping a large mass of payload (rocket + fuel) from an aerostat quickly.

The aerostat will rise. It will float higher in the atmosphere, with decreased pressure around it. It will expand. It will then rise still further.

And there's no ready supply of solid or liquid ballast (as would be available on a near-ground cargo drop) to compensate for the lost mass.

This is untenable for any manned / habitable module, and you'd all but certainly want any of same well outside the danger zone of a rocket malfunction.

One likely consequence is that any launch aerostats would be at best highly unstable in their altitude and station-keeping characteristics. It's quite possible that a disposable, single-use design might be required. Given that materials would likely have to be shipped from Earth, or possibly from near-Venus asteroids via space-mining, this considerably increases cost and complexity of any such missions.

Aerostats, as lighter-than-air craft, have vastly more-tightly constrained mass budgets than any water-based floating structures. Ignoring and/or waving that away is obtuse in the extreme. Particularly given the additional concerns and considerations of launch-capable structures. Existing aerostats and rockets operate at the outer limits of engineering design capabilities, and still go boom with some regularity, often due to exceeded structural tolerances.