A lot of the work was done to a design point of 0.25 seconds, and Dyson's book says the issue there wasn't the blast clearing, but just being able to move the machinery fast enough. I kind of share your puzzlement; I can see the blast clearing this quickly in space (the debris moves at tens of thousands of km/sec) but not in the atmosphere.
The blast clears that quickly in the atmosphere because the shockwave and debris move at-or-faster-than the speed of sound, so a few hundred meters is tenths of a second or less. You sit inside the mushroom cloud, of course, but the important part is gone quickly.
The Rapatronic camera was used to take these kinds of pictures, and you can see that the actual blast front is around 20 meters across after 1 millisecond (!!) https://en.wikipedia.org/wiki/Rapatronic_camera