The funny thing is that looking backwards, I would never use a grid of squares for a raycaster like wolfenstein3d did.
If I were to do a raycaster today, I would use convex sectors with portals, basically like duke nukem, but constant wall heights. You can do drawing very simply by just doing a linear pass across the sector, recursively stepping into other sectors.
Then you can at least do arbitrary level geometries.
Yeah, that is basically what i did around 2009 when i was making a 3D engine in Haxe for Flash, RayFaster 2 (it was the successor to another Flash game engine, RayFaster, which did Wolf3D-like raycasting and i used to make a simple FPS[0] though with Flash's input limitations i couldn't do mouselook and didn't feel that great).
The engine used Build/Doom-like maps and would simply raycast against the camera's current sector (sectors could be any shape not just convex) and used connections between sectors as portals to recursively follow the (2D) map until it hit a solid wall - so basically Build's portal rendering approach except using raycasting instead of trapezoid rasterization. It could also do sprite rendering (you could have both sprites facing the camera and "aligned" sprites that could be used for decals) and even had some simple 3D model rendering (the triangles were sorted and clipped against the portal during rasterization). Unfortunately Flash isn't available in browsers anymore and Ruffle isn't compatible with it (it can run the engine -much slower than real Flash- but the palette is all wrong), so i took some shots using the standalone Flash "projector"[1][2][3][4]. Also making maps for it was certainly much more laborsome than making maps for a Wolf3D-like game and that combined with me losing interest in making Flash games meant i never made a game with it.
[0] https://www.youtube.com/watch?v=Z81NhEbl3q8
[1] http://runtimeterror.com/pages/iv/images/01feb493af6dd3184b8...
[2] http://runtimeterror.com/pages/iv/images/136a8753b211ed7e3d1...
[3] http://runtimeterror.com/pages/iv/images/84c2012258982b82053...
[4] http://runtimeterror.com/pages/iv/images/e94cc334c7735e1aacb...
A grid of squares makes sense for the target hardware at the time though (286 or better CPU)
It really doesn't. I would argue that convex sectors with portals could be quicker than casting rays across a grid for every pixel column, except for some degenerate cases. This is because for any pixel column in-between the boundaries of wall segments, there's virtually no work to be done (O(1)), compared to the work of casting a ray along the grid O(n).
Btw, 286 played wolfenstein rather poorly. The 386 is rather more appropriate.
Wolfenstein was built in a couple of months by a then 21-year-old Carmack. He didn't focus on optimizing levels until Doom.