Fun. At a very high level this is similar to what Gaussian splats do. You nudge Gaussians, that start out as random blobs in 3D space, in a direction that minimizes a cost between their projections into known camera poses and input images.