Maybe a dumb question, but when navigating the worlds on the site, I notice that the space is 3D, but the objects (the drum set, or the vending machine) are flat 2D layers that one cannot walk around or examine from a different angle. Is this the inherit limitation of the approach or future features? Thanks.
Could this be used to replace photogrammtry when accuracy is needed? Photogrammetry requires lots of images and can be brittle, and is slow to compute.
One of the biggest differences is the conditioning signal. Genie 3 and similar input raw keyboard commands (WASD + arrow keys), while Atlas inputs camera poses. This small difference means that Genie 3 has no 3D whatsoever; the model needs to learn an internal mapping between keyboard commands, world states, and pixels; and with Genie 3 there is no clear way to control the generated world aside from the input image and text prompt. Since Atlas makes camera pose explicit it can use posed input frames to shape the generated world, giving you a lot more creative control.
Another big differentiator is multimodality. Genie 3 only outputs pixels. Atlas also outputs pixels, but it can also output explicit 3D for the cases where you need it (such as plugging into game engines, simulators, or VFX workflows)
Unfortunately that's a complex question... this depends on the number of diffusion steps, the size of the context, the image resolution, and the type and number of inference devices we use. There are lots of knobs to trade off speed, quality, latency, throughput, and cost.
An ideal workflow would be something semi-interactive that you can use to quickly iterate on an idea, followed by a longer offline bake-out to generate final production-quality assets.
spacial context feature is cool - what are the limitations, if any? What would it take to geo and rotation tag every photo ever taken , combine it into a mass spatial context, run it through atlas and build an entire 3D model of the world?
Atlas is an auto-regressive diffusion model, so context length limitations apply similar to LLMs and video models.
Where Atlas has an edge is that its context comprised of an arbitrary sequence of images with camera poses, which lends itself to managing the context in creative ways (we called this "context juggling" in our RTFM blog, https://www.worldlabs.ai/blog/rtfm). So yes through clever context management you could potentially build an entire 3D model of the world.
Yes, as long as the input images are "poseable" -- if they were taken in the same space they need to have some overlap, where the same object or part of the scene is visible in multiple views so the pose can be predicted.
You can also manually position the input images in 3D space to create scenes generatively; we show examples of this in the "generating with spatial context" section
Maybe a dumb question, but when navigating the worlds on the site, I notice that the space is 3D, but the objects (the drum set, or the vending machine) are flat 2D layers that one cannot walk around or examine from a different angle. Is this the inherit limitation of the approach or future features? Thanks.
Could this be used to replace photogrammtry when accuracy is needed? Photogrammetry requires lots of images and can be brittle, and is slow to compute.
Are y'all using any sort of self-distillation similar to https://self-evo.github.io/ to sharpen representations?
How does it compare to Google's Genie 3 or other similar models? At first glance it does look much more impressive visually
One of the biggest differences is the conditioning signal. Genie 3 and similar input raw keyboard commands (WASD + arrow keys), while Atlas inputs camera poses. This small difference means that Genie 3 has no 3D whatsoever; the model needs to learn an internal mapping between keyboard commands, world states, and pixels; and with Genie 3 there is no clear way to control the generated world aside from the input image and text prompt. Since Atlas makes camera pose explicit it can use posed input frames to shape the generated world, giving you a lot more creative control.
Another big differentiator is multimodality. Genie 3 only outputs pixels. Atlas also outputs pixels, but it can also output explicit 3D for the cases where you need it (such as plugging into game engines, simulators, or VFX workflows)
Thanks, those are great points, now I'm definitely excited for the full release to try it out
How long does it take to process from input images to model output?
Unfortunately that's a complex question... this depends on the number of diffusion steps, the size of the context, the image resolution, and the type and number of inference devices we use. There are lots of knobs to trade off speed, quality, latency, throughput, and cost.
An ideal workflow would be something semi-interactive that you can use to quickly iterate on an idea, followed by a longer offline bake-out to generate final production-quality assets.
spacial context feature is cool - what are the limitations, if any? What would it take to geo and rotation tag every photo ever taken , combine it into a mass spatial context, run it through atlas and build an entire 3D model of the world?
Atlas project lead here.
Atlas is an auto-regressive diffusion model, so context length limitations apply similar to LLMs and video models.
Where Atlas has an edge is that its context comprised of an arbitrary sequence of images with camera poses, which lends itself to managing the context in creative ways (we called this "context juggling" in our RTFM blog, https://www.worldlabs.ai/blog/rtfm). So yes through clever context management you could potentially build an entire 3D model of the world.
Would it be more reasonable to take images from movies and create worlds of various IPs?
My first thought is a detailed Hogwarts that is fully explorable using scenes from the movies (or even descriptions from the books?)
can atlas also generate 3D without pose information attached to the input images?
Yes, as long as the input images are "poseable" -- if they were taken in the same space they need to have some overlap, where the same object or part of the scene is visible in multiple views so the pose can be predicted.
You can also manually position the input images in 3D space to create scenes generatively; we show examples of this in the "generating with spatial context" section
Please give us access :) We are doing super useful things for museums and travelers and this would be a huge unlock!
Also, big fan of Dr.Li!