Here is thought, this is a fixed 3d environment and you lack training data or at least an algorithm to train. Why not use RL to learn good trajectories?
Like build a 3d environment of your home/room and generate images and trajectories in a game engine to generate image data to pretrain/train it, then for each run hand label only promising trajectories i.e. where the robot actually did better cleaning. That might make it a good RL exercise. You could also place some physical flags in the room that when the camera gets close enough it gets rewarded to automate these trajectory rewards.
If mass produced, no part of a robot vacuum is expensive. Blower fans are ~$1. Camera is $1. Cheap wifi MCU with a little ML accelerator + 8 Mbytes of ram is $1. Gyro is $1. Drive motors+gearboxes together are $1. AC charger $2. Plastic case $2. Batteries are the most expensive bit (~$3), but you can afford to have a battery life of just 10 mins if you can return to base frequently.
The hard part is the engineering hours to make it all work well. But you can get repaid those as long as you can sell 100 Million units to every nation in the world.
I don't really see how the vacuum can effectively clean a whole room or flat using only a CNN of the current image in front of the robot. This would help detect obstacles, but a bumper sensor would do that as well.
All but the most basic vacuum robots map their work area and devise plans how to clean them systematically. The others just bump into obstacles, rotate a random amount and continue forward.
Don't get me wrong, I love this project and the idea to build it yourself. I just feel like that (huge) part is missing in the article?
Not saying that it’s viable here to build a world map since things like furniture can move but some systems, e.g. warehouse robots do use things like lights to triangulate on the assumption that the lights on the tall ceiling are fixed and consistent.
The classic Roombas from a decade or so ago worked without any sort of mapping or camera at all -- they basically did a version of the "run and tumble" algorithm used by many bacteria -- go in one direction until you can't any more then go off in a random new one. It may not be efficient but it does work for covering territory.
I think the only reason for mapping is to be able to block off 'no go' areas (no escaping out the front door!) and to be able to go home to the charger.
Cool project! That validation loss curve screams train set memorization without generalization ability.
Too little train data, and/or data of insufficient quality. Maybe let the robot run autonomously with an (expensive) VLM operating it to bootstrap a larger train dataset without needing to annotate it yourself.
Or maybe the problem itself is poorly specified, or intractable with your chosen network architecture. But if you see that a vision llm can pilot the bot, at least you know you have a fighting chance.
Check out using maybe some kind of monocular depth estimation models, like Apple's Depth Pro (https://github.com/apple/ml-depth-pro) and use the depth map to predict a path?
Here is thought, this is a fixed 3d environment and you lack training data or at least an algorithm to train. Why not use RL to learn good trajectories? Like build a 3d environment of your home/room and generate images and trajectories in a game engine to generate image data to pretrain/train it, then for each run hand label only promising trajectories i.e. where the robot actually did better cleaning. That might make it a good RL exercise. You could also place some physical flags in the room that when the camera gets close enough it gets rewarded to automate these trajectory rewards.
I would begin in one room to practice this.
If mass produced, no part of a robot vacuum is expensive. Blower fans are ~$1. Camera is $1. Cheap wifi MCU with a little ML accelerator + 8 Mbytes of ram is $1. Gyro is $1. Drive motors+gearboxes together are $1. AC charger $2. Plastic case $2. Batteries are the most expensive bit (~$3), but you can afford to have a battery life of just 10 mins if you can return to base frequently.
The hard part is the engineering hours to make it all work well. But you can get repaid those as long as you can sell 100 Million units to every nation in the world.
I don't really see how the vacuum can effectively clean a whole room or flat using only a CNN of the current image in front of the robot. This would help detect obstacles, but a bumper sensor would do that as well.
All but the most basic vacuum robots map their work area and devise plans how to clean them systematically. The others just bump into obstacles, rotate a random amount and continue forward.
Don't get me wrong, I love this project and the idea to build it yourself. I just feel like that (huge) part is missing in the article?
https://opencv.org/structure-from-motion-in-opencv/
Not saying that it’s viable here to build a world map since things like furniture can move but some systems, e.g. warehouse robots do use things like lights to triangulate on the assumption that the lights on the tall ceiling are fixed and consistent.
The classic Roombas from a decade or so ago worked without any sort of mapping or camera at all -- they basically did a version of the "run and tumble" algorithm used by many bacteria -- go in one direction until you can't any more then go off in a random new one. It may not be efficient but it does work for covering territory.
I think the only reason for mapping is to be able to block off 'no go' areas (no escaping out the front door!) and to be able to go home to the charger.
For the actual cleaning, random works great.
You are right. The original Roomba was discussed on HN 3 months ago:
https://news.ycombinator.com/item?id=46472930
Surely mapping also helps reducing the time it takes to achieve the task?
Cool project! That validation loss curve screams train set memorization without generalization ability.
Too little train data, and/or data of insufficient quality. Maybe let the robot run autonomously with an (expensive) VLM operating it to bootstrap a larger train dataset without needing to annotate it yourself.
Or maybe the problem itself is poorly specified, or intractable with your chosen network architecture. But if you see that a vision llm can pilot the bot, at least you know you have a fighting chance.
I don’t understand why we don’t have smarter vaccuums yet. Mine just makes a beeline to get stuck under a chair.
It could easily understand so much about the environment with even a small multimodal model.
We do, deebot T20 maps the whole house, knows what type of floors there are, maps furniture etc
My Roborock uses lidar and rarely if ever bumps into things.
get the wyze robot vacuum, it's pretty smart.
Check out using maybe some kind of monocular depth estimation models, like Apple's Depth Pro (https://github.com/apple/ml-depth-pro) and use the depth map to predict a path?
Very cool project though!
The trick is to make a robot that has a Lidar and a camera, then train a model that can replace the Lidar.
(Lidar can of course also be echolocation).
The harder trick is to do it cost effectively. I picked up my Roborock for $200 and it has LiDAR. Works really well.
I thought the trick is just to use an xbox kinect. But lidar got a lot cheaper in the recent years.
There’s things like SLAM, optical flow etc, read up on things instead of being so defeatist IMO even for a hobby project, seems so forced