> sort LEGO bricks by colour and size

I just looked into this out of idle curiosity, after watching some guy build a LEGO sorting machine. (They work in a warehouse that sells used bricks for model builders.)

Interestingly, this is on the cusp of viability, but training the ML model would still be cost-prohibitive (for me). With $17M, it's within reach, but there's still the obvious mechanical hurdles: Kids don't disassemble their Lego, the conditions are "less than ideal", and even vibrating belts in a warehouse scenario have a lot of trouble keeping bricks separated for the camera to get a clear image.

Robot hands are nowhere near the point where they can reliably (or even unreliably!) take apart two arbitrary Lego bricks that are joined, let alone anything of even mild complexity. This is hard for most humans, and often requires the use of tools! See: https://www.lego.com/en-us/service/help-topics/article/lego-...

The machine vision part is... getting there! You could pull some clever tricks with modern hardware such as bright LED lights, multi-spectral or even hyper-spectral sensors, etc. The algorithms have improved a lot also. Early attempts could only recognise a few dozen distinct shapes, and the most recent models a few hundred, but they're about 2-3 years old, which means "stone ages".

A trick several Lego recognition model training runs used was to photo realistically render 3D models of bricks in random orientations and every possible color, which is far faster than manually labelling photos of real bricks.

These days you could use the NVIDIA Omniverse libraries to heavily accelerate and automate this.