Molmo released recently and is able to provide point coordinates for objects in images. I’ve been testing it out recently and am currently building an automation tool that allows users to more easily control a computer. Looks like Anthropic built a better one.

Edit: it seems like these new features will eliminate a lot of automated testing tools we have today.

Code for molmo coordinate tests https://github.com/logankeenan/molmo-server