Great to see people thinking about this. But it feels like a step on the road to something simpler.
For example, web accessibility has potential as a starting point for making actions automatable, with the advantage that the automatable things are visible to humans, so are less likely to drift / break over time.
Any work happening in that space?
There is a proposed extension in the repo that is getting some traction that automatically converts forms into tools. There is trouble in linking this to a11y though, since that could lead to incentivize sites to make really bad decisions for human consumers of those surfaces.
In theory you could use a protocol like this, one where the tools are specified in the page, to build a human readable but structured dashboard of functionality.
I'm not sure if this is really all that much better than, say, a swagger API. The js interface has the double edge of access to your cookies and such.
As someone heavily involved in a11y testing and improvement, the status quo, for better or worse, is to do it the other way around. Most people use automated, LLM based tooling with Playwright to improve accessibility.
I certainly do - it’s wonderful that making your site accessible is a single prompt away!
We're building an app that automatically generates machine/human readable JSON by parsing semantic HTML tags and then by using a reverse proxy we serve those instead of HTML to agents
Chris Shank & Orion Reed doing some very nice work with accessibility trees. https://bsky.app/profile/chrisshank.com/post/3m3q23xpzkc2u
I tried to play along at home some, play with rust accesskit crate. But man I just could not get Orcas or other basic tools to run, could not get a starting point. Highly discouraging. I thought for sure my browser would expose accessibility trees I could just look at & tweak! But I don't even know if that's true or not yet! Very sad personal experience with this.