At least in this case (not so sure about the Prompt API case mentioned in another thread) the agent is "in" the page. And that means that the agent is constrained by the same CORS limits that constrain the behavior of the page's own JS.

If you think about it, everything we've done to make malicious webpages unable to fiddle around with your state on other sites using XHRs, are exactly and already the proper set of constraints we'd want to prevent models working with webpages from doing the same thing.