I have Copilot buttons sprinkled everywhere on my work computer, and every time I have tried to use them I get something saying "Oh, I can't do that". It's truly baffling.
Copilot button on my email inbox? I try "Find me emails about suchandsuch", and get the response "I don’t have direct access to your email account. If you’re using Outlook (desktop, web, or mobile), here are quick ways to find all emails related to...". Great, so it doesn't even know what program it's runnning in, let alone having any ability to do stuff in there! Sigh.
Using the paid M365 Copilot ($30/mo) Chat and Researcher agent, I recently discovered an interesting limit: Copilot is technically unable to retrieve more than 24 email messages. Ever.
We can't know if the answers I got from it are reliable but it seems like the Microsoft Graph API calls it makes and the tools Copilot has are missing the option to call the next page. So, a paginated response is missing all data beyond the first page.
I vibe coded this page as "documentation" since obviously no official MS docs exist for anything like this: https://vibes.jukkan.com/copilot-search-gotchas.html
I tried copilot agent once, and it just claimed that it accessed a website that should have been blocked by corporate firewalls and uploaded a bunch of proprietary data. Lots of very specific information about how it clicked on specific buttons of the website etc.
We raised a high priority ticket with MS and turns out that Copilot Agent lied about the entire thing because the website was blocked. It completely made it up.
The fact that we are supposed to use Copilot Agent for open-ended "research" is mind-boggling.
Copilot uses the Bing search index to access public content. Your corporate firewall is irrelevant.
How did it know about the buttons? Or were they so generic that it could hallucinate them as well?
I wonder if the site you mentioned was earlier harvested through some firewall hole during Copilot's training.
A whole new toolbar appeared in Outlook on my work computer with nothing but a single button to open a copilot chat window. I tried asking it a few simple questions and it completely failed at all of them. Copilot didn't even know if I was using the web or desktop version of the very app it was embedded in!
Wasting UI space for a useless tool it's just a waste of time, it actively makes it harder to get work done. But I guess the important thing is the number of times that AI button gets clicked is going up on some PMs telemetry dashboard.
I'm baffled by this as well, Microsoft seems to have lost the plot almost completely.