Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.
I figured it was lack of MCP support. I would just include a note why it was excluded. Have you checked out FX? It seems to be like Pi but will more things out of the box like MCP.
There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark? And who should be the tie breaker when someone thinks they should use a different plugin? Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
OhMyPi is essentially Claude like bloated and Pi is so barebones that many will just get frustrated trying to add bricks after bricks to make it usable for complex workflows - so I don't really see any point of it being in such benchmarks. Pi should have added option to easily add "packs" (and then also tweak tehm) sort of things. When I tried it I actually thought that's what OMP will be, but it was not. It was a different harness altogether in a way and not a nice way.
I wish we could use just the frontend of omp harness-agnostically, like the provider switching, stats dashboard, subscription pooling etc. are excellent features, but it doesn't seem inconceivable that we can have all that with a swappable "actual" harness. I've tried to make omp more pi-like by lazy loading most of the tools rather than dumping them into system prompt. haven't benchmarked this yet.
Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.
hi Albert from the Strands team. We can look into a Pi run. Our researcher is working on a deep dive for our benchmarks, so we got resources to test against other harnesses
Base Pi doesn’t seem like an apt comparison here. Strands comes with MCP servers, subagents, and web fetch tools built in. To get those on Pi, you have to add addons, and then you get into the space of “which addons should be added to make it an apples-to-apples comparison? Which MCP addon do I use, the fastest one, or the most popular one?”. My guess is they chose OMP because it’s the defacto standard for answering those “which addons should be considered out of the box” questions for Pi.
I figured it was lack of MCP support. I would just include a note why it was excluded. Have you checked out FX? It seems to be like Pi but will more things out of the box like MCP.
If "pi install npm:pi-mcp-adapter" is preventing them from including pi in benchmarks, then they might not be competent to run trustworthy benchmarks.
There are four different Pi plugins with the name “pi-mcp-adapter”, not including forks. Which one should they use for the fairest benchmark? And who should be the tie breaker when someone thinks they should use a different plugin? Now answer the same questions for all of the other plugins necessary for Pi to reach parity with Strands.
It’s like you didn’t even read my comment.
Deepseek harness is not actually built on Pi harness. It's an independent project.
This is true, but the team said on Twitter that they were heavily inspired by Pi and this whole team used it extensively:
https://x.com/tianyi/status/2088306143772946499
OhMyPi is essentially Claude like bloated and Pi is so barebones that many will just get frustrated trying to add bricks after bricks to make it usable for complex workflows - so I don't really see any point of it being in such benchmarks. Pi should have added option to easily add "packs" (and then also tweak tehm) sort of things. When I tried it I actually thought that's what OMP will be, but it was not. It was a different harness altogether in a way and not a nice way.
I wish we could use just the frontend of omp harness-agnostically, like the provider switching, stats dashboard, subscription pooling etc. are excellent features, but it doesn't seem inconceivable that we can have all that with a swappable "actual" harness. I've tried to make omp more pi-like by lazy loading most of the tools rather than dumping them into system prompt. haven't benchmarked this yet.
Do you have any write up on that - how you went about it? Also, in that case why not just make Pi slightly OMP like? Really curious.
[dead]
Deepseek was cheaper, but also less accurate. "Beats" isn't a fair assessment.
That’s fair, but Strands is advertising their harness needing way less tokens which does relate to cost.
In that same vein, Pi is less bloated then Deepseek and Oh My Pi, which are built on top of Pi. Isn’t it dubious to leave it out?
Came here to say this. They have oh-my-pi in the benchmark but not pi, but those are very different animals. pi is lightweight out of the box so has very little start-time overhead. (And will not spin up agents like crazy.) pi might do worse if those things are actually important for solving the problem, but it certainly has a shot at being most efficient.
hi Albert from the Strands team. We can look into a Pi run. Our researcher is working on a deep dive for our benchmarks, so we got resources to test against other harnesses