When first encountering these ecosystems and looking at the various pieces they contain, one may repeatedly ask: "why didn't they just use <off-the-shelf solution> for this problem instead of writing this component/subsystem from scratch"?
The answer is often that the battle-hardened mature off-the-shelf solution did not exist at the time the code was written. You're doing software archaeology.
In my experience it’s extremely difficult for a highly resourced corporate engineering team to get married to an open source project run by volunteers, consensus, or both. It is possible but you need to have a first class relationship with an upstream who will take your patches.
Every patch delay puts more pressure on you and your team to fork the codebase and go it alone. You and your team sit down and promise you’ll rebase over upstream releases and everyone nods wisely. Then you skip a release, and another, and presto: you now you have Bank Redis or Bank Selenium or Bank Hadoop trapped on the last version of upstream before the fork but to which you can patch changes as fast as you like. I’d liken this to crossing an event horizon except the astronaut sees the universe freeze and fade away instead of the outside observer.
It’s possible to make it work if the upstream project either gives you a majority vote (or at least a substantial share of the vote) on project direction, or you’re working on a project large enough to have lots of corporate (ie funded, high velocity) stakeholders already.
Yes, agreed. And it's not just delays - when the upstream decides that they actively don't want to take a patch that you have a burning need for (and maybe you already have systems in production depending on it), that can accelerate the process to Bank Redis a lot.
I've rolled with the "we'll keep a local patch against upstream" for small changes before, which helps keep on track with upgrades, but depends how feasible that is.
That's only half the answer. These large investment banks' value-add is partly that they can integrate everything they know into these closed-world environments (kind of like a Smalltalk image), which is something that simply isn't done in the wider world because you can't accrete it out of smaller pieces and it doesn't make sense at all for smaller entities.
[flagged]
Very much so, I worked on Quartz at BAML for a few years.
The whole idea was actually to use as much existing Open Source technology as possible. Hence Python and it's rich library ecosystem, instead of something home grown like SecDB/Slang. This was supplemented with proprietary infrastructure and libraries only where there was a clear need. For example a Directed Acyclic Graph library to ease migrations from the Excel sheets used by Quants. The distributed object store was pretty neat.
You could code up a basic web service with minimal functionality and have it running in nonprod in an afternoon, and then production the day after. All that boilerplate stuff was super low friction, so you could spend much more of your time on solving the actual problem.