Ofc. I'm not claiming credit for Apache Arrow. Lots of people contributed to it.
I didn't write the graph algorithms in networkit. Somebody else did.
But if you're looking for 100-200 graph algorithms running efficiently on columnar memory, I haven't found a working implementation that's permissively open source.
Happy to work with cugraph or anyone else who builds on top of Parquet/Arrow ecosystem.
cudf/cugraph is open source and helped fund the arrow ecosystem , I think it's good to give teams credit when they do work they didn't have to do and not misrepresent them. And yes, it works great, graph500 leaderboard level.
I'm happy to credit these people.
https://arrow.apache.org/docs/r/authors.html
CUDA and the ecosystem around it is more complicated.
I'll use it. I agree that it advanced the state of the art and helped fund some of the truly OSS projects.
Don't feel the need to bring it up on a HN comment. Neither does the parent article by Sem.
In fact, it's in the vendor's interest to transparently route the algorithms in Icebug to the GPU in a compatible way instead of writing their own APIs.
cudf+cugraph are the table+graph algorithms that started around that same time to align with it, that's what helped get arrow funded from the nvidia side
can't reply on the below, but re:networkx, it was more of the reverse, they did a nice job of building a standalone embeddable table+graph arrow-friendly library over many years, and with networkx compatibility from the beginning. Later, they collaborated with the networkx team to eventually upstream it so networkx users can benefit more easily
as it is embeddable, you don't need to go through networkx to use it -- eg, we use it directly, as do various databases
and I'm not sure why you're saying people do not use networkx->cugraph because of performance, again, this is graph500 level performance that even graph database vendors now support because it is literally magnitudes faster than their cpu alternatives. We have had projects like court cases where the data science team switched to sitting on top of pygraphistry/gfql -> cudf/cugraph and using the GPU versions were the difference between hours and minutes, which for iterating over interactive analysis, is important. It's night and day going from CPU -> GPU, and a lot of butts were saved because of this.
We see the same thing with databricks+neo4j migrations.. most tools have their sweet spots, but also their comparative weak spots.
The reason for not recommending networkx is the choice it offers:
I'm looking to offer something that's best in class on both CPU and GPU, so people don't have to choose.Now that nvidia is becoming a major CPU vendor, they may like the idea too.
They use networkx today: https://github.com/rapidsai/nx-cugraph/ because it's more popular than networkit or icebug. But its popularity is based on ease of use, not performance:
https://github.com/timlrx/graph-benchmarks
Not clear if the author or anyone else has an updated version of these benchmarks. Opened an icebug issue on the repo.
see graph500
we use igraph cpu / cugraph gpu in pygraphistry/gfql projects, and our experience is cugraph is ~10X+ on tiny cheap GPUs over igraph. looking at that table, where all seem same magnitude as igraph, I'd therefore expect much better than all the options listed. And when things get bigger and it merits a bigger GPU, even better. The team has been doing a great job over the last decade, and all free.