The databases we use today in production have severe limitations and are not even close to what is theoretically possible. Many traditional parts of a database (indexing, caching, scheduling, et al) are AI-complete algorithm problems. Entire sub-classes of database (e.g. graph or spatial) famously have persistently poor scalability and performance because of open questions in the foundational computer science.
Just the fact that increasing the generality, scalability, and performance of databases asymptotically converges on designing AGI suggests that it is, in fact, "deep tech". And this property has to mesh with other practical constraints on database behavior. Many problems in databases are hard with little forward progress in decades.
It is true that most database research is not deep tech but there is ample room for it to be if one is sufficiently ambitious.
I personally built quite several specialized hyper-performant DB engines, and studied most popular OSS projects, and believe most DB questions are theoretically answered long ago(decade back). The puzzle is mostly to assemble pieces together to fit specific tradeoff of performance/simplicity/functionality and not overengineer system.
Could you explain what practical research there is to be done? The heavy theory I know does not seem to be very useful in practice. Optimal join algorithms, Yannakakis adjacent algorithms, tree decomposition of queries all seem to be worse than well implemented naive algorithms. But maybe the implementations of the new algorithms just are not good? I really don’t know.
There are several interesting foundational problems around scalability, generality, efficiency, and adaptability. Historically use cases haven't always been there to justify focusing on these problems but that is changing. In my opinion, some of the more important open problems:
Ideally a table should be index-organized across all relevant columns. No public system works anything like this. We don't have single indexing structures that work for a collection of arbitrary types each with possibly unpredictable distributions, never mind ones that mix temporal, geometric, and other difficult types. The AI-complete nature of indexing becomes evident when you dig into this. Downstream from this is an implication of extremely granular and adaptive storage management that current storage engines aren't designed for.
Tractable cache replacement algorithms are broken for many workloads and data models. These algorithms need to be very fast for search, update, and eviction selection but they are also AI-complete; improvements to generality have impractically high computational cost. Storage growth is decoupled from RAM availability thanks to disaggregation, aggravating the problem even for workloads that worked well under tractable cache replacement. In theory we know that cache admission (read: fancy latency-hiding schedules) is more robust and scales better but is so difficult to implement in non-trivial real systems that I don't think anyone has figured out how to reduce that concept to practice yet.
At exabyte scales, conventional database internals have embedded assumptions that no longer hold true. For example, you cannot guarantee even "small" internal control structures are resident in memory. What used to be fairly boring internals bits in databases suddenly have to be redesigned from first principles. This is more applied than theoretical but it suggests a major change in the way we do internal architecture.
Traditionally we've treated spatial and temporal locality as architecturally separate concerns. This is extremely convenient from a building real systems standpoint. Optimizing either one in isolation is adversarial to the efficiency of the other, which becomes increasingly visible as you scale up. Converging these concerns into a single "thing" almost certainly has solving the above problems as a prerequisite. If you squint, you can kind of see this as the last step before databases become literal AI.
All of these have really broad scope. If we could solve even half of these open research problems the resulting database engines would be unrecognizable. There are ton of other narrower interesting research problems around data layouts, compression, join parallelism, etc that still have potential for substantial improvement.
It is a great time to be doing database research, we've barely scratched the surface.
To be honest none of these problems feels remotely "solvable" beyond having a ton of knobs and engineering tradeoffs like we already have now.
In a sense they don't seem very exciting because all of this required generality.
how do you optimally balance code generation and virtual function calling patterns for the best performance? Is there a standardized OLTP format on object storage? How do you performantly implement differential privacy policies within the db engine?
Are there re-usable query primitives for extremely large scale multi-modal data? how do you scale such queries or make them efficient?
The list goes on.
There is research as of last year which brings unoptimised impls of those join algorithms to “on-par” with optimized non-optimal industry impls.
There’s still research to be done in optimizing these algorithms in both theory and practice.