You could even generate a bloom filter for every block, and when you have a full block of bloom filters, write them out to an index file.
You could even generate a bloom filter for every block, and when you have a full block of bloom filters, write them out to an index file.
My personal idle walking-with-dog kind of design was a linear binary record file with regular marks letting you resynchronize where you are (and stamp cryptographically) with minimal seeks, and separate indexing files with bloom filters and the like. If the indexes are corrupted or deleted, they can be reconstructed from the main log, main log is single-writer/multiple-readers with no locking in any form necessary, and easier to survive kernel/hw failure