I think S3 is a good match for storing database pages as long as they are immutable. The Vectorized query processing model seems to fit this approach very well (e.g., http://oai.cwi.nl/oai/asset/14075/14075B.pdf) Anyone out there from Snowflake care to answer how Snowflake stores database pages in AWS?
I haven't used BookKeeper or Pulsar myself so I can't comment on how well they might work for distributing an intent log.
I'm not really sure how Noms versions stuff. It seems they use a variant of a B-tree index. Or maybe only for secondary indexes?
I think if you can use a simple monotonically increasing sequence number SirixDB has indexing advantages as for instance when storing XML and JSON documents or graph data.
The cool thing also is that SirixDB not only copies changed database pages but it implements a sliding window algorithm for versioning the database pages itself along with well known backup versioning strategies. Furthermore user-specified, typed secondary index structures are also naturally versioned.
One downside is that SirixDB doesn't support branching, even though it would be relatively easily possible to implement I guess, but I'm not convinced that it's needed. I don't want that anyone has to merge merge conflicts. I think automatic algorithms to do this are also not the right thing. But of course it's really interesting and I also thought about it :-) maybe someone has a really good use case? :-)
BTW: Everything in SirixDB is immutable regarding updates of resources in databases. Of course you can revert to an old revision and change stuff, but the revisions in-between will still be accessible.
I think S3 is a good match for storing database pages as long as they are immutable. The Vectorized query processing model seems to fit this approach very well (e.g., http://oai.cwi.nl/oai/asset/14075/14075B.pdf) Anyone out there from Snowflake care to answer how Snowflake stores database pages in AWS?
I haven't used BookKeeper or Pulsar myself so I can't comment on how well they might work for distributing an intent log.