Michael Stonebraker | Big Data is (at least) Four Different Problems

I first heard of this 3 V's in a Michael Stonebreaker's talk[1]. For the uninitiated he's a legend in DBMS space, Turing award winner[2]. Highly recommend this and related talks by him, most of them are in YouTube. [1] ▶ this video [2] link

Firstly, disclosure: I work on XTDB, where we are in the process of building out a new columnar SQL engine that also (still) supports edn-flavoured Datalog, see link Perhaps the most fundamental aspect here is the inherent tension between (A) typical columnar relational schemas which primarily target efficient OLAP across very wide & sparse tables, and (B) the nature of 'graph' systems (whether OLAP or OLTP) which tend towards handling a huge number of tables and/or recursive self-joins + incremental view maintenance. I believe Michael Stonebraker's conjecture continues to be that a "one size fits all" database engine is essentially unachievable, and therefore you can't expect a single database system to be competitive across both domains simultaneously. However some databases (SQL Server springs to mind) are apparently pretty good at running multiple engines in parallel. With the v2 XTDB engine design we are sacrificing some of our graph capabilities in the short term (WCOJ and self-join optimisations) for what we feel are more pressing advantages of the columnar model (low-cost retention of history and vectorized processing). We are also aiming to retain reasonable OLTP performance as well, in a similar vein to SingleStore etc., since the target audience is still application developers rather than data analysts. If you are looking for a way of using Datalog on existing columnar storage engines today, you should definitely look at link Also, this Stonebraker talk on "Big

I read the Red Book when I was figuring out data architectures for my company. (the other book was Designing Data-Intensive Archictures by Martin Kleppmann) Stonebraker et al are very opinionated (especially about SQL and relational databases), but in a way I can accept because a lot of his thinking is based on first-principles. Stonebraker also has a string of database successes (Ingres, Postgres, Vertica etc.) which gives him enough street cred to prove he's not just spewing theory from an ivory tower. He's not necessarily correct in every opinion, but his opinions are interesting and worth considering. I enjoyed his talk on "Why Big Data is at least 4 different problems" ▶ this video

Whether that actually matters is arguable: No one has performant distributed (i.e. cross-partition) ACID. (Except maybe Google.) For the thing that does matter, which is ACID within the same partition/entity group, I would argue that RethinkDB already does have that. Mike Stonebraker says that there are no fast distributed transactions [1]. I'm inclined to believe him, and that's not just an appeal to authority. [1] ▶ this video

Whenever I see a young gullible engineer, all twinkly-eyed and bedazzled by the MongoDB hype (or any other NoSQL hype), I'll send him a link [1] on a rather interesting talk by a very thoughtful Turing award winning engineer named Michael Stonebraker. It takes a couple of days for the charm of NoSQL to wane off, but eventually they'll always see how they have been fooled into believing without a sane level of critique. [1] ▶ this video

Yes, that slide deck in isolation looks an ad so it's probably better to get some background on Michael Stonebraker's "big picture" beliefs and core premises: ++ He believes SQL is the "right" and most proven access model. (Note that SQL does not mean RDBMS like Oracle/MSSQL.) ++ He believes in ACID ++ He believes that other db engines' (NoSQL, etc) approaches to gain speed and scaling of clusters by sacrificing SQL and ACID is wrong and his proof is that those other products end up eventually bolting on SQL and ACID anyway. I think a better presentation to flesh out these ideas is the following youtube video: ▶ this video It's 55 minutes long so to save time, HN readers may prefer to browse it by pressing "L" key to step through it quickly. Just read the slides shown behind him and if the bullet points look interesting, rewind and listen to his presentation. One can dwell on those SQL/ACID merits in isolation and consider separately, the question of whether VoltDB-as-a-product fulfills those ideals.

Stonebraker recently gave a talk at my school on where he thinks databases are headed in the future[0]. Definitely worth a watch. [0] ▶ this video