Scaling the Linear Sync Engine

> Previous sync engines really want you to sync all data Linear had to do all sorts of shenanigans to be able to sync all data, for orgs with lots of it – there's a talk on that here: ▶ this video

(this should probably be a post) CRDTs and local first are ideas that is perpetually in the hype cycle for the last decade or so, starting around the time Riak CRDTs became a thing and continuing all the way to today. Niki's post is a perfect illustration: CRDTs offer this "magical" experience that seems perfectly good until you try building a product with them. Then it becomes a nightmare of tradeoffs: - state-based or operation-based? do you trim state? how? - is it truly a CRDT (no conflicts possible), a structure with explicit conflict detection, or last-writer-wins/arbitrary choice in a trench coat that will always lose data? Case in point: Automerge uses pseudo-random conflict resolution, so Niki's solution will drop data if it's added without a direct causal link between the edits. To learn this, you have to go to "under the hood" section in Automerge docs and read about merge rules very attentively. It might be acceptable for a particular use case, but very few people would even read that far! - what is the worst case complexity? Case in point: yjs offers an interface that looks very much like JSON, but "arrays" are actually linked lists underneath, which makes it easy to accidentally become quadratic. - how do you surface conflicts and/or lost data to the user in the interface? What are the user expectations? - how do you add/remove replication nodes? What if they're offline at the time of removal? What if they come online after getting removed? - what's user experie

You may not need CRDT per-se, but building a collaborative experience is so difficult. I worked on collaborative systems for a bit, and also have read a bit about how Figma and Notion do it (this is a good read: link ) -- it's still super hard to get right. This talk by Karri about Linear's "sync engine" is also a good watch: ▶ this video .

This article has been quite the inspiration for many projects and progress on this front. I think we're seeing more and more developments around CRDTs and local-first frameworks / applications. I'm working on a few projects in this area: - link - Notion meets Notebook-style live programming for TypeScript / React - link - a rich text editor built on TipTap / Prosemirror that supports Yjs for local-first collaboration - link - a wrapper around Yjs for easier development In my experience so far, some things get more complicated when building a local-first application, and some things get a lot easier. What gets easier is that once you've modeled and implemented the data-layer (which does require you to rethink / unlearn a few principles), you don't need to worry about data-fetching, errors etc. as much as in a regular "API-based" app. Another interesting video I recommend on this topic is about Linear's "Sync Engine" which employs some of the local-first techniques as well: ▶ this video

Heads-up: This is a video ( ▶ this video ) embedded in a blog post; there doesn't seem to be an accompanying text-only version.