Works for Linear because the tab stays open, and worse case if tab is closed you can recover later when the tab is opened again and deal with conflict resolution. Won't work if:
1. user clicks a button and closes the tab thinking transaction is done and it's important that transaction is done
2. conflict resolution is difficult or impossible in future client wake up
The user clicks the button, the mutation is stored in local storage. The user closes the tab, but it's not a problem.
A background worker picks up the mutation and sends it to the remote backend. It takes time, retries, etc.
Similarly, any errors reported by the background worker go to local store, and the next time the UI tab is loaded / activated, they are shown. A service worker can show a notification outright to let the user easily load the main UI. Normally this would be a rare occasion.
1. What if the browser gets closed/killed?
2. Error messages around syncing issues are notoriously worse than those of a sync request to the backend that failed. So the UX in the end is worse.
More generally: You can't circument the trade-offs of a distributed database, which such products are, conceptually.
Expose the sync queue to end user and train them to understand if they attempt to close the tab with a pending queue they will get the ugly prompt warning them
Yes and for linear (if you break it down strictly in a theoretical CS sense, is roughly equivalent to TodoMVC in terms of application complexity assuming non-collaborative text editing) they have clearly defined states for most items and few if none truly destructive actions. The hardest part is anything text related.
I agree. To each their own, but the UI updating automatically doesn't really add much value to me. I would prefer that the view I am seeing is a snapshot in time of what the ground truth server was, not some mixed state that forces me to consider the possibility that seeing my request go through on the screen doesn't actually mean it went through and has been sent to the server.
Well that "make a mutation client-side" phrase is doing a lot of work.
Make a mutation to what?
The classic server rendered web-app doesn't have any data to make a mutation to. You could try to patch the UI but that would be a huge pita and not really a scalable (in effort) solution.
If you have an SPA, you still don't really have data on the client-side. You have a bunch of cached query responses. You can update those, but (a) it will be a pita to do correctly, (b) you'll have to do it to every possibly affected query, and (c) you have to remember to undo it at the right time (way more subtle than it appears - think it through!).
A sync engine creates the client-side normalized datastore that allows you to "do a mutation client side". In fact, you're kind of right that once you have a sync engine, just doing a mutation is really easy. The real challenge is all the infra required to enable you to do so.
Indeed. I have to say, I hate this. Suppose you are in a meeting, you update something and you see the result, but the rest of the team does not. Ok, a couple of hundred ms does not play into this but if the update does not make it through? And yes, it happens.
Changes go through and synced to everyone on your team in almost realtime. If there's a conflict on the server and your change cannot be applied (almost never happens), your change is rolled back on your client, again, almost in realtime. If servers cannot be reached, we will show you a syncing badge within 4 seconds to tell you that you have made changes that haven't been sent to others yet.
Strange that we can be so be polar opposites on this. You hate it, I would never write an app in any other way, ever again.
(curious) What if a user closes it before 4 seconds? Ctrl+enter, it optimistically locally updates within 1 second. I close ctrl+w. But my wifi goofed and it didn't reach the server.
I have mysteriously lost comments/descriptions I wrote on issues. I figured it was related to a failed and lost opportunistic update like this, although I suppose it could have been caused by a fixable bug.
The HTTP request is fired off instantly, so chances are that the request is already written to the socket and closing the page won't cancel the request. Should your wifi-router drop it, your client will retain the transaction on disk and retry it the next time you come online.
Yeah that's the issue isn't it?
I see in the UI it's sent. But actually it's sent only the next morning.
To be fair. It's fine for an issue tracker. Anything actually important i'd spend a few seconds going over what I just sent. In which case I'd see it's not synced. And what's not that important it's really fine if in some random wifi edge case it's phantom sent. So makes sense.
That's really gross behaviour; users like it because they don't understand it and don't know to blame it for their issues when weird things happen to them, and weird things to them all the time.
‹giant argument breaks out before people realize a bunch of messages went missing and were posted out of order› “Oh, it's just ‹app› being weird again. I really hate that.”
As a user, I like when things appear to sync instantly and perfectly, such as in Google Docs.
As a developer, I hated the article and many of the comments I read thus far because:
- Having clients and a server properly sync and not lose data in the event of a network failure amounts to having a consistent distributed system which is not easy to do, and the commenters don't seem to have understood that
- I hate having written a long document and then losing it because the sync code is buggy, so the previous point becomes even more important.
So reading many of the things here has been mildly infuriating.
That being said, none of these people are likely affiliated with Linear, and given the overall quality of the product I'm pretty sure it works properly.
In the case of a partition the client nodes get temporarily out of sync but the system will then synchronise to one state again once the partition is resolved if it’s written correctly.
So no violation of CAP theorem it just prioritises liveness over consistency
There's guaranteed persistence, but there's no guarantee that the host will be up anytime soon. E.g.: I might leave a final reply with all the details on an issue before going on vacation (or maybe I don't work the next day but my colleagues abroad do!). I see that it's properly posted and close the laptop.
The reply with be delayed by days or weeks, but the UI indicated that it had been properly saved.
Edge case but playing devil’s advocate: a user can also uninstall the native app at any time, and might still expect their last change before they closed the app to be reflected in the web version.
You can never truly trust anything about a client because by definition you don’t control it
Corner case: actually, it can. Also, thats how auto-updating works ; depending on local state as a source of truth using browser apis is a terrible idea IMO.
The whole concept of "assume it is committed while we sync in the background" is, in the most cases, a terrible architectural decision, unless it is coupled with explicit feedback (eg. A small visual indicator indicating if the background queue is empty or syncing). Also, it breaks temporality: last-update-wins no longer holds, because update time and sync time are decoupled. And you also create a new problem, which is local cache coherence.
It may be a good fit for some systems (though I cannot think of a single one), but in general is just a horrible solution.
Though its depressing how few actually use it to its full extent. My team is one of the few where I work; heavy declarative mutation directives with optimisticResponse (and optimisticUpdaters because some of our APIs are not very Relay-compatible, annoyingly)
Relay does optimistic updates well. However, frustratingly, Relay does not do any persistent caching to disk, like Linear does. This means, first page load will always have to fetch data from the server.
I think that’s an acceptable trade off, personally… but! You can implement persistent caching if you’d like :) you create the record store when you stand up the environment, so it’s possible to have that hydrated with data from somewhere else
But I’ve only done that in toy examples not prod, I’m sure there’s something I’m missing haha
No, because if a transaction fails it still needs to be handled by the user in many cases.
E.g. if you buy a book, but it turns out the book was already sold, then you will first get a message "Your book is on its way!" and then "Oops, sorry, the book was already sold to someone else".
Eventual consistency is just a property of the database.
Works for Linear because the tab stays open, and worse case if tab is closed you can recover later when the tab is opened again and deal with conflict resolution. Won't work if:
1. user clicks a button and closes the tab thinking transaction is done and it's important that transaction is done
2. conflict resolution is difficult or impossible in future client wake up
The user clicks the button, the mutation is stored in local storage. The user closes the tab, but it's not a problem.
A background worker picks up the mutation and sends it to the remote backend. It takes time, retries, etc.
Similarly, any errors reported by the background worker go to local store, and the next time the UI tab is loaded / activated, they are shown. A service worker can show a notification outright to let the user easily load the main UI. Normally this would be a rare occasion.
1. What if the browser gets closed/killed? 2. Error messages around syncing issues are notoriously worse than those of a sync request to the backend that failed. So the UX in the end is worse.
More generally: You can't circument the trade-offs of a distributed database, which such products are, conceptually.
Yeah this pattern can be made to work fine.
Main downside is it significantly complicates the front end code compared to just waiting for FE to sync with BE before updating
What if we had a local server running on the same PC, which then relays the request to some shared server on the internet?
That's what a background worker is: a local server managed by the browser and only accessible to pages of the origin domain.
Expose the sync queue to end user and train them to understand if they attempt to close the tab with a pending queue they will get the ugly prompt warning them
Still works if you use beacon requests, they survive tab close
Bear in mind that beacon is something of a hail Mary, though. No way to tell if it was successful or not or react if it wasn't.
Yes and for linear (if you break it down strictly in a theoretical CS sense, is roughly equivalent to TodoMVC in terms of application complexity assuming non-collaborative text editing) they have clearly defined states for most items and few if none truly destructive actions. The hardest part is anything text related.
I agree. To each their own, but the UI updating automatically doesn't really add much value to me. I would prefer that the view I am seeing is a snapshot in time of what the ground truth server was, not some mixed state that forces me to consider the possibility that seeing my request go through on the screen doesn't actually mean it went through and has been sent to the server.
The goal is - and I think they have achieved it - is that you don't have to think about it. They handle sync, and they do it reliably.
Then they have solved one of the fundamental hard things in computer science.
Well that "make a mutation client-side" phrase is doing a lot of work.
Make a mutation to what?
The classic server rendered web-app doesn't have any data to make a mutation to. You could try to patch the UI but that would be a huge pita and not really a scalable (in effort) solution.
If you have an SPA, you still don't really have data on the client-side. You have a bunch of cached query responses. You can update those, but (a) it will be a pita to do correctly, (b) you'll have to do it to every possibly affected query, and (c) you have to remember to undo it at the right time (way more subtle than it appears - think it through!).
A sync engine creates the client-side normalized datastore that allows you to "do a mutation client side". In fact, you're kind of right that once you have a sync engine, just doing a mutation is really easy. The real challenge is all the infra required to enable you to do so.
I think whole generations are constantly discovering that the client is really, really fast.
so.. optimistic ui? like upvotes on hacker news since forever?
Indeed. I have to say, I hate this. Suppose you are in a meeting, you update something and you see the result, but the rest of the team does not. Ok, a couple of hundred ms does not play into this but if the update does not make it through? And yes, it happens.
Changes go through and synced to everyone on your team in almost realtime. If there's a conflict on the server and your change cannot be applied (almost never happens), your change is rolled back on your client, again, almost in realtime. If servers cannot be reached, we will show you a syncing badge within 4 seconds to tell you that you have made changes that haven't been sent to others yet.
Strange that we can be so be polar opposites on this. You hate it, I would never write an app in any other way, ever again.
> If there's a conflict on the server and your change cannot be applied, your change is rolled back on your client
Do you lose your data then? Or are you thrown back into the form after it closed?
(curious) What if a user closes it before 4 seconds? Ctrl+enter, it optimistically locally updates within 1 second. I close ctrl+w. But my wifi goofed and it didn't reach the server.
I have mysteriously lost comments/descriptions I wrote on issues. I figured it was related to a failed and lost opportunistic update like this, although I suppose it could have been caused by a fixable bug.
The HTTP request is fired off instantly, so chances are that the request is already written to the socket and closing the page won't cancel the request. Should your wifi-router drop it, your client will retain the transaction on disk and retry it the next time you come online.
> next time you come online
Yeah that's the issue isn't it? I see in the UI it's sent. But actually it's sent only the next morning.
To be fair. It's fine for an issue tracker. Anything actually important i'd spend a few seconds going over what I just sent. In which case I'd see it's not synced. And what's not that important it's really fine if in some random wifi edge case it's phantom sent. So makes sense.
That's really gross behaviour; users like it because they don't understand it and don't know to blame it for their issues when weird things happen to them, and weird things to them all the time.
‹giant argument breaks out before people realize a bunch of messages went missing and were posted out of order› “Oh, it's just ‹app› being weird again. I really hate that.”
As a user, I like when things appear to sync instantly and perfectly, such as in Google Docs.
As a developer, I hated the article and many of the comments I read thus far because:
- Having clients and a server properly sync and not lose data in the event of a network failure amounts to having a consistent distributed system which is not easy to do, and the commenters don't seem to have understood that
- I hate having written a long document and then losing it because the sync code is buggy, so the previous point becomes even more important.
So reading many of the things here has been mildly infuriating.
That being said, none of these people are likely affiliated with Linear, and given the overall quality of the product I'm pretty sure it works properly.
In the case of a partition the client nodes get temporarily out of sync but the system will then synchronise to one state again once the partition is resolved if it’s written correctly.
So no violation of CAP theorem it just prioritises liveness over consistency
Which is to say: in case of partition, it loses data.
For native apps this is less of an issue since they have access to persistent storage but with browsers there's no guaranteed persistence.
There's guaranteed persistence, but there's no guarantee that the host will be up anytime soon. E.g.: I might leave a final reply with all the details on an issue before going on vacation (or maybe I don't work the next day but my colleagues abroad do!). I see that it's properly posted and close the laptop.
The reply with be delayed by days or weeks, but the UI indicated that it had been properly saved.
> There's guaranteed persistence
There's not. Browsers can delete "persistent" storage at any time.
https://developer.mozilla.org/en-US/docs/Web/API/Storage_API...
It depends. From the link:
> If, for any reason, developers need persistent storage [...] they can do so by using the navigator.storage.persist() method of the Storage API.
This makes a request for guaranteed permanent storage ... which can be approved (or denied) by the user or by browser defaults.
Edge case but playing devil’s advocate: a user can also uninstall the native app at any time, and might still expect their last change before they closed the app to be reflected in the web version.
You can never truly trust anything about a client because by definition you don’t control it
But the os can't uninstall the native app at any time unprompted right
Corner case: actually, it can. Also, thats how auto-updating works ; depending on local state as a source of truth using browser apis is a terrible idea IMO.
The whole concept of "assume it is committed while we sync in the background" is, in the most cases, a terrible architectural decision, unless it is coupled with explicit feedback (eg. A small visual indicator indicating if the background queue is empty or syncing). Also, it breaks temporality: last-update-wins no longer holds, because update time and sync time are decoupled. And you also create a new problem, which is local cache coherence.
It may be a good fit for some systems (though I cannot think of a single one), but in general is just a horrible solution.
No, it’s definitely a lot less likely and probably an edge case you can ignore in practice
AKA what Relay does out of the box haha
Though its depressing how few actually use it to its full extent. My team is one of the few where I work; heavy declarative mutation directives with optimisticResponse (and optimisticUpdaters because some of our APIs are not very Relay-compatible, annoyingly)
> AKA what Relay does out of the box haha...
Relay does optimistic updates well. However, frustratingly, Relay does not do any persistent caching to disk, like Linear does. This means, first page load will always have to fetch data from the server.
I think that’s an acceptable trade off, personally… but! You can implement persistent caching if you’d like :) you create the record store when you stand up the environment, so it’s possible to have that hydrated with data from somewhere else
But I’ve only done that in toy examples not prod, I’m sure there’s something I’m missing haha
I believe this is called “eventual consistency”.
No, because if a transaction fails it still needs to be handled by the user in many cases.
E.g. if you buy a book, but it turns out the book was already sold, then you will first get a message "Your book is on its way!" and then "Oops, sorry, the book was already sold to someone else".
Eventual consistency is just a property of the database.