The default is FULL
https://sqlite.org/compile.html#default_synchronous
>SQLITE_DEFAULT_SYNCHRONOUS=<0-3> This macro determines the default value of the PRAGMA synchronous setting. If not overridden at compile-time, the default setting is 2 (FULL).
>SQLITE_DEFAULT_WAL_SYNCHRONOUS=<0-3> This macro determines the default value of the PRAGMA synchronous setting for database files that open in WAL mode. If not overridden at compile-time, this value is the same as SQLITE_DEFAULT_SYNCHRONOUS.
Many wrappers for sqlite take this advice and change the default, but the default is FULL.
From this (linked https://sqlite.org/pragma.html#pragma_synchronous), does anybody understand EXTRA?
> EXTRA provides additional durability if the commit is followed closely by a power loss.
means?
How can one have "additional" durability, if FULL already "ensures that an operating system crash or power failure will not corrupt the database"?
Is it that FULL only protects against "corruption" as stated, but will still lose committed transactions?
It seems so from the points on https://stackoverflow.com/questions/58113560/during-power-lo...
Which is also quite nasty. I want my databases to be fully durable by default, and not lose anything once they have acknowledged a transaction. The typical example for ACID DBs are bank transactions; imagine a bank accidentally undoing a transaction upon server crash, after already having acknowledged it to a third-party over the network.
This is what the documentation (https://sqlite.org/pragma.html#pragma_synchronous) says about EXTRA:
> EXTRA synchronous is like FULL with the addition that the directory containing a rollback journal is synced after that journal is unlinked to commit a transaction in DELETE mode
So it only has an effect in DELETE mode; WAL mode doesn't use a rollback journal.
That said, the documentation about this is pretty confusing.
Yes, I'm talking about the fact that sqlite in its default (journal_mode = DELETE) is not durable.
Which in my opinion is worse than whatever may apply to WAL mode, because WAL is something a user needs to explicitly enable.
If it is true as stated, then I also don't find it very confusing, but would definitely appreciate if it were more explicit, replacing "will not corrupt the database" by "will not corrupt the database (but may still lose committed transactions on power loss)", and I certainly find that a very bad default.
> sqlite in its default (journal_mode = DELETE) is not durable.
Not true. In its default configuration, SQLite is durable.
If you switch to WAL mode, the default behavior is that transactions are durable across application crashes (or SIGKILL or similar) but are not necessarily durable across OS crashes or power failures. Transactions are atomic across OS crashes and power failures. But if you commit a transaction in WAL mode and take a power loss shortly thereafter, the transaction might be rolled back after power is restored.
This behavior is what most applications want. You'll never get a corrupt database, even on a power loss or similar. You might lose a transaction that happened within the past second or so. So if you cat trips over the power cord a few milliseconds after you set a bookmark in Chrome, that bookmark might not be there after you reboot. Most people don't care. Most people would rather have the extra day-to-day performance and reduced SSD wear. But if you have some application where preserving the last moment of work is vital, then SQLite provides that option, at run-time, or at compile-time.
When WAL mode was originally introduced, it was guaranteed durable by default, just like DELETE mode. But people complained that they would rather have increased performance and didn't really care if a recent transaction rolled back after a power-loss, just as long as the database didn't go corrupt. So we changed the default. I'm sorry if that choice offends you. You can easily restore the original behavior at compile-time if you prefer.
>If you switch to WAL mode, the default behavior is that transactions are durable across application crashes (or SIGKILL or similar) but are not necessarily durable across OS crashes or power failures. Transactions are atomic across OS crashes and power failures. But if you commit a transaction in WAL mode and take a power loss shortly thereafter, the transaction might be rolled back after power is restored.
How is this behavior reconciled with the documentation cited in my comment above? Are the docs just out of date?
For DELETE (rollback) mode, and given the way fsync works, FULL should not lose committed transactions (plural) but it might lose at most one committed transaction.
Because DELETE depends on deleting a file (not modifying file contents), it much depends on the specific file system's (not SQLite's) journaling behavior.
I don't see how the journal delete depends any more on a specific file system's behaviour than the main data write: If a specific file system can decide to automatically fsync unlink(), it can equally decide to automatic fsync write().
In either case it is clear that (on Linux), if you want guarantees, you need to fsync (the file and the dir respectively).
> That said, the documentation about this is pretty confusing.
I now filed a suggestion to clarify the docs on this:
https://sqlite.org/forum/forumpost/ec171a77a3
>EXTRA synchronous is like FULL with the addition that the directory containing a rollback journal is synced after that journal is unlinked to commit a transaction in DELETE mode. EXTRA provides additional durability if the commit is followed closely by a power loss.
It depends on your filesystem whether this is necessary. In any case I'm pretty sure it's not relevant for WAL mode.
> The typical example for ACID DBs are bank transactions; imagine a bank accidentally undoing a transaction upon server crash
That's why they don't try to do it that way! But it's still an informative way to think about it.
Also, while we're discussing defaults, your ACID db is probably running at READ COMMITTED by default, meaning that your bank transactions are vanishing/creating money:
* You read accounts A and B($30) in order to move $5 between them. The new balance for B should be $35. Just before you write the $35, someone else's transaction sets B to $100. Your transaction will proceed and blindly set it to $35 anyway.
But to your overall point, I'm also frustrated that these systems aren't as safe as they look on the box.
> your ACID db is probably running at READ COMMITTED by default
You're probably refering to PostgreSQL. Yes, I am also frustrated that that doesn't default to SERIALIZABLE.
I do wish the top two open-source "ACID" DBs (Postgres and SQLite) used guaranteed-safe, zero-surprise defaults.
It isn’t worth it. Mostly financial transactions are done via append only ledgers, not updating; two phase auth and then capture; a settlement process to actually move money, and a reconciliation process to check all the accounts and totals. Even without DB corruptions they have enough problems (fraud and buggy code) with creating money and having to go back and ask people to more money or to give them more money so they have those systems in place any ways.
> Mostly financial transactions are done via append only ledgers, not updating;
Well, financial institutions will act as you describe, I presume, but lowly web shops will update 'shopping cart' and 'inventory' using the default settings of whatever DBMS the system came with.
Which is reasonable I guess, except that even with modern hardware, updating the same record in an ACID database has a surprising low capacity in terms of txn / second; if you have some popular item in your inventory (or popular merchant in your 2-sided shopping thing), you'll be forced to create multiple receiving accounts or other ugly stuff if you do the balance based transactionality.
yes. most folks don't seem to understand this. but, you can get something approaching such guarantees if you are able to limit yourself to something as (seemingly) simple as updating a ledger. this approach is used in a lot of places where high performance and strong consistency is needed (see e.g. LMAX disruptor for similar). https://tigerbeetle.com/
SERIALIZABLE isn't zero-surprise, since applications must be prepared to retry transactions under that isolation level. There is no silver bullet here.
A CRUD architecture with proper ACID is an OK contender against other possible architectures. Personally I always go for event-sourcing (a.k.a. WAL per the article's title).
But a CRUD that doesn't do ACID properly is crap. And since the people making these decisions don't understand event-sourcing or that they're not being protected by ACID, CRUD gets chosen every time.
The DB also won't be set to SERIALIZABLE because it's too slow.
No corruption doesn't imply no data loss. Reverting to an earlier, consistent, state is in some situations acceptable (think Unix fsck), in others one might depend on committed transactions to have indeed been recorded as such.
I'd think SQLite isn't typically used for bank applications, but rather to keep your web browser's bookmarks and such.
Even worse because browser bookmarks are not exactly a high volume use case where an extra fsync matters at all.
This makes me so sick. For years and years I’ve gotten the vibe from SQLite that it never took being a reliable database seriously, but I bought into the hype for the past several years that it was finally a great DB for using in production, and then this. I swear. Sure, change the default config for now and make it actually behave in a sane way so that it doesn’t lose your data. But later- use a real database.
hey, I just tested and `NORMAL` is default:
edit: fresh installation from homebrew shows default as FULL: I will update the post, thanks!Just checked debian/ubuntu/alpine/fedora/arch docker images. All are FULL by default.
Is this sqlite built from source or a distro sqlite? It's possible the defaults differ with build settings.
The one which avinassh shows is MacOS's SQLite under /usr/bin/sqlite3. In general it also has some other weird settings, like not having concat() method, last I checked.
The Apple built macOS SQLite is something.
Another oddity: misteriously reserving 12 bytes per page for whatever reason, making databases created with it forever incompatible with the checksum VFS.
Other: having 3 different layers of fsync to avoid actually doing any F_FULLFSYNC ever, even when you ask it for a fullfsync (read up on F_BARRIERFSYNC).
> it also has some other weird settings
You also can't load extensions with `.load` (presumably security but a pain in the arse.)
Same with macports here - 2 (opt/local/bin/sqlite3)
and /usr/bin/sqlite3 is 1