If you mix a small SSD and a large HDD into a single bcachefs pool, and set it to 2 replicas, do writes have to succeed on both devices before returning success? I.e. is performance constrained by the slowest device? And what happens when the small SSD fills up? Does it carry on writing to the HDD but with only one replica, or does it report no space left?
You explicitly configure the cache, as laid out here: https://bcachefs.org/Caching/
In the common case that you mentioned, data present on the full SSD would be overwritten "in standard LRU fashion"; meaning the "Least Recently Used" data would no longer be cached. New data would be written to the SSD while a background "rebalance thread" would copy that data to the HDD. I assume that the "sync" command would wait for the "rebalance thread" to finish, though I will admit my own ignorance on that front.
Are you trying to ask if bcachefs lets you set up stupid configurations? If you only have two devices and you configure it use replication to store two copies of all your data, then you are unavoidably constrained by the capacity of the smaller device. Whether the smaller device is a hard drive or SSD is irrelevant, because neither copy can be regarded as a discardable cache when they're both necessary to maintain the requested level of redundancy.
I am trying to understand bcachefs by asking about an edge case that might be illustrative.
> you are unavoidably constrained by the capacity of the smaller device
Sure, so what does bcachefs actually do about it? ENOSPC?
Answers to the question on synchronous write behavior also welcome.
> I am trying to understand bcachefs by asking about an edge case that might be illustrative.
It looks more like you're questioning whether the filesystem that Linus just merged does obviously wrong things for the simplest test cases of its headline features. Has something given you cause to suspect that this filesystem is so deeply and thoroughly flawed? Because this doesn't quite look like trying to learn, it looks like trying to find an excuse to dismiss bcachefs before even reading any of the docs. Asking if everything about the filesystem is a lie seems really odd.
> Asking if everything about the filesystem is a lie seems really odd.
This person was asking pertinent questions, no need to bash it.
I lost some data with XFS after it was declared "stable" in the linux kernel. Every day at around 00:00 the power will fail for around 2 seconds. Other filesystems (ext2, jfs, reiser) will do a fsck, but xfs was smarter. After 2 or 3 crashes the xfs volume will not be usable anymore (no fsck possible).
So yes, some of us do need more than a "trust me, it's ok" when we are talking about our data.