> I also occasionally rsync large directories to/from cloud storage and that can also saturate
For convenience, the rclone tool is nice for most cloud storage like google and stuff that make rsync annoying[0]
rsync also offers compression[1], and you might want to balance it depending if you want to be CPU bound or IO bound. You can pick the compression and level, with more options than just the `-z` flag. You can also increase speed by not doing the checksum, or by running without checksum and then running again later with. Or some intervaling like daily backups without and monthly you do checksums.
If you tar your files up first I have a function that is essentially `tar cf - "${@:2}" | xz -9 --threads $NTHREADS --verbose > "${1}"` which uses the maximum `xz` compression level. I like to heavily compress things upstream because it also makes downloads faster and decompression is much easier than compression. I usually prefer being compute bound.
Also, a systemd job is always nice and offers more flexibility than cron. It's what's helped me most with the wack-a-mole game. I like to do on calendar events (e.g. Daily, Weekly) and add a random delay. It's also nice that if the event was missed because the machine was off it'll run the job once the machine is back on (I usually make it wait at least 15 minutes after machine comes online).
Thank you! I do love rclone a great deal, and need to start using it more. I mostly use rsync right now because I've got some Contabo storage instances and ssh is already set up, so it's nice and easy, and I've already got the rsync commands burned into memory from decades of use :-) I also typically rsync with the same command to an external hard drive for an on-site backup.
Great tips! I'll definitely be using your tar command
Oh, that reminds me. When I had an Android (I already regret my iPhone) I used termux to write a very basic script to rsync data to machines and do so based on if connected to WiFi (with whitelisted SSIDs). Pretty easy to write and then schedule a cron job. Can be nice to put that onto other peoples devices and give them automated backups. Makes it FAR easier to do the 321 backup strategy. Also pretty easy to build a tracking app for a lost phone that way (like if haven't connected to home WiFi in x days send you an email). Both work really well when also using tailscale.