> But my server could be shut down because of a power outage or another reason. I might be at work or even on holidays when it happens, and even wireguard can’t solve this.
A 'power outage' incident doesn't seem to have been mitigated. My homelab has had evolving mitigations: I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime, which got replaced by a dedicated inverter/charger/transfer-switch attached to a big-ass AGM caravan battery (which on a couple of occasions powered through two-to-three hour power outages), and has now been replaced with these recent LiFePo4 battery power station thingies.
Of course, it's only a homelab, there's nothing critically important that I'm hosting, but that's not the dang point, I want to beat most of "the things", and I don't like having to check that everything has rebooted properly after a minor power fluctuation (I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet).
For anyone thinking of doing this, please please don't. A car battery is probably never a sealed deep cycle battery, and the UPS's charging circuitry is not intended to charge a battery of this size (this is assuming you're using a lead based battery, and not something even more crazy and dangerous like Li-Po or LiFePO4). God forbid you have a cell fail on a car battery and that charger starts cooking the battery. I've had actual car lead acid batteries explode because of poor choices someone else made trying to do something like this, and man when they go, they're dangerous and scary. You really need to pick hardware that's all properly specced and sized for the job...there's a reason Eaton and APC charge what they do.
The better approach (if you have EE skills) is to build your own UPS with LiFePO4 batteries, a proper BMS, and a bunch of USB-C PD ports. Skip the lossy inverter entirely and pick equipment that runs on USB-C PD directly.
I don't know why nobody sells these as COTS yet.
Or buy equipment targeting alirack compatibility, i.e., with 240V PSUs that are minimal effort designed to also specifically work on the appropriate DC voltage so that Alibaba, the alirack standard's originator, could delete the inverter from their UPS and feed PV MPPTs directly into the batteries.
I agree entirely, and wouldn't do it again.
To each their own. I'd personally sleep far more soundly with even a car battery UPS under my bed than with one of those consumer ready lithium ion portable power station batteries they sell on Amazon.
But if you can't explain the difference between voltage and current, or know what "short circuit" means, then this isn't something to poke at.
> I have a few things that mount remote file stores and these mounts usually fail upon boot due to the speed at which certain devices boot up - and I've decided not to solve that yet
If your OS is using systemd, you can fix that pretty easily by adding an After=network-online.target (so the ExecStart doesn't even try to check if there is no networking yet) and an ExecCondition shell script [1] to actually check if nfs / smb on the target host is alive as an override to the fs mounts.
Add a bunch of BindsTo overrides to the mounts and the services that need the data, and you have yourself a way to stop the services automatically when the filesystem goes away.
I've long been in the systemd hater camp, but honestly, not having to wrangle with once-a-minute cronjobs to check for issues is actually worth it.
[1] https://forum.manjaro.org/t/for-those-who-use-systemd-servic...
Here's a deeper article on ordering things around network startup: https://systemd.io/NETWORK_ONLINE/
It doesn't conflict with anything you've said, just a very handy document.
You can also use _netdev in the mount options, then systemd mount generator will generate the dependency on network automatically.
You can also use RequiresMountsFor to mark a mount (directory) as a hard dependency for the unit.
Even then, that doesn't resolve the power outage primarily because the local nodes that your Modem transmits to will also be down in the area from same said power outage, most only have backup power for 10-30m, if even that many say there is no backup power for those which is why they disclaim that in their service agreement with regards to emergency phone calling over voip (for services that include unified communications).
So even if your local node could transmit, none of the others could, and they can't buffer either.
To mitigate power outage, you would need both power, and a cellular connection, and that connection would only be good for 2-3 hours (Cell tower backups), and those would require a Cradlepoint.
Author here, indeed I didn't install a UPS. I've tried to keep my setup fairly minimal, and I'm consciously accepting that if there's a power outage my services will be down. I self-host exclusively for myself, not for others.
What I don't want though is a power outage putting my server offline while I'm on holidays, and not be able to access my services at all.
My ISP-provided router supports Wireguard, so I can use that to connect to my KVM and send the Wake on LAN packages.
Out of curiosity, did you look through the BIOS options on your computer? Mine is much less capable than yours (it is a used mini-pc) but it has options to boot itself up upon resuming power.
I use UPS for my internet and then remote access such as intel AMT will get you back into your systems if you've specced your hardware to have such features.
You REALLY should not expose AMT to the internet.
of course not, but you would vpn in, your router (or server behind your router but that may be more risky being able to get in if something fails) should run a vpn if you plan to use something like AMT.
I just use a small UPS to make sure all data is written to the drives properly before the battery runs out.
Do you have power outages often? Even if I have one, my services can come up automatically without doing anything, when the power is restored.
Same. I have two small UPS. The first I connect to my computers and it lasts about 15 minutes. Enough for me to save/checkpoint whatever in a restorable way. The second I connect to the wifi router. This lasts for a while longer and it's pretty useful. In those 15 minutes local network still works. And if the power comes back in a minute or two (which happens more often than outages that last hours where I am), then I don't have to wait for the wifi routers painfully slow boot time.
Speaking of... does anyone know how to speed up wifi router boot time? Stupid thing takes 5 minutes almost.
> Speaking of... does anyone know how to speed up wifi router boot time? Stupid thing takes 5 minutes almost.
This is probably due to the access point having minimal hardware for the task, and it's startup not being particularly well optimised, so "buy a better AP/router" use likely the most practical answer.
As an alternative, you could buy a small device (perhaps a recent rPi model) with more umph (or add this task to an existing machine in your home lab setup), give it a wireless NIC if it doesn't already have one, and run hostapd to turn it into an AP. That might startup a lot faster.
> Speaking of... does anyone know how to speed up wifi router boot time? Stupid thing takes 5 minutes almost.
Maybe try using OpenWRT if your router hardware is supported
I would be happy with smaller/lighter UPS that will only provide 10 seconds of juice. I only purchased the thing because the lights in my apartment would flicker a couple of times a week.
I’ve spend a decent amount on EcoFlow units (Delta 2) that I use as online UPS for my servers and networking. They work great and I also recently installed dual 220watt EcoFlow solar panels on my roof that pump in solar during the day. Works nicely though the ROI admittedly is not there at all, just a cool thing.
> just a cool thing
I think that says it all. It's gone beyond practicality for me, and I'm OK with that. I'm also satisfied with the current setup; I don't need to spend more.
I have a couple of Ecoflow's and Bluetti, and a Segway LFP battery. They all work fine so far.
> I cut a hole in the side of a small UPS so I could connect it to a larger (car) battery for longer uptime
Can you share more about this? I have a APC Back UPS PRO USV 1500VA (BR1500G-GR) and it would be nice to know if this is possible with that one as well.
That UPS eventually died, and I'm not sure if it was because it was hooked up to a larger battery than it was designed for, but it's still only 12 volts so I don't think the electronics would notice. What they may notice is extended run-time in the event of a power failure.
It was a crude mod. Take the cover off and remove the existing little security alarm battery, use tin snips to cut a hole in the side of the metal UPS cover (this was challenging, it was relatively thick metal, I'd recommend using an angle grinder in an appropriately safe environment far away from the internals of the UPS), and feed the battery cables out through the hole. I probably got some additional cables with appropriately sized terminations to effectively extend the short existing ones (since they were only designed to be used within the device). And then connect it up to a car battery.
Cover any exposed metal on the connectors with that shrink rubber tubing or electrical tape. Be very careful with exposed metal around it anywhere, especially touching the RED POSITIVE pole of the battery. Get a battery box - I got one for the big-ass AGM battery.
Test it out on a laptop that's had it's battery removed or disconnected that, just in case, you don't care too much about losing.
Get a battery charger that can revive a flat battery, and do a full refresh/renew charge on the car battery once a year or after it's had to push through a power outage that may have used more than a few percent of its capacity.
Personally, I think it's safer a less hassle to go for a LiFePo4 (LFP) Power Station style device that has UPS capabilities. LFP batteries have 3,000-ish cycle lifetimes, which could be nearly ten years with daily use.
> use tin snips to cut a hole in the side of the metal UPS cover (this was challenging, it was relatively thick metal, I'd recommend using an angle grinder in an appropriately safe environment far away from the internals of the UPS)
Why not just drill a hole? Drill bits large enough to drill a hole for 120A cables exist.
> Get a battery charger that can revive a flat battery, and do a full refresh/renew charge on the car battery once a year or after it's had to push through a power outage that may have used more than a few percent of its capacity.
If you're going this route I'd recommend a marine battery. Car batteries don't handle deep cycles well, and, TBH, UPS chargers aren't designed for failed car batteries (nor marine batteries) and can possibly cause an explosion in the lead-acid battery has a few dead cells.
No, don't do it. I understand his thought process because they are both 12v batteries with more capacity, but car batteries are made for high burst of energy which a car engine ignition requires, whereas UPS batteries are made for slow drains. Also, these UPS are made for charging battery cells in a certain way, if you start to stack a bank of batteries of the same model in parallel hoping for more capacity, even then its a problem for the UPS's charger, they won't charge evenly and eventually becoming a problem.
Marine deep cycle batteries might work better, but at some point I'm pretty sure lithium would be price competitive.
I like to keep my hardware competence sufficiently low so that I’m never cursed with the false confidence to even consider “drilling a hole in a UPS,” nevermind wiring it to a car battery in my closet…
You seem like the kind of guy who doesn't enjoy a nice sulfuric acid spill on the floor, haha
I will mess with all kinds of hardware, especially mini PCs and routers.. I once had a few hundred iPhones in my closet… but I draw the line at anything that uses batteries or electricity in a non-standard way. If the wire can’t carry data, I’m not touching it.
Maybe it’s because when I was a kid, I fancied myself an experimenter, and I had a wire ripped off a lamp, and touched the two ends together…
It isn't quite that bad. the batteries are close enough that it will work.
the real worry is these are already a fire hazzard and so something goes wrong insurance will blame the mod even if not at fault
It's a bit trickier than you think. And can be dangerous.
The discharging circuitry is fine, but the _charger_ might overheat because a larger battery can draw more current while charging for longer periods. I discovered that when I tried to attach a "lead-acid compatible" LFP battery to an UPS.
These days, it's just easier to buy a dedicated rack-mountable LFP battery originally meant for solar installations, an inverter/charger controller, and a rectifier. The rectifier output will serve as a "solar panel" input for the battery. You get a double-conversion UPS with days-long holdover time for a fraction of a lead-acid UPS.
This really doesn't seem like something one would want to mess around with if they don't know what they're doing (fire hazards and all...)
My commercial UPS already scares me at the fire potential. No way I would take on the risk of some DIY on something that could burn down the place or electrocute me.
There's not much to it, you just take the small 12V sealed lead-acid cell out from the bottom of the UPS, extend the two leads, and connect a larger capacity lead-acid battery of the same voltage.
If you don't recognize the terms "sealed", "lead-acid", "battery", "capacity", or "voltage" then you shouldn't do this.
About the only advantage of it is that it's cheap (free if you find a UPS in the trash with an already dead battery), but those cheap UPSs make really crap quality power, and for some of them the only reason they don't overheat is because their stock battery is so small. It's a bit like how you can cook a whole turkey in the microwave, but you probably don't want to.