I'm writing this comment at night, on my phone with its absolute lowest brightness; it is still too bright.
Maybe, just maybe, people aren't "holding it wrong"
Unlike screens produced light. To make them the same brightness as ambient would make them unreadable in my current situation. Yet a black background is much more pleasant. Why? Because the light being produced is less. You're right that the contrast matters. Black background and white text gets us closer to ambient light level while providing contrast. You simply cannot do same with black text on white background. I mean go to a dark room, put down the phone and ask yourself what the natural background is. It's black. That's the ambient level
Night Filter app is great
[1] https://nightfilter.app/Treat yourself to a new phone: they go as low as 1 nit nowdays.
Either you lower your display brightness or up your room lights — goal is the same: equal ambient and display brightness
My phone already goes down to 1 nit. 1 nit is still too bright