Solid motors are simpler, but I wouldn't say they're more reliable. It was the solid booster failure that caused the Challenger disaster, and solid booster failures that have caused ULA's Vulcan Centaur rocket to be grounded. As you say, solids can't be throttled or turned off easily. They also can't be test fired. Yes, you can static fire a solid rocket and reuse the housing/nozzle/etc, but solids often fail due to imperfections in the propellant. Pockets of air or fractures in the propellant can cause a sudden increase in propellant surface area, which then generates more pressure, potentially blowing up the booster. With a liquid rocket, the engine can shut down if there's an anomaly. But with a solid, there's no option but to let the reaction continue.
There's also the issue that a solid booster must be "fueled" before it reaches the pad, meaning you have ground crew working around a large quantity of explosive material. A Brazilian solid rocket exploded on the launch pad, killing 21 people.[1] Liquid rockets can be made inert until everyone is far away, then loaded with propellant.
A big advantage of solid rockets is that they can be stored for long periods and quickly launched. This is handy for use cases like ICBMs, but not particularly important for commercial launches.