The endocrine system is indeed extremely complicated, but this is one of the simplest and best-understood parts of it. We know relatively little about the mechanism behind the psychological effects of sex hormones (for example, we have no idea why they seem to have different effects in different people, with some people being severely affected, and other people barely noticing), but we have a lot of data showing that there is an effect.

"Drug A makes X go down, and X does Y, so A decreases Y" is a good description of the operation of finasteride and dutasteride (if we disregard the unexplained differences between the effects of the two drugs (we'd naïvely expect one to be strictly "better" than the other, but this is not the case)) on everything except the brain. Everything else responds as you'd expect a priori from modelling hysteresis with pencil and paper. But there's a lot we don't understand about the brain.