Hang on, can you actually do something nefarious with just the bank account number?

If someone has your bank account and bank’s routing number (which is also not secret), they can make fraudulent ACH transfers and payments from your account. Of course it will most likely be caught as fraud some time after the fact, but just those two bits of not-secret info are enough to grief someone.

Knuth had to stop sending real checks for errors spotted in his books because they would post pics of the check and thieves abused the account https://www-cs-faculty.stanford.edu/~knuth/news08.html

And both numbers, plus your name and address and a convenient sample of your signature, are on every check you’ve ever written.

I suddenly feel very clever for signing everything with “Shamu T. Whale”

AFAIK that's US thing. In normal countries bank account numbers are not a secret. The worst thing that can happen is someone sending you money.

Yes but there are steep penalties for bank fraud so it is not especially common