ORMs tend to make things more complicated and usually slower. I'd recommend using SQLX. It is a nice half-way point. It eliminates most boilerplate, is going to be sufficiently performant for 99% of cases, and it reduces a lot of cases down to between 1 and 5 lines of code for simple CRUD operations. (Where it isn't fast enough you can easily fall back to using the SQL drivers directly.
I agree 100% with you, and this is how I build my stuff too!
I was just pointing out that framework-less Go projects can have SQLX, ORM, etc, if you want to!