If the notes are good it can be a gem. If not, then it also really bothers me.

Another way of saying it is discussing a book with some people will be really rewarding. But others you will rather prefer not to. This is just an instance of a more generalizable rule: people develop preferences regarding who they want to spend their time with.

FWIW, I switched over to the post-it note method (mostly) in the 90s for real live books. Grab a few colors and shapes. Easy to remove if you don't want to share your thoughts with someone else if the book is something you later choose to get rid of.