I think we agree on the conclusion, but differ on the premise. As a lifelong manga reader, I can say that the footnotes/endnotes made me more focused and invested in what was going on in the panels, not less. It's a different flavor of footnotes than the one you would find in a textbook or reference manual. One way to look at it is as an extended dialogue box for context that can't fit in-panel, but is "vital" from the author's perspective (since they aren't english speakers, they can't really be expected to know just how badly their work is altered sometimes). It's an optional aid for those that are interested in getting more engrossed into the world building.

At the end of the day, when a phrase has multiple meanings (as intended by the author), it can be impossible to translate it into a single, tidy, english catchphrase. This is why in manga raw's (the original, non-translated versions), when authors write kanji characters, they will often superscript them with smaller, hiragana characters. This is essential because in japanese, those kanji characters usually have multiple meanings, so it's necessary to guide the reader towards the intended ones. And even from there, the re-interpreted hiragana characters/words themselves can have multiple - sometimes conflicting - meanings. It's in these conflicts that puns/dual meanings can arise. It's all part of the beauty of the language, and mangaka's are obsessed with showing that (as they should be).

So this sort of nuance will always be lost in translation, but can be essential.