Footnotes and endnotes always carry a hidden burden: They interrupt the reading flow and direct the mind of the reader away from the characters and the plot and towards linguistics. If the translator follows a philosophy that prioritizes the subjective experience of the reader over a deeper understanding of the material, this "mode switch" is something he will try to avoid.
Personally, I tend to agree that a need to explain a translation represents a failure to come up with a satisfying equivalent in the target language.
The key to good endnotes is to make them a nice bonus for those interested rather than required reading for everyone. Basically, make the main text work on its own, then get into the weeds of translation details separately. It's the best of both worlds, though admittedly requires quite a bit of extra work to pull off.
I think we agree on the conclusion, but differ on the premise. As a lifelong manga reader, I can say that the footnotes/endnotes made me more focused and invested in what was going on in the panels, not less. It's a different flavor of footnotes than the one you would find in a textbook or reference manual. One way to look at it is as an extended dialogue box for context that can't fit in-panel, but is "vital" from the author's perspective (since they aren't english speakers, they can't really be expected to know just how badly their work is altered sometimes). It's an optional aid for those that are interested in getting more engrossed into the world building.
At the end of the day, when a phrase has multiple meanings (as intended by the author), it can be impossible to translate it into a single, tidy, english catchphrase. This is why in manga raw's (the original, non-translated versions), when authors write kanji characters, they will often superscript them with smaller, hiragana characters. This is essential because in japanese, those kanji characters usually have multiple meanings, so it's necessary to guide the reader towards the intended ones. And even from there, the re-interpreted hiragana characters/words themselves can have multiple - sometimes conflicting - meanings. It's in these conflicts that puns/dual meanings can arise. It's all part of the beauty of the language, and mangaka's are obsessed with showing that (as they should be).
So this sort of nuance will always be lost in translation, but can be essential.