Being accessible to the intended users always matters. If you think it doesn't, that probably means it's currently accessible to those users (or that those who are it is inaccessible to have filtered themselves out, and are no longer users).

For example - in your debug UI, colorblind-friendly colors don't matter, until you hire your twelfth member of the team, who struggles to tell red and green apart.