I agree. Additionally, both 0.0 and 1.0 don't really exist for dithered signals, so a byte should map to [0.5, 255.5] before division by 256. This also solves the signed integer asymmetry, as a signed byte maps to [-127.5, 127.5] before division by 128. I wonder if audio DSP folks have done this already.
Thinking about this more, dithering requires negative values to cancel out when adding. Works for audio, but color doesn't have negative numbers.
It is still frequency, where it would have negative values. but I doubt any color handling algorithms deal with it as a frequency. Rightfully so, the physical wetware for decoding images is very different than that for decoding audio. Well... not that different if you think of audio as a single pixel monochrome image.
Now I am imagining a weird alternate history where we treat audio like we treat color. OK take three bytes which encode how loud the sound is, one for lows, one for mids and one for highs where lows mids and high frequencies are picked to match human ear response.