>First, the average pixel is not what I would expect it to be at all

It looks like the averaging was done in default sRGB color space, with:

magick "$f" -resize 1x1 txt:-

Downscaling should instead be done in a linear colorspace. Human vision is non-linear, but the filtering required for downscaling is equivalent to blurring, which is linear because it's done optically not within the retina or brain. Using ImageMagick:

magick "$f" -colorspace RGB -resize 1x1 -colorspace sRGB txt:-

Additionally, JPEG supports chroma subsampling, which is usually enabled by default. I don't know what sips does, but with these small files you might as well use PNG and avoid the risk of losing color information this way.

This should produce results closer to human perception.