>The average pixel was not what I expected.

The average pixel doesn't look correct because human vision does not interepret shadowed colors as different colors. We first guess at the shadows, and then do some kind of inverse mapping from the shaded color space to the illuminated one before we "perceive a color". This is why the black,blue/white,gold dress illusion exists.