> When you lower the speed limit, the average will also go down?

Yes, but by much less than OP might naively expect.