But who says that the numbers are representing the points, rather than representing the intervals between the points?

It doesn't even need to represent intervals. A 13 inch ruler with 13 markings at 0.5, 1.5, etc inches is still a valid ruler, albeit an odd construction.