All I know is I used to have full on colds during winter in my 20s to early 30s, like every winter I would get two weeks of runny nose till the skin became red and irritated, sore and painful throat, coughing, etc. Since I take 25000 IU (D-Cure in Belgium) like 2-3 times a month during winter for the past 15+ years my cold last 2-3 days and symptoms are much more mild. For me the data is there.
The problem is that you don't know what your situation would be like if you'd not taken Vitamin D. There are too many variables, generally -- e.g. in your early 20s and 30s you're likely socialising with many more people (so, more exposure chances), and you may also have young children (even worse disease vectors).
I'm similar to you in experience, btw, except that I don't believe correlation is causation. But as it won't do any harm, most of us don't regularly monitor our Vitamin D levels, and Vitamin D supplements are cheap -- it becomes a harmless form of Pascal's wager that I'm happy to keep making. Nothing to lose except a very, very tiny amount of money.