The downside is obviously that uncached queries take much longer (adding >100ms) and more queries are uncached since you can't share the cache with a large user-base. Unless you just visit the same websites over and over again, this results in worse overall performance.
I've never felt this. Most large services run or delegate to anycast DNS services.
If you have knowledge of TCP, you know you will occasionally get stalls much greater than that beyond control.