I think it is because of the popularity of riding, people are used to seeing horses in that role. Not quite pets, but in a relationship to people closer to that of dogs and cats than that of farm animals.