Isn't that a good thing? It would encourage Joe Homeowner to sell the house to someone who could use it as a primary residence rather than leaving it empty and speculating on the real estate value.

Joe Homeowner already is incentivized to sell grandma's home. Homes need upkeep. No-one keeps an old home around to rot and fall down - that tends to bee bad for the evaluation. The issue is that those who can't finance a primary residence now won't be able to finance a primary residence then.