The Osprey has a reputation, for sure, but it's mid-pack. They called the F-104 the widow maker for a reason, for example. And the F-16 has a fairly high accident rate, too, slightly higher than the Osprey. Though I think the F-16's history is a bit more lopsided, they made some changes after early production airframes proved pretty accident prone.
Maybe the Osprey's reputation is due not only to the accident rate but also to the fatality rate. A fatal accident in a standard F-16 (not the 2 seater), assuming no one outside the plane is killed, means 1 death. A fatal accident in a V-22 with the same assumptions would have a minimum of 2 deaths (pilot and copilot) at a soft maximum of 26 deaths (2 crew + 24 passengers, possibly more if overloaded).
In Turkey, F-104 was called “flying coffin”.