IIUC, an untrusted inline SVG is bad. An image tag pointing to an SVG is not.

    <img src="untrusted.svg"> <!-- this is ok -->
    <svg from untrusted src>  <!-- this is not ok -->
I feel like this is common knowledge. Just like you don't inject untrusted HTML into your page. Untrusted HTML also has scripts. You either sanitize it. OR you just don't allow it in the first place. SVG is, at this point, effectively more HTML tags.

Also remember that if the untrusted SVG file is served from the same origin and is missing a `Content-Disposition: attachment` header (or a CSP that disables scripts), an attacker could upload a malicious SVG and send the SVG URL to an unsuspecting user with pretty bad consequences.

That SVG can then do things like history.replaceState() and include <foreignObject> with HTML to change the URL shown to the user away from the SVG source and show any web UI it would like.

how is that special/different from an HTML URL?

Because displaying user-submitted images is pretty common and doesn't feel like a security footgun, but displaying user-submitted HTML is less common (and will raise more careful security scrutiny).