I don't think anyone has a problem with saying "my family came from Ireland", or even "my family was forced to flee Ireland because the British are bastards". Or even "Irish American" would be OK.
The problem we all see is that you're saying that you are Irish. If you weren't born in Ireland, your parents weren't born in Ireland, you don't speak Irish, you don't pay taxes in Ireland, you can't vote in Irish elections, you wouldn't join the Irish military, you don't understand Irish culture, or know anything about Irish history, then in what way are you Irish?
You're not. But you have redefined "Irish" to mean something else. And that's what pisses people off. There are actual Irish people out there. Invent your own identity.
So, if you are living in London and not paying taxes(to Ireland), you are not Irish? Dude, you are clearly mixing ethnicity and nationality. Plenty of Irish in UK, that does not know Gaelic - same situation in Ireland. There are in fact more Irish Gaelic speakers living in USA/Canada than in Ireland.
And Americans are claiming ethnic ancestry - not national ancestry. And many Irish migrated out of Ireland, when it was not a country - and paying taxes is irrelevant in this gatekeeping of identities.