In the broad picture of engineering, I would consider this to be technically difficult. Many pieces of the puzzle need to interact correctly to remove latency, from physical location, network gear, decision about where software runs, removing unnecessary layers of everything, to algos and data structures, and doing razor-tight tradeoff analysis favoring low-latency at every step along the way. It’s also expensive. So it’s hard to agree that this is not hard. On the flip side, if you find this easy and want a job, msg me.
The OP mentioned that "most parties there take minutes to process messages". My point is really that if transactions are taking minutes to complete, that's likely to be a social/structural problem, not a technical one.
With today's technology and all the work in this space, it's not particularly technically difficult to get such times down into the seconds or milliseconds, depending on the constraints.
Yes, HFT has gone to extremes to get trade execution times down as low as microseconds. And sure, that involves technical challenges. But they're mostly sort of obvious ones - you mentioned some of the major categories - that yield to throwing engineering talent at. But I was referring to the Pareto 80% of gains here.