If someone comes to you and says: "I want to build a platform that does WhizzyWhatsIt for my customers, it has to be on AWS so it's mingled with my existing infrastructure. It needs to provide an admin portal so that I can set WhizzyWhatsIt prices and watch user acquisition make my bank account go brrrrrtt. It needs the ability for my quazi-illegal affiliate marketing ring to be able to whitelabel and brand it as their own for a cut of the profits."
This is obviously satire but there's a clear ask, some features, from there you know what you need to have to even achieve those features, what project management process would you employ? Agile? Waterfall? Agile-fall? Kanban? Call me in 6 months?
Probably waterfall stuff that have actual clear functions and integrations (if you can extract all that system gets and what it does with it there is no reason to agile it) then slowly get thru the current mess, documenting it at each step while trying to replace it with something better.
Replacing existing system (and especially one you didn't write) is pretty much always the hardest case.