I've built agents that call different LLMs and keep separated memories. Remember, agents are just long-running workflows with some nodes calling LLMs and that sometimes can be started as tool from other "agent".

There are times when I wonder if couldn't just draw then in a BPMN designer that allowed me to write custom code for nodes. Is BPMN still a thing?