Can I have graphs as the default and fall back to agentic behavior if the prompt / task can't be mapped to a known process graph?

Sure. You could easily have the model choose a workflow/graph for execution and then call a tool that implements the execution engine for that. The tool then makes calls for model inferencing at various points in the processing.