If we will struggle to create reward functions for AI, then how different is that from the struggles we already face when divvying up product goals into small tasks to fit our development cycles?

In other words, to what extent does Agile's ubiquity prove our competence in turning product goals into de facto reward functions?