Is there anything like this but for analyzing thousands of models? Most of this type of tool seem to work best for iterating and experimenting, but I'd actually like to use it for tracking and debugging something in production.

We have a service that runs thousands of custom, user-defined models and when something goes awry having this level of insight would be pretty useful.