There a lot of ways this could work.

1) the hardest, custom silicon + MCU to manage the USB interface

2) not as hard, shared memory, NPU + MCU to manage inference and USB interface

Theoretically you could do 2 with the right MCU, NPU, and memory combo. You'd stream/DMA the weights from memory into the NPU and then read the results with the MCU. From a user's perspective, it might take the form of an openAI API compatible endpoint that enumerates when they plug the USB device in. There would likely be some host-side software to ease the pain of trying to use a USB device as an HTTP API.