I've played with diffusion models on and off since the first release of Stable Diffusion - just for amusement, without a particular goal.
Recently, I've been helping a friend's wife with some basic vector images for her sewing hobby (she has what is essentially a CNC sewing machine) and have been super-impressed with FLUX.1-Kontext, which I've been running on my Macbook Pro with mflux. Its ability to (for example) take a photo of a human or an animal and return a line drawing which is recognisably them (rather than just a generic similarish image as I've experienced with other models) is excellent.
It's an older model now, but (AIUI) has the text-handling features baked in, and in my various testing is very reliable at giving me the outputs that I want, without the randomness I've experienced previously. It's big and relatively slow (~3 mins per 512x512 image edit on my M1 Max Mac) but excellent to work with. It's also very straightforward to set up, without the harness complexity of e.g. comfyui.
How are you converting the bitmaps into vector images?
This is currently the weakest part of the workflow. I'm tracing the output images with VTracer [0] and then finishing them manually.
Through a combination of careful initial prompting (e.g. requesting line drawings, requesting limited colours ["posterization" works] if you need colour at all, requesting background removal) and then using the settings in VTracer to limit the colours used for the resulting SVG to the minimum needed, you can end up with an SVG that doesn't take too long to clean up in Inkscape. It usually takes a few minutes of deletions and path unions to produce something usable.
Diffusion models outputting SVGs would obviously be very interesting for this, but this is probably a small niche compared to everything that diffusion models can do with raster images, and I've not found anything usable online so far. LLMs (understandably, given their core technology) are still pretty bad at creating SVG art.
[0] https://github.com/visioncortex/vtracer
Claude and I have spent most of the summer on exactly this problem for Skechworks (skechworks.com), a free little Mac design app I made to help my own workflow that is very similar to yours. I use it for my custom coin shop The Achieve Mint (theachievemint.com).
It runs VTracer under the hood too. Most of the cleanup you're doing in Inkscape could possibly be pushed to before and after the trace?
Skechworks picks the palette before the trace and forces it again after. I quantize to 8 colors with pngquant, trace, then snap every fill back to the nearest palette color. VTracer fills each region with its own average color, otherwise you get hundreds of nearly identical grays. One design went from 279 fill colors to 3.
I upscale 2x before I quantize. The anti-aliasing is what tells the tracer where an edge really sits. Quantize at 1x and you throw that away, and small text comes out lumpy. Related gotcha: filter_speckle is an area, so don't double it when you double the image.
I also flatten transparency onto white first. VTracer counts every alpha value as a different color. I had a 5 color picture that it saw as 688 colors, and the trace took 6 GB of memory. Flattened, the same picture took 300 MB.
For the background I sample the whole edge of the picture. If 85% of it is one color, I delete every path in that color that touches the edge. White inside the drawing never touches the edge, so it stays.
For black and white I threshold with Otsu, then trace in color mode with hierarchical stacked instead of bw mode. bw mode hands you one giant black path full of holes. Stacked gives you every region as its own shape, which is much easier to edit. You have to pad the image with the background color first, or VTracer decides your subject is the canvas.
Your FLUX step is the same trick I use for photos and messy art. I have an image model redraw the picture as flat art, then trace the redraw. That came out about a quarter the file size of the best direct trace I could get. The catch is that the model moves and resizes the subject a little every run. I measure the box around the ink in both pictures and fit the trace back onto the original.
What I haven't solved is curve fitting. vectorizer.ai gives you real arcs and straight lines. VTracer gives you splines, so circles are never quite circles.
is the cnc sewing machine an off the shelf model or something DIY? I'd love to hear more
Off the shelf - it’s a Brother. It prints via a proprietary file format (.PES) but there’s an extension for Inkscape that supports creation and export.