Surprised by your comment, I took a gander at graphviz; it's about a quarter of a million lines of code if you discount lib/sparse/color_palette.c and lib/edgepaint/lab_gamut.c, which are hundreds of thousands of lines of data values. This is much more than I expected.

If you want something that builds diagrams from textual descriptions, you might want to check out TikZ, which includes a subset of graphviz but also lets you draw anything you want. See https://en.wikipedia.org/wiki/PGF/TikZ. On the other hand, you won't have the rapid visual feedback you get with WYSIWYG drawing editors like FigJam.