That's how I think about it too.
E-readers are vertically integrated devices: the hardware, software, UI, and refresh behavior are all tailored around reading. E-ink tablets like reMarkable are similar, but optimized around writing and annotation.
A traditional monitor is much more general-purpose, so it doesn't get the same kind of end-to-end optimization for the display medium. I think there's room for an in-between category: a more interactive e-ink device where both the hardware and software are designed around the strengths and limits of the panel.
There's some related work happening in this direction: