Talk to hardware directly

Most "from scratch" would be some sort of microcontroller - write directly to hardware, program registers directly. It's the best feeling when you know every single instructions on the CPU is under your control, there is no 20 million lines of kernel code that do something.

But the MCU needs special hardware (MCU itself, display, programmer) so it is not a good starting project. This brings us to good old MS-DOS. Sure, you need to call BIOS to set up graphics (mode 13h, "320x200x256 colors" is the best for beginners), but after that, you are talking directly to hardware.