Had the pleasure of working with Jason Garrett-Glaser (better known as DarkShikari) circa 2011 on a patch for x264 to better utilize more cores (slice-threaded lookahead). At the time he was working with Sony to optimize their PlayStation game streaming service, and this had the potential to lower encoding latency (critical to game streaming) even on same-core systems by spreading the frame over more cores to do more parallel work, and doing more intelligent scheduling and convergence after that tile-based work was reconverged into a frame. At the time we were using servers with 48 physical cores to do transcoding, a bit of a rarity in a single NUMA space back then and there were no instances to test this on so I flamegraph traced a few encodes at very high thread parallel counts and send back the instrumented profiles (from a custom compiled version) showing the core work being done and he used that to optimize the motion estimation code which worked on the first shot to perfectly level out cpu usage and increase encoding speed across ~24 cores (was previously topping out around 8 and running one core hot iirc). I got a look into how exotic and complex this project was, particularly in terms how low level and advanced things like motion estimation, quantizing and perceptual compression were... it was and still is sort of magic how many smart techniques are baked into something that we barely pay attention to. They were, even at that time, working on x265.
His blog, Diary of an x264 developer, was a great read too for anyone wanting to get a vantage point on how crazy that codebase was.
[1]https://github.com/mstorsjo/x264/blob/master/doc/threads.txt
(edit: was Gaikai that was acquired by Sony in 2012 and is the basis for PlayStation Now)
Using multiple CPUs for encoding goes way back. The early (1994) C-Cube MPEG-1 and MPEG-2 real-time encoders for DirecTV were built with 8 chips running in parallel (out of necessity since they were only 60 MHz processors) to encode 704x480. Each chip encoded a slice of the image and an auxiliary bus was used to send reconstruction video between chips so that motion vectors could cross chip image boundaries.
Here's a picture of the VME bus development board. It was a two board sandwich to support 10 chips in parallel (only one board pictured).
https://www.w6rz.net/scorpio.jpg
The first real-time HDTV encoders for terrestrial digital television (ATSC) were built by Harris using an IBM encoder chip. It took 6 chips to encode 1920x1080. The IBM chips didn't have an auxiliary bus, so you could see motion vectors limited to chip image boundaries (in this case, 6 tiles arranged like so).
Yeah at this time intel was just releasing their quick sync tech and a few vendors were putting 4 intel cpus on a pcie board and using it for video encoding! Matrox also had a considerable piece of this business doing video compression and editing/compositing using bespoke processors and boards.
Despite being more "power efficient" - both nvidia and intel's hardware solutions left a lot to be desired in terms of final quality for the same bitrate, compared to x264 that seemed to just put out substantially better quality video for the same bitrate even at the same h264 profile levels. There are still lots of x264 design elements, some even verbatim ported, into modern av1 and hevc encoders.
As someone doing a lot of DVD transcoding at the time, I remember being excited for Quick Sync on my new Sandy Bridge build - only to find that, while fast, it had absolutely _horrendous_ quality. I revisited it on my previous main Plex server build, which I think was a Haswell, and it was still unusable for transcoding compared to x264.
It took Nvidia until the encoder they bundled into 20xx series cards and Intel until the 6th or 7th generation of Quick Sync, but their hardware encoders are _finally_ at a point you can get decent quality at reasonable bitrates. Even so, x264 (and x265!) beat the pants off of them if one can spare the time and electricity; they're still that far ahead.
Today I learned about Intel Quick Sync. Thanks. I retired from video compression in 2011, so I've been out of the loop for a long time.
https://en.wikipedia.org/wiki/Intel_Quick_Sync_Video
Very cool. Took a similar approach to optimizing openutv by looking at flamegraphs. UTV is a fork of openrv which is an older release of the commercial RV. Like many things it uses ffmpeg and also native readers and writers for certain formats. Allowed me to get it to 8k@60fps buttery smooth playback with hardware acceleration. If you are still in the video/imaging space would love your feedback on UTV. openutv.com