2026-03-22
version eight from the last post was the monolith that worked. when I split it up, the server got a name. cathode. seemed fitting.
what cathode does is playout. take sources in, keep a stream running, never let it go down. sounds simple. it’s the hardest problem in live streaming.
casparCG has been the gold standard for 20 years. swedish public television built it, european broadcasters run it daily. it solved playout by pre-loading the next clip in the background while the current one plays and auto-transitioning when it finishes. obs can do it too but you need a full desktop environment. I don’t want to run a vm with a gui just to keep a stream alive. I wanted headless. I wanted docker. I wanted something I could deploy and forget.
I tried for years to build the full stack. ui, api, admin panel, scheduling, content management, playout engine. all at once. it took many restarts to realize I needed to strip everything down and focus on the core tech. and the core tech is the switch.
the stream must never go down. never. in 24/7 streaming the output encoder runs continuously. the question is how do you change what’s feeding it without killing it.
ffplayout solved it with named pipes and multiple ffmpeg processes. I tried that. obs solved it with scenes. voctomix solved it with a gstreamer compositor.
I used voctomix in the second rewrite. that’s where I first saw interpipeline communication. separate gstreamer pipelines connected through shared memory. the pattern stuck with me even after voctomix didn’t work out. every time ffplayout finished a clip and started the next one, voctocore saw a source disconnect and threw up its failsafe screen. so every clip transition was a flash of the failsafe. not great for 24/7.
cathode runs each source in its own isolated gstreamer pipeline. four by default. failover, two content inputs, and a blinder. they connect to a central mixer through shared memory. if a source crashes, the mixer keeps going. you switch inputs by changing compositor properties. the pipeline never stops.
that solves the switch. but there’s a harder problem hiding behind it.
people have been asking about gapless playlist playback in gstreamer for years. as recently as may 2025 a gstreamer core developer suggested the exact interpipeline architecture cathode uses but didn’t address how to make playlist transitions seamless within it. gstreamer’s own design docs acknowledge what they call the “snowball problem.” as far as I can tell, nobody had documented a working solution.
here’s the problem. when you’re playing files through interpipeline shared memory, there’s nothing constraining how fast the decoder runs. gstreamer’s about-to-finish signal fires when the file is done being read, not when it’s done being displayed. for a local file that can be less than a second. so the signal fires, the next clip loads, its signal fires immediately, the next one loads. clips cascade into each other.
raw uridecodebin3 in the interpipeline. about-to-finish cascades. clips skip.
concat element. works for hours then runs out of memory. elements accumulate and never get cleaned up. oom killer took it down at 24gb after about 15 hours.
per-clip pipeline rebuild. two-frame black gap on every transition. 66 milliseconds of nothing. visible.
clocksync element to constrain the decode rate. deadlocks during gstreamer preroll. pipeline never reaches playing state.
tee and fakesink with raw uridecodebin3. no playsink backpressure. about-to-finish still fires wrong.
hot-swapping elements in a running pipeline. timestamp misalignment. audio and video drift apart.
the fix was playbin3 with custom sink bins. playbin3 wraps uridecodebin3 with playsink, which gives you backpressure control. I built custom video and audio sink bins with a tee splitting to two outputs. one branch goes to a fakesink with sync=true, which provides the real-time clock reference. the other branch goes to intervideosink, which feeds the mixer. the fakesink creates backpressure that propagates upstream through playsink all the way back to uridecodebin3. about-to-finish finally fires at the right wall-clock time. the next clip pre-rolls. seamless transition. no black frames.
the switch problem and the gapless playlist problem. those were the two walls. cathode got past both of them. 31 tests making sure they stay solved.
— pfarnsworth