About a Stream

Alas – I am not really a gamer, I played Doom in the 90’s and that’s about it. As a hobby music producer and wannabe superstar DJ the live music broadcasts on Twitch were of much more interest to me, and as a web developer, programmer, the technology behind the live streams also interested me.

Most game streamers on Twitch require at least two fairly high powered Windows PC’s to stream their content live to Twitch, one PC for game play and one PC to capture the game play video content and encode it into a format supported by the Twitch ingest media servers. OBS Studio is a popular free and open source software for video recording and live streaming and most of the top streamers have created a very professional looking stream with OBS using overlay graphics and dynamic alerts to provide interaction between the streamer and the viewers. Top streamers regularly attract viewing audiences in the 10s of thousands.

Using OBS Studio on my trusty old Macbook Pro I was able to live stream my latest DJ set using a couple of webcams, some audio visualisation graphics and moi the superstar DJ on the decks! After playing for a few hours I had attracted a grand total of 1 viewer and his dog – but hey it was a lot of fun.

A few weeks later a friend of mine said to me “man you just don’t stream enough” and it was clear that to attract viewers and followers I would have to put in a lot of streaming hours which I just didn’t have time for.  I thought that some kinda 24/7 automated stream would be cool – I could stream all day long! I wasn’t about to go out and buy a new Mac (or god forbid a gaming PC) and attempt to stream 24/7 from my home so I started thinking about other ways to develop a streaming platform that would run (economically) on a hosted unix server.

ffmpeg

I’ve been involved with media streaming projects in the past and was familiar with FFMPEG “A complete, cross-platform solution to record, convert and stream audio and video” A quick Google for “ffmpeg twitch” revealed that FFMPEG can also stream it’s output to Twitch (and other live stream services – YouTube, Facebook etc.) and after a few test streams I set about developing my own streaming platform using Docker service containers.

In November 2017 I started streaming Techno DJ sets 24/7. My goal was to create an interactive Twitch music channel (mostly playing Techno DJ sets) where viewers could interact with the channel by “liking” the DJ that was playing and change what was playing by making requests. Other DJ’s could upload their own sets which would then be reviewed and added to a dynamically changing daily playlist. I wanted the channel to be visually interesting with constantly changing and interesting visuals and dynamic overlays showing what was currently playing and how viewers had recently interacted with the channel. I also wanted to be able to feed audio into the stream relatively dynamically either from static local media files, live feeds – i.e. me on the sofa or from online sources such as Soundcloud and YouTube.

My Docker live stream environment looks a bit like this:

The media servers create the main audio and background video streams. The stream manager creates dynamic graphic and text overlays based on what is playing, viewer stats, ticker tape messages etc. It also monitors the encoder log files for errors and can restart encoders when required. Similarly the chat bot server interacts with viewers on Twitch accepting commands from the Twitch stream chat IRC channel allowing viewers to see what’s playing, “like” the current DJ or make requests. All the stream data required by the manager and chat bot is either stored in an SQL database or cached using memcached. The encoders – running ffmpeg in small debian Docker containers – encode all the stream data – audio, video, graphic and text overlays into a format accepted by the streaming service provider, i.e. Twitch.

I started developing the docker images and code for my live streaming platform on a VMWare Ubuntu server with access to a fairly good Intel processor. The most processor intensive containers are the media encoders running FFMPEG. Without a dedicated graphics GPU all the rendering needs to be done by the main processor/s. As the quality and frame rate of the stream increase so does the processing power required to encode the stream fast enough to satisfy the streaming service provider. For example YouTube expects a 720p HD stream to have a resolution of 1280×720 and a video bitrate range of  1,500 – 4,000 Kbps. As my content was relatively static – no high frame rate game action – I was able to lower my frames per second to 15 and still produce an acceptable looking 720p HD stream and with some other FFMPEG tweaks I was ready to move the stream to a live server.

Going Live

My live server is an 8GB VPS with 4 CPU cores. As soon as I started up the streaming containers everything else running on the server ground to a halt and it was pretty clear that I would need a dedicated server for live streaming.

With RAM and bandwidth not a real issue the VPS just needed enough CPU power to run at least 1 encoder. It took a few upgrades before I found a VPS that could handle the stream without completely maxing out the available processing power and that came within my monthly budget. The live server has been running one Twitch stream for over 6 months without any real problems. Occasionally the CPU maxes out causing the encoder threads to fail usually resulting in a stream “hang” or crash – this is no big problem Docker restarts the crashed server instantly and the stream manager will restart the encoder when errors in the logs are detected.

As I developed the stream I started added more overlays, more graphics and shiny knobs. All things that require more encoder power until I was at a point where I knew I would need to consider a new VPS, especially if I wanted to simultaneously stream to other services such as YouTube.

The Live Stream on Twitch @ 720p HD (15fps)

Introducing the Packet c2.medium.x86

About this time the nice people at Packet sent me an email inviting users to test their new AMD EPYC c2.medium.x86 bare metal servers and I jumped at the opportunity of trying out my streaming platform on a big beast of a server.

Installing my streaming environment on the c2.medium.x86 was a real breeze. The Packet c2.medium.x86 Ubuntu server deployed in Amsterdam in just a few minutes. I installed Docker and transferred all my data via a remote tar over ssh from my live streaming server to the c2.medium.x86. All my Docker containers built in a few minutes and then it was simply a case of stopping my live stream containers in London and starting the c2.medium.x86 containers in Amsterdam. Et Voila  – I was streaming from Amsterdam!

The c2.medium.x86 really puts the BEAR into bare metal server – because it is a beast of a machine with 24 physical cores / 48 hyperthreaded cores at a 2.0 Ghz base clock speed (3.0 Ghz max with turbo). With one encoder running my live Twitch stream the processor overall load was about 1%. You can see ffmpeg using 76% of one hyperthreaded  core below.

I asked a few questions in the Packet slack community about processor usage and the opinion was that I could probably run another 47 encoders before I maxed out the AMD EPYC processor. That would be 48 unique streams! To test this theory I created another three ffmpeg encoders to give me four concurrent live streams encoding on the AMD EPYC – 2 on Twitch, 1 on YouTube, 1 on Facebook. With all four encoders running processor utilisation was around 4%. Unfortunately I didn’t have any more YouTube, Facebook or Twitch accounts to stress the CPU with!

FFMPEG uses multi threading to take advantage of multi core processors, with 4 encoders running you can see lots of ffmpeg processes nicely spread across AMD EPYC hyperthreaded cores.

Conclusions

I am no expert when it comes to understanding the architecture of modern servers and processors but I do know that the AMD EPYC running in the Packet c2.medium.x86 was perfect for live stream encoding with ffmpeg.

Live streaming is becoming very popular across many social media platforms and whilst there are online services available that will multicast your live stream to multiple services there is no reason why you cannot create your own on demand streams broadcasting live content to various platforms either for a live event or for a one time marketing project.

Simply spin up a c2.medium.x86 and let it fly.

I want to thank Packet for giving me the opportunity of testing the c2.medium.x86 Server as part of their AMD EPYC challenge.

You can find the latest installation instructions here

Commands

sudo apt-get update
sudo apt-get -y install autoconf automake build-essential libass-dev libfreetype6-dev libgpac-dev \
  libsdl1.2-dev libtheora-dev libtool libva-dev libvdpau-dev libvorbis-dev libx11-dev \
  libxext-dev libxfixes-dev pkg-config texi2html zlib1g-dev
mkdir ~/ffmpeg_sources

sudo apt-get install yams

sudo apt-get install libx264-dev

sudo apt-get install unzip
cd ~/ffmpeg_sources
wget -O fdk-aac.zip https://github.com/mstorsjo/fdk-aac/zipball/master
unzip fdk-aac.zip
cd mstorsjo-fdk-aac*
autoreconf -fiv
./configure --prefix="$HOME/ffmpeg_build" --disable-shared
make
make install
make distclean

sudo apt-get install libmp3lame-dev

sudo apt-get install libopus-dev

cd ~/ffmpeg_sources
wget http://webm.googlecode.com/files/libvpx-v1.3.0.tar.bz2
tar xjvf libvpx-v1.3.0.tar.bz2
cd libvpx-v1.3.0
PATH="$PATH:$HOME/bin" ./configure --prefix="$HOME/ffmpeg_build" --disable-examples
PATH="$PATH:$HOME/bin" make
make install
make clean

cd ~/ffmpeg_sources
wget http://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2
tar xjvf ffmpeg-snapshot.tar.bz2
cd ffmpeg
PATH="$PATH:$HOME/bin" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure \
  --prefix="$HOME/ffmpeg_build" \
  --extra-cflags="-I$HOME/ffmpeg_build/include" \
  --extra-ldflags="-L$HOME/ffmpeg_build/lib" \
  --bindir="$HOME/bin" \
  --enable-gpl \
  --enable-libass \
  --enable-libfdk-aac \
  --enable-libfreetype \
  --enable-libmp3lame \
  --enable-libopus \
  --enable-libtheora \
  --enable-libvorbis \
  --enable-libvpx \
  --enable-libx264 \
  --enable-nonfree
PATH="$PATH:$HOME/bin" make
make install
make distclean
hash -r

cp ~/bin/ffmpeg /usr/bin/

After compilation FFMPEG version is

ffmpeg version git-2014-01-21-91f4394 Copyright (c) 2000-2014 the FFmpeg developers
built on Oct 22 2014 03:13:56 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)

It’s 2014 and I am still using the Apple Darwin Media server, but after years of being neglected by Apple it is starting to show it’s age. One of the main problems is that for Apple i devices there is no native RTSP support so streaming content from the Darwin server to portable devices is not so easy.

Apple have largely replaced RTSP with HTTP Live Streaming (HLS). HLS basically just cuts up video files into small MPEG video segments that can be downloaded/streamed via a normal website using the HTTP protocol.

This works pretty well for video content that you want to convert from say MP4 to HLS, you run a conversion process that transcodes the video into multiple 10 second segments that you serve up to clients via an .m3u8 reference file on your web server.

If you want to do this with live content then the process is similar but you will also need a client to transmit the live video to your server to transcode it into segments on the fly.

When I am streaming live RTSP content to my Darwin media server I use the Apple Quicktime Broadcaster (on my MAC) as its free and kinda works well. So lets use Quicktime Broadcaster (QTB) to stream live video to a server running FFMPEG which will transcode the video into HLS segments which can be served up by Apache (or your web seriver of choice) to HLS clients.

You can download the Quicktime Broadcaster for Mac here. It hasn’t been updated since 2009, so obviously isn’t high on Apples list of supported software anymore, but hey ho, it is free.

Fire up QTB and configure your video and audio settings.

We will be creating a manual unicast connection and the screen shots below show the settings I used.

Set transmission to Manual Unicast. The address is the TCP/IP address of the server that will run FFMPEG. Note the port numbers used for audio and video, this info is saved in the .SDP file.

For this demo I am using my webcam as the source video. If you have an external device select it as the video source. Choose your compression options depending on the bandwidth available to you. Enable and configure audio compression in the same way.

Click broadcast and the client will attempt to unicast the video data to the server (even though nothing is listening yet).

Export these settings to a .sdp file by clicking File -> Export -> SDP save the SDP file and then upload it to your server.

Now we want to run FFMPEG so that it listens on the ports we specified, captures the unicast video and audio data, transcodes it into HLS segments and stores the HLS data to a publicly accesible area of your web server.

We do this by running FFMPEG with all the options we need, Here is an example command line :

ffmpeg -i  /Dropbox/stream/hls.sdp -acodec libfaac -ac 2 -b:a 128k -vcodec libx264 -vpre libx264-ipod640 -b:v 500k -threads 0 -g 75 -level 3.1 -map 0 -vbsf h264_mp4toannexb -flags -global_header -f segment -segment_time 10 -segment_list hls.m3u8 -segment_list_flags +live -segment_list_entry_prefix http://tv.server.co.uk/media/ -segment_format mpegts stream%05d.ts

Here we are telling FFMPEG to use the SDP file we exported from QTB as the input, I saved it to my Dropbox. Then we are specifying the compression parameters for the transcoded MPEG transport stream (TS) files. You can easily find other examples of various FFMPEG compression options and use them depending on your requirements.

The segmenter options define how big the segments will be and where the .m3u8 file will be created – in this case I specified 10 second segments and the m3u8 file is being created in the folder where I ran FFMPEG from. The .m3u8 file and TS segments should be created in, or later moved to a folder accessible from the internet.

The segment_list_entry_prefix defines the prefix to the TS files in the m3u8 file, this is the configured url of your web server including the uri to your HLS files.

Finally the name used for the segment files is defined. Note the +Live segment_list_flags option, this tells ffmpeg that we will be transcoding live content. The official documentation for all these options can be found here.

Configure your ffmpeg options, start the QTB broadcast and run FFMPEG. If all is working you will see FFMPEG starting to receive and transcode the unicast video (and audio) data.

Now point your i-Phone/Pad/Pod or HLS client (i.e. VLC, Safari) to the m3u8 file on your webserver, i.e. from the options I used above the url would be :

http://tv.server.co.uk/media/hls.m3u8

You should see your live video stream.

My video stream had about a 60 second lag time. Disconnect and reconnect and you will see you are receiving live content – with the 60 second lag. Note that FFMPEG doesn’t automatically remove the old TS segments, you need to handle that yourself.

If it doesn’t work check FFMPEG for errors, note that some of the segment options need a relatively new version of FFMPEG to function. I downloaded and compiled the latest version of FFMPEG and it’s dependencies on my Ubuntu server using the guide found here.

Make sure the m3u8 file and MPEG TS segments are accessible from the web, open the m3u8 file in a text editor to check the url being used for each segment.

If FFMPEG starts dropping files it means your live data is coming in too fast for FFMPEG to process, you need to look at your compression settings. Try reducing the length of the TS segments from 10 seconds to 2 seconds, Remember your server must be able to transcode and save the live data within these times otherwise the stream will start to stutter.

With the release of OS X 10.8 Mountain Lion, ported X11 (Unix) Window and Terminal applications are no longer supported in the native Mac operating system, this means that any applications ported to OS X using Mac Homebrew will no longer work after upgrading to Mountain Lion until you install new versions of X11 and XCode.

Here are the steps required to get your ported Homebrew apps including ffmpeg working again under OS X 10.8 Mountain Lion.

REINSTALL XCODE

Goto the App Store and reinstall the latest version of the XCode development software. When the software is installed, run XCode and goto Xcode > Preferences > Downloads and click Install next to the Command Line Tools package to reinstall XCode command line tools.

Let OS X know where XCode is with the following command

sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

INSTALL X11

Follow the install link on the X11 popup as shown above or goto http://xquartz.macosforge.org/trac/wiki to download the latest version of the XQuartz Project X Windowing System for OS X. “The XQuartz project is an open-source effort to develop a version of the  X.org X Window System that runs on OS X. Together with supporting libraries and applications, it forms the X11.app that Apple has shipped with OS X since version 10.5.”

Open the X11 dmg and run the install package. Restart your Mac and then update the X11 symlink with this command

sudo ln -s /opt/X11 /usr/X11

UPDATE HOMEBREW

From the command line run

brew doctor

to check the status of your home brew installation. And follow any recomendations there. Update your Homebrew software using

brew update

You may need to install the git repository software for the update to run:

brew install git

Your Homebrew apps including ffmpeg should now run on OS X 10.8 Mountain Lion, and you should be able to update and install new software using the brew install command.