Youtube audio streaming service
Not looking for easy solutions
I think we can all agree that youtube ads are out of control. Sometimes it plays several ads in a row, sometimes at least one of them is unskippable and 30 seconds long, and sometimes there are just too many ad segments in a single video. I usually listen to DJ sets while I work, and I can’t even put into words how annoying it is to have music constantly interrupted by an ad. And this is not all – the iOS app frequently throws up irritating subscription banners and doesn’t allow background audio playback without a subscription. All in all, if you don’t have a premium/youtube music subscription, or if you don’t use an ad-block, your experience is pretty terrible. This made me think: can I “fix” youtube so that I could listen to music without ads both on mobile and desktop? I decided to make it a coding challenge for myself. Here’s how it went. Before we begin, all the sources are available here.
Do not repeat this at home
The basic outline for this project came to me almost instantly. It had to be a web service, one that could retrieve a video from youtube and then process it so that it could be streamed to any client with a web browser. I already knew that downloading video from youtube without a premium subscription was possible, since that’s exactly what youtube-dl does. I also had some idea of how I was going to transform the video into a “streamable” format, because I already had some experience with FFmpeg. And since I only wanted to use this service to listen to music, I could throw away all of the video data and keep just the audio. Not only would this make the file size smaller, which is good for streaming over cellular, it would also allow for audio playback in the background on iOS (that is, when the browser app is closed, or even when the phone is locked). To be honest, though, I don’t know if the streamed content absolutely has to be audio-only for that to work.
Technicalities
Important! The service requires FFmpeg to be installed! In my experience, it’s best to install it via a package manager, e.g. homebrew on a mac.
Compiling is pretty straightforward. Run $ go install and $ go build to make a binary. Tested with go version 3.17.
The binary requires a config.yml file to be present in the same folder to run. Here’s what the configuration might look like:
env: "debug"
port: ":1323"
version: "v.0.0.3"
output-route: "output"
output-directory: "output"
source-directory: "source"
completion-marker: ".complete"Now let’s go over what every config option means.
env– Current environment. Specify “prod” for production. Use the-dcommand flag to override this option and launch the service in debug mode.port– Service port. To override it, use the-pcommand flag followed by the port you want.version– Service version.output-route– URL route for statically served m3u8 files, so that the streaming URL looks like this{domain}/{output-route}/{m3u8 filename}.output-directory– Directory path where transmuxing output files are saved. To override it, use the-ocommand flag followed by the directory you want.source-directory– Directory path where downloaded youtube files are saved. It’s called “source” because these are the source files for transmuxing. To override it, use the-scommand flag followed by the directory you want.completion-marker– Completion marker file name. You can find out what this is in the Transmuxing and Anatomy of the request sections.
The binary can also download audio from youtube without launching the service. Simply run $ ./{binary name} {video URL}. By default the file is downloaded into the same folder as the binary. Please keep in mind that this download functionality is pretty limited. The details of the process are covered in the Downloading section.
Run $ ./{binary name} serve to launch the service.
Under the hood.
Service command-line interactions and configuration are handled with Cobra and Viper. Both are amazing libraries and just such a pleasure to work with. The setup is pretty conventional. Using cobra, we check the command options and select the corresponding handler. If the user didn’t provide the serve command, we simply use the download handler to attempt to obtain the audio. If serve is provided, we start the server. The service API is built with the echo framework.
Downloading video file
Before we attempt a download, we need to extract the unique video id from the youtube video URL. This is pretty easy to do, as it’s always present in every video URL as the “v” query parameter. For example, in the URL https://www.youtube.com/watch?v=Zr9wsgUIF6o the video id is Zr9wsgUIF6o. Next, we create an instance of the Downloader struct. This is our reference to a single download process, something akin to a stateful helper object. It provides methods for controlling the download, and it stores the state and context of the download being performed by that particular instance. To be instantiated, a downloader requires an object that conforms to the Client interface. The way HTTP requests are handled is completely decoupled from the downloader implementation, which gives us maximum control and flexibility. To conform to the Client interface, an object should expose HTTPGet and HTTPGetBodyBytes methods. The service uses a simple wrapper around the Go HTTP transport by default, but you can easily write an implementation of your own, or extend the existing one. The default client wrapper also has rudimentary socks proxy support.
To create a downloader object we also need to provide that video id and a reference to the progress update feed (more on this later). Once the downloader is created, we’re ready to download the video from youtube. I won’t go too deep into the details here – if you’d like to know more about the process, I recommend checking out this amazing article.
The download process can be broken down into several steps. First, we retrieve the video info, so that we can look through all the file formats available for that video and select the most suitable one. Pretty much all videos are available in multiple file formats, and higher quality videos have separate files for the video and audio streams. Since we only need the audio, our perfect case scenario is being able to download the audio track alone. Unfortunately, the current version of the service doesn’t support any other file formats, so if a video doesn’t have its audio available as a separate file, the download will fail. The video info metadata can be obtained from the https://youtube.com/get_video_info endpoint. Once we’ve retrieved the video info, parsed it, and selected the file format we want, we proceed with the actual download. To obtain the file, we need to find out its URL, which is hidden in the video info’s cipher param. If you take a look at a cipher, you’ll notice that it looks like some kind of hashed string. And it is – cipher is a string containing another set of parameters, one of them being the video URL. The only problem is that youtube doesn’t use any conventional hashing algorithm. Instead, the contents of the string are randomly shuffled several times. So how do we recover the original? This is the trickiest part of the entire process. To do it, we need to download the embedded video player js bundle and parse it to find the decipher function that would allow us to generate the URL. The decipher function is a sequence of simple string operations, which we’ll call steps here. These steps are “hidden” within the minimized player code, and each video has its own unique randomized set of them. Finding them all takes some serious regexp magic – take a look at decipher.go if you want to see what that looks like. Youtube has made it pretty difficult to figure out the deciphering logic from the minimized player code, but it’s not impossible. After all, the player itself relies on this function to obtain the source file for streaming. The operations, their order, all the info we need is already there. Once recovered, the resulting decipher function might look like this:
splice:function(a,b){a.splice(0,b)},
reverse:function(a){a.reverse()},
EQ:function(a,b){var c=a[0];a[0]=a[b%a.length];a[b%a.length]=c}};
a=a.split("");
Mt.splice(a,3);
Mt.EQ(a,39);
Mt.splice(a,2);
Mt.EQ(a,1);
Mt.splice(a,1);
Mt.EQ(a,35);
Mt.EQ(a,51);
Mt.splice(a,2);
Mt.reverse(a,52);
return a.join("")As you can see, these operations are easy to reproduce in Go once we find out what we actually need to do. Regexps aren’t my strongest suit, so I’m using the decipher code I found in this great repo. Once we have deciphered the cipher and assembled the URL, all that’s left is to download the file. The download progress updates are either streamed to the progress feed (more on this later) or printed out to the command line. Once the file has been downloaded, the downloader object is safe to delete.
Converting video file - transmuxing
The next step is transmuxing the file we just downloaded into HLS format. For this purpose we create a transmuxer object. It’s very similar to the downloader in concept, and its purpose is to take care of everything related to converting the file. To convert it, we simply run an ffmpeg command-line command using exec.Command. Once the file has been converted, we create a special empty “marker” file to indicate that it was processed successfully and is ready for streaming. The existence of the converted m3u8 file alone isn’t enough to tell us that the process completed successfully, since that file can be created even if an error occurred during conversion. At this point we’re pretty much done, and our file is available for streaming.
Anatomy of a request
As I mentioned above, the service uses echo to handle client requests. Once a client makes an empty get request to an endpoint, the server upgrades the connection to WebSockets. The client then sends a video URL over the socket. We extract the video id from the URL, construct an output file path using that id, and check for a completion marker file. As was mentioned previously, the existence of this file signifies that the video has already been downloaded, converted into HLS format, and is available for streaming. If a completion marker exists, we notify the client and close the connection. Otherwise, we let the client know that processing is about to begin. Next, we create a new process handler for the video, or, if a handler for that video already exists at that point in time, we simply return the existing instance. I’ll go over how a process handler works in the next section. For now, it’s enough to note that from here on all that’s left is to subscribe to the updates from the process handler and deliver them to the client. Here’s what that looks like:
func DownloadAndProcessVideo(ctx echo.Context) error {
ws, err := upgrader.Upgrade(ctx.Response(), ctx.Request(), nil)
if err != nil {
log.Print(err)
return nil
}
defer ws.Close()
_, url, err := ws.ReadMessage()
if err != nil {
log.Print(err)
if err := ws.WriteJSON(models.ProgressUpdate{Type: models.ERROR}); err != nil {
log.Print(err)
}
return nil
}
videoID, err := utils.ExtractVideoID(string(url))
if err != nil {
log.Print(err)
if err := ws.WriteJSON(models.ProgressUpdate{Type: models.ERROR}); err != nil {
log.Print(err)
}
return nil
}
if files.CheckIfWasProcessed(viper.GetString(consts.OutputDir), videoID) {
if ws.WriteJSON(models.ProgressUpdate{Type: models.AUDIO_IS_AVAILABLE, VideoID: videoID}); err != nil {
log.Print(err)
}
return nil
}
if ws.WriteJSON(models.ProgressUpdate{Type: models.REQUEST_ACCEPTED, VideoID: videoID}); err != nil {
log.Print(err)
return nil
}
progressUpdates := processhandler.GetOrCreateSubscription(client.Get(), videoID)
for update := range progressUpdates {
if err := ws.WriteJSON(update); err != nil {
log.Print(err)
break
}
}
return nil
}Progress update feed and process handler
So what exactly are these mysterious process handler and progress update feed? I’ll start with the latter. The progress feed is a thread-safe one-to-many data stream container. If that sounds pretty complex, don’t worry, I’ll explain everything in detail. Let’s take a look. ProgressUpdateFeed is a struct with the following properties: a mutex, an array of Go channels called updaters, and a boolean flag signifying whether the feed has been closed.
type ProgressUpdateFeed struct {
mu sync.RWMutex
updaters []chan models.ProgressUpdate
closed bool
}Then, it has the Subscribe method.
func (f *ProgressUpdateFeed) Subscribe() <-chan models.ProgressUpdate {
f.mu.Lock()
defer f.mu.Unlock()
if f.closed {
return nil
}
ch := make(chan models.ProgressUpdate)
f.updaters = append(f.updaters, ch)
return ch
}As you can see, all it does is create a new channel, add it to the array of updaters, and return it. This way, whenever we send data to the feed, it goes out to all the “listeners”, or “subscriptions”, whichever you prefer to call them. That’s exactly what the Send method does.
func (f *ProgressUpdateFeed) Send(update models.ProgressUpdate) {
f.mu.RLock()
defer f.mu.RUnlock()
if f.closed {
return
}
for _, ch := range f.updaters {
go func(ch chan models.ProgressUpdate) {
ch <- update
}(ch)
}
}It iterates over all the available channels and propagates the data to each of them. It’s important to note that the data is sent in a non-blocking manner: if one of the channels isn’t ready to receive new data, we don’t block waiting for it and simply move on to the next channel instead. This also means that the data “inside” the updaters can be out of sync, with one channel empty while another has several updates pending, and this is by design. Lastly, it implements the Close method.
func (f *ProgressUpdateFeed) Close() {
// TODO: This is a hack to make sure all active channels send updates to clients
// before closing
time.Sleep(500 * time.Millisecond)
f.mu.Lock()
defer f.mu.Unlock()
if !f.closed {
f.closed = true
for _, ch := range f.updaters {
go func(c chan models.ProgressUpdate) {
// We go through all pending channel updates and "pop" the values
// before closing channel in order to avoid panic
for {
noValuesPending := false
select {
case _, ok := <-c:
if !ok {
noValuesPending = true
}
default:
noValuesPending = true
}
if noValuesPending {
break
}
}
close(c)
}(ch)
}
}
}If we invoke Send and Close one right after another, we might end up in a situation where not all of the updaters receive the data, due to the async nature of the update. To make the closing mechanism more “fair”, I made it wait an arbitrary number of milliseconds, just to ensure that as many channels as possible, preferably all of them, receive the data. Next, we iterate over all the channels and “pop” any remaining updates using select. This is to ensure that every channel is empty before we close it.
Ok, this is pretty much it for the progress update feed, now onto the process handler. This is a set of helper methods that use the progress feed to handle video processing. First, we store feeds, a dictionary of progress feeds where the video id acts as a key, and second, a mutex to ensure there are no race conditions when accessing those feeds.
var (
feeds map[string]*feed.ProgressUpdateFeed
mutex sync.RWMutex
)Here we also have several helper methods. The first is GetOrCreateSubscription, which either creates a new feed, begins video processing (I’ll explain what this means in a bit) by invoking handleProcessing as a goroutine, and returns a subscription to the newly created feed, or simply creates a new subscription to the feed if it already exists.
func GetOrCreateSubscription(client client.Client, videoID string) <-chan models.ProgressUpdate {
mutex.Lock()
defer mutex.Unlock()
if feeds == nil {
feeds = make(map[string]*feed.ProgressUpdateFeed)
}
p, ok := feeds[videoID]
if !ok {
feed := feed.New()
feeds[videoID] = feed
newDownloader := downloader.New(client, videoID, feed)
go handleProcessing(videoID, newDownloader, feed)
return feed.Subscribe()
}
return p.Subscribe()
}By “video processing” I mean the entire thing: downloading the video and converting it to HLS format, all the steps. handleProcessing takes care of just that, as well as sending updates on the state of the process to the corresponding feed at every step. Once the processing is done, the feed is closed and the process handler is removed.
func handleProcessing(videoID string, d *downloader.Downloader, f *feed.ProgressUpdateFeed) {
defer removeFeed(videoID)
defer f.Close()
ctx := context.Background()
if viper.GetBool(consts.Debug) {
log.Printf("Downloading video with id: %s
", videoID)
}
if !files.CheckIfWasProcessed(viper.GetString(consts.SourceDir), videoID) {
f.Send(models.ProgressUpdate{Type: models.DOWNLOAD_BEGUN, VideoID: videoID})
if err := d.RetrieveVideoInfo(ctx); err != nil {
log.Print(err)
f.Send(models.ProgressUpdate{Type: models.ERROR, Error: err})
return
}
if err := d.DownloadVideo(ctx); err != nil {
log.Print(err)
f.Send(models.ProgressUpdate{Type: models.ERROR, Error: err})
return
}
f.Send(models.ProgressUpdate{Type: models.DOWNLOAD_FINISHED})
}
f.Send(models.ProgressUpdate{Type: models.TRANSMUXING_BEGUN})
sourceFilePath, err := files.GetSourceFilePath(videoID)
if err != nil {
log.Print(err)
f.Send(models.ProgressUpdate{Type: models.ERROR, Error: err})
return
}
t := transmuxer.New(videoID, sourceFilePath)
if err := t.ConvertVideo(); err != nil {
log.Print(err)
f.Send(models.ProgressUpdate{Type: models.ERROR, Error: err})
return
}
f.Send(models.ProgressUpdate{Type: models.TRANSMUXING_FINISHED})
f.Send(models.ProgressUpdate{Type: models.AUDIO_IS_AVAILABLE, VideoID: videoID})
if viper.GetBool(consts.Debug) {
log.Printf("Video %s
is ready for streaming!", videoID)
}
}
func removeFeed(videoID string) {
mutex.Lock()
defer mutex.Unlock()
if _, ok := feeds[videoID]; ok {
delete(feeds, videoID)
}
}Now I’d like to provide a little bit of context. Progress feeds let us manage multiple video processes intelligently. If one client initiates the processing of a video, and then another client asks for the same video, the existing process will be used. The second client receives all the same updates as the first, starting from the point it made the request. For example, if the first client has initiated a download and 50% of it is already done by the time the second client makes its request, the second client will receive download progress updates from 51% and up. Also, if any or all of the clients disconnect, the updates will still be delivered, because, as I mentioned before, Send is non-blocking and doesn’t sync data between channels. This does lead to a situation where we might end up with “stale” channels: a client has disconnected, but its channel still exists and still awaits data to be “popped” by the <- operator. Thankfully, we handle this when we close the feed by making sure every channel is empty first. To sum up, a process is initiated and runs in a “detached” manner, and it allows subscribing and unsubscribing to updates on the progress of the video processing at any point in time.
Conclusion
And that’s pretty much all there is to it. In the end, I’m pretty proud of this project. Researching the youtube player internals was so much fun, and I enjoyed designing the progress update feed and process handler quite a lot too. Feel free to use my code for your own personal projects, and let me know if you have any ideas or suggestions. You can reach out to me via email. And here’s a link to the project repo again.
Thank you very much for reading!
2021-02-06