Distributed systems are fascinating; it seems that this paradigm is the only answer to the challenge when a system faces an extraordinary workload that a single computer cannot handle. Such workloads are also increasingly common or at least more frequently heard of. We can see how YouTube is required to serve 2.49 billion active users per month, how Discord must manage the storage of trillions of messages, how Apple feels the need to operate more than 160,000 Cassandra nodes to meet their needs, and how Netflix must handle hundreds of millions of daily active users. These massive workloads make it impossible for the aforementioned companies to rely on a single computer, ultimately necessitating the implementation of distributed systems.
However, distributed systems are not free either. Adopting distributed systems means having to deal with the complexity that is inherently part of them. Operating a single PostgreSQL instance may not be very difficult, but what about operating 32 PostgreSQL instances, each with 15 logical shards? It's certainly not that simple. Similarly, operating thousands of services that communicate with each other is not as straightforward as operating a single backend instance. Combine that with the inherent unreliability of the network, and the plethora of tools, patterns, and things that can go wrong only adds to the complexity of implementing such systems.
"Distributed systems are hard," as some people say.
A New Series
This article is the first in a series in which I'll be documenting my journey of learning distributed systems, specifically by implementing an event-driven system. I will document each phase and stage in my process of building a video-sharing service like YouTube, where users can upload videos, watch them, stream them, and more. Why create a YouTube clone? Besides wanting to learn about distributed systems, I am also curious about how to create a video streaming system like YouTube. My curiosity was particularly sparked during the last Indonesian presidential campaign in which one of the candidates frequently went live on TikTok. This made me wonder how TikTok can smoothly handle livestreams watched by millions of people. And because I don't particularly like TikTok, I chose to create a YouTube clone instead, which has the same functionality I'm interested in, which is live streaming. Of course, the system I'll be creating is not designed to handle millions of users like TikTok right away, but at least I can get an idea of what the implementation looks like by trying to create a simpler version.
It should be noted that this series is mainly intended to document my learning process, not as a guide on how to properly implement distributed systems, as I do not feel qualified to write such an article. This also means that I will not only document my successes but also my failures, mistakes, and adjustments throughout my journey of building this system. This includes architectural changes, where there is a 100% chance that the architecture I describe at the beginning of the series will end up different from the final result when I decide it’s time to conclude the series. However, I will ensure that for every mistake I correct and adjustment I make, I will provide a comprehensive explanation of why it was done.
The Project's Overview
It would be good to provide a brief overview of the system I aim to build in this kickstarter article.
Here are the features I have in mind at the time of writing this article:
Users can register, log in, and log out, including logging in with OAuth
Users can watch videos in various resolutions
Users can leave comments on videos
Users can control the video in details while watching, including muting audio, seeking, and adjusting video quality on-the-fly
Users can upload videos, which then can be served in various resolutions
Users can livestream
Users can chat in real-time during livestream sessions
The list above is non-exhaustive and is just an initial draft, I just put it there to give a general description of the thing I'll be building. Aside of that, I expect the list to change as the series progresses. I will document every addition or removal of features through articles in this series.
As for the tech stack, I plan on mainly using Go as the programming language, several different databases and message queues, MinIO for the object storage, and Kubernetes for the infrastructure. Aside of that, I'm also planning on using the LGTM stack for monitoring needs. The adoption of new technologies will be documented in new articles in the series.
End Words
I am curious to see what kind of system I will ultimately build. Nonetheless, I am excited to start this new series and hope to learn a lot from it. I also hope that the writings I produce during my journey of building this system will be useful to anyone who reads them.
This writing is intended as a kickstarter for the new series. Part 1 of this series will be coming soon, and I will begin with a discussion of the initial system architecture design.
