Learn more about computerscience with this collection
Understanding machine learning models
Improving data analysis and decision-making
How Google uses logic in machine learning
Bonus Challenge: Multiple data-centers
Expanding to other regions requires two steps:
- Replicate the setup in each data-center.
- Have the Dispatcher broadcast the events to its peer Dispatchers from the other data-centers.
8
59 reads
MORE IDEAS ON THIS
Challenge 2: Connection Management
- Each connection is managed by an Akka actor.
- Actors are so lightweight that there can be millions of them on a single system. Moreover, all of them can be served by a small number of threads, proportional to the number of cores. This...
8
44 reads
Challenge 3: Multiple Live Videos
- Clients are subscribing to events for a particular live video i.e. they are telling the server which live video they are watching.
- The Frontend server stores all subscriptions in an in-memory table.
- Every time a new event is published, the su...
8
61 reads
Final architecture
- A viewer likes a video and sends an HTTP request to the Likes backend, which stores it in a database.
- The backend forwards the like to any random Dispatcher node using an HTTP request.
- Dispatcher looks up in the key-value store to find out which Frontend nodes are subscribed ...
12
315 reads
Interactive Live Videos
Having a lot of people interact on live videos poses many technical challenges. Mainly because viewers generate a lot of interactions that need to be delivered fast.
To get a sense of the scale, the top live streams in the world gathered millions of concurrent users:
8
121 reads
The Realtime Platform
LinkedIn has built the Realtime Platform to distribute multiple types of data in real-time such as:
- Likes, comments and viewer count for Live Videos
- Typing indicators and Read receipts for Instant Messaging
- Presence i.e. the green online indicators ...
7
118 reads
Related posts
- How LinkedIn displays Presence indicators in real-time: https://engineering.linkedin.com/blog/2018/01/now-you-see-me--now-you-dont--linkedins-real-time-presence-platf
- How LinkedIn measures end-to-end latency across systems: https://engineering.linked...
10
206 reads
Challenge 4: 10K Concurrent Viewers
- Scale horizontally to handle more concurrent viewers -> Add multiple Frontend nodes and coordinate them using a Dispatcher node.
- In a similar fashion to the Frontend node, the Dispatcher has a subscriptions table to know which frontend nodes should r...
9
54 reads
Challenge 1: The Delivery Pipe
- User devices have a persistent connection to the Realtime Platform servers.
- The servers use server-sent events to stream data fast on this connection via the EventSource interface.
A persistent connection is an HTTP L...
8
87 reads
Challenge 5: 100 Likes/second
- Scale horizontally again to handle more events -> Add multiple Dispatchers and move the subscription table into a key-value store so it's accessible to all Dispatchers.
- Dispatchers are independent from Frontend nodes and don't have persistent connect...
9
76 reads
Performance and scale
- Each Frontend node handles 100k persistent connections. It handles only this many connections because the server is doing a lot of work processing multiple types of data (likes, comments, instant messaging etc.).
- Each Dispatcher can publish 5k events per seco...
8
49 reads
CURATED FROM
CURATED BY
Alt account of @ocp. I use it to stash ideas about software engineering
Related collections
More like this
Performance and scale
- Each Frontend node handles 100k persistent connections. It handles only this many connections because the server is doing a lot of work processing multiple types of data (likes, comments, instant messaging etc.).
- Each Dispatcher can publish 5k events per seco...
Setting up the feedback loop for the problem statements.
Once the teams have generated a few problem statements, each member should vote on one problem statement that best refects the needs discovered from the data analysis process.
It’s recommended that teams employ the technique of dotmocracy, where each member of the team has two votes. Each m...
The Search For The Beauty Center
The brain, according to scientists, has a ‘beauty center’ which works in tandem with the other areas like the pleasure center, memory center, or visual center, with many functions complementing each other.
New, extensive studies show that beautiful visuals or faces have si...
Read & Learn
20x Faster
without
deepstash
with
deepstash
with
deepstash
Access to 200,000+ ideas
—
Access to the mobile app
—
Unlimited idea saving & library
—
—
Unlimited history
—
—
Unlimited listening to ideas
—
—
Downloading & offline access
—
—
Personalized recommendations
—
—
Supercharge your mind with one idea per day
Enter your email and spend 1 minute every day to learn something new.
I agree to receive email updates