Learn more about computerscience with this collection
Understanding machine learning models
Improving data analysis and decision-making
How Google uses logic in machine learning
Phase 3: Python + Cassandra + Postgres (2)
Even though the performance of Cassandra was great, it was complex to optimize it and hard to diagnose it during slowdowns and other issues.
Also, after a point they often found themselves delegating logic to the database layer because it was hard to speed up the performance of Python.
1
5 reads
MORE IDEAS ON THIS
Phase 2: Python + Redis + Postgres
The next step was to pre-compute the feeds in Redis.
- They stored a feed for every user and populated them by fanning out new activities to all followers.
- To minimize memory usage they stored only activity IDs in Redis and queried Postgres wh...
1
5 reads
Phase 1: Python + Postgres
The first solution was to store all activities in a single Postgres table and build the feed at read-time with a query like:
SELECT * FROM love WHERE user_id IN (...)
This solution ran smoothly up to 10M rows and with some fine tuning it held...
1
4 reads
Phase 3: Python + Cassandra + Postgres (1)
Next they moved the pre-computed feeds into Cassandra and stored the full activity data in them to avoid having to query Postgres when loading the feed.
- To reduce memory usage and cost they stored the last 3,600 activities for active users, and only 18...
1
5 reads
Phase 4: Go + RocksDB
In 2018 they upgraded their architecture again. They haven't disclosed how their data is organized, but they mentioned what technologies they are using.
They replaced Cassandra with Rock...
1
6 reads
The challenge of scaling activity feeds
Activity feeds show activities from the people you follow and they represent the core of many popular products such as Facebook, Twitter, Instagram, LinkedIn, and Pinterest.
Feeds are hard to scale because there is no clear way to shard the data across multiple machines. On...
1
8 reads
CURATED FROM
IDEAS CURATED BY
Alt account of @ocp. I use it to stash ideas about software engineering
Related collections
Other curated ideas on this topic:
Trouble accepting positive feedback
One 2017 study published in the Journal of Experimental Social Psychology found that low self-esteem is directly correlated to not being able to accept or capitalize on compliments from others.
- Because the...
Improving your life with motivation
While motivation is a huge topic, and the science on it is not in agreement, there are many takeaways we can use to understand how motivation operates and use it to improve our lives.
- Rewards and punishments are at the centre of motivation. For example, we may not b...
7 Things You Need To Understand About Anxiety And Depression
No, we are not over yet. After discussing the topic with my friend’s Dad (he is a doctor), I’ve made a list of things you need to understand and avoid anxiety attacks (hard to breathe moments).
- Think of your anxiety as a biological problem, rather than thinking there is something wron...
Read & Learn
20x Faster
without
deepstash
with
deepstash
with
deepstash
Personalized microlearning
—
100+ Learning Journeys
—
Access to 200,000+ ideas
—
Access to the mobile app
—
Unlimited idea saving
—
—
Unlimited history
—
—
Unlimited listening to ideas
—
—
Downloading & offline access
—
—
Supercharge your mind with one idea per day
Enter your email and spend 1 minute every day to learn something new.
I agree to receive email updates