The recommendation pipeline has four steps:<ol><li>Candidate sourcing - Fetch the best tweets from different recommendation sources</li><li>Ranking - Rank tweets such that the most relevant ones appear first in the feed.</li><li>Filtering - Apply heuristics and filters to create a balanced and diverse feed.</li><li>Mixing non-tweet content like Ads, Follow Recommendations, and Onboarding prompts.</li></ol>The recommendation pipeline runs 5 billion times per day and completes in under 1.5 seconds on average.

Recommendation algorithm

Gets the best 1,500 recent tweets:<ul><li>50% from people you follow (In-Network)</li><li>50% from people you don't follow (Out-of-Network)</li></ul>In-Network<ul><li>Uses a logistic regression model to rank the recent tweets of people you follow. The higher the likelihood of engagement between you and another user, the more of their tweets will be included.</li></ul>Out-of-network<ul><li>Uses a social graph of engagements to find tweets that people you follow have recently engaged with, and tweets liked by people who are similar to you.</li><li>Uses an embedding space of users and tweets to determine what tweets and users are similar to your interests.</li></ul>

Candidate sourcing

Ranking is achieved with a 48M parameter neural network that is continuously trained on tweet interactions to optimize for positive engagement (e.g. Likes, Retweets, and Replies). This ranking mechanism takes into account thousands of features and outputs ten labels to give each tweet a score, where each label represents the probability of an engagement.

Ranking

<ul><li>Visibility Filtering: Remove tweets from accounts you block or mute. </li><li>Author Diversity: Avoid too many consecutive tweets from a single author.</li><li>Content Balance: Deliver a fair balance of In-Network and Out-of-Network tweets.</li><li>Feedback-based Fatigue: Lower the score of tweets that you provided negative feedback around.</li><li>Social Proof: Exclude Out-of-Network tweets without a second degree connection to the tweet i.e. ensure someone you follow engaged with the tweet or follows the tweet’s author.</li><li>Co﻿nversations: Provide more context to a Reply by threading it together with the original tweet.</li></ul>

Apply heuristics and filters

Twitter aims to deliver you the best of what’s happening in the world right now. This blog is an introduction to how the algorithm selects Tweets for your timeline.

computerscience

Computer Science

artificialintelligence

Artificial Intelligence

Twitter's Recommendation Algorithm

Cassandra stores columns on disk in the clustering order of your table. Therefore, to get the performance, your queries with ORDER BY should match the table's clustering order.For example, if you often query the newest data in the table, then you should set your clustering order to be (insertion_time DESC):<code>CREATE TABLE timeseries ( event_type text, insertion_time timestamp, event blob, PRIMARY KEY (event_type, insertion_time) ) WITH CLUSTERING ORDER BY (insertion_time DESC); </code>

Match your query's ORDER BY to the clustering order

Cassandra allows you to sort your query results in reverse clustering order, but it comes at a cost. Cassandra has to read all partition rows/columns and sort them in-memory, making your queries slower and increasing load on the cluster.The slowdown is proportional to the number of rows and columns you have, and it can become a serious problem if your partitions grow to thousands of rows and dozens of columns.See more info on <a href="https://stackoverflow.com/questions/57931713/cost-of-order-by-in-cassandra">StackOverflow</a>.

Reversing clustering order at query time scans the entire partition

Ordering query results to make use of the on-disk sorting of columns.

Cassandra - Clustering Order

DROP COLUMN only makes the column invisible to SQL operations by setting attisdropped=True in the system catalog table <a href="https://www.postgresql.org/docs/current/catalog-pg-attribute.html">pg_attribute</a>. Existing rows keep the original value, while new rows will store NULL on that column.Thus, dropping a column is quick, but it doesn't reduce the disk size of your table. That space will be reclaimed over time as existing rows are updated. However, if you want to reclaim disk space immediatelly, you can execute VACUUM FULL to force a rewrite of the table.See more info on <a href="https://stackoverflow.com/a/15700213">StackOverflow</a>.

DROP COLUMN doesn't physically remove columns

PostgreSQL - DROP COLUMN doesn't physically remove columns

The XY problem is asking about your attempted solution rather than your actual problem . This leads to wasted time and energy for both the people asking for help and those providing help.<ol><li>User wants to do X.</li><li>User doesn't know how to do X, but thinks they can fumble their way to a solution by doing Y.</li><li>User doesn't know how to do Y either.</li><li>User asks for help with Y.</li><li>Others try to help with Y, but are confused because Y seems like a strange problem to solve.</li><li>Finally, after much interaction, it becomes clear that the user really wants help with X, and that Y wasn't even a suitable solution for X.</li></ol>

What is the XY problem?

<ol><li>Always include information about a broader picture along with any attempted solution.</li><li>If someone asks for more information, do provide details.</li><li>If there are other solutions you've already ruled out, share why you've ruled them out. This gives more information about your requirements.</li></ol>

How to avoid the XY problem

Asking about your attempted solution rather than your actual problem

problemsolving

Problem Solving

communication

Communication

The XY Problem

Activity feeds show activities from the people you follow and they represent the core of many popular products such as Facebook, Twitter, Instagram, LinkedIn, and Pinterest.Feeds are hard to scale because there is no clear way to shard the data across multiple machines. Once you have a large user base, almost everyone ends up being connected to everyone else.The next ideas show how Stream scaled their feed infrastructure over time.

The challenge of scaling activity feeds

The first solution was to store all activities in a single Postgres table and build the feed at read-time with a query like:SELECT * FROM love WHERE user_id IN (...)This solution ran smoothly up to 10M rows and with some fine tuning it held up to 100M rows and 1M users. After that point the performance dropped and users occasionally waited multiple seconds for their feeds to load.

Phase 1: Python + Postgres

The next step was to pre-compute the feeds in Redis.
<ul>
<li>They stored a feed for every user and populated them by fanning out new activities to all followers.</li>
<li>To minimize memory usage they stored only activity IDs in Redis and queried Postgres when loading the feed to get the full activity data.</li>
<li>To distribute load they sharded feeds across multiple Redis machines.</li>
</ul>
This solution was easy to setup and maintain, but it got expensive when they needed to store more data in Redis. As the user base grew, the queries to Postgres became slower and it would have been too expensive to move more data into Redis to speed up queries.

Phase 2: Python + Redis + Postgres

Next they moved the pre-computed feeds into Cassandra and stored the full activity data in them to avoid having to query Postgres when loading the feed.<ul><li>To reduce memory usage and cost they stored the last 3,600 activities for active users, and only 180 for inactive users (falling back to Postgres after that point).</li><li>To reduce compute costs they split fanout tasks in two: active users got a high priority queue, and inactive users a low priority one. The high priority queue had a higher capacity buffer to cope with spikes, whereas the low priority queue relied on autoscaling and spot instances.</li></ul>

Phase 3: Python + Cassandra + Postgres (1)

Even though the performance of Cassandra was great, it was complex to optimize it and hard to diagnose it during slowdowns and other issues.Also, after a point they often found themselves delegating logic to the database layer because it was hard to speed up the performance of Python.

Phase 3: Python + Cassandra + Postgres (2)

In 2018 they <a href="https://getstream.io/blog/stream-and-go-news-feeds-for-over-300-million-end-users/">upgraded their architecture again</a>. They haven't disclosed how their data is organized, but they mentioned what technologies they are using.They replaced Cassandra with RocksDB because it's simpler to maintain, has better performance, and has more consistent performance.They replaced Python with Go to:<ul><li>Speed up features such as aggregation, ranking and serialization</li><li>Simplify their infrastructure and improve latency. For the same number of requests, they managed to use 10 times less servers thanks to the lower memory and CPU usage of Go.</li></ul>

Phase 4: Go + RocksDB

Scaling Activity Feeds to 300M Users @ Stream

If you need to delete all rows of a table, use TRUNCATE instead of DELETE followed by VACUUM.TRUNCATE removes the entire content of the table and reclaims disk space immediately. Therefore it doesn't require a subsequent VACUUM or VACUUM FULL to reclaim the now-unused disk space. This is most useful on large tables.Depending on your use case, it might be useful to also reset the sequences. For this you would run: TRUNCATE table RESTART IDENTITY.From personal experience, TRUNCATE executed instantly on a table with 280M rows. 😁

Delete all rows of a table using TRUNCATE instead of DELETE

PostgreSQL - How to efficiently delete all rows from a table

It's safe to interrupt a VACUUM command, but you may lose some or all the work that was done:<ul><li>If you interrupt a VACUUM FULL, then all work will be lost as the database will revert to using the previous version of the table.</li><li>If you interrupt a VACUUM, then only the in-progress batch will be lost. All batches that were fully cleaned won't need to be cleaned again.</li></ul>See also <a href="https://dba.stackexchange.com/questions/159647/does-cancelling-an-autovacuum-process-in-postgresql-make-all-the-work-done-use">this Stack Overflow thread</a>.

It's safe to interrupt VACUUM

PostgreSQL - Is it safe to interrupt VACUUM?

Training a large language model (LLM) from scratch is getting complex and expensive.In the future, there will be a middle layer between companies that train their own LLMs and end-users that build applications on top of them.This middle layer will have startups that take existing LLMs and train on top of them (not just fine-tuning) to create models for different verticals. These startups will provide value through the data flywheel they manage to build, which will also be their key differentiator from other startups.

Future startups will offer specialized LLMs

Greylock general partner Reid Hoffman interviews OpenAI CEO Sam Altman. The AI research and deployment company's primary mission is to develop and promote AI technology that benefits humanity. Founded in 2015, the company has most recently been noted for its generative transformer model GPT - 3, which uses deep learning to produce human-like text, and its image-creation platform DALL-E. This interview took place during Greylock’s Intelligent Future event, a day-long summit featuring experts and entrepreneurs from some of today’s leading artificial intelligence organizations. You can read a transcript of this interview here: https://greylock.com/greymatter/sam-altman-ai-for-the-next-era/

technologyandthefuture

Technology & The Future

startup

Startups

AI for the Next Era - Sam Altman, OpenAI CEO

As engineers, we put a lot of value on our skills. We tend to judge others and decide if we respect them based on their skills and expertise.This becomes more apparent when you're in a technical leadership role. Your engineering expertise has a major influence on whether people respect and follow you.It's important to stay up to date and remain an expert in a domain of your preference. If you don't stay up to date, you run the risk of becoming an expert with 10-20 years of experience who gets technically challenged by junior people, which in turn costs you the trust of the engineers you lead.

Remain a technical expert as you grow into leadership roles

📣Join us for a new edition of Talks, Talks #134! ⚙️The journey starts with Daniel Rizea, Director of Engineering @ Fitbit, that will introduce us in Technical Leadership. He will be followed by Marian Staș, Chairman @ Leaders for the Third Millennium Club, that will drive us through Real Leadership in Action.

career

Career

leadershipandmanagement

Leadership & Management

Talks #134 Let's Talk About Leadership with Daniel Rizea

IC track<ol><li>Engineer</li><li>Senior engineer</li><li>Staff engineer</li><li>Senior staff engineer</li><li>Principal engineer</li><li>Distinguished engineer</li></ol>Technical leadership track<ol><li>Engineering manager</li><li>Senior manager</li><li>Director</li><li>VP</li><li>CTO</li></ol>Both tracks have similar levels of prestige, influence and power.

The IC and technical leadership tracks mirror each other all the way up

On the leadership track you don't have enough time to manage a large organization and maintain the same level of technical depth. However, you need to stay technical in order to maintain your technical decision-making sharp, and to gain the credibility of your team. This credibility helps your team understand and respect you enough to want to follow the vision and strategy you want to lead them toward.

Leaders must stay technical to gain the credibility of their teams

<ol><li>Be a voracious reader — Consume technical articles, books, videos, or podcasts.</li><li>Hire well — Hire good technical people because they will teach you things faster than if did on your own. Also, they will eat less of your time than bad hires that are not aligned with the team.</li><li>Review PRs & Clear out bugs — This gives you a ground view on the processes and development environment your team uses daily.</li><li>Create a team of tech advisors — Have a group of people you can discuss technical topics with.</li><li>Code as a hobby</li><li>Participate in hackathons</li></ol>

Tips to stay technical as an engineering leader

Shopify CTO Allan Leinwand explains how to stay technical as an engineering leader while balancing the demands of people leadership.

Twitter's Recommendation Algorithm

Cassandra - Clustering Order

PostgreSQL - DROP COLUMN doesn't physically remove columns

The XY Problem

Scaling Activity Feeds to 300M Users @ Stream

Ovidiu Podariu (Tech)