The goal was not to create a new programming language, but to provide a better way to write software. Specifically, the Go team wanted to address the complexities that arise when building modern server software, such as:<ul><li>controlling dependencies</li><li>programming with large teams with changing personnel</li><li>ease of maintainability</li><li>effective use of multicore CPUs and networking</li><li>efficient testing</li></ul>

Why Go was created

<ul><li>Portability. It allowed programmers to work on whatever platform they liked, and compile for whatever platform was required.</li><li>Gofmt. It showed that autoformatting could be done well and it could save valuable time by not arguing over spaces and newlines.</li><li>Strong standard library. It kept the community working with the same toolkit and prevented variant libraries from arising.</li><li>Concurrency. It helped popularize concurrency as a way to structure server software by making concurrency a first-class component of the language.</li><li>Fast builds 😁</li></ul>

What Go got right

computerscience

Computer Science

softwareengineering

Software Engineering

Reflections on Go's 14th Anniversary - Rob Pike

<ul><li>Primary → Server that can modify data.</li><li>Standby → Server that replicates the data changes on the primary server.</li><li>Warm standby → Standby server that doesn't accept connections until it's promoted to a primary server.</li><li>Hot standby → Standby server that accepts connections and serves read-only queries.</li></ul>

Server roles in a Postgres cluster

There are two ways in which data is synchronized from the primary server to the standby servers:<ol><li>Synchronous → Write transactions are not committed until all servers commit them.</li><li>Asynchronous → Write transactions are first committed on the primary server and then copied to the standby servers.</li></ol>See the image for pros and cons of each type.

How data is synchronized in a Postgres cluster

Postgres has two built-in asynchronous replication solutions:<ol><li>Streaming replication, which is a type of WAL Shipping that streams WAL records to standby servers as they are generated.</li><li>Logical replication, which constructs logical data modifications (<a href="https://www.postgresql.org/docs/current/logicaldecoding-example.html">SQL-like commands</a>) from the WAL and streams them to standby servers.</li></ol>

Built-in replication solutions

PostgreSQL Replication

Set the followings settings in /etc/postgresql/15/main/postgresql.conf:"Error Reporting and Logging"<ul><li>logging_collector = on</li><li>log_min_duration_statement = 1s</li><li>log_lock_waits = on</li></ul>"Run-time Statistics"<ul><li>track_activity_query_size = 8192</li></ul>

Enable slow query log

<ol><li>Download the log files on your computer. The default location is <code>/var/lib/postgresql/15/main/log/</code>.</li><li>Install <a href="https://pgbadger.darold.net/">pgBadger</a></li><li>Analyze the logs: <code>pgbadger -o stats.html --prefix '%m \[%p\] %q%u@%d' /dir\_with\_logs/\*.log</code></li><li>Open the HTML output file in your browser to review the results</li></ol>

Analyze slow query log

PostgreSQL - How to Configure Slow Query Log

The recommendation pipeline has four steps:<ol><li>Candidate sourcing - Fetch the best tweets from different recommendation sources</li><li>Ranking - Rank tweets such that the most relevant ones appear first in the feed.</li><li>Filtering - Apply heuristics and filters to create a balanced and diverse feed.</li><li>Mixing non-tweet content like Ads, Follow Recommendations, and Onboarding prompts.</li></ol>The recommendation pipeline runs 5 billion times per day and completes in under 1.5 seconds on average.

Recommendation algorithm

Gets the best 1,500 recent tweets:<ul><li>50% from people you follow (In-Network)</li><li>50% from people you don't follow (Out-of-Network)</li></ul>In-Network<ul><li>Uses a logistic regression model to rank the recent tweets of people you follow. The higher the likelihood of engagement between you and another user, the more of their tweets will be included.</li></ul>Out-of-network<ul><li>Uses a social graph of engagements to find tweets that people you follow have recently engaged with, and tweets liked by people who are similar to you.</li><li>Uses an embedding space of users and tweets to determine what tweets and users are similar to your interests.</li></ul>

Candidate sourcing

Ranking is achieved with a 48M parameter neural network that is continuously trained on tweet interactions to optimize for positive engagement (e.g. Likes, Retweets, and Replies). This ranking mechanism takes into account thousands of features and outputs ten labels to give each tweet a score, where each label represents the probability of an engagement.

Ranking

<ul><li>Visibility Filtering: Remove tweets from accounts you block or mute. </li><li>Author Diversity: Avoid too many consecutive tweets from a single author.</li><li>Content Balance: Deliver a fair balance of In-Network and Out-of-Network tweets.</li><li>Feedback-based Fatigue: Lower the score of tweets that you provided negative feedback around.</li><li>Social Proof: Exclude Out-of-Network tweets without a second degree connection to the tweet i.e. ensure someone you follow engaged with the tweet or follows the tweet’s author.</li><li>Co﻿nversations: Provide more context to a Reply by threading it together with the original tweet.</li></ul>

Apply heuristics and filters

Twitter aims to deliver you the best of what’s happening in the world right now. This blog is an introduction to how the algorithm selects Tweets for your timeline.

artificialintelligence

Artificial Intelligence

Twitter's Recommendation Algorithm

Cassandra stores columns on disk in the clustering order of your table. Therefore, to get the performance, your queries with ORDER BY should match the table's clustering order.
For example, if you often query the newest data in the table, then you should set your clustering order to be <code>(insertion_time DESC)</code>:
<code>CREATE TABLE timeseries ( event_type text, insertion_time timestamp, event blob, PRIMARY KEY (event_type, insertion_time) ) WITH CLUSTERING ORDER BY (insertion_time DESC);</code>

Match your query's ORDER BY to the clustering order

Cassandra allows you to sort your query results in reverse clustering order, but it comes at a cost. Cassandra has to read all partition rows/columns and sort them in-memory, making your queries slower and increasing load on the cluster.The slowdown is proportional to the number of rows and columns you have, and it can become a serious problem if your partitions grow to thousands of rows and dozens of columns.See more info on <a href="https://stackoverflow.com/questions/57931713/cost-of-order-by-in-cassandra">StackOverflow</a>.

Reversing clustering order at query time scans the entire partition

Ordering query results to make use of the on-disk sorting of columns.

Cassandra - Clustering Order

DROP COLUMN only makes the column invisible to SQL operations by setting attisdropped=True in the system catalog table <a href="https://www.postgresql.org/docs/current/catalog-pg-attribute.html">pg_attribute</a>. Existing rows keep the original value, while new rows will store NULL on that column.Thus, dropping a column is quick, but it doesn't reduce the disk size of your table. That space will be reclaimed over time as existing rows are updated. However, if you want to reclaim disk space immediatelly, you can execute VACUUM FULL to force a rewrite of the table.See more info on <a href="https://stackoverflow.com/a/15700213">StackOverflow</a>.

DROP COLUMN doesn't physically remove columns

PostgreSQL - DROP COLUMN doesn't physically remove columns

The XY problem is asking about your attempted solution rather than your actual problem . This leads to wasted time and energy for both the people asking for help and those providing help.<ol><li>User wants to do X.</li><li>User doesn't know how to do X, but thinks they can fumble their way to a solution by doing Y.</li><li>User doesn't know how to do Y either.</li><li>User asks for help with Y.</li><li>Others try to help with Y, but are confused because Y seems like a strange problem to solve.</li><li>Finally, after much interaction, it becomes clear that the user really wants help with X, and that Y wasn't even a suitable solution for X.</li></ol>

What is the XY problem?

<ol><li>Always include information about a broader picture along with any attempted solution.</li><li>If someone asks for more information, do provide details.</li><li>If there are other solutions you've already ruled out, share why you've ruled them out. This gives more information about your requirements.</li></ol>

How to avoid the XY problem

Asking about your attempted solution rather than your actual problem

problemsolving

Problem Solving

communication

Communication

The XY Problem

Activity feeds show activities from the people you follow and they represent the core of many popular products such as Facebook, Twitter, Instagram, LinkedIn, and Pinterest.Feeds are hard to scale because there is no clear way to shard the data across multiple machines. Once you have a large user base, almost everyone ends up being connected to everyone else.The next ideas show how Stream scaled their feed infrastructure over time.

The challenge of scaling activity feeds

The first solution was to store all activities in a single Postgres table and build the feed at read-time with a query like:SELECT * FROM love WHERE user_id IN (...)This solution ran smoothly up to 10M rows and with some fine tuning it held up to 100M rows and 1M users. After that point the performance dropped and users occasionally waited multiple seconds for their feeds to load.

Phase 1: Python + Postgres

The next step was to pre-compute the feeds in Redis.
<ul>
<li>They stored a feed for every user and populated them by fanning out new activities to all followers.</li>
<li>To minimize memory usage they stored only activity IDs in Redis and queried Postgres when loading the feed to get the full activity data.</li>
<li>To distribute load they sharded feeds across multiple Redis machines.</li>
</ul>
This solution was easy to setup and maintain, but it got expensive when they needed to store more data in Redis. As the user base grew, the queries to Postgres became slower and it would have been too expensive to move more data into Redis to speed up queries.

Phase 2: Python + Redis + Postgres

Next they moved the pre-computed feeds into Cassandra and stored the full activity data in them to avoid having to query Postgres when loading the feed.<ul><li>To reduce memory usage and cost they stored the last 3,600 activities for active users, and only 180 for inactive users (falling back to Postgres after that point).</li><li>To reduce compute costs they split fanout tasks in two: active users got a high priority queue, and inactive users a low priority one. The high priority queue had a higher capacity buffer to cope with spikes, whereas the low priority queue relied on autoscaling and spot instances.</li></ul>

Phase 3: Python + Cassandra + Postgres (1)

Even though the performance of Cassandra was great, it was complex to optimize it and hard to diagnose it during slowdowns and other issues.Also, after a point they often found themselves delegating logic to the database layer because it was hard to speed up the performance of Python.

Phase 3: Python + Cassandra + Postgres (2)

In 2018 they <a href="https://getstream.io/blog/stream-and-go-news-feeds-for-over-300-million-end-users/">upgraded their architecture again</a>. They haven't disclosed how their data is organized, but they mentioned what technologies they are using.They replaced Cassandra with RocksDB because it's simpler to maintain, has better performance, and has more consistent performance.They replaced Python with Go to:<ul><li>Speed up features such as aggregation, ranking and serialization</li><li>Simplify their infrastructure and improve latency. For the same number of requests, they managed to use 10 times less servers thanks to the lower memory and CPU usage of Go.</li></ul>

Phase 4: Go + RocksDB

Scaling Activity Feeds to 300M Users @ Stream

If you need to delete all rows of a table, use TRUNCATE instead of DELETE followed by VACUUM.TRUNCATE removes the entire content of the table and reclaims disk space immediately. Therefore it doesn't require a subsequent VACUUM or VACUUM FULL to reclaim the now-unused disk space. This is most useful on large tables.Depending on your use case, it might be useful to also reset the sequences. For this you would run: TRUNCATE table RESTART IDENTITY.From personal experience, TRUNCATE executed instantly on a table with 280M rows. 😁

Delete all rows of a table using TRUNCATE instead of DELETE

PostgreSQL - How to efficiently delete all rows from a table

It's safe to interrupt a VACUUM command, but you may lose some or all the work that was done:<ul><li>If you interrupt a VACUUM FULL, then all work will be lost as the database will revert to using the previous version of the table.</li><li>If you interrupt a VACUUM, then only the in-progress batch will be lost. All batches that were fully cleaned won't need to be cleaned again.</li></ul>See also <a href="https://dba.stackexchange.com/questions/159647/does-cancelling-an-autovacuum-process-in-postgresql-make-all-the-work-done-use">this Stack Overflow thread</a>.

Reflections on Go's 14th Anniversary - Rob Pike

PostgreSQL Replication

PostgreSQL - How to Configure Slow Query Log

Twitter's Recommendation Algorithm

Cassandra - Clustering Order

Ovidiu Podariu (Tech)