Today, companies live and die based on how well they harness data. Almost every app these days, once it reaches a certain scale, leverages a data pipeline to ensure that data can be collected, stored, and analyzed. A typical pipeline has three main stages:

1. The app sends data, typically...

• Job queues store a list of jobs that need to be run asynchronously. The simplest are first-in-first-out (FIFO) queues though most applications end up needing some sort of priority queuing system. Whenever the app needs a job to be run, either on some sort of regular schedule or as determined...

A caching service provides a simple key/value data store that makes it possible to save and lookup information in close to O(1) time. 

Applications typically leverage caching services to save the results of expensive computations so that it’s possible to retrieve the results from the cache ...

• DNS stands for “Domain Name System” and it’s a backbone technology that makes the world wide web possible. 
• At the most basic level DNS provides a key/value lookup from a domain name (e.g., google.com) to an IP address (e.g., 85.129.83.120), which is required in order for your comput...

They’re the magic sauce that makes scaling horizontally possible. They route incoming requests to one of many application servers that are typically clones/mirror images of each other and send the response from the app server back to the client. 

Any one of them should process the request t...

Once an app reaches a certain scale, there will likely be certain “services” that are carved out to run as separate applications. 

They’re not exposed to the external world but the app and other services interact with them. 

And that’s a wrap on Web Architecture 101. I hope you found this useful. I’ll hopefully post a series of 201 articles that provide deep dives into some of these components over the course of the next year or two. 

Every modern web application leverages one or more databases to store information. Databases provide ways of defining your data structures, inserting new data, finding existing data, updating or deleting existing data, performing computations across the data, and more.

In most cases the web...

The technology provides a way of serving assets over the web much faster than serving them from a single origin server. It works by distributing the content across many "edge" Servers around the world. In general a web App should always use a CDN to serve CSS, Javascript, images, videos.