What is a server?
A server is a computer that provides something to other computers over a network. That's it, that's the whole definition. “Server” can mean the physical or virtual machine itself, or it can mean the software running on that machine that actually answers requests. Both uses are common, and context usually makes it clear which one is meant.
The computer asking for something is usually called the client. The client sends a request; the server receives it, does some work, and sends back a response. A web browser requesting a page, a mobile app fetching a profile, and a script calling an API are all clients talking to servers.
Real examples of requests a client might send: GET /products, GET /profile, POST /checkout. Each one asks the server for a different piece of work.
A server is not inherently a giant, special computer. Your laptop can technically run server software. Production servers are simply machines configured to reliably serve real workloads, at scale, all day, every day.
Physical, virtual, and cloud servers
A physical server is one real machine sitting in a data center. Most modern infrastructure instead runs on virtual machines: software-defined slices of a physical machine, each with its own share of CPU, RAM, and disk, behaving like an independent computer. A cloud instance is simply a virtual machine (or a smaller, faster-to-start container) that a cloud provider manages for you.
Application A, its own slice of CPU and RAM
Application B, isolated from VM 1
For the architecture reasoning in this course, “server” generally means a unit of compute capacity, regardless of whether it happens to be physical, virtual, or a cloud instance. The distinction rarely changes the sizing math you'll use.
A computer that accepts requests or jobs and provides a service to other programs. It may be a physical machine, virtual machine, container host, or cloud instance.
One running copy of a server process or virtual machine. Two instances can run the same application while using separate CPU and memory.
A server instance that runs business logic: validating requests, applying rules, calling databases, and producing responses.
A compute process that handles background jobs outside the interactive request path. Workers commonly process images, reports, emails, and scheduled jobs.
A group of server instances performing the same role. A fleet can be uniform, with identical sizes, or heterogeneous, with different capacities.
The network entry point that distributes incoming requests across healthy application-server instances.
The processing resource that executes instructions. CPU-bound work slows down when computation consumes all available processor time.
Fast temporary storage used by running processes and in-flight work. Memory-bound systems fail when active work requires more RAM than a host owns.
The amount of work completed per unit of time, such as requests per second (RPS) or jobs per second.
The time one request or job takes from start to finish. Throughput describes volume; latency describes waiting time experienced by one item.
The ability to continue serving requests when components fail. Multiple servers can improve availability only when traffic can avoid unhealthy instances.
A service that does not keep request-specific state only in one server's local memory, allowing later requests to run on another instance.
From browser to server, and back
Type shop.example.com/products into a browser and a whole chain of events fires off in a fraction of a second: the browser sends a request, that request travels across the network, a server receives it, application code executes, that code may query a database, and finally a response travels all the way back.
What are we intentionally skipping?
DNS lookups, TCP connection setup, TLS/HTTPS negotiation, CDNs, and reverse proxies all sit inside this journey too. They matter, but they're networking concerns, not compute-capacity concerns, so this section deliberately keeps them out of view. Later material can go deeper on that networking layer.
Every request that reaches your application costs real compute time: CPU cycles to run code, and RAM to hold the request's data while it's being processed. That cost is exactly what this whole section teaches you to measure and provision for.
Web servers vs. application servers
A web server primarily handles HTTP connections: it can serve static files directly, or act as a proxy that forwards requests onward. Nginx and Apache are common examples. An application server runs your actual business logic: authentication, validation, calling a database, building a response. A Node.js app, a Java/Spring service, or a Python/FastAPI app are all application servers.
Modern systems handle this split in different ways. Sometimes the two roles run as genuinely separate machines. Sometimes an application framework serves HTTP directly, folding both roles into one process. Sometimes a load balancer or reverse proxy sits in front and takes on some of the web server's job. What matters for system design is which role a component performs, not whether that role happens to live on its own physical machine.
“Web server and application server always mean two separate machines.”
They describe two different responsibilities that traffic passes through, not two mandatory pieces of hardware. Many production systems collapse them into one process; others deliberately split them. Both are valid, and this course's exercises focus on the request path's shape, not on forcing every role onto its own box.
Static vs. dynamic requests
Not every request costs the same. A static request asks for a file that already exists, like an image; the server just hands it over. A dynamic request runs real application code: it authenticates you, applies business rules, and often talks to a database. That code execution is exactly what shows up as CPU and memory usage on your application servers.
CPU, RAM & server resources
CPU performs computation: executing application code, serializing JSON, encrypting or decrypting data, validating input, transforming data, running calculations. Every one of those actions takes measurable processor time.
Reading a utilization percentage
CPU utilization is a simple percentage of how busy the processor is over some window of time.
Mostly idle. Plenty of spare capacity, but also plenty of unused (paid-for) capacity.
Roughly half of compute capacity is in use. Comfortable, sustainable.
Limited headroom remains. A common operating ceiling in this course's exercises.
Fully occupied. New work queues up instead of starting immediately.
CPU saturation doesn't just slow down the current request: it increases queueing, so every subsequent request waits longer too, and overall latency climbs even before anything technically “fails.”
What RAM does
RAM (memory) stores data actively needed by running processes: the application runtime itself, in-flight request data, in-memory caches, active background jobs, and buffers. Unlike CPU, memory isn't reclaimed the instant work finishes; it stays occupied for as long as that work is in progress.
CPU and RAM are different bottlenecks. A server can have plenty of spare CPU and still crash from running out of memory, and vice versa. Always check both, independently.
CPU-bound, memory-bound, and I/O-bound
“The server is overloaded” doesn't always mean CPU is high. Every workload has a resource that actually limits it first.
Computation itself is the limit. The server is busy calculating, encrypting, or transforming data.
Working data cannot comfortably fit in RAM. Concurrency, not raw request rate, is usually the driver.
The application spends most of its time waiting on the network, disk, or a database call to return.
Throughput, latency & concurrency
RPS stands for requests per second: simply how many requests complete in a one-second window.
When this course says a server's maximum throughput is 800 RPS, it means the server can process about 800 requests every second, under the measured workload, before it reaches its tested limit. That workload qualifier matters: RPS is not a universal, portable number.
“Server capacity is a universal RPS number.”
A simple GET /health endpoint and an expensive image-processing endpoint do not cost the same amount of compute per request. The same server might handle 800 RPS of the first and only 40 RPS of the second. Always attach a capacity number to the specific workload it was measured against.
Latency is not throughput
Throughput asks “how much work can we complete?” Latency asks “how long does one request take?” A system can process a huge volume of requests per second while each individual request still takes a noticeable amount of time to finish; both are true at once.
Concurrency: how many, not how fast
Throughput counts completions per second. Concurrency counts how many requests are active at the same moment, in flight, holding resources, regardless of the rate they arrive at.
This distinction matters more than it looks: a request that stays open for a long time (say, generating a report) ties up memory for its entire duration, even at a modest arrival rate. You'll use this idea directly later, when sizing memory with Little's Law.
A server handles 500 RPS maximum. Sustained CPU target is 80%. What is its safe sustained capacity?
Server capacity & headroom
A server's tested maximum, say 800 RPS, is the ceiling measured under ideal conditions. Running there continuously leaves zero margin for anything unexpected. An operational target, commonly 80% of the tested maximum in this course's exercises, deliberately reserves the rest as headroom.
Why headroom exists
Real traffic is never perfectly uniform. Headroom absorbs:
- • Traffic variance: request rates fluctuate second to second.
- • Uneven distribution: a load balancer rarely splits traffic perfectly.
- • Slow requests: a handful of expensive requests can dominate a server's time.
- • Background work: deploys, health checks, and log shipping all use some capacity too.
- • Unexpected spikes: nothing in production behaves exactly like a spec sheet.
“80% utilization is a universal industry law.”
It isn't. 80% is a common, reasonable operational target, and it's the specific constraint used in many of this course's simulations, but real systems use different ceilings depending on workload volatility, failure tolerance, and cost sensitivity. Treat the number in each problem's requirements as the actual contract, not a universal constant.
Capacity planning
Capacity planning is arithmetic before it is architecture. Three formulas cover almost every sizing decision you'll make in this course.
Worked example: traffic of 1,700 RPS, a 600 RPS server, and an 80% target gives a safe capacity of 480 RPS per server. 1,700 ÷ 480 = 3.54, which rounds up to 4 servers, since a fractional server doesn't exist. Four servers provide 2,400 RPS of fleet capacity, so sustained utilization actually lands at 1,700 ÷ 2,400 ≈ 70.8%.
| Size | vCPU | Memory | Rated capacity | Monthly cost |
|---|---|---|---|---|
| Small | 4 | 8 GB | 220 RPS | $180 |
| Medium | 8 | 16 GB | 430 RPS | $330 |
| Large | 16 | 32 GB | 800 RPS | $620 |
| XL | 32 | 64 GB | 1,350 RPS | $1,120 |
| 2XL | 64 | 128 GB | 1,850 RPS | $2,100 |
The work arriving at the system: requests per second, jobs per second, or concurrent sessions.
The maximum work a component can complete per unit of time under the model being used.
The fraction of available capacity currently demanded. At 80%, one-fifth remains as headroom.
Unused capacity intentionally reserved for variability, measurement error, slow requests, and short bursts.
The number of requests or jobs in progress at the same time, not the number arriving each second.
The first constrained resource that prevents the system from meeting its requirement.
A server's maximum capacity is not automatically its safe sustained capacity. Every sizing calculation in this course starts by applying a utilization target, not the raw tested maximum.
Traffic is 2,000 RPS. Safe server capacity is 400 RPS per server. Minimum number of servers?
Vertical vs. horizontal scaling
Scaling up (vertical) replaces a server with a larger shape. Scaling out (horizontal) adds more servers of the same or a similar shape. Each solves a different constraint.
Move to a larger machine. Operationally simple, but every size has a hard ceiling and the largest shapes can have a poor price-to-capacity ratio.
Add machines and distribute work. Capacity grows incrementally, but every instance must actually be reachable, and the routing policy has to match the fleet.
Prefer scaling up when
- • The workload can't be partitioned safely.
- • Operational simplicity matters more than redundancy.
- • A larger shape fits both the capacity and budget limits.
- • Coordination cost grows with server count.
Prefer scaling out when
- • One machine can't meet aggregate demand.
- • Work can be distributed across stateless instances.
- • Capacity should grow in smaller increments.
- • No single compute failure should take everything down.
“More servers always means more usable capacity.”
Only if traffic can actually reach them. A server sitting on the canvas with no connection from the load balancer contributes zero usable capacity, no matter how large it is.
Adding a node is not the same as adding a reachable path. Every application server needs a real connection from the entry point before it counts toward fleet capacity.
Load balancing & traffic distribution
A load balancer is one ingress point that distributes incoming traffic across a set of backend servers. It converts several individual machines into one usable fleet, routes only to healthy backends, and is what makes horizontal scaling actually work.
Sends the same number of requests to each server, in turn. A strong default for a uniform fleet where every server has the same capacity.
Sends work in configured proportions. Use it when servers have different capacities; weights should represent whichever resource actually limits the workload.
Prefers the server with fewer active connections. Useful when request duration varies significantly and connection count approximates current load.
Uniform vs. mixed fleets
Equal routing is only safe when equal work produces roughly equal utilization. In a mixed fleet, one request may consume the same CPU or memory regardless of host size, so a smaller server needs a proportionally smaller traffic share. Always evaluate the hottest server after routing, never the average server.
Local bottlenecks & fleet averages
A fleet average is computed across every server. A local bottleneck lives on exactly one of them. The two numbers can tell completely different stories.
“If total fleet CPU is below 80%, every server is healthy.”
Averages can conceal a badly overloaded individual machine sitting behind two idle ones. Always inspect the busiest server specifically, not the mean across the fleet.
Fleet totals can hide overloaded individual machines. Teach yourself to check the hottest server or resource first, and only then look at the fleet-wide picture.
Little's Law & memory sizing
Throughput measures how quickly work arrives or completes. Memory is often held for a request or job's entire lifetime. A modest arrival rate can therefore still create large memory demand, if each item stays active for a long time.
Arrival rate, written as λ
Average time in the system, written as W
Average work in progress, written as L
RAM is local, not a shared bucket
Three servers may collectively own enough memory while one of them still crashes. Each job runs on exactly one host, and that host alone must hold the job's working set. Load distribution, not fleet totals, determines whether memory placement actually succeeds.
Round Robin gives each host one-third of the jobs. The fleet owns enough total RAM, but the Compact server still fails.
Weights follow the 6:96:96 RAM ratio. Every host carries roughly the same percentage of its own memory capacity.
“The fleet has 198 GB and demand is 158.2 GB” is an incomplete argument by itself. You must also prove that no single server receives more memory demand than its own safe limit, which is exactly what the routing weights above are doing.
Workload isolation
Interactive APIs optimize for latency and immediate availability. Batch systems optimize for throughput and completion deadlines. Running both on one finite pool lets a scheduled batch run consume capacity that customers need right now.
The worker pool must sit outside the API traffic path, with enough of its own throughput to meet its own deadline. The API fleet must independently satisfy its own request target. If either pool relies on borrowed capacity from the other, the two workloads are not actually isolated, they're just drawn as if they were.
Baseline, peaks & burst traffic
When autoscaling isn't available, the same fleet stays online all day. A good design covers the highest specified demand while avoiding excessive idle capacity during ordinary traffic.
Efficient most of the day, but rejects work during a known peak.
Meets peak demand and keeps useful baseline utilization.
Meets the peak, but pays for idle capacity the other 21 hours.
Ask three questions before sizing a fixed fleet: must the full peak be handled, and for how long? Can temporary utilization run hotter than the sustained target? And what does the extra idle capacity cost during every hour that isn't the peak? A baseline utilization floor exists precisely to reject a technically valid but wasteful design, one that buys a huge fleet for a short daily spike and then sits mostly idle.
Cost vs. capacity
Cost should never be the first filter. A valid architecture has to satisfy its technical constraints, capacity, utilization, redundancy, first. Only once several designs are all valid does cost help you choose between them.
Serve at least 2,000 RPS for a budget of no more than $2,300/month.
Option A · 2× 2XL2 servers × 1,850 RPS + load balancerMeets capacityOver budget
Generous headroom, but blows the budget for no capacity benefit at this traffic level.
Option B · 3× Large3 servers × 800 RPS + load balancerMeets capacityMeets budget
Meets capacity and budget with reasonable headroom to absorb variance.
Option C · 5× Medium5 servers × 430 RPS + load balancerMeets capacityMeets budget
Cheapest option, but leaves almost no headroom for traffic spikes or slow requests.
Capacity planning is constraint solving, not simply buying bigger servers. The cheapest option that satisfies every requirement is a good choice; the cheapest option overall, ignoring requirements, is not.
The compute-sizing workflow
Before touching a design canvas, find the actual bottleneck. Scaling a resource that isn't constrained does nothing for end-to-end throughput.
Application tier is at 96% CPU. Database is at 31% CPU. Where should you add capacity first?
A repeatable process
Do the arithmetic before placing components. This keeps the design canvas from becoming a guessing exercise, and gives you a short, defensible explanation for every decision.
- 1
Understand demand
Record baseline rate, peak rate, and the time unit for every value.
- 2
Identify the workload
Decide if it's request-driven, job-driven, CPU-bound, or memory-bound.
- 3
Find per-server capacity
Look up (or measure) how much one instance of a given size can do.
- 4
Apply headroom
Multiply rated capacity by the target utilization ceiling before sizing anything.
- 5
Calculate server count
Divide required demand by safe per-server capacity, then round up.
- 6
Check local CPU / RAM
Verify no single server, not just the fleet average, exceeds its limit.
- 7
Check traffic distribution
Confirm the routing policy actually matches the fleet, uniform or mixed.
- 8
Check peak behavior
Re-run the same checks against the peak, not only the baseline.
- 9
Check failure & redundancy
Confirm the design still holds if one instance is unavailable.
- 10
Check cost
Compare viable designs by monthly cost only after they all satisfy the constraints above.
- 11
Simulate / load test
Validate the design against realistic traffic before trusting the arithmetic.
- 12
Iterate
Revisit any step once traffic, workload shape, or budget changes.
Don't average away failure
A healthy fleet average can conceal one overloaded server. Inspect per-server values.
Don't confuse rate and concurrency
A handful of jobs each second can still mean hundreds running simultaneously.
Don't optimize one constraint
A design must pass topology, capacity, utilization, deadlines, and budget together.
Practice labs
Use Build to construct a design from a blank canvas. Use Fix to diagnose a concrete failure in an existing design and make the smallest effective repair. Read each problem's requirements carefully; the numbers in the labs are not the same as the examples above, and reasoning through them is the point.
Campaign Capacity
A retail API must absorb a major marketing campaign's traffic without falling over.
CPU Isn't the Problem
A document-rendering service looks fine on CPU, but memory tells a different story.
Background Jobs Are Starving the API
Batch work keeps eating into the capacity customers need right now.
Burst Without Waste
One fixed fleet has to satisfy both a daily baseline and a predictable afternoon peak.
You are ready when you can calculate the required fleet before touching the canvas, and explain why every component, connection, size, and routing choice exists.