HTTP up close

Every web page, every app that "talks to the cloud", and most of the tools DevOps people use run on one simple idea: a request goes out, a response comes back. Both are plain text you can read. Once you can read them, "the site is broken" turns into "the API answers 503 because the app behind the proxy isn't running", which is a problem you can fix.

A request and a response

After the TCP handshake (lesson 4), the browser sends a few lines of text. This is the whole request for the club's home page:

GET /about HTTP/1.1            # method, path, protocol version
Host: club.example.org         # which website (one server can host many)
User-Agent: curl/8.5.0         # what's asking
Accept: */*                    # what it can handle
                               # an empty line ends the headers

And the server answers in the same shape: a status line, headers, an empty line, then the body:

HTTP/1.1 200 OK                         # protocol, status code, reason
Server: Apache/2.4.58 (Ubuntu)
Content-Type: text/html; charset=UTF-8  # what the body is
Content-Length: 104                     # how long it is

<h1>About the Coding Club</h1>           # the body

Look at a URL with that in mind: http://club.example.org:80/about?lang=en is the scheme (http), the host (sent in the Host header), the port (80 unless you say otherwise), the path and the query string.

Methods

Status codes

The first digit tells you who to blame:

Headers worth knowing

• Host: which site you want. One server, one IP address, many websites: the web server picks the right virtual host by this header.
• Location: where a 3xx is sending you.
• Content-Type and Content-Length: what the body is, and how long.
• Authorization / WWW-Authenticate: logging in, and the server asking you to.
• Set-Cookie / Cookie: HTTP forgets you after every request. Cookies are how a site remembers you between them.
• Cache-Control, ETag, Last-Modified: whether a copy may be kept and reused.

Proxies, and the 5xx trio

Real sites rarely answer from a single program. A web server or load balancer (a reverse proxy) takes the request and passes it on to an app behind it:

browser  →  Apache / nginx (port 80/443)  →  app on 127.0.0.1:8080

When the app is the problem, the proxy tells you so, with slightly different words:

• 502 Bad Gateway: the app answered with garbage, or (on nginx) didn't answer at all.
• 503 Service Unavailable: nothing is there to take the request. Apache says this when the app isn't running.
• 504 Gateway Timeout: the app took too long.
• 500 is different: the app itself answered, and it crashed. Look at the app's logs.

So the first question for any 5xx is "who said it: the proxy or the app?" The Server header usually tells you.

HTTP versions

HTTP/1.1 is the readable text version above. HTTP/2 sends the same requests and headers in a compact binary form, many at once over one connection. HTTP/3 does the same over QUIC, which runs on UDP (lesson 4) to save time. The meaning (methods, status codes, headers) stays exactly the same, and it's always HTTPS (Security basics, lesson 8).

curl, the HTTP debugger

On the server side, every request ends up as one line in the web server's access log. Side by side, as usual:

Try it: the club's web stack 🌍

The club server runs two websites and an API behind Apache. Members say "the sign-up page is broken". Read the conversations, find out what each part says, and get the API answering.

Quick check