Old-School Networking

• UDP — an 8-byte header, boundaries that survive, empty ports, fragments (measured)
• Head-of-line blocking — what happens to later messages after one is lost, TCP vs UDP (measured)
• Sockets — socket, connect, accept, seen through strace (measured)
• Behind the server — the accept queue, and epoll for many clients (measured)

The practice network is the same as in part 6: a mischief wire with 20 ms each way between the PC (192.168.1.10) and the server (203.0.113.20). The wire got a small addition so it can drop UDP packets too.

Let's send the submission over UDP. The server runs a small program accepting submissions on UDP port 5000.

# One submission over UDP (20 ms each way)
pc$ python3 cli7.py udp
    sent 59 bytes, got 46 bytes in 40.9 ms: {"id": "EXP-2026-0100", "status": "submitted"}
# Only two packets crossed the wire, there and back (part 6's TCP took ten)
  0.000  pc  → srv  UDP 59437 → 5000  length 67  (frame 101 bytes)
  0.041  srv → pc   UDP 5000 → 59437  length 54  (frame 88 bytes)

Only two packets crossed the wire, there and back. No greeting, no ACK, no goodbye. The reply took 0.041 s: exactly one round trip. Part 6's TCP used ten packets for the same submission and 0.132 s to finish saying goodbye.

In the cover photo's details, the UDP header is just two ports, a length and a checksum. No Seq, no Ack, no window. UDP simply writes a room number on an IP packet, and guarantees nothing about whether it arrives or in what order.

Part 6 said TCP doesn't keep message boundaries. Let's check side by side with UDP. Send 100 bytes three times; the server waits 0.3 s before reading.

# PC: send 100 bytes three times (TCP set to send at once, TCP_NODELAY)
pc$ send(b'A' * 100); send(b'B' * 100); send(b'C' * 100)
# On the wire, both are three 100-byte packets
    TCP  Len=100  Len=100  Len=100      UDP  Len=100  Len=100  Len=100
# Server: wait 0.3 s, then read
    TCP recv() → [300] bytes
    UDP recv() → [100, 100, 100] bytes

On the wire, both were three 100-byte packets. Yet the TCP receiver got all 300 bytes in one recv(). Packet boundaries vanish once the data lands in the receive buffer. With UDP, each send is one letter (a datagram) and is received as one. When you design your own protocol over TCP, decide how to mark boundaries: a length first, newlines, and so on.

With TCP, connecting to an empty port got an RST. UDP has no RST. Instead, the other OS replies with an ICMP “port unreachable”.

# Send UDP to port 5009, where nobody listens. The server replies over ICMP: port unreachable
  0.000 192.168.1.10 → 203.0.113.20 UDP 47 33355 → 5009 Len=5
  0.040 203.0.113.20 → 192.168.1.10 ICMP 75 Destination unreachable (Port unreachable)
# A connect()ed socket receives that as an error; an unconnected one keeps waiting, unaware
    connected socket: ConnectionRefusedError: [Errno 111] Connection refused after 40.7 ms
    unconnected socket: TimeoutError: timed out after 2002 ms

The same notice arrived both times, but the program saw it in two ways. A socket that had connect()ed to that address got a Connection refused error in 0.041 s. A socket that just used sendto() couldn't match the notice to itself and waited for its 2-second timeout. A small tip: with UDP too, if you talk to one peer, connect() first. If the packet is lost on the way nothing comes back at all, though, so you always need a timeout.

A UDP letter can hold up to 65,507 bytes. But the wire's MTU is 1500 bytes (part 2). A letter that doesn't fit is cut into fragments by IP. We sent one 3000-byte letter.

# Send one 3000-byte UDP datagram (the wire's MTU is 1500)
  0.000 192.168.1.10 → 203.0.113.20 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=0, ID=56e0)
  0.000 192.168.1.10 → 203.0.113.20 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=1480, ID=56e0)
  0.000 192.168.1.10 → 203.0.113.20 UDP 82 58842 → 5000 Len=3000
  0.041 203.0.113.20 → 192.168.1.10 UDP 56 5000 → 58842 Len=14
    3000 bytes -> reply {"size": 3000} in 41.2 ms
# Drop the second fragment on the wire
    3000 bytes -> no reply (2 s timeout)
# 30 seconds later the server sends back “reassembly time exceeded”
  30.470 203.0.113.20 → 192.168.1.10 ICMP 590 Time-to-live exceeded (Fragment reassembly time exceeded)

The 3000 bytes plus the 8-byte header, 3008 bytes, went as three fragments of 1480, 1480 and 48 bytes and were reassembled at the server. But with the second fragment dropped on the wire, nothing was delivered, even though the other two arrived. The server waited 30 seconds, discarded the fragments and sent back “reassembly time exceeded” (at 30.47 s). UDP doesn't resend, so that's the end of it.

So with UDP, the rule of thumb is to keep each letter small enough to fit the link. In 2020 the DNS community agreed together on a default that keeps UDP replies to 1232 bytes (DNS Flag Day 2020; part 9).

# 50 messages of 100 bytes every 20 ms; the wire drops only message 10. Time to arrive (ms)
    No.      7      8      9     10     11     12     13     14     15
# UDP: 49 arrived. Message 10 is lost; the rest aren't held up
    UDP   20.6   20.6   20.3     —    20.3   20.4   20.4   20.4   20.4
# TCP: all 50 arrived. Message 10 was late (resent), and 11 and 12 waited for it
    TCP   20.4   20.4   20.3   80.9   60.6   40.5   20.6   20.3   20.6

With UDP, message 10 never arrived, and the other 49 all took about 20 ms. TCP delivered all 50, but message 10 waited for its resend: 80.9 ms. Worse, messages 11 and 12, which had already arrived, weren't handed to the app until 10 was in place: 60.6 and 40.5 ms. Keeping order means one missing message at the front holds up everything behind it: head-of-line blocking.

In voice, a missing 20 ms is barely noticeable; a late voice is the real nuisance. So voice and video calls and games use UDP and patch over gaps themselves. DNS is a short one-round-trip question, so it uses UDP too (part 9). And QUIC, used by HTTP/3, builds its own resending and ordering on top of UDP. It keeps order per stream inside the connection, so a loss in one stream doesn't block the others (part 10).

A server makes a doorway with socket(), picks a port with bind(), starts listening with listen() and takes clients one at a time with accept(). A client connects with connect(). Part 6's three-way handshake happens between connect and accept. In Python, that's all it takes.

# Server (the smallest form)
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)   # a TCP doorway; SOCK_DGRAM for UDP
s.bind(('0.0.0.0', 5000))
s.listen()
while True:
    c, addr = s.accept()          # take one client whose handshake is done
    data = c.recv(4096)
    c.sendall(b'ok')
    c.close()

# Client
c = socket.create_connection(('203.0.113.20', 5000))
c.sendall(b'hello'); print(c.recv(4096)); c.close()

To see whether the calls really come in this order, we used strace, a tool that lists what a program asks the OS to do. Top: the Python that sends one submission from the PC. Bottom: the Node.js that receives it.

# PC: strace on a Python script that sends one submission using a socket directly (right: seconds the call took)
  socket(AF_INET, SOCK_STREAM) = 3                 0.0001
  connect(3, {203.0.113.20:3000}, 16) = 0          0.0408
  sendto(3, "POST /api/ex…", 181) = 181            0.0001
  recvfrom(3, "HTTP/1.1 201…", 4096) = 175         0.0411
  recvfrom(3, "", 4096) = 0                        0.0001
  close(3) = 0                                     0.0001
# Server: what happens inside Node.js (epoll_pwait waits for “anything?”, then handles it)
  epoll_pwait(15, [fd 21 IN]) = 1                  1.4631
  accept4(21) = 22                                 0.0001
  epoll_pwait(15, [fd 22 IN]) = 1                  0.0000
  read(22, "POST /api/ex…", 65536) = 199           0.0000
  writev(22, ["HTTP/1.1 201…"]) = 203              0.0002
  epoll_pwait(15, []) = 0                          0.0001
  epoll_pwait(15, [fd 22 IN]) = 1                  0.0391
  read(22, "", 65536) = 0                          0.0001
  shutdown(22, SHUT_WR) = 0                        0.0002
  epoll_ctl(15, EPOLL_CTL_DEL, 22) = 0             0.0000
  close(22) = 0                                    0.0000

The 3 that socket() returned on the PC is a “file descriptor”, the same kind of number you get when opening a file. To the OS, a socket is a kind of file. connect() took 0.041 s, blocking the program for the handshake's round trip. Waiting for the reply, recvfrom() also took 0.041 s. The final, empty recvfrom() is the sign that the other side said goodbye (FIN).

The OS handles the greeting; the app just takes connections with accept(). So what if the app never takes them? We made four connections to a server that called listen(1) and never accepted.

# Four connections to a server that called listen(1) but never accept()s (clients give up after 3 s)
    client 1: connected in 42 ms
    client 2: connected in 41 ms
    client 3: TimeoutError after 3003 ms
    client 4: TimeoutError after 3003 ms
# Server side: two have finished the handshake and wait to be taken
srv$ ss -tn sport = :5005 ; ss -tln sport = :5005
    State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
    ESTAB 0 0 203.0.113.20:5005 192.168.1.10:48164
    ESTAB 0 0 203.0.113.20:5005 192.168.1.10:48160
    State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
    LISTEN 2 1 0.0.0.0:5005 0.0.0.0:*

The first two were told they were connected in 0.04 s, though the app had done nothing. The OS finished the handshake and queued them (Recv-Q 2 on the LISTEN line). Once the queue was full at 2 (the listen number plus one), the SYNs of the third and fourth were silently dropped; after resending, they timed out at 3 s.

When a server is too busy to keep up with accept, clients don't see “can't connect” but “slow” or “timed out”. The default listen number is 511 in Node.js and 128 in Python; Linux's cap (somaxconn) was 4096 on this kernel.

The Python server above can't serve anyone else while it waits in recv(). The old answer was a process or thread per client, but with thousands of clients that's far too heavy. In 1999, handling ten thousand connections on one machine was dubbed “the C10K problem”.

Today's answer is to let a watcher do the waiting. Register the listening socket and every client socket with Linux's epoll; one thread asks “anything ready?” and handles only what is. The bottom half of the strace above is exactly this: Node.js waited 1.46 s in epoll_pwait, a client arrived on the listening socket (21), accept4 took out 22, 22 became readable so it read, wrote the reply, and went back to epoll. Node.js's event loop and nginx both work this way.

Run netstat -an -p udp on a Mac and, even when you think you're doing nothing, you should see several UDP ports open, such as 5353 (mDNS, for finding machines on the same LAN).