Old-School Networking

• Port numbers — an address is a building, a port a room; a connection is four values
• Greetings and numbering — the handshake, Seq and Ack, refused vs. left waiting (measured)
• Retransmission — drop one packet, swap two (measured)
• Windows and congestion — a receiver that won't read, and how much is in flight on a crowded link (measured)
• Goodbyes — FIN, TIME-WAIT, RST

This part's practice network puts a small program the author wrote between the PC (192.168.1.10) and the server (203.0.113.20). It relays traffic and can delay it, limit bandwidth, drop chosen packets and reorder them: a “mischief wire”. Unless stated otherwise it adds 20 ms each way (40 ms round trip). Part 5's routers and NAT are left out to keep things focused.

A port number is 16 bits: 0 to 65535. Server programs listen on a fixed port, and the PC borrows a free one each time it connects.

# Ports the server is listening on (LISTEN)
srv$ ss -tln
    State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
    LISTEN 0 128 0.0.0.0:4000 0.0.0.0:*
    LISTEN 0 511 0.0.0.0:3000 0.0.0.0:*
# The range the PC picks its own port from (Linux default)
pc$ sysctl net.ipv4.ip_local_port_range
    net.ipv4.ip_local_port_range = 32768 60999

On the server, the Node.js expenses API waits on 3000 and the “slow server” made for this part on 4000. On the PC side, Linux picks from 32768–60999 (Windows and Mac use 49152–65535). This submission got 35594.

A connection is identified by four values: the address and port at each end. Two browser tabs on the same PC connecting to port 3000 on the same server are two connections, because the PC-side ports differ. If you see “Address already in use” while developing, something is already listening on that port.

Before sending any data, TCP sets up a connection with three greetings (the three-way handshake): “I'd like to connect” (SYN), “Go ahead, and so would I” (SYN+ACK), “Got it” (ACK). Here is one submission.

# One submission (20 ms each way; Seq and Ack counted from 0, as Wireshark shows them)
    0.000  pc  → srv  SYN      Seq=0    Ack=0    Len=0
    0.040  srv → pc   SYN+ACK  Seq=0    Ack=1    Len=0
    0.041  pc  → srv  ACK      Seq=1    Ack=1    Len=0
    0.041  pc  → srv  PSH+ACK  Seq=1    Ack=1    Len=198  POST /api/expenses
    0.081  srv → pc   ACK      Seq=1    Ack=199  Len=0
    0.090  srv → pc   PSH+ACK  Seq=1    Ack=199  Len=203  201 Created
    0.090  pc  → srv  ACK      Seq=199  Ack=204  Len=0
    0.090  pc  → srv  FIN+ACK  Seq=199  Ack=204  Len=0
    0.132  srv → pc   FIN+ACK  Seq=204  Ack=200  Len=0
    0.132  pc  → srv  ACK      Seq=200  Ack=205  Len=0
# the first three are the greeting, the last three the goodbye

The cover photo has that first SYN opened. A SYN also carries requests. MSS 1460 means “keep each segment's contents to 1460 bytes”: part 2's MTU of 1500 minus 20 bytes each for the IP and TCP headers. The two sides also agree here on SACK (reporting exactly which ranges arrived), timestamps and the window scale.

# curl's timings: 1 round trip to connect, 2 to the first byte of the reply
pc$ curl -w '…' -X POST … http://203.0.113.20:3000/api/expenses
    connect 0.0407 s   first byte 0.0900 s   total 0.0901 s   local port 35594
# the PC closed first, so it keeps a TIME-WAIT for 60 seconds
pc$ ss -tano dst 203.0.113.20
    TIME-WAIT 0 0 192.168.1.10:35594 203.0.113.20:3000 timer:(timewait,59sec,0)

curl's numbers show one round trip (0.041 s) to connect and two (0.090 s) to the first byte of the reply. The greeting alone costs a round trip. With a server 100 ms away, that's 0.1 s before anything happens. So reusing one connection is faster.

# Two submissions, connecting each time
    0.0827 s + 0.0824 s = 0.1652 s
# Two submissions over one reused connection (connects 0 the second time)
    0.0825 s + 0.0421 s = 0.1246 s

The PC's POST is 198 bytes, numbered 1 to 198. The server's Ack=199 means “I have everything up to 198; send 199 next”. The server's 203-byte reply has its own numbering, and the PC answered Ack=204. Each direction has its own sequence.

Wireshark starts them at 0 for readability, but real sequence numbers start at a random value (Sequence Number (raw) in the photo's details). That stops stray packets from an old connection slipping in, and stops outsiders guessing the numbers and injecting forged packets.

A greeting can fail in two ways. Connect to a port where nothing is listening and the server's OS sends back an RST (nobody here). If the SYN is dropped on the way, nothing comes back at all. We had the mischief wire drop SYNs to port 3002 and compared.

# To closed port 3001: the server answers RST (nobody here); refused in one round trip
pc$ curl http://203.0.113.20:3001/
    curl: (7) Failed to connect to 203.0.113.20 port 3001 after 40 ms: Couldn't connect to server
# To port 3002, where the wire drops SYNs: retries continue, and curl gives up at 20 s
pc$ curl --connect-timeout 20 http://203.0.113.20:3002/
    curl: (28) Failed to connect to 203.0.113.20 port 3002 after 20002 ms: Timeout was reached
# times the SYN was sent (s)
    0.0  1.0  2.0  3.0  4.1  5.1  7.1  11.3  19.5

The refusal took 0.041 s: one round trip. With no answer, the PC resent the SYN every second, then doubled the gap to 2, 4 and 8 seconds. This Linux (6.18) spaces the first few one second apart (tcp_syn_linear_timeouts = 4) and by default gives up after about two minutes.

We set up an upload of a 1 MB receipt image with the submission, limited the mischief wire to 10 Mbit/s, and dropped only the 40th data segment from the PC.

# Uploading a 1 MB receipt image, the wire drops the PC's 40th data segment (10 Mbit/s, 20 ms each way)
wire: up drop data#40
# The PC-side capture (numbers are Wireshark's packet numbers)
    58  0.1282  pc → srv  Seq=55169 Len=1448
   123  0.1759  srv → pc  Ack=55169  SACK 56617-58065
   128  0.1771  srv → pc  Ack=55169  (Dup ACK #1)
   130  0.1783  srv → pc  Ack=55169  (Dup ACK #2)
   131  0.1784  pc → srv  Seq=55169 Len=1448  (Fast Retransmission)
# Time taken: three runs without loss, one with
    1.007 s  1.007 s  1.006 s   /   1.008 s

Seq=55169 never arrives, so each later segment makes the server repeat the same Ack, “still missing from 55169”, with a SACK listing what did arrive. After three identical Acks, the PC resent just 55169 without waiting for a timeout (Fast Retransmission): 0.050 s after the loss, about one round trip. The whole upload took 1.008 s, barely different from three runs without loss (1.006–1.007 s).

Each segment carries 1448 bytes: MSS 1460 minus the 12-byte timestamps option. If no later segments follow, so the repeated Acks never come, TCP waits for a timeout (RTO) instead. Linux's minimum timeout is 0.2 s, so that case is noticeably slower.

When the order changes

# The wire delays the 40th segment by one place. Server-side capture:
    76  0.1375  pc → srv  Seq=56617 Len=1448  (arrived ahead of 55169)
    77  0.1376  srv → pc  Ack=55169  SACK 56617-58065
    78  0.1388  pc → srv  Seq=55169 Len=1448  (Out-Of-Order)
# The later segment arrives first; the server says “still missing from 55169, but I have 56617–58065”. Then 55169 arrives and fills the gap
# Retransmissions by the PC: 0

This time the 40th segment wasn't dropped, just moved back one place. The server put 56617, which arrived first, aside and replied “still missing 55169”, but 55169 arrived right after and filled the gap. The PC resent nothing. The app (Node.js) only ever sees the bytes back in order, so it never knows anything was swapped.

# Sending 1 MB to a server that doesn't read for 3 seconds (32 KB receive buffer)
    48  0.163  srv → pc  Win=0  Ack=40689  (ZeroWindow)
# The buffer fills: window 0. The PC peeks at intervals: “ready yet?”
    49  0.414  pc → srv  Len=0  (probe)
    51  0.966  pc → srv  Len=0  (probe)
    53  2.022  pc → srv  Len=0  (probe)
# After 3 seconds the server starts reading and the window opens
    55  3.082  srv → pc  Win=36200  (Window Update)
    curl: total 4.188 s

The server on port 4000 reads nothing for 3 seconds after connecting. Its 32 KB receive buffer filled at 40,688 bytes, and at 0.163 s the window hit 0 (ZeroWindow). The PC stopped sending and probed at growing intervals, 0.41, 0.97 and 2.02 s: “open yet?” At 3.08 s the server started reading, the window opened and sending resumed.

The window field is 16 bits, so on its own it can only say 64 KB. The SYN agrees a multiplier (Window scale: 10 in the cover photo means ×1024). This flow control keeps a fast sender from swamping a slow receiver. When Node.js's write() returns false, this window is ultimately behind it.

The window is about the receiver. There's also the road in between. Send too much down a thin link and packets overflow the queue in front of a router and are dropped. TCP infers how crowded the road is from the losses and adjusts how much it sends (congestion control).

With the mischief wire at 10 Mbit/s and a queue of at most 30 packets, we downloaded a 2 MB CSV from the server. The chart counts, every 10 ms from the server-side capture, the segments sent but not yet acknowledged.

# Downloading a 2 MB CSV (10 Mbit/s, 20 ms each way, at most 30 packets queued at the wire)
# Segments the server sent in each round trip (41 ms)
     10   20   40   51   36   36   35   36  …
# The server mid-transfer (ss -ti; cwnd is in segments)
    cubic  mss:1448  cwnd:56  ssthresh:46  rtt:66.713 ms  minrtt:40.738 ms
    queue drops 4  retransmissions 4   curl: 1.884 s, 8.90 Mbit/s

The first round trip sent 10 segments, the next 20, then 40: doubling every round trip, called slow start (the initial 10 comes from RFC 6928). When the link and queue could hold no more, four overflowed; after resending them the server cut its sending to about 70 percent (ssthresh 46), then grew slowly. That's cubic, Linux's default.

Note the gap between rtt:66.7 and minrtt:40.7 mid-transfer. Waiting in the queue added 26 ms to the round trip. Queues so deep that delay balloons are called bufferbloat. Some schemes, like Google's BBR, don't wait for losses but read congestion from rising round-trip times.

The last three packets of the submission are the goodbye. The PC sent FIN (nothing more to send), the server answered with its own FIN, and the PC finished with an ACK. Each direction is closed separately, so it's really four messages, but the server's ACK and FIN went together in one.

The side that closes first keeps a TIME-WAIT state for 60 seconds (the timer:(timewait,59sec) above). It's there to answer again if the last ACK is lost, and to stop late packets from an old connection leaking into a new one with the same ports. Programs that open huge numbers of short connections can pile up tens of thousands of TIME-WAITs and run out of PC-side ports: another reason to reuse connections.

An abrupt end with no goodbye is an RST. Besides connecting to a port nobody listens on (section 5), you also get one when the program at the other end crashes. Connection reset by peer means an RST came from the other side.

Run netstat -an with a few websites open and you should see a row of ESTABLISHED connections to port 443, with recently closed ones in TIME_WAIT.