Skip to content
Tutorials Center
My learning

Topics

Lesson 69 of 79 ยท Tools and Troubleshooting

Troubleshooting Network Problems

Learn a layered method to troubleshoot networks: tell DNS, routing and firewall problems apart, diagnose MTU and MSS issues, and separate latency from loss.

"The network is down" can mean a hundred different things: a loose cable, a dead DHCP server, a wrong DNS entry, a firewall rule, a failed route, a path that drops big packets, or just a server that is slow. Good troubleshooting is not about memorizing fixes. It is about using a method that narrows the possibilities quickly, using the tools from essential network tools and packet capture.

This lesson gives you that method, built on the layers of the OSI and TCP/IP models, then covers the failure patterns that fool people most often: DNS versus routing versus firewall, MTU and MSS problems, and telling latency from packet loss. Command output in this lesson is sample output, since a web page cannot probe a real network.

A layered method

Start by defining the problem precisely: What exactly fails? Since when? For whom (one user, one subnet, everyone)? What changed recently? "Only the laptop on Wi-Fi cannot open the intranet since this morning" already excludes many causes.

Then choose a direction and test one layer at a time:

  • Bottom-up: physical, link, network, transport, application. Good when nothing works at all.
  • Top-down: start with the application error and work down. Good when most things work.
  • Divide and conquer: begin in the middle (can you ping an IP address?) and move up or down depending on the result. Often the fastest.
flowchart TD
    A["Start: a site will not load"] --> B{"Link up and has<br/>an IP address?"}
    B -->|no| B1["Layers 1-2 and DHCP:<br/>cable, Wi-Fi, switch port,<br/>169.254 address"]
    B -->|yes| C{"Can you ping the<br/>default gateway?"}
    C -->|no| C1["Local LAN problem:<br/>VLAN, ARP, gateway down"]
    C -->|yes| D{"Can you ping a public<br/>IP such as 1.1.1.1?"}
    D -->|no| D1["Routing, ISP or firewall:<br/>traceroute to find the break"]
    D -->|yes| E{"Does the name<br/>resolve?"}
    E -->|no| E1["DNS: resolver, search domain,<br/>record or cache"]
    E -->|yes| F{"Does the port<br/>connect?"}
    F -->|no| F1["Firewall, service down,<br/>wrong port or security group"]
    F -->|yes| G["Application layer:<br/>TLS, HTTP status, logs, load"]

Each question rules out a whole group of causes. Stop at the first "no" and investigate there.

A few habits make the process reliable: change one thing at a time, write down what you tried, compare with a machine that works, check the simplest explanation first (cables, power, typos in addresses), and undo changes that did not help.

Step by step with commands

TEXT
# 1. Link and address
$ ip addr show              # does the interface have an IP address?
inet 192.168.1.20/24 ...

# 2. Local network and gateway
$ ping -c 2 192.168.1.1     # reaches the router?
2 packets transmitted, 2 received, 0% packet loss

# 3. Beyond the gateway, by number (no DNS involved)
$ ping -c 2 1.1.1.1
2 packets transmitted, 0 received, 100% packet loss      <- stop here, routing/ISP

# 4. Name resolution
$ dig example.org +short
;; connection timed out; no servers could be reached     <- DNS server unreachable

# 5. The service itself
$ curl -sv --max-time 5 https://example.org -o /dev/null
* connect to 203.0.113.25 port 443 failed: Connection timed out

Testing by IP address first is a powerful trick: if ping 1.1.1.1 works but ping example.org does not, the network is fine and the fault is DNS.

Telling DNS, routing and firewall apart

Three different problems all appear to a user as "cannot reach it". Their symptoms differ if you look carefully.

Symptom Likely cause Clue
Name or service not known, NXDOMAIN DNS record missing or wrong name Fails immediately, before any connection is attempted
Long wait, then could not resolve host DNS server unreachable or slow dig @8.8.8.8 name works but your default resolver does not
Network is unreachable or No route to host No route for that destination, or ARP failed on the local segment Immediate error, ip route has no default gateway
Packets go out and stop at some hop Routing break or ISP fault traceroute ends with * * * and never reaches the target
Connection refused Host reached, nothing listening on that port (RST received) Fast failure, different from a timeout
Connection timed out Packets dropped silently: firewall, security group, or host down Slow failure; SYN sent, no answer
Connects, then resets or hangs Application, TLS or MTU issue Handshake completes but data stalls

The difference between refused and timed out is the single most useful distinction. A refusal means the packet arrived and the host's TCP stack said no, so the network path works. A timeout means no answer came back, so something dropped the packet and the path (or a firewall) is the suspect. A rejected packet from a firewall (REJECT) looks like a refusal; a dropped one (DROP) looks like a timeout.

Other classic faults: a wrong subnet mask (the host thinks a neighbor is remote, or the reverse), a wrong default gateway, a duplicate IP address (intermittent, ARP flapping), an expired DHCP lease, a stale DNS cache (the TTL has not expired yet), and IPv6 versus IPv4 differences (the name has an AAAA record but IPv6 is broken, so connections hang before falling back).

Python A diagnostic checklist that walks the layers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

MTU and MSS problems

The MTU is the largest IP packet a link can carry (1,500 bytes on Ethernet); the MSS is the largest TCP payload, which is the MTU minus 40 bytes of IPv4 and TCP headers (so 1,460). A path crossing a tunnel, VPN, PPPoE link or similar encapsulation has a smaller MTU, because the extra header takes room. Normally this is handled automatically by path MTU discovery: a router that cannot forward a packet marked "do not fragment" sends back an ICMP "fragmentation needed" message and the sender shrinks its packets.

The failure happens when a firewall blocks ICMP and discards those messages. The result is a very confusing symptom, sometimes called a PMTU black hole:

  • Small packets work: the TCP handshake completes, DNS works, a short request is answered.
  • Large packets vanish: a page loads halfway, an upload hangs, an SSH session works until you list a big directory, a TLS handshake stalls at the certificate.

You can test directly with pings that set the do-not-fragment bit:

TEXT
$ ping -M do -s 1472 example.org        # Linux: 1472 + 28 header bytes = 1500
ping: local error: message too long, mtu=1400       <- the path (here a VPN) allows only 1400
$ ping -M do -s 1372 example.org
64 bytes from 203.0.113.25: icmp_seq=1 ttl=57 time=19.3 ms

(On Windows use ping -f -l 1472; on macOS ping -D -s 1472.) The largest size that works plus 28 bytes gives the path MTU. The fixes are to allow the needed ICMP messages through firewalls, to lower the MTU on the tunnel interface, or to clamp the MSS on the router so TCP never builds packets that are too big (the router rewrites the MSS option in SYN packets, which you can see in a capture).

Python Find the MSS to use for a tunnel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Latency versus loss

Slow is not one problem but two, with different causes and fixes.

  • High latency means each round trip takes long. Causes: distance (speed of light in fiber is roughly 200,000 km/s), queuing in congested routers (bufferbloat), or a slow server. Latency hurts anything that waits for replies: page loads that need many round trips, remote shells, games, calls.
  • Packet loss means some packets never arrive. Causes: congestion, bad cables or Wi-Fi interference, overloaded devices. TCP reacts to loss by shrinking its window (congestion control), so even 1% loss can cut throughput dramatically, and loss makes everything feel stalled.
  • Jitter is variation in latency, which hurts voice and video.

A useful rule of thumb (the Mathis formula) says TCP throughput is at most about MSS / (RTT x sqrt(loss)) multiplied by a constant near 1.2. Notice what it implies: throughput falls with RTT and with the square root of loss.

TEXT
$ mtr -rwc 50 example.org
Host                   Loss%  Snt  Last   Avg  Best  Wrst StDev
1. 192.168.1.1          0.0%   50   1.1   1.2   1.0   2.0   0.2
2. 10.20.0.1            0.0%   50   8.4   8.6   8.0  12.1   0.8
3. 198.51.100.9        24.0%   50  14.2  14.5  13.9  15.0   0.3     <- loss starts here...
4. 203.0.113.1         24.0%   50  18.1  18.4  17.8  19.5   0.4     <- ...and continues to the end
5. 203.0.113.25        23.0%   50  18.6  18.8  18.2  20.1   0.5

When loss appears at a hop and continues at all later hops, that hop (or the link after it) is the real culprit. If loss shows at one hop only and later hops are clean, that router just rate-limits its replies to probes and is not dropping your traffic.

Python How latency and loss limit TCP throughput

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table that prints shows the lesson in numbers: a path with 1% loss at 150 ms round trip carries only about 1 Mbps per connection, however fast the link is.

Other common failure patterns

  • Works for some sites only: DNS filtering, a split-tunnel VPN, an IPv6 problem, or a proxy requirement.
  • Intermittent drops every few minutes: Wi-Fi roaming or interference, a duplicate IP address, DHCP lease problems, an overloaded NAT table (NAT).
  • Slow first connection, fast afterwards: DNS timeouts (a dead first resolver in the list), cold caches, or IPv6 fallback delays.
  • Internal works, Internet does not: default route, NAT, ISP outage.
  • Works by IP, not by name: DNS, search domains, or a hosts-file entry overriding the real address.
  • Everything stalls under load: flow control (receiver window), congestion, bufferbloat, a saturated link.

When you are stuck, capture packets on both ends. The capture rarely lies, and it usually points at the layer where the conversation stops.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A connection attempt fails instantly with 'Connection refused'. What does that tell you?

    A refusal is a TCP RST from the destination, so packets travel both ways. A timeout is what silent drops produce.
  2. 2. ping 1.1.1.1 works but ping example.org says 'Name or service not known'. Where is the fault?

    The network path works since an IP address answers; only the name lookup fails.
  3. 3. Small web pages load but large downloads and uploads hang. Which cause fits best?

    Small packets fit through the narrow link while full-size packets are dropped, and blocked ICMP prevents the sender from learning the smaller MTU.
  4. 4. In an mtr report, loss of 24% starts at hop 3 and is also seen at hops 4 and 5. Where is the problem?

    Loss that persists on all later hops is real forwarding loss starting at that point. Loss at one hop only usually means the router de-prioritizes probe replies.
  5. 5. According to the Mathis relationship, what happens to maximum TCP throughput if the RTT doubles and loss stays the same?

    Throughput is proportional to 1/RTT, so doubling the round-trip time halves the achievable rate.

Summary

  • Define the problem, then test layer by layer (bottom-up, top-down or divide and conquer), changing one thing at a time.
  • Ping by IP address separates network faults from DNS faults. "Refused" means a host answered; "timed out" means packets were dropped.
  • DNS, routing and firewall problems differ in speed of failure, error text and where traceroute stops.
  • MTU black holes cause "small works, large hangs". Test with do-not-fragment pings and fix with ICMP, lower tunnel MTU or MSS clamping.
  • Latency and loss are different problems; TCP throughput falls with RTT and with the square root of loss. Loss that continues down a path marks the real culprit.

Now put it all together in one story: what happens when you type a URL.