Skip to content
Tutorials Center
My learning

Topics

Lesson 80 of 95 ยท System Administration

Network Troubleshooting: ip, ss, tcpdump, nmap, dig

Diagnose Linux network problems layer by layer with ip, ping, traceroute, ss, dig, curl, tcpdump and nmap, and learn to read routes, sockets and packet traces.

"The site is down" and "I cannot connect" are the two sentences that start most network investigations. The temptation is to try random things. A calmer method works far better: go through the layers from the bottom up, and at each layer ask one yes-or-no question. Is the cable or Wi-Fi link up? Do I have an address? Can I reach my gateway? Can I resolve the name? Is the port open? Does the application answer? The first "no" tells you where the problem lives, and the right tool for that layer tells you why.

In Networking Basics you met ping, curl and the idea of ports. This lesson adds the diagnostic toolbox of a system administrator: ip, ping, traceroute, ss, dig, tcpdump and nmap. The sandbox has no network and none of these tools installed, so the real commands are read-only examples that show typical output. We still practise the reading skills with data you can process yourself: a routing-table lookup, an ss listing, a packet capture and a layer-by-layer decision script.

flowchart TD
    A["1. Link up?<br/>ip link"] -- yes --> B["2. Address?<br/>ip addr"]
    B -- yes --> C["3. Gateway reachable?<br/>ip route, ping"]
    C -- yes --> D["4. Name resolves?<br/>dig, getent hosts"]
    D -- yes --> E["5. Port open?<br/>nc, ss, curl"]
    E -- yes --> F["6. App correct?<br/>curl -v, logs"]
    A -- no --> X["Fix cable, Wi-Fi, driver"]
    B -- no --> Y["Fix DHCP or static config"]
    C -- no --> Z["Fix route, VLAN, gateway"]
    D -- no --> W["Fix resolver or DNS record"]
    E -- no --> V["Fix firewall or service"]

Work from the bottom layer up; the first failure is the layer to investigate.

ip replaced the old ifconfig, route and arp. Add -br for a brief table and -c for colour.

CONSOLE
$ ip -br link
lo      UNKNOWN   00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0    UP        52:54:00:ab:cd:ef <BROADCAST,MULTICAST,UP,LOWER_UP>

$ ip -br addr
lo      UNKNOWN   127.0.0.1/8 ::1/128
eth0    UP        192.168.1.10/24 fe80::5054:ff:feab:cdef/64

$ ip -s link show eth0        # error and drop counters
$ ip neigh                    # the ARP/neighbour table (IP to MAC addresses)

Two details matter. UP with LOWER_UP means the interface is enabled and a carrier (cable, Wi-Fi association) is present; UP without LOWER_UP means no link. And an address like 192.168.1.10/24 also states the network: the /24 says that all addresses 192.168.1.0 to 192.168.1.255 are on the same local segment. An address starting 169.254. is a link-local fallback, a classic sign that DHCP failed.

Layer 3: routes

When the system sends a packet it looks up the destination in the routing table and uses the longest matching prefix: the most specific route wins, and default (0.0.0.0/0) matches everything as a last resort.

CONSOLE
$ ip route
default via 192.168.1.1 dev eth0 proto dhcp metric 100
10.0.0.0/8 via 10.1.0.1 dev eth1
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.10

$ ip route get 8.8.8.8
8.8.8.8 via 192.168.1.1 dev eth0 src 192.168.1.10 uid 1000

ip route get ADDRESS is the most useful single command here: it shows the route, the interface and the source address the kernel would pick. We can model the lookup ourselves, which shows why a missing or overly broad route misdirects traffic:

Bash Longest-prefix route lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Note that 10.1.7.7 matches both 10.0.0.0/8 and 10.1.0.0/16, and the /16 wins because it is more specific. When a host can reach the local network but nothing beyond, look for a missing default route. When it can reach some remote networks but not others, look for a more specific route that sends traffic the wrong way. Add a temporary route with sudo ip route add 172.20.0.0/16 via 192.168.1.254 (gone after a reboot).

Reachability: ping, traceroute and mtr

ping sends ICMP echo requests. Always limit the count in scripts (-c 3) and test in a sensible order: yourself, the gateway, an outside IP address, and finally a name.

CONSOLE
$ ping -c 3 127.0.0.1          # is the network stack working at all?
$ ping -c 3 192.168.1.1        # the gateway
$ ping -c 3 1.1.1.1            # the internet, no DNS involved
$ ping -c 3 example.com        # the internet, with DNS
$ traceroute -n example.com    # each router on the path (or: tracepath)
$ mtr -rwn -c 10 example.com   # traceroute plus loss and latency per hop

Reading the results: pinging an IP address that works while a name fails points to DNS. A ping that works while the web page does not usually means a service or firewall issue, since ICMP and TCP are separate. Many networks block ICMP, so a failed ping alone proves nothing. In a traceroute, * * * for one hop in the middle is usually harmless (a router that ignores probes); what counts is where the path stops for good and where the latency suddenly jumps.

More ping and traceroute options

Option Meaning
ping -c 5 Stop after 5 requests (without it, ping runs until Ctrl+C)
ping -i 0.5 Wait 0.5 seconds between requests (the default is 1; intervals below 0.2 s need root)
ping -W 2 Give up on each reply after 2 seconds; -w 10 ends the whole run after 10 seconds
ping -s 1400 Payload size in bytes; with -M do (do not fragment) it finds the largest packet that fits, i.e. the MTU
ping -a Audible ping: a beep for every reply
ping -q Quiet: print only the summary
ping -4 / -6 Force IPv4 or IPv6
traceroute -n No name lookups (faster, clearer)
traceroute -m 20 Maximum 20 hops (default 30)
traceroute -q 1 One probe per hop instead of three
traceroute -I / -T Use ICMP or TCP probes when UDP probes are filtered

The summary line 3 packets transmitted, 3 received, 0% packet loss and the rtt min/avg/max/mdev line carry the facts: loss means a broken or congested path; a high mdev (jitter) means unstable latency. What drives hop times: physical distance (about 5 ms per 1,000 km one way in fibre), network congestion, router load (a slow reply from a middle router does not mean the path is slow, because routers deprioritise answers to probes), and the number of hops.

Network card details with ethtool

ip shows the logical interface; ethtool talks to the network card and its driver. It is not installed in our sandbox and needs root for most functions.

CONSOLE
$ ethtool eth0                  # speed, duplex, link detected, auto-negotiation
Settings for eth0:
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Link detected: yes
$ sudo ethtool -s eth0 speed 100 duplex full autoneg off    # force the link mode
$ ethtool -i eth0               # driver name and version, firmware, bus address
$ ethtool -S eth0               # the card's own error and drop counters
$ ethtool -k eth0               # offload features (checksum, TSO, GRO)
$ sudo ethtool -p eth0 5        # blink the port LED for 5 s to find the cable

A link that shows 100Mb/s or Half duplex when you expect gigabit full is a classic cause of slow transfers: bad cable, bad port or a failed auto-negotiation. Link detected: no means the cable or the switch port.

Reading ifconfig and the older tools

You will meet the net-tools commands in old books and on old systems. They still work where installed, but ip replaces them:

Old Modern Purpose
ifconfig ip addr, ip link Interfaces and addresses
route -n ip route Routing table
netstat -tulpn ss -tulpn Listening sockets
arp -a ip neigh ARP (neighbour) table
iwconfig iw Wireless

ifconfig output has counters worth reading. RX is received and TX transmitted. errors and dropped should be 0 or tiny; overruns means the card's buffer filled before the kernel emptied it (too much traffic or too slow a host); collisions should be 0 on modern full-duplex links. ifconfig eth0 up and ifconfig eth0 down enable and disable an interface (today: ip link set eth0 up). To find your public address, ask an outside service: curl -s https://icanhazip.com or dig +short myip.opendns.com @resolver1.opendns.com.

route manages the routing table in the old way. These lines map to ip route:

CONSOLE
$ route -n                                     # table without name lookups
$ sudo route add default gw 192.168.1.1        # ip route add default via 192.168.1.1
$ sudo route add -net 10.8.0.0 netmask 255.255.255.0 gw 192.168.1.5 dev eth0 metric 10
$ sudo route del -net 10.8.0.0 netmask 255.255.255.0

DNS: dig, host and getent

Names are resolved before connecting, so DNS is a very common culprit. The tools to know:

CONSOLE
$ dig example.com                  # full answer, with TTL and the server used
$ dig +short example.com           # just the address
$ dig example.com MX +short        # mail servers
$ dig @1.1.1.1 example.com         # ask a specific resolver
$ dig -x 93.184.216.34             # reverse lookup
$ dig +trace example.com           # walk from the root servers down
$ getent hosts example.com         # what the SYSTEM resolves (uses /etc/hosts too)

The difference between dig and getent hosts matters: dig talks to DNS servers directly, while getent (and every normal program) follows /etc/nsswitch.conf, which can consult /etc/hosts first. If dig works but curl fails by name, check getent hosts and /etc/hosts for a stale entry. Look at the status: line of a dig answer: NOERROR is fine, NXDOMAIN means the name does not exist, and SERVFAIL means the DNS server itself failed. The number in an answer line is the TTL, how many seconds resolvers may cache it, which explains why a changed record seems to take effect "randomly".

nslookup, host and the record types

nslookup is the older, simpler lookup tool. It has a one-shot mode and an interactive prompt, and it asks for record types with -type=:

CONSOLE
$ nslookup example.com                      # A and AAAA records via the configured resolver
$ nslookup -type=mx example.com             # mail exchangers (with priority numbers)
$ nslookup -type=ns example.com             # the authoritative name servers
$ nslookup -type=soa example.com            # start of authority: serial, refresh, retry, expire, minimum TTL
$ nslookup -type=txt example.com            # text records (SPF, domain verification)
$ nslookup -type=any example.com            # everything the server is willing to list (often refused today)
$ nslookup -port=5353 example.com 127.0.0.1 # another port and another server
$ nslookup 93.184.216.34                    # reverse lookup (PTR)
Record Holds dig equivalent
A / AAAA IPv4 / IPv6 address of a name dig example.com A
MX Mail servers, lowest priority number is tried first dig example.com MX +short
NS Name servers responsible for the zone dig example.com NS +short
SOA Zone serial number and timers dig example.com SOA +short
TXT Free text: SPF, DKIM, verification codes dig example.com TXT +short
CNAME An alias to another name dig www.example.com CNAME +short
PTR Name for an IP address (reverse DNS) dig -x 93.184.216.34 +short

host example.com prints the same facts in one friendly line per record (host -t mx example.com for one type). Use dig for details you need to read closely (flags, TTLs, authority section) and host or nslookup for a quick answer; dig is installed from bind9-dnsutils (Debian) or bind-utils (Red Hat). arp -a or ip neigh shows the layer-2 neighbours the host has talked to recently, i.e. the IP to MAC address mapping.

nmcli output can be tuned the same way: -t (terse, colon-separated, for scripts), -p (pretty), -m tabular|multiline and -f NAME,DEVICE,STATE (choose the fields), for example nmcli -t -f NAME,DEVICE connection show --active. See Network Configuration.

Layer 4: sockets and ports with ss

ss (socket statistics) replaced netstat. Use it to see what is listening on this machine and which connections exist.

CONSOLE
$ ss -tulpn                    # TCP + UDP, listening, with process, numeric
$ ss -tn state established     # open TCP connections
$ ss -tn '( dport = :443 )'    # filter by port
$ ss -s                        # summary counts

The flags: -t TCP, -u UDP, -l listening only, -p show the process (needs root for other users' processes), -n numbers instead of names. The most important thing to read is the local address. 127.0.0.1:6379 accepts connections only from the same machine, while 0.0.0.0:3306 or *:3306 accepts them on every interface. A database that should be private but listens on 0.0.0.0 is a security issue, and a service that listens only on 127.0.0.1 explains why remote clients cannot connect. Here is a script that classifies a captured ss -tulpn listing:

Bash Reading ss output: who is exposed?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

To test a remote port without special tools, Bash itself can open TCP connections through /dev/tcp. A refused connection fails at once, while a filtered one hangs until the timeout, and that difference is a clue (see the capture below):

Bash Probing a port with /dev/tcp

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

On a real network the usual choices are nc -zv host 443 (netcat) or curl -v telnet://host:443.

Layer 7: curl for the application

When the port is open, check what the application says. curl is the diagnostic Swiss army knife:

CONSOLE
$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com
$ curl -I https://example.com                 # headers only
$ curl -v https://example.com                 # connection, TLS and request/response
$ curl --connect-timeout 5 -m 15 URL          # do not hang forever
$ curl --resolve example.com:443:203.0.113.5 https://example.com   # bypass DNS for one test
$ openssl s_client -connect example.com:443 -servername example.com  # certificate details

--resolve is especially useful: it lets you test a new server before DNS points to it. With -w you can print timing for each phase (%{time_namelookup}, %{time_connect}, %{time_appconnect}, %{time_starttransfer}), showing whether time goes to DNS, the TCP handshake, TLS or the application.

Seeing packets with tcpdump

When tools disagree, look at the wire. tcpdump prints packets as they pass an interface. It needs root, and filters keep the output manageable:

CONSOLE
$ sudo tcpdump -i eth0 -nn host 192.168.1.20 and port 3306
$ sudo tcpdump -i any -nn -c 20 port 53          # DNS traffic, 20 packets
$ sudo tcpdump -i eth0 -nn -w capture.pcap       # save for Wireshark
$ sudo tcpdump -nn -r capture.pcap 'tcp[tcpflags] & tcp-syn != 0'

-nn suppresses name lookups (slow, and confusing when DNS is the problem). The most informative field is the TCP flags: [S] is a SYN (connection attempt), [S.] a SYN-ACK (accepted), [.] an ACK, [P.] data, [F.] a close, and [R.] a reset (refused). The three patterns to recognize:

  • SYN, SYN-ACK, ACK: the connection works.
  • SYN, then a reset: nothing listens on that port (or a firewall rejects).
  • SYN, SYN again, SYN again, no answer: a firewall silently drops the packets, or routing is broken.

We can automate that reading. The capture below is sample text in the format tcpdump prints; the script classifies each flow:

Bash Classifying connection attempts in a capture

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Scanning with nmap

nmap discovers hosts and open ports from the outside, which is how you check what your firewall really exposes. Only scan machines you own or have written permission to test; scanning others can be illegal and sets off alarms.

CONSOLE
$ nmap -p 22,80,443 server.example.com      # specific ports
$ nmap -sT -p- 192.168.1.20                 # every TCP port, full connect scan
$ nmap -sV -p 80,443 192.168.1.20           # detect service versions
$ nmap -Pn 192.168.1.20                     # skip the ping step (host ignores ICMP)
$ nmap -sn 192.168.1.0/24                   # who is alive on the network

Results use three main states: open (something answers), closed (the host replies "nothing here", an RST), and filtered (no reply: a firewall drops the probe). That is the same distinction as in the tcpdump patterns above, and it is why scanning from outside and comparing with ss -tulpn inside is a fast way to find a firewall gap. See Firewalls: ufw, iptables and nftables.

Putting it together

The decision script below encodes the layered method. It takes a described situation (what each check reports) and prints the first failing layer and what to look at. In real life each variable would be filled by one of the commands above.

Bash Find the first failing layer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Common patterns and what they mean

Symptom Likely cause Check
ping 1.1.1.1 works, ping example.com fails DNS dig, /etc/resolv.conf, Network Configuration
Connection refused (instant) Nothing listens, or REJECT rule ss -tulpn on the server
Connection times out Firewall DROP, wrong route, server down tcpdump, traceroute, firewall rules
Works from some clients only Service bound to a specific interface, ACL or MTU issue ss, source-based rules
Small requests work, large downloads hang MTU / path MTU discovery blocked ping -M do -s 1472 host
Works by IP, certificate error by name TLS name mismatch, wrong virtual host openssl s_client, curl --resolve
Intermittent slowness Packet loss, DNS timeouts, saturated link mtr, ip -s link, ss -s

Keep notes as you go: write down each check and its result. That record stops you from repeating tests and gives a colleague something concrete to read.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. ping 1.1.1.1 succeeds but ping example.com fails with "Temporary failure in name resolution". Which layer should you investigate first?

    Reaching an IP address proves link, address and routing work. Only the name lookup fails, so check the resolver configuration and DNS records.
  2. 2. What does 0.0.0.0:3306 in the Local Address column of ss -tulpn tell you?

    0.0.0.0 means all interfaces. A service meant to be private should listen on 127.0.0.1 instead.
  3. 3. In a tcpdump capture you see three [S] packets to port 443, spaced 1, 2 and 4 seconds apart, and no reply. Most likely:

    SYN retransmissions with exponential back-off and no SYN-ACK or RST mean the packets never get an answer: filtered or unroutable.
  4. 4. Which of these routes does ip route get 10.1.7.7 choose if both 10.0.0.0/8 via 10.1.0.1 and 10.1.0.0/16 dev eth1 exist?

    Routing picks the most specific (longest prefix) match, and a /16 is more specific than a /8.
  5. 5. Which tool shows what the system itself resolves for a name, including entries in /etc/hosts?

    getent hosts follows nsswitch.conf like normal programs. dig queries DNS servers directly and ignores /etc/hosts.

Summary

  • Troubleshoot bottom-up: link, address, gateway, DNS, port, application. The first failing check names the layer.
  • ip -br link, ip -br addr, ip route and ip route get show interfaces, addresses and the longest-prefix route a packet would take.
  • ping and traceroute/mtr test reachability; remember that ICMP may be blocked. dig queries DNS directly, getent hosts shows what programs actually resolve.
  • ss -tulpn lists listeners and their scope; 127.0.0.1 is private, 0.0.0.0 is exposed. curl -v and --resolve test the application layer.
  • tcpdump -nn shows flags: SYN with SYN-ACK is success, SYN with RST is refused, SYN with silence is filtered.
  • nmap reports open, closed and filtered ports; scan only your own systems.

Next: Network Configuration: NetworkManager, netplan, DNS