Skip to content
Tutorials Center
My learning

Topics

Lesson 87 of 147 ยท Standard Library

URLs and Network Basics

Parse and build URLs with net/url, handle query strings and escaping, check IP addresses and CIDR ranges, and write a TCP echo server and client on loopback.

Almost every Go program that talks to the outside world starts with the same two questions: where is the thing I want to reach, and how do I reach it? The first question is answered by URLs and IP addresses, the second by sockets. The standard library covers both with three packages: net/url for URLs and query strings, net (together with the newer net/netip) for addresses and connections, and, built on top of them, net/http, which later lessons cover in depth.

This lesson stays below HTTP. You will learn how to take a URL apart and put one together without string concatenation bugs, how escaping works, how to relate a relative link to a base URL, how to check IP addresses and CIDR ranges, and how a TCP server and client talk to each other. Everything runs on the loopback address 127.0.0.1, because the sandbox on this site (like many servers) can only connect to itself.

Parsing a URL

A URL such as https://shop.example.com:8443/books?page=2#reviews has a well-defined anatomy: scheme, optional user information, host, optional port, path, query and fragment. url.Parse splits a string into a *url.URL whose fields mirror those parts:

Go The parts of a URL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

A few details are worth noticing:

  • Host includes the port (shop.example.com:8443); use Hostname() and Port() to split them. They also handle IPv6 literals such as [::1]:8080 correctly.
  • Path is the decoded path (/books/go in depth). EscapedPath() returns the encoded form that belongs on the wire. RawPath stays empty unless the original encoding differs from the default encoding of Path, which is rare.
  • RawQuery is the query string as written, without the ?. It is not parsed for you; that is the job of url.Values, shown next.
  • Redacted() replaces the password with xxxxx, so it is the safe version to log. Never print u.String() for URLs that may contain credentials.
  • The fragment (#reviews) is for the browser. Clients do not send it to the server, so a Go HTTP server never sees it.

url.Parse is lenient. It accepts a relative reference like /books?page=2 or even a bare word, and it reports errors only for things that are definitely malformed. If you need an absolute URL, check u.IsAbs() (is there a scheme?) and u.Host != "" yourself. A user-supplied string is not trustworthy just because it parsed.

Query strings with url.Values

url.Values is a map[string][]string: every key can have several values, which is how the web has always worked (?tag=go&tag=c). It comes with methods for the common operations, and url.ParseQuery builds one from a raw query string.

Go Reading and writing query strings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The important rules:

  • Get returns the first value, or an empty string if the key is missing. It cannot tell "missing" from "present but empty"; Has can.
  • Set replaces all values for a key, Add appends, Del removes. To read every value, index the map directly (q["q"]).
  • Encode produces a query string sorted by key, with every key and value escaped. That makes its output deterministic, which is handy for tests and for signing requests.
  • A space becomes + in a query string but %20 in a path. That is why there are two escape functions: QueryEscape for query components and PathEscape for one path segment. Note how PathEscape leaves = and & alone (they are legal in a path) but encodes / and ?.
  • Parsing a malformed escape, such as a lone %, is an error. ParseQuery returns the pairs it could read and an error for the rest.

Never build URLs with "https://api.example.com/search?q=" + userInput. A value containing &, # or a space changes the meaning of the URL, and with the wrong input, attackers can inject extra parameters. Put the value into url.Values and call Encode, or build a url.URL as shown in the next section.

Building and resolving URLs

You can create a url.URL struct directly, set the parts you know and let String() assemble them with the right escaping:

Go Building, resolving and joining URLs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

ResolveReference implements the same rules a browser uses for links on a page: a relative path replaces the last segment of the base, .. goes up one directory, a path starting with / replaces the whole path, a reference starting with // keeps only the scheme, and a lone ?query or #fragment modifies the base. Web crawlers rely on this to turn every href on a page into an absolute address (the web crawler project does exactly that).

url.JoinPath joins path elements and cleans up the slashes between them, so you do not end up with // or a missing /. It also resolves . and .. elements, so do not pass untrusted segments to it and assume they stay inside a directory; validate them first.

Errors and the opaque surprise

url.Parse reports problems with a *url.Error-style message that includes the offending input. Here are a few real ones, followed by a famous trap:

Go Bad URLs and the scheme trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Two lessons here. First, a URL without // is not an authority-based URL: example.com/path is a path with no host at all, so always include the scheme when you parse something you want to connect to. Second, localhost:8080 parses without an error but as scheme localhost with the opaque part 8080, exactly like mailto:[email protected] is scheme mailto and opaque [email protected]. The parser sees word:rest and cannot know you meant a host and port. Writing http://localhost:8080 removes the ambiguity.

IP addresses and CIDR ranges

The net package represents an IP address as net.IP, a byte slice that holds either 4 bytes (IPv4) or 16 bytes (IPv6). net.ParseIP returns nil for text that is not an address, so always check the result. Methods such as IsLoopback, IsPrivate and To4 answer common questions.

A CIDR block like 192.168.1.0/24 describes a whole range: the first 24 bits are fixed, the remaining 8 vary. net.ParseCIDR returns the address, the network (*net.IPNet) and an error.

The newer net/netip package (Go 1.18) provides netip.Addr, netip.Prefix and netip.AddrPort. They are small value types that you can compare with ==, use as map keys and copy freely, which net.IP slices cannot do. Prefer them in new code.

Go IP addresses, CIDR and netip

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Look at the second block of output: parsing 192.168.1.77/24 gives the network 192.168.1.0/24, because ParseCIDR zeroes the host bits for the IPNet it returns. The mask prints in hex (ffffff00) and Size() reports 24 fixed bits out of 32.

Use net.SplitHostPort and net.JoinHostPort for host:port strings instead of splitting on :. IPv6 addresses contain colons themselves, so JoinHostPort adds the square brackets and SplitHostPort removes them. SplitHostPort also complains if the port is missing, as the last line shows.

Allow-lists and block-lists of IP ranges (for example "only accept requests from 10.0.0.0/8") are best written with netip.Prefix.Contains. When you check a client address behind a proxy, remember the real address may be in a header such as X-Forwarded-For, which you must only trust when it comes from your own proxy.

TCP in one program: an echo server

HTTP, databases, caches and chat servers are all built on TCP, a reliable ordered stream of bytes between two endpoints. In Go the pattern is always the same:

  1. Server: net.Listen("tcp", addr) returns a net.Listener. Accept() blocks until a client connects and returns a net.Conn.
  2. Client: net.Dial("tcp", addr) returns a net.Conn too.
  3. A net.Conn is an io.Reader, an io.Writer and an io.Closer, so everything you know from os, io and bufio applies: wrap it in bufio.Scanner, bufio.Reader or fmt.Fprintf.

Asking for port 0 tells the operating system to pick any free port, and ln.Addr() tells you which one it chose. That makes the example below safe to run anywhere:

Go A TCP echo server and client on loopback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Walk through what happens. The accept loop runs in its own goroutine and starts one more goroutine per connection, the classic Go server shape (sync.WaitGroup.Go needs Go 1.25). The client writes a line, then waits for the reply line; TCP guarantees the bytes arrive in order, but it does not preserve message boundaries, which is why the protocol uses newlines to frame each message. Closing the listener makes Accept return an error, which ends the loop; closing the client connection makes the handler's Scan return false, which ends the handler. After that wg.Wait() returns and the program exits cleanly.

The TCP Servers and Custom Protocols lesson builds on this skeleton with framing, deadlines and graceful shutdown.

Failures and timeouts

Network code fails in ways ordinary code does not: nobody listening, a peer that never answers, a connection that drops. Two habits protect you.

  • Always set timeouts. net.DialTimeout limits how long connecting may take, and conn.SetDeadline, SetReadDeadline and SetWriteDeadline set an absolute point in time after which a read or write fails. Without a deadline, a read from a silent peer blocks forever.
  • Inspect errors with the right tools. Network errors implement net.Error, which has a Timeout() method. Use errors.As (or a type assertion) and errors.Is as in Wrapping and Inspecting Errors, not string matching.
Go Connection refused and read deadlines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The listener accepts the connection in the kernel even though our program never calls Accept, so the dial succeeds. The client then reads from a peer that never writes; after 50 milliseconds the deadline fires and Read returns an error whose Timeout() is true. A refused connection (the first case) is not a timeout: it fails immediately because nothing listens on that port.

DNS lookups

Names such as example.com are turned into addresses by DNS, which needs network access and therefore cannot run in the sandbox. This read-only snippet shows the API:

Go
addrs, err := net.LookupHost("example.com")   // []string of IP addresses
ips, err := net.LookupIP("example.com")       // []net.IP
cname, err := net.LookupCNAME("www.example.com")

var r net.Resolver                            // custom resolver with context and timeouts
ips2, err := r.LookupNetIP(ctx, "ip", "example.com")

net.Dial("tcp", "example.com:80") performs the lookup for you and tries each returned address in turn. Prefer the context-aware forms (net.Dialer.DialContext, Resolver methods) in real programs, so a cancelled request also cancels the lookup, as covered in The context Package.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does u.Host contain for the URL https://example.com:8443/a?

    Host includes the port when the URL has one. Use u.Hostname() for the name alone and u.Port() for the port.
  2. 2. What does url.Values{"a": {"1"}}.Get("b") return, and can you tell that the key is absent from the result alone?

    Get returns the first value or an empty string. To distinguish a missing key from an empty one, call Has.
  3. 3. Which call is the correct way to add a user-supplied search term to a URL?

    Values.Encode escapes every key and value. Concatenating raw input lets characters such as & and # change the meaning of the URL.
  4. 4. What happens when you call url.Parse("localhost:8080")?

    A word followed by a colon is read as a scheme. Include the scheme, as in http://localhost:8080, to get a real host and port.
  5. 5. In a TCP server, why does the echo example end each message with a newline?

    TCP delivers an ordered stream of bytes, not messages. Applications add framing, such as newline-terminated lines or length prefixes.

Summary

  • url.Parse splits a URL into Scheme, User, Host, Path, RawQuery and Fragment. Use Hostname(), Port() and Redacted() rather than slicing strings yourself.
  • url.Values is a map[string][]string; Get reads the first value, Has tests presence and Encode produces a sorted, escaped query string.
  • Build URLs from a url.URL plus url.Values, never by concatenating user input. QueryEscape and PathEscape differ (+ versus %20).
  • ResolveReference applies browser link rules and url.JoinPath joins path elements; localhost:8080 without a scheme is parsed as an opaque URL.
  • net.ParseIP, net.ParseCIDR and the value types in net/netip handle addresses and ranges; use SplitHostPort and JoinHostPort for host:port strings.
  • net.Listen, Accept and Dial give you a net.Conn that works with bufio, fmt and io. TCP has no message boundaries, so protocols add framing.
  • Always set dial timeouts and deadlines, and inspect failures with net.Error and errors.As.

Next: Unicode and Text Processing