Skip to content
Tutorials Center
My learning

Topics

Lesson 105 of 135 ยท Advanced C

I/O Multiplexing: select, poll and epoll

Handle many sockets and file descriptors in one C thread with select, poll and Linux epoll - learn fd_set, pollfd, level versus edge triggering and an epoll echo server.

A server that talks to one client can simply call recv and wait. A server that talks to a thousand clients cannot: while it sits blocked in recv for client 1, clients 2 to 1000 are ignored. In Network Programming with Sockets you saw one answer, a thread per client. Threads work, but each one costs memory and scheduling time, and a server with ten thousand idle connections should not need ten thousand threads.

The other answer is I/O multiplexing: one thread gives the operating system a list of file descriptors and asks, "wake me when any of these is ready." Then it handles only the ones that are ready, never blocking on the rest. C programmers have three classic tools for this: select, poll and, on Linux, epoll. This lesson explains how each works, how they differ and how to build a single-threaded echo server with epoll. Everything applies to any file descriptor (pipes, terminals, sockets), not only networking; see Low-Level I/O with System Calls for the basics of descriptors. The examples run in the sandbox using pipes and the loopback address.

The event loop

Every multiplexing server has the same shape:

flowchart TD
    A["register descriptors<br/>(listener + clients)"] --> B["wait until at least one<br/>descriptor is ready"]
    B --> C{"which one?"}
    C -->|"listener"| D["accept new client<br/>register it"]
    C -->|"client"| E["recv / send<br/>without blocking long"]
    E -->|"connection closed"| F["unregister and close"]
    D --> B
    E --> B
    F --> B

One thread, one loop. The only place it sleeps is the wait step, and it wakes only when there is work.

"Ready" has a precise meaning. A descriptor is readable when a read or recv would return immediately: either data is waiting, or the other side closed the connection (a read then returns 0). A listening socket is readable when a client is waiting to be accepted. A descriptor is writable when a write would not block, because there is room in its buffer.

Multiplexing tells you that an operation would not block right now. It does not make the operation itself non-blocking. If you read from a ready descriptor, the call returns what is there and does not wait for more, so that is safe. But if you try to write a megabyte to a socket that only has room for 4 KB, a blocking send will wait. Production servers therefore also set their sockets to non-blocking mode with fcntl(fd, F_SETFL, flags | O_NONBLOCK), which makes send and recv return -1 with errno set to EAGAIN instead of waiting.

select

select is the oldest and most portable of the three; it exists on Linux, every BSD, macOS and Windows (Winsock). You pass up to three sets of descriptors (to read, to write, with errors) as fd_set objects, which you fill with four macros:

Macro Meaning
FD_ZERO(&set) empty the set
FD_SET(fd, &set) add fd
FD_CLR(fd, &set) remove fd
FD_ISSET(fd, &set) after the call: is fd ready?
C
int select(int nfds, fd_set *readfds, fd_set *writefds,
           fd_set *exceptfds, struct timeval *timeout);

Two details trip up nearly everyone:

  1. nfds is the highest descriptor number plus one, not the count of descriptors.
  2. select overwrites the sets with the ready subset. You must rebuild them before every call.

The return value is the number of ready descriptors, 0 on timeout and -1 on error. The timeout is a struct timeval (seconds and microseconds); a NULL pointer waits forever and a zeroed struct polls without waiting.

The example below creates two pipes, writes to only one, and asks select which is readable:

C select with two pipes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

In round 1, select reports one ready descriptor (pipe B). In round 2 the data is gone, so nothing becomes ready and the call returns 0 after the 0.2 second timeout.

select is simple but has real limits. The fd_set is a fixed-size bit array, and FD_SETSIZE is typically 1024: you cannot even put a descriptor numbered 1024 or higher into a set (doing so is undefined behavior). The kernel and your code also both scan all nfds bits on every call, so the cost grows with the highest descriptor, and you must rebuild the sets each time. It is fine for a handful of descriptors and for portability, but not for thousands of connections.

poll

poll fixes the biggest annoyances. Instead of bit sets you pass an array of structs, one per descriptor:

C
struct pollfd {
    int   fd;        // descriptor to watch (negative = ignore this entry)
    short events;    // what you want to know about
    short revents;   // what happened; filled in by poll
};

int poll(struct pollfd *fds, nfds_t count, int timeout_ms);

Requested events are mainly POLLIN (readable) and POLLOUT (writable). Some events are reported in revents even if you did not ask for them: POLLERR (error), POLLHUP (the other side hung up) and POLLNVAL (the descriptor is not open). The array has no size limit, and because the input (events) and output (revents) are separate fields, you do not rebuild anything between calls. The timeout is in milliseconds; -1 waits forever and 0 returns at once. The return value is the number of entries with a non-zero revents, 0 on timeout and -1 on error.

C poll and the states of a pipe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

The last two lines of output show the standard way a server learns a client has disconnected: the socket becomes readable, recv returns 0 and POLLHUP may be set. Always handle the "readable but zero bytes" case by closing the connection; otherwise poll will report the same descriptor again immediately, and the loop spins at 100 percent CPU.

Even poll has a cost. For every call the kernel must examine every entry in your array, and your code must scan the array to find the ready ones. With 50,000 mostly idle connections that is a lot of wasted work per wake-up.

epoll (Linux)

epoll solves the scaling problem by keeping the list of watched descriptors inside the kernel. You register each descriptor once; afterwards epoll_wait returns only the descriptors that are ready, so the cost depends on the number of events, not on the number of connections. Three calls make up the API:

Call Purpose
epoll_create1(0) create an epoll instance (itself a file descriptor)
epoll_ctl(ep, op, fd, &event) EPOLL_CTL_ADD, EPOLL_CTL_MOD or EPOLL_CTL_DEL a watched descriptor
epoll_wait(ep, events, max, timeout_ms) wait; fills the array with ready events and returns how many
C
struct epoll_event event = { .events = EPOLLIN, .data.fd = client_fd };
epoll_ctl(ep, EPOLL_CTL_ADD, client_fd, &event);

The data member is a union that the kernel hands back untouched; most programs store the descriptor in data.fd, while larger programs store a pointer (data.ptr) to their per-connection struct.

flowchart LR
    subgraph poll_["poll or select"]
        P1["you pass N descriptors<br/>every call"] --> P2["kernel checks<br/>all N"] --> P3["you scan all N<br/>for revents"]
    end
    subgraph epoll_["epoll"]
        E1["register once<br/>(epoll_ctl)"] --> E2["kernel records<br/>readiness"] --> E3["epoll_wait returns<br/>only ready ones"]
    end

poll and select do work proportional to the number of watched descriptors on every wake-up; epoll's work is proportional to the number of ready ones.

Level-triggered and edge-triggered

By default epoll is level-triggered, the same behavior as select and poll: as long as data is left unread, every epoll_wait reports the descriptor as ready. That is forgiving. If you read only part of the data this time, you are told again next time.

Adding EPOLLET makes it edge-triggered: you are notified only when the state changes (new data arrives). If you do not read everything then, you may never be told again. Edge-triggered code must therefore use non-blocking descriptors and read in a loop until recv fails with EAGAIN. It can save system calls on very busy servers, but it is easy to get wrong. Start with level-triggered mode.

Other useful flags: EPOLLOUT (writable), EPOLLRDHUP (the peer closed its writing side), EPOLLONESHOT (disable the descriptor after one event, handy when worker threads process connections) and EPOLLERR/EPOLLHUP (always reported).

An epoll echo server

The next example is a complete single-threaded server. It listens on the loopback address and uses one epoll instance to watch the listening socket and every client at once. In main, three clients connect and stay connected at the same time, then talk in a scrambled order (2, 0, 1). A thread-per-client design would use three threads; here one thread handles all three. As in the earlier lesson, the server runs in a thread so that both ends fit in one program:

C A single-threaded echo server with epoll

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with scanf or fgets), it gets the next line from this box.

Follow one request through the loop. The first epoll_wait wakes up because the listening socket became readable, so the server calls accept and registers the new descriptor. Later, a client socket becomes readable, recv returns the message, the server answers and goes back to waiting. When main closes its sockets, each client descriptor becomes readable again, recv returns 0, and the server removes it from the epoll set and closes it. After the third disconnect the loop ends.

A few honest simplifications to fix in a real server: accept, recv and send results should all be checked (including EINTR and EAGAIN), send can write fewer bytes than asked, and the 5000 ms timeout is a safety net for the demo. For long-lived connections, add a per-connection buffer and use non-blocking sockets together with EPOLLOUT for pending writes.

Choosing between them

select poll epoll
Portability everywhere, including Windows all Unix systems Linux only
Descriptor limit FD_SETSIZE (usually 1024) none none
Rebuild per call yes no no (registered once)
Cost per wait O(highest descriptor) O(descriptors watched) O(ready events)
Good for few descriptors, portable code a few hundred descriptors thousands of connections

Other platforms have their own equivalents: kqueue on macOS and the BSDs, I/O completion ports on Windows, and io_uring on newer Linux kernels. Portable libraries such as libevent, libev and libuv (the engine under Node.js) hide these differences behind one event-loop API, which is usually the right choice for a production server. Understanding the three calls in this lesson tells you what those libraries do underneath.

Multiplexing also does not replace threads when work is CPU-heavy. A single loop that spends 50 ms computing a response stalls every other client for that time. Real servers often run one event loop per CPU core, or hand slow work to a thread pool.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the first argument (nfds) of select()?

    select scans descriptors from 0 up to nfds - 1, so it must be one more than the largest descriptor in any set.
  2. 2. Why must you rebuild the fd_set before every call to select()?

    On return, the sets contain only the descriptors that are ready, so the original list is lost unless you copy or rebuild it.
  3. 3. A readable client socket returns 0 from recv(). What does it mean, and what should the server do?

    Zero bytes means end of stream. If the server keeps the descriptor registered, it stays readable and the loop spins.
  4. 4. Why does epoll scale better than poll for 50,000 mostly idle connections?

    With poll, every call passes and scans the entire list. With epoll you register once, and the work per wake-up depends on the number of ready events.
  5. 5. Which statement about edge-triggered epoll (EPOLLET) is correct?

    Edge-triggered mode notifies on changes only, so unread data can be left unnoticed. It needs non-blocking I/O and read loops, and epoll is Linux-only.

Summary

  • I/O multiplexing lets one thread wait on many descriptors and handle only the ready ones; the usual structure is a loop of wait, then accept or read or write.
  • select takes bit sets, needs nfds = highest fd + 1, is limited by FD_SETSIZE and overwrites its sets, so rebuild them on every call.
  • poll uses an array of struct pollfd with separate events and revents, no fixed limit and no rebuilding; watch for POLLHUP and zero-byte reads.
  • epoll (Linux) registers descriptors in the kernel with epoll_ctl and returns only ready events from epoll_wait, so it scales to thousands of connections.
  • Level-triggered mode (default) repeats notifications while data remains; edge-triggered (EPOLLET) needs non-blocking sockets and read-until-EAGAIN loops.
  • Readiness does not make a slow send non-blocking: use O_NONBLOCK, check return values, and keep CPU-heavy work out of the event loop.

Next: _Static_assert and C23 Attributes