Skip to content
Tutorials Center
My learning

Topics

Lesson 53 of 79 ยท Application Layer: Names and Web

VoIP, RTP and Streaming Media

Learn how calls and video travel over IP: SIP signalling, the RTP packet and RTCP reports, jitter buffers, codec bandwidth, and adaptive bitrate streaming with HLS and DASH.

A web page can wait. If a packet is late by half a second, the browser simply shows the page half a second later. A voice call cannot wait: a word that arrives too late is useless, because the listener has already moved on, and the same is true of a live video. At the same time, a lost packet in a call is tolerable (a tiny click, a smeared frame), while resending it would take too long to matter. Real-time media therefore breaks the assumptions that TCP was designed around: it wants timing more than reliability.

This lesson shows how the Internet carries calls and streams. SIP sets up and tears down sessions. RTP carries the audio and video with timestamps and sequence numbers over UDP, and RTCP reports quality. A jitter buffer hides the variations in delay, and adaptive bitrate streaming (HLS and DASH) fits video to the available bandwidth when the video is not live-interactive. You will run a model of each.

Two kinds of media traffic

Interactive (calls, conferencing) Streaming (video on demand, live broadcast)
Delay budget About 150 ms one way for natural conversation Seconds are acceptable
Transport RTP over UDP (or WebRTC) HTTP over TCP (or QUIC)
Loss handling Conceal or tolerate, minimal retransmission TCP retransmits, the buffer absorbs the delay
Buffering Tiny (tens of milliseconds) Large (many seconds)
Protocols SIP, RTP, RTCP, WebRTC HLS, DASH over HTTP

The network performance lesson defined delay, jitter and loss. In this lesson they stop being abstract numbers: they decide whether a call sounds good.

Setting up a call: SIP

The Session Initiation Protocol (SIP) is the signalling protocol of most voice-over-IP systems. It is a text-based protocol shaped much like HTTP, with requests and numeric responses, usually over UDP or TCP port 5060 (5061 with TLS). SIP does not carry the voice. It only finds the other party, negotiates what the media will look like, and ends the call.

sequenceDiagram
    participant A as Caller
    participant S as SIP server
    participant B as Callee
    A->>S: INVITE (with SDP offer)
    S->>B: INVITE
    B-->>S: 180 Ringing
    S-->>A: 180 Ringing
    B-->>S: 200 OK (with SDP answer)
    S-->>A: 200 OK
    A->>B: ACK
    Note over A,B: RTP media flows directly in both directions
    A->>B: BYE
    B-->>A: 200 OK

A call: the INVITE carries an offer, the 200 OK carries an answer, and the media flows directly between the endpoints.

The body of the INVITE and the 200 OK is a Session Description Protocol (SDP) text that lists the media, the addresses, the ports and the codecs. A tiny example:

TEXT
v=0
o=alice 2890844526 2890844526 IN IP4 203.0.113.5
s=Call
c=IN IP4 203.0.113.5
t=0 0
m=audio 49170 RTP/AVP 0 8
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

It says: "send me audio to 203.0.113.5 on UDP port 49170, and I can use codec 0 (G.711 mu-law) or 8 (A-law), both sampled at 8000 Hz". The other side answers with the codec it picked. Other pieces help in practice: a SIP server (proxy and registrar) knows where each user currently is, STUN and TURN servers and the ICE procedure help the endpoints to reach each other through NAT, and the media is secured with SRTP, the encrypted variant of RTP.

RTP: carrying the media

The Real-time Transport Protocol (RTP) wraps each chunk of audio or video in a small header and sends it in a UDP datagram. It does not guarantee delivery or order, but it gives the receiver what it needs to rebuild the stream:

Header field Size Purpose
Version 2 bits Always 2
Padding, extension, CSRC count 1 + 1 + 4 bits Optional features
Marker 1 bit Marks something special, such as the first packet after silence or the last packet of a video frame
Payload type 7 bits Which codec: 0 for G.711 mu-law, 8 for A-law, and dynamic values for codecs such as Opus
Sequence number 16 bits Increases by one per packet, so the receiver can detect loss and reordering
Timestamp 32 bits The sampling instant, in ticks of the media clock (8000 per second for telephone audio, 90,000 for video), so the receiver can replay with the original timing
SSRC 32 bits Identifies the source of the stream

The fixed header is 12 bytes. On the wire, an RTP packet also has UDP and IP headers. RTP usually uses an even UDP port, and its companion RTCP uses the next odd one. RTCP sends periodic sender and receiver reports: how many packets were lost, how much interarrival jitter was seen, and enough timing information for the sender to compute the round-trip time. Applications use these reports to adapt, for example lowering the bit rate when loss rises. RTCP is deliberately limited to a few percent of the session bandwidth.

Python The RTP header and the cost of headers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three headers differ only in the sequence number (increasing by one) and the timestamp (increasing by 160 ticks, which is 20 ms at 8000 Hz). The table of bandwidths reveals an important fact: headers are a big part of voice traffic. A G.711 call needs 64 kbit/s of audio, but 80 kbit/s on the wire, since each 160-byte payload carries 40 bytes of headers. For a low-rate codec such as G.729, the headers are 67% of every packet. Sending bigger packets (40 ms instead of 20 ms) saves headers but adds delay, and makes each lost packet more audible.

Jitter and the playout buffer

Packets leave the sender at perfectly regular intervals, every 20 ms, but arrive with varying delays, because they queue behind other traffic in routers. This variation is jitter. If the receiver played each packet the moment it arrived, the audio would stutter and warble.

The fix is a playout buffer (jitter buffer). The receiver deliberately waits a little: it holds incoming packets and plays packet i at a fixed time after its sending time, say send_time + P. A packet that arrives before its slot waits in the buffer, and the stream sounds smooth. A packet that arrives after its slot is too late to be useful and is discarded, as if it was lost. The size of the delay P is a trade-off:

  • A small P keeps the conversation lively but loses many late packets.
  • A large P loses almost nothing, but adds delay for the whole conversation. A delay beyond about 150 ms makes people talk over each other.

An adaptive jitter buffer measures the jitter continuously and adjusts P. The standard estimator from the RTP specification keeps a running average of how much the spacing between arrivals differs from the spacing between sending times: J = J + (|D| - J) / 16. The buffer then adds a multiple of this to the smallest delay it has seen. When a buffer must grow or shrink, voice decoders stretch or shorten the audio slightly, or do it during silence, so the change is not heard.

Python Playout delay, late packets and the jitter estimator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For 500 packets with a 40 ms base delay and occasional queueing spikes, the table shows the trade-off. A 50 ms playout delay makes more than a quarter of the packets late, 60 ms still drops about 5%, and at 130 ms nearly everything arrives in time. The adaptive buffer finds a good balance on its own, with a delay near 100 ms and a few percent late packets. A real decoder hides the missing packets with packet loss concealment, repeating or interpolating the previous sound, which works for single missing 20 ms frames.

Streaming media and adaptive bitrate

Video on demand and live broadcasts (a sports match, a conference stream) tolerate seconds of delay, and that changes the design. They use plain HTTP, which has big advantages: it passes every firewall and proxy, uses the existing CDN infrastructure, and gets TCP's reliability for free.

The two dominant technologies are HLS (HTTP Live Streaming, from Apple) and DASH (Dynamic Adaptive Streaming over HTTP, an international standard). They work in the same way:

  1. The video is encoded several times at different bit rates and resolutions (the ladder: for example 400 kbit/s, 1 Mbit/s, 2.5 Mbit/s, 5 Mbit/s).
  2. Each version is cut into short segments of 2 to 10 seconds, stored as ordinary files on a web server or CDN.
  3. A manifest (an .m3u8 playlist for HLS, an .mpd file for DASH) lists the versions and the URLs of their segments.
  4. The player downloads the manifest, then fetches segments one after another with ordinary HTTP GET requests, and puts them in its buffer, often 10 to 30 seconds ahead.
  5. Before each segment, the player's adaptive bitrate (ABR) algorithm chooses which version to fetch next, based on what it measured.

Two families of algorithms exist. A throughput-based rule estimates the available bandwidth from the last downloads and picks the highest bit rate below it, with a safety margin. A buffer-based rule ignores the throughput estimate and uses the buffer level: low buffer means pick a low bit rate to avoid a stall, a full buffer means it is safe to take higher quality. Real players combine both. The goal is to maximize quality while avoiding rebuffering (the dreaded spinning wheel) and avoiding constant switching.

Python Adaptive bitrate rules on a bumpy network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The capacity trace dips to 900 kbit/s in the middle. A player that insists on 5 Mbit/s freezes for more than two minutes in total. A player that always picks the lowest rate never stalls, but the picture is poor all the time. Both adaptive rules never freeze, and average between 1.9 and 2.3 Mbit/s: the throughput rule is cautious (its 20% margin keeps it at 2.5 Mbit/s even when 6 Mbit/s is available), and the buffer rule climbs to 5 Mbit/s whenever its buffer is full and drops back when it drains.

Latency is the cost of this design: with 4-second segments and a buffer of several segments, a live stream arrives 10 to 30 seconds after the event. Low-latency HLS and DASH shorten it to a few seconds by splitting segments into smaller chunks that the player can fetch before the whole segment is finished. For true interactive use, with delays below one second, the world uses WebRTC (see the real-time web lesson), which carries RTP over UDP with encryption, congestion control and NAT traversal built in.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do voice calls usually use UDP and RTP instead of TCP?

    Real-time media can tolerate small losses but not delay. TCP's retransmissions and head-of-line blocking would add delay that ruins the conversation.
  2. 2. What are the RTP sequence number and timestamp used for?

    Sequence numbers increase by one per packet. Timestamps count ticks of the media clock, so playback timing can be rebuilt regardless of network jitter.
  3. 3. A G.711 call sends 160 bytes of audio every 20 ms. With RTP, UDP and IPv4 headers (40 bytes), how much bandwidth does one direction use at the IP level?

    Each packet is 200 bytes, 50 packets per second, so 200 x 8 x 50 = 80,000 bit/s.
  4. 4. What does a larger playout (jitter buffer) delay do?

    More waiting time lets slower packets still make their slot, but every extra millisecond of delay is heard in the conversation.
  5. 5. In HLS and DASH, what does the manifest contain?

    The player reads the manifest to learn which versions exist and where each segment can be downloaded, then chooses a version for every segment.

Summary

  • Real-time media needs low and steady delay more than perfect delivery, so calls use RTP over UDP rather than TCP. Streaming uses HTTP and large buffers.
  • SIP sets up sessions with INVITE, 180, 200 OK, ACK and BYE, and SDP describes the media, ports and codecs. STUN, TURN and ICE help with NAT.
  • RTP adds a 12-byte header with a sequence number, a media-clock timestamp and a source identifier. RTCP reports loss, jitter and round-trip time.
  • Headers can be a big share of a voice packet: G.711 needs 80 kbit/s on the wire for 64 kbit/s of audio.
  • A jitter buffer plays packets at a fixed offset to smooth out delay variation. Larger buffers lose fewer packets but add latency, so adaptive buffers follow the measured jitter.
  • HLS and DASH cut video into segments encoded at several bit rates. An ABR algorithm, based on throughput, buffer level or both, picks the version of each segment. WebRTC covers interactive use.

Next, move on to how email is delivered in Email: SMTP, IMAP and POP3.