Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 79 ยท Application Layer: Names and Web

Email: SMTP, IMAP and POP3

Follow an email from sender to inbox: MUA, MTA and MDA roles, the SMTP command dialogue, MIME attachments, IMAP vs POP3, and how SPF, DKIM and DMARC fight forgery.

Email is one of the oldest services on the Internet, and it still works on the same basic design: store-and-forward text messages passed between servers. Unlike the web, email is asynchronous. The recipient does not need to be online when you send, because a server holds the message until they collect it.

Several protocols cooperate to do this. One protocol pushes mail to servers (SMTP), others let you read it (IMAP, POP3), and a family of standards (MIME, SPF, DKIM, DMARC) describe what the message looks like and prove who sent it. In this lesson you read a real SMTP conversation, run a small mail server on loopback, build a message with an attachment and simulate the checks that decide whether mail is genuine.

Who does what: MUA, MTA, MDA

Email software is split into roles:

  • MUA (Mail User Agent): the program you use to read and write mail, such as a desktop client, a phone app or a webmail page.
  • MSA (Mail Submission Agent): the server that accepts mail from your MUA, usually on port 587, after you log in.
  • MTA (Mail Transfer Agent): the server that moves mail between organizations using SMTP. Postfix, Exim and Exchange are MTAs.
  • MDA (Mail Delivery Agent): the component that finally drops a message into the recipient's mailbox on the destination server.
  • Mailbox access (IMAP or POP3): how the recipient's MUA reads that mailbox.
flowchart LR
    A["Alice's MUA"] -- "SMTP submission<br/>(587, login)" --> M1["Alice's mail server<br/>(MSA + MTA)"]
    M1 -- "DNS MX lookup, SMTP (25)" --> M2["Bob's mail server<br/>(MTA + MDA)"]
    M2 --> BOX[("Bob's mailbox")]
    BOX -- "IMAP / POP3" --> B["Bob's MUA"]

SMTP carries the message from the sender to the recipient's server. IMAP or POP3 are a separate, later step for reading it.

How does the sending server find Bob's server? It asks DNS for the MX records of the domain after the @. An MX record lists mail servers with a preference number; the sender tries the lowest number first and falls back to the others. If a server is down, the sending MTA does not give up: it queues the message and retries for days, which is the store-and-forward model at work.

SMTP: the Simple Mail Transfer Protocol

SMTP (RFC 5321) is a text protocol over TCP, by default on port 25 between servers, 587 for submission from clients (with STARTTLS), and 465 for submission wrapped in TLS from the start. It is a line-based dialogue much like the toy protocols in Application Layer Protocols: the client sends a command, the server answers with a three-digit reply code.

The essential commands:

Command Purpose
EHLO name Greets the server and asks for its extensions (HELO is the old form).
STARTTLS Upgrades the connection to encryption.
AUTH Logs the user in (on submission).
MAIL FROM:<addr> Starts a message and names the envelope sender (where bounces go).
RCPT TO:<addr> Adds one recipient. Repeat for more.
DATA Starts the message content, which ends with a line containing only a dot.
RSET Cancels the current transaction.
QUIT Closes the session.

Reply codes follow the same pattern as HTTP statuses: 2xx success, 3xx "go on" (for example 354 after DATA), 4xx temporary failure (try again later), 5xx permanent failure (give up and bounce). A server that is busy can answer 451 and the sender will retry; a nonexistent mailbox gets 550.

Two details are worth noting. First, the envelope (MAIL FROM, RCPT TO) is separate from the headers inside the message (From:, To:). The envelope drives delivery, while the headers are what a mail program displays. They can differ, which is how blind copies (Bcc) and mailing lists work, and also how forgers fake the visible sender. Second, a line in the body consisting of a single . ends the message, so a body line that starts with a dot gets an extra one added ("dot-stuffing") and the receiver removes it.

The program below runs a small SMTP server and client on loopback. The server keeps a state machine (you cannot send RCPT before MAIL), stores the message, and rejects unknown commands. Watch the replies.

Node.js A mini SMTP dialogue over loopback

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Real servers add a lot: authentication, STARTTLS, size limits, spam checks, queues and retries. But every one of those is still this conversation.

MIME: more than plain text

SMTP and the original message format (RFC 5322) were designed for 7-bit ASCII text. MIME (Multipurpose Internet Mail Extensions) makes email carry international text, HTML, images and attachments while still passing through old text-only systems. The key idea is to describe the content with headers and, where needed, encode it into safe ASCII:

  • MIME-Version: 1.0 marks the message as MIME.
  • Content-Type says what a part is: text/plain; charset=utf-8, text/html, image/png, application/pdf, or a multipart type holding several parts.
  • Content-Transfer-Encoding says how the bytes were made safe: 7bit, quoted-printable (readable text with =C3=A9-style escapes for non-ASCII) or base64 (any binary data, about 33% bigger).
  • Content-Disposition: attachment; filename="..." marks a part as a file to save.
  • Header values with non-ASCII characters use encoded words such as =?UTF-8?B?...?=.

Multipart messages use a boundary string, chosen not to appear in the content, to separate parts: multipart/mixed for a body plus attachments, multipart/alternative for the same content in several forms (plain text and HTML).

The example builds such a message and then parses it back, decoding the subject, the quoted-printable text and the base64 attachment.

Node.js Build and parse a MIME message

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

More MIME headers, and what SMTP's two ends are called

Two more MIME headers are worth knowing. Content-ID gives a part a unique label, so an HTML body can point at an embedded image with <img src="cid:logo123"> and the image travels inside the same message. Content-Description is a free-text label for humans, such as "Quarterly report". MIME also lets mail carry text in any language and rich HTML with CSS, but its drawbacks are real: messages grow (base64 adds about a third), mail clients may interpret the same message differently, rich content can hide tracking images or malicious attachments, and old systems that do not understand MIME show the raw parts.

In the SMTP conversation, the program that sends is the sender-SMTP (the client) and the program that accepts is the receiver-SMTP (the server). A server can play both roles when it forwards mail on. A relay accepts mail not meant for itself and passes it toward the destination. An open relay (one that forwards for anyone) is abused by spammers, so servers must only relay for authenticated users. A gateway connects SMTP to a different mail system, translating formats on the way. Besides port 25 (server to server) and 587 (submission with login), you will sometimes see port 2525, an unofficial alternative that mail providers offer because many home and cloud networks block outbound port 25. SMTP's disadvantages are that it is text-based and was not built for authentication (hence SPF, DKIM and DMARC below), that it has no built-in encryption (STARTTLS was added later), and that it only pushes mail, so you need IMAP or POP3 to read it.

Python An inline image with Content-ID, parsed back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading mail: IMAP and POP3

Once mail sits in the recipient's mailbox on the server, the user's program must fetch it. SMTP cannot do that, because it only pushes. Two protocols exist.

POP3 (Post Office Protocol, port 110, or 995 with TLS) is the simple one. The client logs in (USER, PASS), lists messages (LIST), downloads them (RETR n) and, usually, deletes them from the server (DELE n). It is built for one computer that downloads everything and keeps the only copy. It has no folders on the server and no idea which messages you have read.

IMAP (Internet Message Access Protocol, port 143, or 993 with TLS) keeps the mail on the server and lets many devices view it consistently. Key features:

  • Folders (mailboxes) on the server, which all devices share.
  • Flags such as \Seen, \Answered, \Flagged, \Deleted, stored per message, so reading on your phone marks it read on your laptop.
  • Partial fetch: download just headers, or just one MIME part, so a huge attachment is not transferred until you open it.
  • Server-side search and IDLE, which holds a connection open so the server can notify the client at once about new mail (a push mechanism).
POP3 IMAP
Mail lives Mostly on the client after download On the server
Multiple devices Awkward Designed for it
Folders and flags No Yes
Bandwidth use Downloads whole messages Can fetch parts
Offline reading Everything is already local Needs caching by the client

Today almost everyone uses IMAP (or a proprietary API behind a webmail). A typical client setup has an incoming server (IMAP, port 993) and an outgoing server (SMTP submission, port 587 or 465).

Proving who sent it: SPF, DKIM and DMARC

SMTP has no built-in way to check that MAIL FROM or From: tells the truth. Anyone can connect to a mail server and claim to be [email protected]. That is why phishing and spam are so easy, and three DNS-based standards were added to fight forgery.

SPF (Sender Policy Framework) lets a domain publish, in a DNS TXT record, which IP addresses may send mail for it:

TEXT
example.net.  TXT  "v=spf1 ip4:203.0.113.0/24 ip4:198.51.100.7 -all"

The receiving server compares the connecting IP with that list for the domain in the envelope sender. -all means "everything else should fail". SPF breaks when mail is forwarded, because the forwarder's IP is not in the list.

DKIM (DomainKeys Identified Mail) adds a digital signature. The sending server signs selected headers and a hash of the body with a private key and adds a DKIM-Signature header naming the domain and a selector. The receiver fetches the public key from DNS (selector._domainkey.example.net) and verifies. A valid signature proves that the signed parts were not altered and that the signing domain authorized the mail, and it survives forwarding. It breaks if a mailing list rewrites the subject or body.

DMARC ties them together and links them to the address people actually see. A domain publishes a policy at _dmarc.example.net:

TEXT
"v=DMARC1; p=reject; rua=mailto:[email protected]"

A message passes DMARC if SPF or DKIM passes and is aligned, meaning the authenticated domain matches the domain in the visible From: header. If neither does, the policy tells receivers what to do (none to just report, quarantine for the spam folder, reject to refuse) and rua asks them to send aggregate reports so the owner can see who is sending as their domain.

The simulation implements these ideas with real cryptography: an Ed25519 key pair for DKIM (a modern algorithm that DKIM allows; most deployments still use RSA), a CIDR match for SPF and an alignment check for DMARC. Four messages arrive.

Node.js SPF, DKIM and DMARC decisions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Case 3 shows why DMARC accepts either SPF or DKIM: a mailing list fails SPF but keeps the DKIM signature. Case 4 shows the odd result that SPF alone can pass while the content was tampered with; DMARC accepts it because SPF is aligned. Real DMARC deployments therefore rely on DKIM for integrity and use SPF for sender authorization, and add TLS between servers (MTA-STS) so that mail is not read or modified on the wire.

Authentication proves the domain, not the person. A criminal can register examp1e.net, set up perfect SPF, DKIM and DMARC for it, and still send convincing phishing. These checks stop others from forging your domain, not look-alike domains.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which protocol moves a message from the sender's server to the recipient's server?

    SMTP pushes mail between servers (and from clients to their server). IMAP and POP3 are for reading a mailbox, and MIME describes content.
  2. 2. An SMTP server replies 451. What should the sending server do?

    4xx codes mean a temporary problem, so MTAs retry for days. 5xx codes are permanent and cause a bounce.
  3. 3. You read a message on your phone and later open your laptop. The message is already marked read. Which protocol makes this possible?

    IMAP keeps mail and flags such as \Seen on the server, so all devices share one state.
  4. 4. What does Content-Transfer-Encoding: base64 do?

    Base64 only changes the representation so binary data survives text-based transport. It makes data about a third larger, and it provides no secrecy.
  5. 5. Which standard publishes a public key in DNS so receivers can verify a message signature?

    DKIM signs the message and the public key lives in a DNS TXT record under selector._domainkey.domain. SPF lists allowed sending IPs, and DMARC sets the policy.

Summary

  • Email is store-and-forward. A MUA submits to its server, MTAs relay with SMTP (found through DNS MX records), and the recipient reads the mailbox with IMAP or POP3.
  • SMTP is a text dialogue: EHLO, MAIL FROM, RCPT TO, DATA, a lone . and QUIT, with 2xx, 3xx, 4xx (retry) and 5xx (fail) replies. The envelope is separate from the message headers.
  • MIME describes content with Content-Type, Content-Transfer-Encoding (quoted-printable, base64) and multipart boundaries, making attachments and non-ASCII text possible.
  • IMAP keeps mail on the server with folders, flags and partial fetch; POP3 downloads and usually removes it.
  • SPF authorizes sending IPs, DKIM signs the message and DMARC requires aligned SPF or DKIM and states what to do with failures.

Next, a tour of other long-lived protocols in FTP, SSH and Other Protocols.