Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 66 ยท Protection, Security and Virtualization

Cryptography in the Operating System

Learn the cryptography an operating system relies on: symmetric and public-key encryption, hashes, MACs, digital signatures, full-disk encryption, TLS and secure boot, with toy Python demos.

Access control (protection) works only while the OS is in charge. Take the disk out of a stolen laptop, listen to a Wi-Fi network, or swap in a modified boot loader, and the OS never gets the chance to say no. Cryptography protects data even when the attacker controls the medium: it makes stolen data unreadable, makes tampering detectable and lets two parties who have never met prove who they are.

You do not need the mathematics to understand how the OS uses cryptography, but you do need a clear picture of the building blocks.

Four building blocks

Symmetric One shared secret key.
Public key A key pair, one public.
Hash A fingerprint of data.
Signature Proof of origin.
1 / 2

Symmetric encryption

Both sides share one secret key. The same key encrypts and decrypts. Modern ciphers such as AES (128/192/256-bit keys) and ChaCha20 are fast, which makes them the workhorses for bulk data: files, disks and network streams.

Two ideas to know:

  • A block cipher such as AES works on fixed-size blocks (16 bytes); a mode of operation (CBC, CTR, GCM, XTS) says how to chain blocks. Never use the naive mode where each block is encrypted independently (ECB), because equal plaintext blocks give equal ciphertext blocks and patterns leak.
  • A stream cipher produces a pseudo-random keystream and XORs it with the data.

The weakness is key distribution: how do two strangers agree on a secret key without anyone overhearing?

Python A toy stream cipher (XOR with a keystream)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is why real systems demand a fresh nonce for every message.

Public-key (asymmetric) cryptography

Each party has a key pair: a public key that anyone may know and a private key that stays secret. What one key does, only the other can undo.

  • Encryption: anyone encrypts with your public key; only you can decrypt with the private key.
  • Digital signatures: you sign with your private key; anyone verifies with your public key.
  • Key exchange (Diffie-Hellman, ECDH): two parties derive a shared secret over a public channel.

Public-key operations are far slower than symmetric ones, so real protocols use them only to agree on a symmetric key, then switch to AES or ChaCha20. RSA, elliptic-curve (ECDSA, Ed25519) and X25519 are the common algorithms. Quantum computers could break today's RSA and elliptic curves, so post-quantum algorithms are being standardized.

The toy below uses tiny numbers so you can see the mechanism. Real keys are hundreds of digits long.

Python Toy RSA with tiny primes, and Diffie-Hellman

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both sides arrive at the same shared secret although only A and B ever crossed the wire.

Hash functions

A cryptographic hash (SHA-256, SHA-3, BLAKE3) turns any data into a fixed-size digest. Good hashes are:

  • deterministic (same input, same digest),
  • one-way (the input cannot be recovered from the digest),
  • collision resistant (it is infeasible to find two inputs with the same digest),
  • avalanche-sensitive (changing one bit changes about half of the output bits).

Uses: integrity checks (download checksums), password storage (with salt and slow hashing), content addressing (Git object names) and as the core of signatures. MD5 and SHA-1 are broken for security purposes.

A message authentication code (MAC), such as HMAC, mixes in a secret key, so that only someone with the key can create a valid tag. It provides integrity and authenticity between parties who share a key.

Python Integrity with hashes and HMAC

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Digital signatures and certificates

To sign a file, hash it, then encrypt the digest with your private key. The receiver hashes the file, decrypts the signature with your public key and compares. A match proves the data is unchanged and came from the key owner.

But how do you know a public key really belongs to Bank X? A certificate is a statement "this key belongs to this name" signed by a certificate authority (CA). Your OS ships with a list of trusted root CAs, and a certificate chain links the website's certificate to one of them. The OS also uses signatures to verify software updates and drivers.

Where the operating system uses cryptography

  • Full-disk encryption: BitLocker (Windows), FileVault (macOS), LUKS/dm-crypt (Linux) encrypt every block with AES (typically in XTS mode). The data key is itself encrypted ("wrapped") by a key derived from your password or stored in a TPM (Trusted Platform Module), so a stolen drive is useless.
  • File and folder encryption, and per-app encrypted storage on phones.
  • Network security: TLS (HTTPS), SSH, VPNs and Wi-Fi WPA3 use public-key exchange followed by symmetric encryption and MACs.
  • Authentication: password hashes, Kerberos tickets, passkeys (public-key logins that cannot be phished).
  • Package and update signing, so only genuine updates install.
  • Secure boot and measured boot. Firmware has trusted keys. It verifies the boot loader's signature, the boot loader verifies the kernel and the kernel verifies the drivers, so a rootkit cannot slip in before the OS starts. With measured boot, each stage's hash is recorded in the TPM, so a remote server can check what actually booted (attestation). See also disk management and the boot sequence.
flowchart LR
    F["Firmware<br/>trusted keys"] -->|"verifies signature"| B["Boot loader"]
    B -->|"verifies signature"| K["Kernel"]
    K -->|"verifies signature"| D["Drivers and apps"]
    T["TPM"] -.->|"records hash of each stage"| B

Each stage checks the next before handing over control: a chain of trust.

Common mistakes

  • Rolling your own cipher. Use vetted libraries and modes (AES-GCM, ChaCha20-Poly1305).
  • Reusing a nonce or IV with the same key.
  • Using a fast hash for passwords (use Argon2, scrypt or bcrypt).
  • Storing keys next to the encrypted data.
  • Confusing encoding (Base64) or compression with encryption: they provide no secrecy.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do secure protocols like TLS use public-key cryptography and then switch to symmetric encryption?

    Asymmetric crypto solves key distribution but is slow. Symmetric ciphers do the bulk work quickly once both sides share a key.
  2. 2. Which key creates a digital signature?

    Only the signer holds the private key, so anyone can verify with the public key but nobody else can forge the signature.
  3. 3. What is the purpose of a nonce in stream ciphers and AES-GCM?

    Reusing a nonce with a key leaks the XOR of plaintexts and can break the encryption.
  4. 4. What does a chain of trust in secure boot ensure?

    Firmware verifies the boot loader, which verifies the kernel, so unauthorized code cannot run early.
  5. 5. Which statement about hash functions is correct?

    Hashes are deterministic and avalanche-sensitive, but they are one-way and unrelated to encryption.

Summary

  • Symmetric ciphers (AES, ChaCha20) are fast and need a shared key; public-key systems (RSA, elliptic curves) solve key exchange and enable signatures but are slower.
  • Hashes give fingerprints for integrity; HMAC adds a secret key; signatures plus certificates establish who made data.
  • The OS uses cryptography for disk encryption, TLS, password storage, update signing and secure boot with a TPM.
  • Use vetted algorithms, fresh nonces and slow password hashes; never invent your own.

Next: Virtual Machines and Hypervisors