Skip to content
Tutorials Center
My learning

Topics

Lesson 57 of 66 ยท Protection, Security and Virtualization

Security Threats and Defenses

Study the main operating system security threats (malware, buffer overflows, privilege escalation, social engineering) and defenses such as salted password hashing, ASLR, sandboxing and auditing.

Protection decides who is allowed to do what. Security asks a harder question: what happens when someone tries to break the rules? Attackers may be remote criminals, curious insiders, malicious programs the user ran by mistake, or just buggy software that behaves like an attacker. A system is only as secure as its weakest link, and the weakest link is often a human.

Security is usually described by three goals, the CIA triad:

  • Confidentiality: only authorized parties can read the data.
  • Integrity: data and programs cannot be changed without authorization.
  • Availability: legitimate users can use the system when they need it.

Attacks target one or more of these. Theft of a password file breaks confidentiality, ransomware attacks availability, and a tampered system file attacks integrity.

Common threats

Threat What it is Example
Virus Code that attaches itself to a host program and spreads when it runs An infected macro in a document
Worm Self-replicating program that spreads over the network without a host A worm exploiting a service flaw
Trojan horse Useful-looking program with a hidden harmful function A "free" game that steals passwords
Ransomware Encrypts your files and demands payment Locks documents, demands money
Spyware / keylogger Secretly records activity Captures typed passwords
Rootkit Hides malicious code by subverting the OS itself Hooks the system call table
Backdoor Hidden way in that bypasses authentication An undocumented admin login
Logic bomb Malicious code that waits for a trigger Deletes files on a certain date
Phishing / social engineering Tricks a person into giving up secrets Fake login page
Denial of service (DoS) Exhausts a resource to deny service Flood of requests
Man-in-the-middle Intercepts and possibly alters communication Rogue Wi-Fi hotspot

Authentication: proving who you are

Authentication checks identity using something you know (password), have (security key, phone) or are (fingerprint). Combining two kinds is multi-factor authentication (MFA) and defeats most stolen-password attacks.

Storing passwords safely

A system must never store passwords in plain text, because one leak exposes everyone. It stores a hash: a one-way function of the password. At login, hash what was typed and compare.

But plain hashes are weak:

  • Identical passwords give identical hashes, so an attacker sees who shares a password.
  • Rainbow tables (precomputed hash lookups) crack common passwords instantly.

The fixes are:

  • A salt: a random value stored beside the hash and mixed into it, so every user's hash is different and precomputed tables are useless.
  • A deliberately slow hash (bcrypt, scrypt, Argon2, PBKDF2) so each guess costs real time, making brute force expensive.
Python Salted, slow password hashing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two users share a password, yet their stored values differ, and an unsalted hash falls to a one-line lookup.

Brute-force math

A password's strength grows exponentially with length. This calculation shows why length beats clever substitutions.

Python How long to brute-force a password?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Program threats: the buffer overflow

Many famous attacks exploit a program bug, not the OS. The classic one is the buffer overflow. In a language like C, writing past the end of an array does not raise an error; it overwrites whatever sits next in memory. On the call stack that neighbour can be a saved return address. If an attacker supplies input long enough to overwrite it with the address of their own code, the function "returns" into the attacker's code.

flowchart TB
    subgraph Stack["Stack frame (grows downward)"]
      direction TB
      A["local buffer[16]"] --> B["saved frame pointer"]
      B --> C["return address"]
    end
    I["input longer than 16 bytes"] -.->|"overwrites"| C

Writing beyond buffer smashes the return address that sits just above it.

The next program is a toy model of a stack frame, not real exploitation: it shows the layout and why a missing length check is fatal.

Python A toy stack frame and a missing bounds check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other program threats include integer overflows, format-string bugs, use-after-free, race conditions (TOCTOU: time of check to time of use), SQL injection and cross-site scripting in web code.

Defenses

No single defense is enough, so systems layer several (defense in depth):

  • Safer languages and code review. Memory-safe languages (Rust, Java, Python, Go) eliminate most overflows.
  • Stack canaries. The compiler places a secret value before the return address and checks it before returning; an overwrite is detected and the program aborts.
  • NX / DEP (non-executable memory). Data pages (stack, heap) are marked non-executable, so injected bytes cannot run.
  • ASLR (address space layout randomization). The kernel loads the stack, heap and libraries at random addresses on every run, so an attacker cannot predict where to jump. (Randomization also helps against the return-oriented attacks that reuse existing code.)
  • Sandboxing and isolation. Run risky code (browser tabs, apps, services) with minimal rights: Linux seccomp, namespaces (containers), Android and iOS app sandboxes.
  • Patching. Most real breaches use known, already-fixed flaws.
  • Firewalls and network filtering, intrusion detection systems and antivirus (signature and behavior-based).
  • Logging and auditing. Record logins, privilege use and file access so attacks can be noticed and investigated.
  • Backups, tested and offline, as the final answer to ransomware.
  • Encryption for data at rest and in transit (next lesson).

Privilege escalation and the human factor

Attackers rarely start as root. They get a foothold in an ordinary account and then escalate privilege by abusing a setuid program, a kernel bug, weak file permissions or a stolen token. Keep setuid programs few and small, apply least privilege, and treat every input as hostile.

Humans are also targets: phishing and social engineering bypass technical controls, so training, MFA and hardware keys matter as much as software patches.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which of these breaks the integrity goal?

    Integrity means data and programs change only through authorized actions. Silent modification violates it.
  2. 2. What problem does a salt solve in password storage?

    A unique random salt per user means each hash must be attacked separately, and rainbow tables no longer work.
  3. 3. In a stack buffer overflow, what is the attacker usually trying to overwrite?

    Replacing the return address redirects execution to the attacker's code when the function returns.
  4. 4. What does ASLR do?

    Because memory addresses change on every run, hard-coded jump targets in an exploit stop working.
  5. 5. Which defense best limits the damage if one application is compromised?

    A sandbox confines the compromised program to the few resources it needs.

Summary

  • Security aims to protect confidentiality, integrity and availability against viruses, worms, trojans, ransomware, rootkits, phishing, denial of service and more.
  • Store passwords as salted, slow hashes; use multi-factor authentication.
  • Buffer overflows overwrite memory next to a buffer, notably the return address; canaries, non-executable memory, ASLR and safe languages counter them.
  • Layer defenses: sandboxing, least privilege, patching, auditing, backups and encryption.

Next: Cryptography in the Operating System