Skip to content
Tutorials Center
My learning

Topics

Lesson 72 of 80 ยท Projects

Project: A Mini Web Application Firewall

Build a mini web application firewall in Python: normalize requests, score them with rules, add allow-lists, measure false positives and write tests.

A web application firewall (WAF) sits in front of a web application and inspects each HTTP request before the application sees it. Requests that look like attacks are blocked, logged or flagged. Commercial and open-source WAFs power a lot of the internet's front doors, and they are controversial for a good reason: they are useful, and they are never perfect. A WAF that is too strict blocks real customers; a WAF that is too loose lets attacks through. Learning to build a tiny one teaches you exactly why.

In this project you will build a rule engine that scores simulated requests, add an allow-list (a positive security model) next to the deny-list rules, measure false positives and false negatives on labelled traffic, and protect the whole thing with tests. Everything is simulated: there is no network and no real server. You will use ideas from Input Validation and Sanitization, SQL Injection and Parameterized Queries, Cross-Site Scripting and Output Encoding and Security Code Review, SAST and Testing.

A WAF is a safety net, not a fix. The vulnerability is in the application, and the real cure is parameterized queries, output encoding and proper validation. Use a WAF to buy time (so-called virtual patching), to add visibility and to block bulk automated attacks, never as a reason to ship vulnerable code.

The design

flowchart LR
    REQ["Incoming request"] --> NORM["Normalize<br/>decode, case, comments"]
    NORM --> SCHEMA["Allow-list check<br/>per route and parameter"]
    SCHEMA --> RULES["Deny-list rules<br/>add up scores"]
    RULES --> DEC{"Total score"}
    DEC -- "below 3" --> ALLOW["Allow"]
    DEC -- "3 to 4" --> LOG["Allow and log"]
    DEC -- "5 or more" --> BLOCK["Block with 403"]

Each check adds points instead of deciding alone; the total picks allow, log or block.

Three design choices are worth explaining before you read the code.

1. Anomaly scoring instead of one-rule blocking. Many signs of an attack are also found in innocent text. An apostrophe appears in "O'Brien". A semicolon appears in a forum post about shell commands. If any single match blocked the request, you would block customers constantly. With scoring, weak signals add up slowly (a lone quote is worth 2 points) while strong ones (a UNION SELECT is worth 5) block on their own. The rules for popular open-source WAFs use the same idea, called anomaly scoring.

2. Normalization before matching. Attackers disguise payloads: they URL-encode them twice, mix upper and lower case, hide keywords with SQL comments (UN/**/ION) or use odd whitespace. The engine therefore first converts every value into a canonical form (decode repeatedly, normalize Unicode, remove comments, collapse spaces, lower-case), and matches rules against that. This is the same canonicalization principle used for path traversal in the validation lesson.

3. An allow-list next to the deny-list. Deny-list rules describe what is bad, and attackers constantly invent new bad things. An allow-list describes what is good: the id parameter of /products must be 1 to 9 digits, and anything else is wrong whatever it contains. Allow-lists are far more reliable, but you can only write them for routes whose inputs you understand. Use them wherever you can and use deny-list rules as a net for everything else.

Step 1: the rule engine

The code defines a request as a small dictionary, normalizes values, checks the route schema, runs the rules and returns a decision with the reasons. Keeping the reasons matters: when a customer is blocked, support needs to see which rule fired.

Python A scoring rule engine for simulated requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output from the top. The harmless searches score zero or two points and pass. O'Brien earns 2 points for the quote and is still allowed, which is the entire reason for scoring. The attacks score between 5 and 14 points, usually from more than one rule and sometimes from the allow-list check as well (the id parameter is supposed to be digits). The scanner's user agent alone reaches only 3 points, so the request is allowed but logged, which is a reasonable response to a weak signal.

You should also notice the encoded and obfuscated variants (%27 OR 1=1-- and UnIoN/**/SeLeCt) being caught: that is normalization at work.

Step 2: measuring false positives and false negatives

A WAF without measurement is guesswork. You need labelled traffic: requests where you know in advance whether each one is an attack. Then you can count four outcomes:

Predicted attack (blocked) Predicted normal (allowed)
Really an attack True positive (good) False negative (missed attack)
Really normal False positive (customer blocked) True negative (good)
  • Precision = TP / (TP + FP): of everything blocked, how much was really an attack.
  • Recall = TP / (TP + FN): of all real attacks, how many were blocked.

Moving the block threshold trades one for the other: a low threshold catches more attacks and blocks more innocent users. The only way to find a sensible setting is to try the traffic. The second program includes the engine again (so it runs on its own), then adds:

  1. A labelled data set that includes tricky innocent requests (a developer pasting an injection example in a forum post, a student searching for SQL help) and attacks the rules do not know about.
  2. A threshold sweep showing precision and recall at each setting.
  3. A scoped exception: on one route and one parameter, skip one rule category. Exceptions should be as narrow as possible, with a written reason. Here the forum post body may contain SQL examples, but it still gets the XSS rules.
  4. Tests that pin down the behavior, so a later rule change that accidentally lets an attack through or blocks a customer fails loudly.
Python Evaluate, tune with an exception, and test the firewall

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study the sweep. At a threshold of 2, nearly everything suspicious is caught (recall 0.92) but four innocent requests are blocked, so precision is only 0.73. Raising the threshold to 5 removes two of those false positives for a recall of 0.83, a good balance on this tiny data set. At 8 nothing innocent is blocked (precision 1.00) but two thirds of the attacks get through. There is no universally correct setting; the right value depends on the cost of a missed attack against the cost of a blocked customer, and you must measure it on your traffic.

The scoped exception removes one false positive without opening a hole elsewhere, and the tests prove it: the same SQL-looking text on /search is still blocked, and an XSS payload on the forum route is still blocked. The last test is a deliberate known-gap marker. It records a real limitation (a probe on an unprotected route with no schema) so nobody assumes the WAF covers it, and it will fail, loudly, on the day someone improves the rules.

Deploying a WAF in real life

  • Start in monitor mode. Run the rules in log-only mode on live traffic for days or weeks, review what would have been blocked, tune, then switch rules to blocking one category at a time.
  • Decide fail-open or fail-closed. If the WAF crashes or times out, either all requests pass uninspected (fail-open, available but exposed) or none do (fail-closed, safe but down). Pick on purpose.
  • Keep regexes safe. A WAF that is vulnerable to catastrophic backtracking can be taken down with one request. The rules above use bounded repetitions such as .{0,20}; see the ReDoS discussion in the input validation lesson.
  • Mind what you log. Blocked requests may contain passwords or personal data. Redact sensitive parameters and set retention limits.
  • Inspect bodies, but cap their size. Large uploads and unusual encodings (multipart, JSON, XML) need parsers of their own, and attackers aim at the gap between how the WAF parses and how the application parses.
  • Review exceptions. Every exception is a hole. Record the reason, the owner and an expiry date.

Ideas for extending the project

  • Add rules for SSRF targets (private IP ranges and metadata addresses, see SSRF, Unsafe Deserialization and XXE).
  • Add rate limiting per client with a token bucket, as in API Security, and combine it with the score.
  • Inspect JSON bodies by walking nested values.
  • Add a reputation score that grows when a client is blocked repeatedly.
  • Feed the decisions into the log analyzer from the previous project.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the mini WAF use anomaly scoring instead of blocking on the first rule that matches?

    Scoring lets weak evidence accumulate slowly while strong evidence blocks alone, which keeps names like O'Brien working.
  2. 2. What is the main advantage of an allow-list schema over deny-list rules?

    If id must be 1 to 9 digits, every other shape fails, however cleverly disguised. The cost is that you must know the valid inputs for each route.
  3. 3. A WAF blocks a customer who searched for 'sql union select tutorial'. What kind of error is this?

    Blocking a legitimate request is a false positive. A false negative would be letting a real attack through.
  4. 4. Why normalize (decode, remove comments, lower-case) a value before matching rules?

    Matching on one canonical form defeats many obfuscations, such as double URL encoding or UN/**/ION.
  5. 5. What is the safest way to introduce new blocking rules on a live site?

    Monitor mode shows real false positives before customers feel them. Then you enforce rules gradually, with scoped exceptions where needed.

Summary

  • A WAF inspects requests before the application does; it adds visibility and buys time, but it never replaces fixing the vulnerable code.
  • Normalize inputs, then combine an allow-list for routes you understand with scored deny-list rules for everything else.
  • Anomaly scoring turns many weak signals into a decision: allow, log or block.
  • Measure precision (false positives) and recall (false negatives) on labelled traffic, and tune thresholds with data.
  • Keep exceptions narrow and documented, write tests including known-gap markers, and roll out in monitor mode first.

Next: Project: Secure Token Service, where you will build, sign, verify and rotate tokens with HMAC.