Skip to content
Tutorials Center
My learning

Topics

AWK and sed Tutorial

Learn AWK and sed step by step: 38 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Are AWK and sed? Meet sed and AWK, the two classic Unix text tools: what each one does, how they differ, when to pick which, and how to run every example in this course.
  2. 2 How Text Streams Work: Lines, Fields and Pipes Understand the model behind sed and AWK: standard input and output, lines and records, fields, exit status, and how pipes connect small tools into pipelines.
  3. 3 Regular Expressions for sed and AWK Learn the regular expressions that sed and AWK use: literals, anchors, character classes, repetition, groups, alternation, and the difference between BRE and ERE.
  4. 4 sed and AWK Versions and Portability Know which sed and AWK you have: GNU, BSD, BusyBox, gawk, mawk and one-true-awk, how to check, what differs, and how to write scripts that run everywhere.

sed Basics

  1. 5 sed Substitute: The s Command in Depth Master sed's substitute command: pattern and replacement, the g, N, p and I flags, custom delimiters, the & symbol, escaping, and the most common mistakes.
  2. 6 sed Print, Delete and Quit: p, d, q and -n Use sed to print or delete lines by number and by pattern: the p, d, q and = commands, why -n matters, and how sed's automatic print really works.
  3. 7 sed Addresses and Ranges Tell sed which lines to act on: line numbers, regex addresses, ranges, first~step, addr,+N, the ! operator, nested blocks and the empty-regex shortcut.
  4. 8 sed Options: -n, -e, -f, -E, -i, -s and -z Learn every important sed option: -n, -e, -f, -E, -s, -z, -u, --posix and safe in-place editing with -i and backups, including a dry-run workflow for many files.
  5. 9 sed Capture Groups and Backreferences Rearrange text with sed: capture groups, \1 to \9 in the replacement, backreferences inside patterns to find repeats, and GNU case conversion with \U, \L and \u.

sed Scripting

  1. 10 sed Append, Insert, Change, y and l Commands Add and replace whole lines with sed: the a, i and c commands, y for transliteration, l to reveal hidden characters, and r and w for reading and writing files.
  2. 11 The sed Hold Space: h, H, g, G and x Understand sed's second buffer, the hold space: the five commands h, H, g, G and x, with worked examples such as reversing a file, swapping lines and remembering text.
  3. 12 sed Branching, Loops and Multi-line Editing Write real sed programs: labels with b and t, loops that repeat a substitution, and multi-line editing with n, N, D and P, including joining lines and commas in numbers.
  4. 13 sed Script Files and Classic Recipes Organize sed programs in script files with comments and shebangs, pass shell variables safely, and learn a dozen classic recipes: trim, number, strip tags, CRLF and more.

AWK Basics

  1. 14 Your First AWK Program: Patterns, Actions and BEGIN/END Write your first AWK programs: the pattern-action model, running from the command line or a file, BEGIN and END blocks, the built-in variables NR and NF, and comments.
  2. 15 AWK Fields and Separators: FS, OFS, RS and ORS Control how AWK splits and joins text: the field separator FS and -F, regex separators, OFS, modifying fields and $0, NF arithmetic, and the record separators RS and ORS.
  3. 16 AWK Patterns: Regex, Expressions and Range Patterns Choose which lines an AWK action runs on: regex patterns, comparison and match operators, combining with && || !, field-specific matches, range patterns and the BEGIN/END patterns.
  4. 17 AWK Variables, Types and Operators Learn how AWK variables work: automatic initialization, number and string conversion, arithmetic, concatenation, assignment and increment operators, precedence and passing values with -v.
  5. 18 AWK print, printf and Output Redirection Format AWK output precisely: print versus printf, format specifiers, widths and alignment, redirecting to files, piping to commands such as sort, and using close().
  6. 19 AWK String and Math Functions Use AWK's built-in functions: length, substr, index, split, sub, gsub, match with RSTART and RLENGTH, tolower, toupper, sprintf, and math functions such as int, sqrt, rand.

AWK Programs

  1. 20 AWK Associative Arrays: Counting, Grouping and Summing Use AWK associative arrays to count words, group rows by key, compute totals and averages, test membership, delete entries and loop in a stable order.
  2. 21 String Functions and Regular Expressions in AWK Master AWK's text functions: length, substr, index, split, sub, gsub, match with RSTART and RLENGTH, sprintf, toupper and tolower, plus regex matching rules.
  3. 22 printf and Formatted Reports in AWK Use AWK printf to align columns, pad and round numbers, build headers and footers with BEGIN and END, draw text bar charts and write tidy tabular reports.
  4. 23 User-defined Functions and Recursion in AWK Write your own AWK functions: parameters, return values, local variables, passing arrays by reference, recursion, helper libraries and loading code with -f.

Text Processing Pipelines

  1. 24 Pipelines with grep, sort, uniq, cut and AWK Build Unix text pipelines that combine grep, cut, sort, uniq, tr, head and AWK: top-N reports, frequency tables, filters, and when to replace three tools with one AWK.
  2. 25 Working with Several Files and Joins in AWK Read two or more files in one AWK run: FILENAME, FNR vs NR, the NR==FNR lookup idiom, joining and comparing files, getline, and passing values with -v.
  3. 26 Combining sed and AWK in Scripts Learn when to use sed and when to use AWK, how to chain them in pipelines, share shell variables safely, write executable scripts and avoid quoting pitfalls.
  4. 27 Multi-line Records, Paragraph Mode and CSV in AWK Control how AWK splits input: custom RS and FS, paragraph mode with RS empty, multi-line records, fixed-width data, and parsing quoted CSV fields correctly.

One-liners

  1. 28 sed One-liners: A Practical Collection A tested collection of sed one-liners with explanations: spacing, numbering, deleting and printing lines, trimming, joining, reversing, and selective editing.
  2. 29 AWK One-liners: A Practical Collection A tested collection of AWK one-liners with explanations: print columns, sum and average, filter by field, number and reverse lines, dedupe, transpose and more.
  3. 30 Text Surgery One-liners: sed and AWK Together One-liners for everyday text surgery: insert, append and replace lines, comment out settings, strip HTML tags, extract emails and URLs, convert TSV to CSV, and edit many files.

Real-world Tasks

  1. 31 Analyzing Web Server Access Logs with AWK and sed Parse Apache and nginx access logs: count hits, top URLs and IPs, status code breakdown, requests per hour, bandwidth totals, error hunting and response times.
  2. 32 CSV Reports and Data Cleaning with AWK Clean messy CSV with sed and AWK, then build reports: validate rows, trim and normalise values, fix dates, handle headers, group totals, percentages and top-N lists.
  3. 33 System Administration Text Tasks with sed and AWK Everyday sysadmin jobs with sed and AWK: parse /etc/passwd, df and ps output, find failed SSH logins, edit hosts and cron files, and build reports from command output.
  4. 34 Bulk Edits and Refactoring Source Code with sed and AWK Refactor many files safely: rename functions and variables, update imports and versions, add file headers, report TODOs, rename files in bulk and preview every change first.

Projects and Practice

  1. 35 Project: A Log Dashboard in One AWK Program Build a complete AWK program that turns an access log into a text dashboard: summary, status table, top pages and visitors, hourly traffic chart and an error list.
  2. 36 Project: A Markdown-to-HTML Converter with AWK and sed Write a small Markdown-to-HTML converter: headings, paragraphs, lists, code blocks, bold, italics, links and HTML escaping, using a line-by-line AWK state machine plus sed.
  3. 37 Project: A Template Engine and Mail Merge with AWK and sed Build a tiny template engine: replace {{placeholders}} from a key=value file, support defaults and conditionals, generate sed scripts, and mail-merge CSV rows into letters.
  4. 38 AWK and sed Practice Exercises with Solutions Twenty graded AWK and sed exercises with runnable solutions: line selection, substitution, columns, grouping, joins, debugging broken commands, plus a quick cheat sheet.