Most log questions (top N, count per X, filter by Y) are answered by chaining small Unix filters into one pipeline. This page shows the standard pipeline shape, then the two filters that need the most explanation: awk for picking fields and uniq for collapsing duplicates.
Text-processing pipelines
A handful of small Unix filters, chained into one pipeline over a log file, answer most “top N”, “count per X”, and “filter by Y” questions.1 Each stage does one thing: select a field, group identical values, count them, rank them.
The standard shape
awk '{print $1}' Bash/access.log | sort | uniq -c | sort -nr | head -5 finds the top five client IPs: awk prints field 1, sort groups identical values, uniq -c counts adjacent duplicates, sort -nr ranks numerically in reverse, and head -5 keeps five lines.
| Question | Pipeline |
|---|---|
| Requests per path | cut -d'"' -f2 then cut -d' ' -f2, then sort | uniq -c | sort -nr |
| Distinct IPs | awk '{print $1}' | sort -u | wc -l |
| Exact path match | grep ' /api/ads HTTP/', so /api/ads-v2 does not match |
| Requests per method | cut -d'"' -f2 | cut -d' ' -f1 | sort | uniq -c | sort -nr |
| Top memory processes | ps aux --sort=-%mem | head -6 on Linux |
As given in the quick reference.1
Pitfalls
uniqonly collapses adjacent identical lines, so it normally needs sorted input.2- Quotes do not stop
awkfrom splitting on spaces, and User-Agent values contain spaces, so later field numbers are unstable; cut on the quote character instead.3 - GNU and BSD options differ:
ps --sortis Linux-only, and macOS needsps aux | tail -n +2 | sort -k4 -nr.1
awk
awk processes text one line at a time: it splits each line into fields and runs condition { action } rules against them. The name comes from its authors, Aho, Weinberger, and Kernighan.3
Essentials
| Syntax | Meaning |
|---|---|
$1, $2 | First and second fields (whitespace-separated by default) |
$0 | The whole line |
NF, $NF | Number of fields; the last field |
NR | Current line number |
-F: | Use another delimiter, such as : for /etc/passwd |
As described in the note. Examples: awk '$9 >= 500 {print $1, $9}' prints the IP and status of 5xx requests in the sample log.3
Finding a field number
awk 'NR == 1 {for (i = 1; i <= NF; i++) print i, $i}' file prints every field of the first line with its number. In the sample combined log, $1 is the client IP, $7 the path, $9 the status, and $10 the response size.
When --help is unsupported, use man awk; in restricted environments such as HackerRank, awk --help 2>&1 | less.3
uniq
uniq (from “unique”) combines identical lines only when they are adjacent; it has no memory of earlier lines. That is why it almost always comes right after sort.
| Option | Effect |
|---|---|
-c | Prefix each line with its count |
-d | Show only repeated lines |
-u | Show only lines that occur once |
-i | Ignore case |
As listed in the note. If you only need sorted, deduplicated output, sort -u does both. The macOS/BSD version may not support --help; use man uniq.2
Related
- APT package management: installing and troubleshooting Ubuntu packages.
- Python SRE drills: the same log-parsing tasks in Python.
- Domain index: other pages in this domain.