Log Parsing Reference

__omp_shell("")

Log parsing on the command line is less about memorising six Unix tools and more about understanding how they combine. Each tool does one narrow transformation well. Together they let you reduce gigabytes of noisy text into one small answer you can act on during an incident.

grep is usually the first filter. It selects lines that match a pattern, which makes it ideal for isolating one service name, one request ID, one error class, or one time window. Good incident work often starts by removing 99 percent of irrelevant lines. grep can also invert matches, count occurrences, and search recursively, so it is often the fastest way to establish whether a suspected event really happened.

cut is useful when the log format is regular and field-based. If a space, tab, or comma consistently separates columns, cut can extract the timestamp, status code, or user ID you care about without heavier parsing. It is fast, but brittle when delimiters appear inside quoted values. That is why it works well on simple access logs and much less well on complex JSON payloads.

sed shines when the text needs lightweight rewriting. You can normalise repeated prefixes, strip noise, replace tokens, or print only certain ranges. During incident response, sed is often the fastest way to reshape logs into something the next tool can consume. It is especially handy when timestamps or identifiers need to be cleaned before aggregation.

awk is the step up from simple slicing. It understands records and fields, and it can compute while it scans. That makes it useful for summing counts, grouping by a column, calculating latency buckets, or applying conditions like "show only 5xx responses where duration exceeded 200 ms". If the pipeline needs logic rather than plain filtering, awk is usually the right tool.

sort and uniq are aggregation partners. sort groups identical lines together, and uniq -c can then count them. This is the classic pattern for answering questions such as which exception occurred most often or which endpoint produced the most failures. Because uniq only collapses adjacent duplicates, sorting first is usually essential.

A simple example makes the workflow concrete. Suppose you want the timestamps for exceptions from one service:

grep "xxService" service.log | grep "Exception" | cut -d" " -f2

Suppose instead you want the most common failing endpoint in an access log:

grep " 500 " access.log | awk '{print $7}' | sort | uniq -c | sort -nr

The important lesson is not the exact syntax. It is the sequence of operations: filter, extract, reshape, aggregate. Once you think in those stages, command-line log analysis becomes a reliable habit rather than a bag of one-off tricks. For structured logs, tools like jq may be better, but the Unix text stack remains useful because production systems still emit a lot of plain text.