Skip to main content

Log Parsing Pipelines

grep is the first tool in almost every log analysis workflow. It pre-filters massive log files down to the lines that matter, letting downstream tools (awk, jq, sort, uniq) work on a tiny, relevant subset.

The Core Pipeline Pattern​

log file  →  grep (filter lines)  →  awk/cut (extract fields)  →  sort | uniq (aggregate)

1. Counting Errors Over Time​

# How many ERROR lines today?
grep -c "ERROR" /var/log/app.log

# Errors per hour (if logs contain HH:MM timestamps)
grep "ERROR" /var/log/app.log | grep -o "[0-9]\{2\}:[0-9]\{2\}" | cut -d: -f1 | sort | uniq -c

2. Extracting IP Addresses (-o)​

-o prints only the matched text — one result per line — instead of the whole line.

# All IPs from Nginx access log
grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" /var/log/nginx/access.log

# Top 10 attacking IPs
grep -E -o "([0-9]{1,3}\.){3}[0-9]{1,3}" /var/log/nginx/access.log \
  | sort | uniq -c | sort -nr | head -10

3. Filtering 4xx/5xx HTTP Errors​

# All 404 and 500 lines
grep -E " (404|500) " /var/log/nginx/access.log

# Count by status code
grep -E -o " [45][0-9]{2} " /var/log/nginx/access.log \
  | sort | uniq -c | sort -nr

4. Extracting Java Stack Traces​

Stack traces are multi-line. Use -A (after context) to capture the full trace:

# Capture exception line plus next 25 lines
grep -A 25 "Exception in thread" application.log

# Get only unique exception class names
grep -o "java\.lang\.\w\+Exception" application.log | sort -u

5. Filtering Structured (JSON) Logs​

Many apps write NDJSON logs. Chain grep with jq:

# Pre-filter to error level with grep (fast), then parse with jq
grep '"level":"error"' app.log | jq -r '"\(.timestamp) \(.message)"'

# Find all unique error codes
grep '"status":[45]' events.log | jq '.status' | sort -u

6. Live Log Monitoring​

grep buffers output when writing to a pipe. Add --line-buffered to see results immediately:

# Monitor live for errors — prints each match immediately
tail -f /var/log/app.log | grep --line-buffered -E "ERROR|FATAL"

# Multiple services at once
tail -f /var/log/app.log /var/log/worker.log | grep --line-buffered "ERROR"

7. The Full Incident Response Pipeline​

When an alert fires, run this to get a 60-second picture of what happened:

LOG="/var/log/app.log"

echo "=== Error count last hour ==="
grep "ERROR" "$LOG" | grep "$(date +'%Y-%m-%d %H')" | wc -l

echo "=== Unique error messages ==="
grep "ERROR" "$LOG" | grep -o "ERROR:.*" | sort | uniq -c | sort -nr | head -20

echo "=== Last 10 fatal events ==="
grep "FATAL" "$LOG" | tail -10