Log File Format Troubleshooting: Parsing JSON, Syslog, and Custom Formats
Troubleshoot common log file parsing issues. Covers multiline log entries, timestamp format variations, character encoding problems, and strategies for parsing non-standard log formats.
Key Takeaways
- Log files from different applications and systems use incompatible formats, inconsistent timestamp representations, and varying field delimiters.
- Stack traces, SQL queries, and XML payloads span multiple lines.
- Different applications use different timestamp formats — ISO 8601, Unix epoch, RFC 2822, or locale-specific strings.
- Legacy applications may write logs in Windows-1252, Shift_JIS, or other encodings while modern tooling expects UTF-8.
CSV ↔ JSON Converter
Convert between CSV and JSON formats
Why Log Parsing Breaks
Log files from different applications and systems use incompatible formats, inconsistent timestamp representations, and varying field delimiters. When a log aggregation pipeline silently drops entries or misparses fields, finding the root cause requires understanding both the source format and the parser's expectations.
Common Log Formats
| Format | Structure | Multiline | Typing | Example Source |
|---|---|---|---|---|
| JSON Lines | One JSON object per line | No (by design) | Yes | Docker, structlog |
| Syslog (RFC 5424) | Structured with PRI | No | Limited | Linux system logs |
| Apache/Nginx CLF | Space-delimited | No | No | Web servers |
| Java stack traces | Unstructured | Yes | No | JVM applications |
| Custom delimited | Application-specific | Maybe | No | Legacy applications |
Multiline Log Entries
Stack traces, SQL queries, and XML payloads span multiple lines. Most parsers treat each line as a separate entry, splitting stack traces into dozens of fragments. Solutions include: configuring multiline patterns in the collector (Filebeat, Fluentd), switching to JSON Lines format, or using a codec that recognizes continuation patterns.
Timestamp Normalization
Different applications use different timestamp formats — ISO 8601, Unix epoch, RFC 2822, or locale-specific strings. A single pipeline might receive all of these simultaneously. Define timestamp parsing rules per source and normalize to UTC ISO 8601 at ingestion time.
Encoding Issues
Legacy applications may write logs in Windows-1252, Shift_JIS, or other encodings while modern tooling expects UTF-8. Invalid byte sequences cause silent truncation or parser crashes. Identify the source encoding and convert at the collection layer.