Feat/11856 generic visualisation v4 - #45
Conversation
Add a standalone tool to plot Suricata counter values from eve.json/eve-stats.json files. It reads JSON-lines, keeps the stats event records, optionally filters them with a universal jq filter, and plots the requested counter paths over time (or against the traffic multiplier with --by-multiplier). Key features: - Universal jq filter with automatic leading-dot normalization for bare field identifiers (e.g. 'uptime > 30' -> '.uptime > 30') - Correct series alignment: x and y extracted from the same record - --delta mode to show per-interval rates instead of cumulative values - --by-multiplier mode to plot a summary value against the multiplier - NaN handling in plots; empty/all-NaN series produce no figure - Robust error handling and informative diagnostics (typo detection, warn-once for insufficient --delta points) - Documented in README with usage examples
- _apply_filter: catch ValueError from jq.compile and return [] instead of
crashing on an invalid filter
- _normalize_filter: fix comment example to reflect preserved leading
whitespace (' uptime > 30' -> ' .uptime > 30')
- _extract_series: coerce extracted values to float, warning and skipping
non-numeric values instead of appending raw values that would later crash
There was a problem hiding this comment.
Pull request overview
Adds a new Python utility (util/visualize.py) for plotting Suricata stats counters from eve.json / eve-stats.json (with jq-based filtering, delta mode, and a multiplier-summary mode), and documents this workflow in the README so users can more easily visualize performance metrics.
Changes:
- Added
util/visualize.pyto extract and plot arbitrary Suricata counter paths over time or against traffic multiplier. - Extended
README.mdwith a new “Generic counter visualisation” section (TOC entry, options overview, and example commands).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| util/visualize.py | New CLI tool to read/filter stats records and plot selected counters (time series or multiplier summaries). |
| README.md | Documents util/visualize.py usage, options, and example invocations; adds TOC entry. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| for counter_path in args.path: | ||
| y_compiled = _compile_path(counter_path) | ||
| xs: List[float] = [] | ||
| ys: List[float] = [] | ||
| extracted_any = False | ||
| for multiplier, stats_path in multipliers: | ||
| records = _read_stats_records(stats_path) | ||
| if args.filter: | ||
| records = _apply_filter(records, args.filter) | ||
| # Extract x and y from the same record so they stay aligned | ||
| # even when some records lack the counter path. | ||
| points = _extract_series( | ||
| records, x_compiled, y_compiled, "uptime", counter_path | ||
| ) | ||
| if points: | ||
| extracted_any = True | ||
| summary = _summarize_series(points, args.delta) | ||
| xs.append(multiplier) | ||
| ys.append(summary) | ||
| if not extracted_any: | ||
| logger.warning( | ||
| "Counter path %r extracted no values from any multiplier " | ||
| "run; check for a typo.", | ||
| counter_path, | ||
| ) | ||
| name = f"{counter_path} ({label})" if label else counter_path | ||
| series.append((name, list(zip(xs, ys)))) |
There was a problem hiding this comment.
Does not really matter that much in this use case. Considering that with large graphs with a lot of input data makes it harder to read the graph I think this is not a relevant for this.
There was a problem hiding this comment.
I wouldn't worry about this too much either.
Maybe you could try to tell an LLM to refactor _main_by_multiplier into the normal main, so that we don't have two functions that do almost the same thing and see if you end up with something reasonable.
|
Incase the images thing is not noticed, also putting the link here. #42 |
There was a problem hiding this comment.
I don't have almost any comments about the implementation, somewhat because I have never worked with matplotlib.
The only issue that I have found is that some of the graphs would really benefit from an alternative Y axis. This is mostly visible on graphs with pkts and bytes, where bytes will always be some orders of magnitude larger than pkts and when plotting memuse for flow and tcp, because there is a large empty gap between them. Both of these cases result in a series (or multiple) that are pretty much flat, even though they necessarily don't have to be.
A useful heuristic for this might be the difference between the Y range (highest - lowest point) of all combined series and the Y range of individual series.
(I forgot that replying in a thread will also copy the comment here, so they look weird out of context)
| for counter_path in args.path: | ||
| y_compiled = _compile_path(counter_path) | ||
| xs: List[float] = [] | ||
| ys: List[float] = [] | ||
| extracted_any = False | ||
| for multiplier, stats_path in multipliers: | ||
| records = _read_stats_records(stats_path) | ||
| if args.filter: | ||
| records = _apply_filter(records, args.filter) | ||
| # Extract x and y from the same record so they stay aligned | ||
| # even when some records lack the counter path. | ||
| points = _extract_series( | ||
| records, x_compiled, y_compiled, "uptime", counter_path | ||
| ) | ||
| if points: | ||
| extracted_any = True | ||
| summary = _summarize_series(points, args.delta) | ||
| xs.append(multiplier) | ||
| ys.append(summary) | ||
| if not extracted_any: | ||
| logger.warning( | ||
| "Counter path %r extracted no values from any multiplier " | ||
| "run; check for a typo.", | ||
| counter_path, | ||
| ) | ||
| name = f"{counter_path} ({label})" if label else counter_path | ||
| series.append((name, list(zip(xs, ys)))) |
There was a problem hiding this comment.
I wouldn't worry about this too much either.
Maybe you could try to tell an LLM to refactor _main_by_multiplier into the normal main, so that we don't have two functions that do almost the same thing and see if you end up with something reasonable.
This pull request adds new documentation to the
README.mddescribing how to use theutil/visualize.pyscript for visualizing generic Suricata counter values from stats files. This provides users with detailed instructions and examples for plotting various statistics, making it easier to analyze Suricata performance metrics.Documentation improvements:
util/visualize.py)" to improve discoverability of visualization features.util/visualize.py, explaining how to plot Suricata counter values over time or by traffic multiplier, and how to use filtering, delta plotting, and output options.Images are here.