Skip to content

Commit bc4521a

Browse files
rpuneetclaude
andcommitted
refactor(bench): replace benchmark tools with unified comprehensive solution
Replaced multiple benchmark scripts with a single integrated tool that provides human-readable reports and optional historical performance graphs. **Key Changes**: Removed: - scripts/bench-viz.py (replaced) - scripts/bench-report.sh (replaced) - scripts/bench-compare.sh (replaced) - bench.txt, bench-report.txt, bench_results.json from .gitignore Added: - scripts/bench.py - Comprehensive benchmark tool with: - Human-readable reports grouped by category (API, Parsers, Format) - Multiple performance metrics per benchmark - Summary statistics (fastest/slowest, averages) - Optional historical performance graphs across git commits **Metrics Reported**: - **Ops/Sec** - Operations per second (throughput) - **ns/op** - Nanoseconds per operation (latency) - **MB/s** - Megabytes per second (data throughput) - **B/op** - Bytes allocated per operation (memory usage) - **allocs/op** - Number of allocations per operation **Usage**: ```bash make bench # Run benchmarks with comprehensive report make bench N=50 # Run benchmarks + generate graphs for last 50 commits ``` **Historical Graphs** (when N specified): - out/ops_per_sec.png - Throughput trends over time - out/ns_per_op.png - Latency trends over time - out/throughput.png - Data throughput (MB/s) trends - out/bytes_per_op.png - Memory allocation trends - out/allocs_per_op.png - Allocation count trends **Makefile Updates**: - Simplified bench target to use new tool - Removed bench-all, bench-report, bench-viz targets - Single command for all benchmark needs **Documentation Updates**: - README.md: Updated with new benchmark usage and metrics - CONTRIBUTING.md: Detailed explanation of benchmark tool and graphs - All documentation now reflects unified approach The tool automatically skips commits where benchmarks fail to compile, making it robust for historical analysis across code structure changes. Requirements: Python 3 + matplotlib (only for historical graphs) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 4052c8d commit bc4521a

File tree

8 files changed

+546
-658
lines changed

8 files changed

+546
-658
lines changed

.gitignore

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,6 @@ go.work.sum
2424
debug/
2525

2626
# Benchmark outputs
27-
bench.txt
28-
bench-report.txt
29-
bench_output.txt
30-
bench_results.json
3127
*.prof
3228
*.test
3329
trace.out

CONTRIBUTING.md

Lines changed: 31 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -293,41 +293,44 @@ The `Extractor` type clones configuration per call and is safe for concurrent us
293293
### Running Benchmarks
294294

295295
```bash
296-
make bench # Quick benchmark run
297-
make bench-all # Detailed output (3s timing, saved to bench.txt)
298-
make bench-report # Generate formatted report
299-
make bench-compare OLD=commit1 NEW=commit2 # Compare commits
300-
make bench-viz N=50 # Generate performance graphs across last N commits
301-
```
302-
303-
### Performance Visualization
296+
# Run benchmarks with comprehensive report
297+
make bench
304298

305-
Generate graphs showing how performance evolved over time:
306-
307-
```bash
308-
# Generate graphs for last 50 commits
309-
make bench-viz N=50
310-
311-
# Custom number of commits
312-
make bench-viz N=100
299+
# Run benchmarks + generate historical graphs
300+
make bench N=50 # Last 50 commits
301+
make bench N=100 # Last 100 commits
313302
```
314303

315-
**Requirements**: Python 3 with matplotlib
304+
**Benchmark Report Includes**:
305+
- Human-readable performance metrics grouped by category (API, Parsers, Format)
306+
- Multiple metrics per benchmark:
307+
- **Ops/Sec** - Operations per second (throughput)
308+
- **ns/op** - Nanoseconds per operation (latency)
309+
- **MB/s** - Megabytes per second (data throughput)
310+
- **B/op** - Bytes allocated per operation (memory usage)
311+
- **allocs/op** - Number of allocations per operation
312+
- Summary statistics (fastest/slowest, averages)
313+
314+
**Historical Performance Graphs**:
315+
316+
When you specify `N=commits`, the tool will:
317+
1. Run benchmarks on current commit and show results
318+
2. Checkout each of the last N commits
319+
3. Run benchmarks (skip commits where benchmarks fail)
320+
4. Generate graphs in `out/` directory showing performance trends
321+
322+
**Graph Files Generated**:
323+
- `out/ops_per_sec.png` - Throughput over time
324+
- `out/ns_per_op.png` - Latency over time
325+
- `out/throughput.png` - Data throughput (MB/s) over time
326+
- `out/bytes_per_op.png` - Memory allocations over time
327+
- `out/allocs_per_op.png` - Allocation count over time
328+
329+
**Requirements**: Python 3 + matplotlib (only for historical graphs)
316330
```bash
317331
pip3 install matplotlib
318332
```
319333

320-
The tool will:
321-
1. Checkout each commit in git history
322-
2. Run benchmarks (skip if they fail to compile)
323-
3. Collect performance metrics
324-
4. Generate graphs in `out/` directory showing trends over time
325-
326-
Graphs are generated for each benchmark showing:
327-
- Nanoseconds per operation (ns/op)
328-
- Bytes allocated per operation (B/op)
329-
- Allocations per operation (allocs/op)
330-
331334
### Continuous Benchmarking
332335

333336
Every commit to `main` and every PR automatically runs benchmarks via GitHub Actions:

Makefile

Lines changed: 4 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: build test test-lib lint fmt clean install coverage coverage-html check help bench bench-all bench-report bench-compare bench-viz
1+
.PHONY: build test test-lib lint fmt clean install coverage coverage-html check help bench
22

33
# Go settings
44
GOCMD = go
@@ -90,35 +90,9 @@ example: build
9090
@echo "Running example..."
9191
./$(BIN_DIR)/imx testdata/goldens/jpeg/google_iptc.jpg
9292

93-
# Run benchmarks
93+
# Run benchmarks with comprehensive report
9494
bench:
95-
@echo "Running benchmarks..."
96-
$(GOTEST) -bench=. -benchmem -run=^$$ . ./internal/meta/... ./internal/format/...
97-
@echo "✓ Benchmarks complete"
98-
99-
# Run benchmarks with detailed output
100-
bench-all:
101-
@echo "Running all benchmarks with detailed output..."
102-
$(GOTEST) -bench=. -benchmem -benchtime=3s -run=^$$ . ./internal/meta/... ./internal/format/... | tee bench.txt
103-
@echo "✓ Benchmarks saved to bench.txt"
104-
105-
# Generate detailed benchmark report
106-
bench-report:
107-
@echo "Generating benchmark report..."
108-
@./scripts/bench-report.sh
109-
@echo "✓ Benchmark report generated"
110-
111-
# Compare benchmarks between commits
112-
bench-compare:
113-
@echo "Comparing benchmarks between commits..."
114-
@./scripts/bench-compare.sh $(OLD) $(NEW)
115-
@echo "Usage: make bench-compare OLD=commit1 NEW=commit2"
116-
117-
# Generate benchmark visualizations across git history
118-
bench-viz:
119-
@echo "Running benchmarks across git history and generating graphs..."
120-
@./scripts/bench-viz.py --max-commits $(or $(N),50) --graphs --output out
121-
@echo "✓ Benchmark visualization complete. Graphs saved to out/"
95+
@./scripts/bench.py $(if $(N),-n $(N),)
12296

12397
# Show help
12498
help:
@@ -138,10 +112,6 @@ help:
138112
@echo " install - Install CLI to GOPATH/bin"
139113
@echo " coverage - Show coverage report (100% target)"
140114
@echo " coverage-html- Generate HTML coverage report"
141-
@echo " bench - Run all benchmarks"
142-
@echo " bench-all - Run benchmarks with detailed output"
143-
@echo " bench-report - Generate detailed benchmark report"
144-
@echo " bench-compare- Compare benchmarks between commits"
145-
@echo " bench-viz - Generate benchmark graphs across git history (N=commits)"
115+
@echo " bench - Run benchmarks with report (use N=commits for graphs)"
146116
@echo " example - Build and run example"
147117
@echo " help - Show this help"

README.md

Lines changed: 26 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -196,39 +196,44 @@ BenchmarkParser_JPEG-12 481448 2.5μs/op 48KB/op 24 a
196196
- Automatic regression detection with 120% threshold
197197
- Performance alerts on PRs
198198

199-
**Run Benchmarks Locally**:
199+
**Run Benchmarks**:
200200
```bash
201-
make bench # Quick benchmark run
202-
make bench-all # Detailed output with 3s timing
203-
make bench-report # Generate formatted report
204-
make bench-compare OLD=commit1 NEW=commit2 # Compare commits
205-
make bench-viz N=50 # Generate performance graphs across last 50 commits
201+
# Run benchmarks with comprehensive human-readable report
202+
make bench
203+
204+
# Run benchmarks + generate historical performance graphs
205+
make bench N=50 # Last 50 commits
206206
```
207207

208-
**Performance Visualization**:
208+
The benchmark tool provides:
209+
- **Human-readable report** with performance metrics grouped by category
210+
- **Summary statistics** showing fastest/slowest benchmarks
211+
- **Multiple metrics**:
212+
- **Ops/Sec** - Operations per second (throughput)
213+
- **ns/op** - Nanoseconds per operation (latency)
214+
- **MB/s** - Megabytes per second (data throughput)
215+
- **B/op** - Bytes allocated per operation (memory usage)
216+
- **allocs/op** - Number of allocations per operation
217+
218+
**Historical Performance Graphs**:
209219

210-
Generate performance graphs showing how benchmarks evolved over time:
220+
Generate performance trend graphs across git history:
211221

212222
```bash
213-
# Generate graphs for last 50 commits
214-
make bench-viz N=50
223+
make bench N=50 # Creates graphs in out/ directory
215224
```
216225

217-
This creates visualizations in `out/` directory showing:
218-
- **ns/op** - Nanoseconds per operation over time
219-
- **B/op** - Bytes allocated per operation over time
220-
- **allocs/op** - Number of allocations per operation over time
221-
222-
Example graphs (run `make bench-viz` to generate):
223-
224-
![High-Level API Performance](out/summary_HighLevelAPI.png)
225-
![Parser Performance](out/summary_ParserEXIF.png)
226+
Generates graphs showing trends over time:
227+
- `out/ops_per_sec.png` - Operations per second across commits
228+
- `out/ns_per_op.png` - Latency across commits
229+
- `out/throughput.png` - MB/s throughput across commits
230+
- `out/bytes_per_op.png` - Memory allocations across commits
231+
- `out/allocs_per_op.png` - Allocation count across commits
226232

227-
**Requirements**: Python 3 with matplotlib (`pip3 install matplotlib`)
233+
**Requirements**: Python 3 with matplotlib for graphs (`pip3 install matplotlib`)
228234

229235
**Tools**:
230236
- Results tracked via [github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark)
231-
- Statistical comparison using [benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat)
232237
- Historical visualization using Python + matplotlib
233238

234239
## Contributing

scripts/bench-compare.sh

Lines changed: 0 additions & 52 deletions
This file was deleted.

scripts/bench-report.sh

Lines changed: 0 additions & 40 deletions
This file was deleted.

0 commit comments

Comments
 (0)