Skip to content

Conversation

@johannes94
Copy link

@johannes94 johannes94 commented Oct 8, 2025

Hello there @bep.

One of the projects I'm working on makes heavy use of the hashstructure lib.

Since we're hashing rather big objects, I noticed huge allocation overhead in the library and did some optimizations.
In particular:

  • Avoid using reflect.ValueOf for obvious string values
  • Avoid allocating visitOps object in a loop
  • Changes binary.Write to direct byte operations using a shared buffer for number types
  • Use the unsafe pkg for zero copy binary writes for strings
  • Only wrapping reflect.Value objects with v.Int(), v.Interface() etc. if absolutely necessary

I'm not sure if this is a fork you intent to maintain as a library for others to use, but since I'm building on your previous improvements I thought this might be valuable to you.

Side Note: Those improvements where partly suggested by AI, but I carefully reviewed them and made sure objects we hash within our projects produce hashes consistent with the previous version.

Benchmark:

goos: darwin
goarch: arm64
pkg: github.com/gohugoio/hashstructure
cpu: Apple M4 Pro
                  │   old.txt    │               new.txt                │
                  │    sec/op    │    sec/op     vs base                │
Map-14              3.862µ ± 24%   2.752µ ± 19%  -28.74% (p=0.000 n=10)
String/default-14   63.02n ±  5%   48.54n ±  4%  -22.98% (p=0.000 n=10)
String/xxhash-14    33.85n ±  7%   30.54n ±  9%   -9.79% (p=0.007 n=10)
geomean             202.0n         159.8n        -20.89%

                  │   old.txt   │               new.txt                │
                  │    B/op     │    B/op     vs base                  │
Map-14              1113.0 ± 0%   665.0 ± 0%  -40.25% (p=0.000 n=10)
String/default-14    56.00 ± 0%   24.00 ± 0%  -57.14% (p=0.000 n=10)
String/xxhash-14     16.00 ± 0%   16.00 ± 0%        ~ (p=1.000 n=10) ¹
geomean              99.91        63.44       -36.50%
¹ all samples are equal

                  │   old.txt   │               new.txt                │
                  │  allocs/op  │ allocs/op   vs base                  │
Map-14              120.00 ± 0%   43.00 ± 0%  -64.17% (p=0.000 n=10)
String/default-14    3.000 ± 0%   2.000 ± 0%  -33.33% (p=0.000 n=10)
String/xxhash-14     1.000 ± 0%   1.000 ± 0%        ~ (p=1.000 n=10) ¹
geomean              7.114        4.414       -37.95%
¹ all samples are equal
@bep
Copy link
Member

bep commented Oct 9, 2025

Thanks for this, much appreciated. And the numbers looks even better on my MacBook Pro M1:

cpu: Apple M1 Pro
                  │ master.bench │ jmalsam-upstream-performance-optimizations.bench │
                  │    sec/op    │          sec/op            vs base               │
Map-10              3.831µ ± ∞ ¹                2.315µ ± ∞ ¹  -39.57% (p=0.029 n=4)
String/default-10   55.92n ± ∞ ¹                30.50n ± ∞ ¹  -45.45% (p=0.029 n=4)
String/xxhash-10    17.12n ± ∞ ¹                13.93n ± ∞ ¹  -18.63% (p=0.029 n=4)
geomean             154.2n                      99.45n        -35.51%
¹ need >= 6 samples for confidence interval at level 0.95

                  │ master.bench │ jmalsam-upstream-performance-optimizations.bench │
                  │     B/op     │          B/op            vs base                 │
Map-10              1080.0 ± ∞ ¹               632.0 ± ∞ ¹  -41.48% (p=0.029 n=4)
String/default-10   40.000 ± ∞ ¹               8.000 ± ∞ ¹  -80.00% (p=0.029 n=4)
String/xxhash-10     0.000 ± ∞ ¹               0.000 ± ∞ ¹        ~ (p=1.000 n=4) ²
geomean                        ³                            -51.09%               ³
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ summaries must be >0 to compute geomean

                  │ master.bench │ jmalsam-upstream-performance-optimizations.bench │
                  │  allocs/op   │        allocs/op         vs base                 │
Map-10              116.00 ± ∞ ¹               39.00 ± ∞ ¹  -66.38% (p=0.029 n=4)
String/default-10    2.000 ± ∞ ¹               1.000 ± ∞ ¹  -50.00% (p=0.029 n=4)
String/xxhash-10     0.000 ± ∞ ¹               0.000 ± ∞ ¹        ~ (p=1.000 n=4) ²
@bep bep merged commit 1f6d247 into gohugoio:master Oct 9, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants