libgitcore: add sha1dc as an optional feature

[dscho]

The new Rust crate sha1dc promises not only type safety but also much better performance compared to the collision-detecting SHA-1 library Git uses at the moment. This performance comes mostly from a SIMD-centric design that relies on features provided by many x86_64 and aarch64 CPUs.

Let's optionally use this crate, toggled by the build option DC_SHA1_RS.

In a test on a Ryzen 7 machine with a ~140MB pack, Rust-backed git-index-pack.exe was 2.80x faster by median wall time. Five C/Rust pairs ran in a randomized, balanced order with --verify --no-rev-index --threads=1 --object-format=sha1:

SHA1DC backend	Median	Best of five
C (default)	24.1s	23.4s
Rust (-rs)	8.6s	8.5s

Note that the sha1dc crate requires a significantly newer Rust version than what Git's existing Rust support requires: 1.87 instead of 1.63.

[srijs]

@dscho fyi, just published 0.1.3 which is another 9–11% faster than 0.1.2 on large inputs, and 32–39%
faster on messages of up to 64 bytes. In case you want to pull this instead :)

[dscho]

@srijs very nice!

[dscho]

@srijs hmm. I cannot see any change in speed... On my Ryzen 7 (under higher ambient load than I performed the earlier benchmark):

SHA1DC backend	Median	Best of five
C (default)	32.9s	32.1s
Rust 0.1.2	10.9s	10.4s
Rust 0.1.3	10.7s	10.5s

By the way, my colleague ran this on a Mac and got this:

SHA1DC backend	Median	Best of five
C (default)	8.035s	8.009s
Rust 0.1.2	4.359s	4.338s
Rust 0.1.3	4.361s	4.268s

Naturally, M4s are much faster than Ryzens, even if the difference between C and Rust is not quite as pronounced on the former ;-)

But as you see, the difference in performance between v0.1.2 and v0.1.3 is basically lost in the noise in our experiments. Maybe your speed improvements target different CPUs?

[srijs]

Huh, that's interesting... this is the same git-index-pack benchmark you used before? Do you have a sense on how much % of time is now left being spent on SHA-1 hashing vs. the compression and the rest of the workload, i.e. how we should expect a ~10% hashing improvement to translate to index pack performance proportionally?

I rechecked again multiple times, and I still see the ~10% improvement very clearly. FWIW, my testing set up is the criterion harness that's checked into the sha1dc repo, ran on an Apple M4 laptop as well as Intel Xeon (EC2 c7i.metal-24xl) and AWS Graviton 4 (EC2 c8g.metal-24xl).

Some thoughts:

• Given I am testing the SHA-1 hashing in isolation and over many iterations, not sure how much of that effect could be warm instruction cache, branch predictor, etc vs mixed workload?
• I wonder how differences between AMD and Intel processors could play into it here? I fine-tuned the AVX2 search parameters on Intel only so far, so any differences in SIMD execution, branch prediction, etc may have thrown that off?
• My tests are mostly on random data, whereas presumably your git repo is not. Not sure if that could be an influence too?

In any case, I guess at least it's not gotten worse 🤣

That's a good reality check though; I'll see if I need to improve my benchmarking set up for future experiments.

PS: If you have the time and are inclined, I wonder if you'd get different results if you ran the benchmarks in the sha1dc repo. You can use the --save-baseline flag on Criterion to save results for 0.1.2 and then it'll automatically compare on the next run.

PS2: Oh also, I don't know how much it makes a difference, but I noticed you heap allocate the hasher, vs. both my own benchmark as well as gitoxide stack allocate it. Part of my improvements esp. for smaller inputs was a smaller hasher struct 🤷

[dscho]

I have now performed the same benchmark in a Windows Subsystem for Linux instance on the same Windows machine:

SHA1DC backend	Median	Best of five
C (default)	23.971s	23.435s
Rust 0.1.2	9.123s	8.492s
Rust 0.1.3	9.020s	8.491s

You can clearly see that filesystem performance seems to contribute, as I copied the packfile in question to the ext4 filesystem. But again, the difference between 0.1.2 and 0.1.3 is lost in the noise.

Of course, it's totally possible that I am holding this thing wrong and that simply editing the version number in Cargo.toml followed by touch src/sha1dc_rs.rs && make -j15 DC_SHA1_RS=1 would not pick up the new-and-improved crate...

this is the same git-index-pack benchmark you used before?

@srijs Yes: time ./git index-pack --verify --no-rev-index --threads=1 --object-format=sha1 pack-9bd13bbf01e82d6d29839523b537eba949835fd7.pack. It's a ~140MB packfile that is taken directly from my git-for-windows/git checkout, it was produced some 2 weeks ago (most likely during scheduled maintenance).

Do you have a sense on how much % of time is now left being spent on SHA-1 hashing vs. the compression and the rest of the workload, i.e. how we should expect a ~10% hashing improvement to translate to index pack performance proportionally?

Unfortunately I have no intuition about this. So I started a Linux perf session.

With the default, C version of SHA1DC, 73% of the time is spent in the git_hash_update() call tree, i.e. basically in SHA1DC's sha1_process(). Interesting: a total of 43.66% of that index-pack process are spent in sha1_compression_states(), and 25.88% in ubc_check(). These numbers are from a single run, so take all of that with a grain of salt.

With the Rust sha1dc 0.1.2, it is a totally different picture: resolve_deltas() is the limiting factor, taking 67.63% of the time, and git_hash_update() only takes up 29.58% of the total time (13.42% of the time is spent running sha1dc::ubc_check::avx2::check()).

With 0.1.3, it is a similar picture: 29.30% are spent in the git_hash_update() call tree, i.e. pretty much the same as 0.1.2, maybe a tad less. Again, this is from a single run.

Now, This is a laptop on which I'm running this, and I do like to drive it hard. So the numbers will fluctuate quite a bit, which is why I tried to measure the calls in 5 rounds of randomized order.

I wonder if you'd get different results if you ran the benchmarks in the sha1dc repo. You can use the --save-baseline flag on Criterion to save results for 0.1.2 and then it'll automatically compare on the next run.

That's a good idea! Unfortunately, I am a bit busy with preparations for Git for Windows v2.56 (https://github.com/git-for-windows/git-sdk-64/actions tells me that we're not ready to release), but I will try to remember when things are less hectic.

I don't know how much it makes a difference, but I noticed you heap allocate the hasher, vs. both my own benchmark as well as gitoxide stack allocate it. Part of my improvements esp. for smaller inputs was a smaller hasher struct 🤷

Unfortunately, I cannot use stack allocation here because Git often needs to call git_hash_update() many times for the same stream, and therefore it need the hasher in a form that persists after git_hash_init() returns.

Having said all that, those performance improvements are really exciting to me, both with 0.1.2 and 0.1.3. Thank you so, so much!

[srijs]

With the Rust sha1dc 0.1.2, it is a totally different picture: resolve_deltas() is the limiting factor, taking 67.63% of the time, and git_hash_update() only takes up 29.58% of the total time (13.42% of the time is spent running sha1dc::ubc_check::avx2::check()).

Makes sense, I guess at the point where it's <30%, plus all the mixed workloads plus I/O, you might not expect to see the same improvement in the real world. FWIW, in gitoxide these were the gains we saw on M4:

operation	sha1dc 0.1.2	sha1dc 0.1.3
gix free pack verify (1 thread)	5.61s	5.37s	1.04x
gix free pack index create (1 thread)	7.29s	7.06s	1.03x
gix free pack verify (all cores)	0.94s	0.90s	1.05x
gix free pack index create (all cores)	2.65s	2.61s	1.02x

That's a good idea! Unfortunately, I am a bit busy with preparations for Git for Windows v2.56 (https://github.com/git-for-windows/git-sdk-64/actions tells me that we're not ready to release), but I will try to remember when things are less hectic.

No worries, I'll work on improving my benchmarking in the meantime. I've already added Zen 3 and 4 to my benchmarking roster to make sure I can cover AMD as well.

Having said all that, those performance improvements are really exciting to me, both with 0.1.2 and 0.1.3. Thank you so, so much!

Appreciate the warm words, and looking forward to seeing this land! LMK if I can help with anything.

[dscho]

I now repeated this benchmark in a Windows/ARM64 cloud PC, and got this:

SHA1DC backend	Median	Best of five
C (default)	0m17.926s	0m17.953s
Rust 0.1.2	0m10.707s	0m10.750s
Rust 0.1.3	0m10.737s	0m10.892s

Maybe slightly less pronounced of a speed-up than on Windows/x64, but definitely something to cheer about!