Skip to content

s2: Add 'best' compression option - #310

Merged
klauspost merged 4 commits into
masterfrom
s2-add-best-compression
Jan 13, 2021
Merged

s2: Add 'best' compression option#310
klauspost merged 4 commits into
masterfrom
s2-add-best-compression

Conversation

@klauspost

@klauspost klauspost commented Jan 12, 2021

Copy link
Copy Markdown
Owner

Add compression mode that will compress the data very well, but at a significant performance decrease.

Mainly for offline compression, but where decompression speed should still be high and compatible with other S2 compressed data.

Comparing on 16 core CPU:

* enwik10:
Default... 10000000000 -> 4761467548 [47.61%]; 1.098s, 8685.6MB/s
Better... 10000000000 -> 4225922984 [42.26%]; 2.817s, 3385.4MB/s
Best... 10000000000 -> 3667646858 [36.68%]; 35.995s, 264.9MB/s

* github-june-2days-2019.json:
Default... 6273951764 -> 1043196283 [16.63%]; 431ms, 13882.3MB/s
Better... 6273951764 -> 950079555 [15.14%]; 736ms, 8129.5MB/s
Best... 6273951764 -> 846260870 [13.49%]; 8.125s, 736.4MB/s

* nyc-taxi-data-10M.csv:
Default... 3325605752 -> 1095998837 [32.96%]; 324ms, 9788.7MB/s
Better... 3325605752 -> 960330423 [28.88%]; 602ms, 5268.4MB/s
Best... 3325605752 -> 794873295 [23.90%]; 6.619s, 479.1MB/s

* 10gb.tar
Default... 10065157632 -> 5916578242 [58.78%]; 1.028s, 9337.4MB/s
Better... 10065157632 -> 5650133605 [56.14%]; 2.172s, 4419.4MB/s
Best... 10065157632 -> 5246578570 [52.13%]; 25.696s, 373.6MB/s

* consensus.db.10gb:
Default... 10737418240 -> 4562648848 [42.49%]; 882ms, 11610.0MB/s
Better... 10737418240 -> 4542443833 [42.30%]; 3.3s, 3103.5MB/s
Best... 10737418240 -> 4272335558 [39.79%]; 38.955s, 262.9MB/s
Add compression mode that will compress the data very well, but at a significant performance decrease.

Mainly for offline compression, but where decompression speed should still be high.

Comparing to 'better':

```
github-june-2days-2019.json -> (discarded) 6273951764 -> 950079555 [15.14%]; 3541.4MB/s
github-june-2days-2019.json -> (discarded) 6273951764 -> 846260870 [13.49%]; 661.5MB/s

nyc-taxi-data-10M.csv -> (discarded) 3325605752 -> 960330423 [28.88%]; 3249.5MB/s
nyc-taxi-data-10M.csv -> (discarded) 3325605752 -> 794873295 [23.90%]; 437.5MB/s

10gb.tar -> (discarded) 10065157632 -> 5650133605 [56.14%]; 3137.9MB/s
10gb.tar -> (discarded) 10065157632 -> 5250102961 [52.16%]; 390.1MB/s

consensus.db.10gb -> (discarded) 10737418240 -> 4542443833 [42.30%]; 3008.2MB/s
consensus.db.10gb -> (discarded) 10737418240 -> 4272335541 [39.79%]; 248.8MB/s
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant