Skip to content

Add LogsDB option to route on sort fields#116687

Merged
kkrik-es merged 58 commits intoelastic:mainfrom
kkrik-es:logsdb/route-on-sort-fields-3
Dec 19, 2024
Merged

Add LogsDB option to route on sort fields#116687
kkrik-es merged 58 commits intoelastic:mainfrom
kkrik-es:logsdb/route-on-sort-fields-3

Conversation

@kkrik-es
Copy link
Contributor

@kkrik-es kkrik-es commented Nov 12, 2024

This is a storage optimization, improving codec efficiency. It requires configuring more than one sort fields (excluding @timestamp) to reduce the likelihood for hotspots in shard routing.

Related to #109334
@kkrik-es kkrik-es self-assigned this Nov 12, 2024
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Nov 12, 2024

Hi @kkrik-es, I've created a changelog YAML for you.
@kkrik-es
Copy link
Contributor Author

kkrik-es commented Nov 14, 2024 •
edited
Loading

EDIT: ignore this, more results below.

results.txt
ESBench results look really promising:

| Cumulative indexing time of primary shards |   1263.28        |    591.821       |    -671.462       |    min |    -53.15% |
|                               Dataset size |     35.842       |     14.9673      |     -20.8747      |     GB |    -58.24% |
|                                 Store size |     35.842       |     14.9673      |     -20.8747      |     GB |    -58.24% |
@kkrik-es kkrik-es requested a review from martijnvg November 14, 2024 06:14
@kkrik-es kkrik-es marked this pull request as ready for review November 14, 2024 06:14
@kkrik-es kkrik-es requested a review from a team as a code owner November 14, 2024 06:14
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Nov 14, 2024

Pinging @elastic/es-storage-engine (Team:StorageEngine)
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
4065c63 
# Conflicts:
#	server/src/main/java/org/elasticsearch/common/TimeBasedKOrderedUUIDGenerator.java
#	server/src/main/java/org/elasticsearch/common/UUIDs.java
@kkrik-es
add test
9457a47
@kkrik-es
fix test
3fd9350
@kkrik-es
Copy link
Contributor Author

kkrik-es commented Nov 14, 2024

ESBench results look really promising:

There was an issue with the run above, there were many indexing errors due to missing sort field values. Updated results (sortin on [host.name, host.os.name, @timestamp):

| Cumulative indexing time of primary shards |   1263.28        |  1158.67        |   -104.609       |    min |   -8.28% |
|                               Dataset size |     35.842       |    29.9704      |     -5.87159     |     GB |  -16.38% |
|                                 Store size |     35.842       |    29.9704      |     -5.87159     |     GB |  -16.38% |
|                              Segment count |   1052           |   687           |   -365           |        |  -34.70% |

Smaller but still sizable wins, esp on indexing time.
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
bfd055d 
# Conflicts:
#	server/src/main/java/org/elasticsearch/cluster/routing/IndexRouting.java
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
2ac0021
kkrik-es and others added 20 commits December 16, 2024 16:30
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
07b3308
@kkrik-es
address comments
cfdb91f
@kkrik-es
address comments
72e520e
@kkrik-es
address comments
a6e3ff5
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
52f2dad
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
6f3bbaf
@kkrik-es @martijnvg
Update x-pack/plugin/logsdb/src/main/java/org/elasticsearch/xpack/log…
8e2bdb0 
…sdb/LogsdbIndexModeSettingsProvider.java

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
cce66f0
[CI] Auto commit changes from spotless
b2e2bd3
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
3ca11a8
@kkrik-es
Merge remote-tracking branch 'origin/logsdb/route-on-sort-fields-3' i…
603bd06 
…nto logsdb/route-on-sort-fields-3
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
4dbc3c3
@kkrik-es
update tests
8204849
@kkrik-es
Merge remote-tracking branch 'origin/logsdb/route-on-sort-fields-3' i…
48fb2d8 
…nto logsdb/route-on-sort-fields-3
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
86998ea
[CI] Auto commit changes from spotless
a59070e
@kkrik-es
update tests
58d428d
@kkrik-es
Merge remote-tracking branch 'origin/logsdb/route-on-sort-fields-3' i…
c72effa 
…nto logsdb/route-on-sort-fields-3
@kkrik-es
fix rest compat tests
e134822
@kkrik-es
Merge branch 'main' into logsdb/route-on-sort-fields-3
71db536
martijnvg
martijnvg approved these changes Dec 19, 2024
View reviewed changes
Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🎉
@kkrik-es kkrik-es added auto-backport Automatically create backport pull requests when merged v8.18.0 labels Dec 19, 2024
@kkrik-es kkrik-es merged commit d80cbdd into elastic:main Dec 19, 2024
@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Dec 19, 2024

💔 Backport failed

Status Branch Result
❌ 8.x Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 116687
@kkrik-es kkrik-es mentioned this pull request Dec 19, 2024
Merged
@kkrik-es
Copy link
Contributor Author

kkrik-es commented Dec 19, 2024

💚 All backports created successfully

Status Branch Result
✅ 8.x

Questions ?

Please refer to the Backport tool documentation
kkrik-es added a commit to kkrik-es/elasticsearch that referenced this pull request Dec 19, 2024
@kkrik-es
Add LogsDB option to route on sort fields (elastic#116687)
63eaf7e 
* Add LogsDB option to route on sort fields

* fix encoding

* Update docs/changelog/116687.yaml

* tests

* tests

* tests

* fix mode

* tests

* tests

* tests

* add test

* fix test

* sync

* updates from review

* test fixes

* test fixes

* test fixes

* Move logic to SyntheticSourceIndexSettingsProvider

* fix test

* sync

* merge, no fallback

* comments

* fix test

* address comments

* address comments

* address comments

* Update x-pack/plugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/LogsdbIndexModeSettingsProvider.java

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>

* [CI] Auto commit changes from spotless

* update tests

* [CI] Auto commit changes from spotless

* update tests

* fix rest compat tests

---------

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
(cherry picked from commit d80cbdd)

# Conflicts:
#	rest-api-spec/build.gradle
#	server/src/main/java/org/elasticsearch/common/TimeBasedKOrderedUUIDGenerator.java
elasticsearchmachine pushed a commit that referenced this pull request Dec 19, 2024
@kkrik-es
[8.x] Add LogsDB option to route on sort fields (#116687) (#119113)
1c9ee51 
* Add LogsDB option to route on sort fields (#116687)

* Add LogsDB option to route on sort fields

* fix encoding

* Update docs/changelog/116687.yaml

* tests

* tests

* tests

* fix mode

* tests

* tests

* tests

* add test

* fix test

* sync

* updates from review

* test fixes

* test fixes

* test fixes

* Move logic to SyntheticSourceIndexSettingsProvider

* fix test

* sync

* merge, no fallback

* comments

* fix test

* address comments

* address comments

* address comments

* Update x-pack/plugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/LogsdbIndexModeSettingsProvider.java

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>

* [CI] Auto commit changes from spotless

* update tests

* [CI] Auto commit changes from spotless

* update tests

* fix rest compat tests

---------

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
(cherry picked from commit d80cbdd)

# Conflicts:
#	rest-api-spec/build.gradle
#	server/src/main/java/org/elasticsearch/common/TimeBasedKOrderedUUIDGenerator.java

* Update LogsIndexingIT.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged backport pending >enhancement :StorageEngine/Logs You know, for Logs Team:StorageEngine test-full-bwc Trigger full BWC version matrix tests v8.18.0 v9.0.0

3 participants

@kkrik-es @elasticsearchmachine @martijnvg