Upgrade to Lucene 10 by javanna · Pull Request #114741 · elastic/elasticsearch

javanna · 2024-10-14T15:56:54Z

No description provided.

This temporarily gets us past missing codec that we attempt to load at node startup, even if we don't use it.

Conflicts: build-tools-internal/version.properties modules/analysis-common/src/test/java/org/elasticsearch/analysis/common/EdgeNGramTokenizerTests.java modules/analysis-common/src/test/java/org/elasticsearch/analysis/common/SynonymsAnalysisTests.java modules/reindex/src/internalClusterTest/java/org/elasticsearch/migration/MultiFeatureMigrationIT.java plugins/analysis-phonetic/src/test/java/org/elasticsearch/plugin/analysis/phonetic/AnalysisPhoneticFactoryTests.java qa/rolling-upgrade-legacy/src/test/java/org/elasticsearch/upgrades/RecoveryIT.java server/src/internalClusterTest/java/org/elasticsearch/repositories/IndexSnapshotsServiceIT.java server/src/test/java/org/elasticsearch/action/admin/cluster/stats/VersionStatsTests.java server/src/test/java/org/elasticsearch/cluster/metadata/MetadataCreateIndexServiceTests.java server/src/test/java/org/elasticsearch/cluster/metadata/MetadataTests.java server/src/test/java/org/elasticsearch/index/IndexVersionTests.java server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java server/src/test/java/org/elasticsearch/index/mapper/DateFieldMapperTests.java server/src/test/java/org/elasticsearch/index/mapper/DynamicTemplatesTests.java server/src/test/java/org/elasticsearch/index/mapper/FieldNamesFieldMapperTests.java server/src/test/java/org/elasticsearch/index/mapper/ParametrizedMapperTests.java server/src/test/java/org/elasticsearch/index/mapper/vectors/BinaryDenseVectorScriptDocValuesTests.java server/src/test/java/org/elasticsearch/index/similarity/SimilarityServiceTests.java server/src/test/java/org/elasticsearch/indices/analysis/AnalysisModuleTests.java server/src/test/java/org/elasticsearch/script/VectorScoreScriptUtilsTests.java server/src/test/java/org/elasticsearch/script/field/vectors/DenseVectorTests.java test/framework/src/main/java/org/elasticsearch/index/mapper/MetadataMapperTestCase.java test/framework/src/main/java/org/elasticsearch/snapshots/AbstractSnapshotIntegTestCase.java x-pack/plugin/deprecation/src/test/java/org/elasticsearch/xpack/deprecation/IndexDeprecationChecksTests.java x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/mapper/ShapeFieldMapperTests.java x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/index/query/LegacyGeoShapeWithDocValuesQueryTests.java

cbuescher

I've done a final pass (non-test code only), left a few notes where I am unsure but nothing really sticks out. LGTM

uschindler

Looks fine. 🍺🚀

javanna · 2024-10-20T11:13:48Z

run elasticsearch-ci/part-3-windows

javanna · 2024-10-20T18:11:58Z

run elasticsearch-ci/part-1-windows

javanna · 2024-10-20T18:12:03Z

run elasticsearch-ci/part-2-windows

javanna · 2024-10-20T18:12:33Z

run elasticsearch-ci/part-3-windows

javanna · 2024-10-20T18:12:48Z

run elasticsearch-ci/part-4-windows

javanna · 2024-10-21T08:02:10Z

The checkstyle errors from windows builds are present in main too, and are being fixed by #115185 .

javanna · 2024-10-21T11:29:51Z

Windows tests failed with the following which is unrelated to the changes made in this PR:

REPRODUCE WITH: gradlew ":server:test" --tests "org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testProcessFileChanges" -Dtests.seed=98579A98909F778 -Dtests.locale=ro-MD -Dtests.timezone=America/Kentucky/Louisville -Druntime.java=22
FileSettingsServiceTests > testProcessFileChanges FAILED
    java.nio.file.AccessDeniedException: C:\bk\server\build\testrun\test\temp\org.elasticsearch.reservedstate.service.FileSettingsServiceTests_98579A98909F778-001\tempFile-001.tmp -> C:\bk\server\build\testrun\test\temp\org.elasticsearch.reservedstate.service.FileSettingsServiceTests_98579A98909F778-001\tempDir-024\config\operator\settings.json
        at __randomizedtesting.SeedInfo.seed([98579A98909F778:210DAE66E062FF98]:0)
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:328)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:291)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.move(FilterFileSystemProvider.java:146)
        at java.base/java.nio.file.Files.move(Files.java:1430)
        at org.elasticsearch.reservedstate.service.FileSettingsServiceTests.writeTestFile(FileSettingsServiceTests.java:358)
        at org.elasticsearch.reservedstate.service.FileSettingsServiceTests.testProcessFileChanges(FileSettingsServiceTests.java:253)

The checkstyle errors on kql are also present in main, like I mentioned above.

The most relevant ES changes that upgrading to Lucene 10 requires are: - use the appropriate IOContext - Scorer / ScorerSupplier breaking changes - Regex automaton are no longer determinized by default - minimize moved to test classes - introduce Elasticsearch900Codec - adjust slicing code according to the added support for intra-segment concurrency - disable intra-segment concurrency in tests - adjust accessor methods for many Lucene classes that became a record - adapt to breaking changes in the analysis area Co-authored-by: Christoph Büscher <christophbuescher@posteo.de> Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co> Co-authored-by: ChrisHegarty <chegar999@gmail.com> Co-authored-by: Brian Seeders <brian.seeders@elastic.co> Co-authored-by: Armin Braun <me@obrown.io> Co-authored-by: Panagiotis Bailis <pmpailis@gmail.com> Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>

elasticsearchmachine and others added 30 commits September 9, 2024 06:11

[Automated] Update Lucene snapshot to 10.0.0-snapshot-dc47adbbe73

5ff75bb

[Automated] Update Lucene snapshot to 9.12.0-snapshot-371fa57d9c7

e0e2e1e

Remove IndexVersions.V_7_6_0

3267fd0

Remove IndexVersions.V_7_7_0 - V_7_11_0

74040a5

Remove remainign V_7x IndexVersions

70ee548

Uncomment some tests for better mergability with main

db7463f

Merge branch 'main' into lucene_snapshot_10

95bc9d3

Fix errors in imports

d0a1110

[Automated] Update Lucene snapshot to 10.0.0-snapshot-64f5697f537

afec00b

[Automated] Update Lucene snapshot to 9.12.0-snapshot-ce23e15eb54

304a1c6

Merge branch 'main' into lucene_snapshot_10

d104bc3

Remove usages of random versions before V_8_0_0 in tests

8fe6c3a

More test fixed related to legacy versions

8a01884

Mute CompositeRolesStoreTests.testXPackUserCanAccessNonRestrictedIndices

09b68e2

Fix failing knn yaml test

ca97d86

Fix docs test related to knn queries

84d4e86

Null out missing codec in BWCLucene70Codec

77f24ae

This temporarily gets us past missing codec that we attempt to load at node startup, even if we don't use it.

Partially revert previous commit

3cea216

[Automated] Update Lucene snapshot to 9.12.0-snapshot-7964682ddf5

8bf41b8

[Automated] Update Lucene snapshot to 10.0.0-snapshot-7c529ce092d

fff3fbb

Fix compilation issues after last Lucene 10 snapshot merge

0ee5e4f

Merge branch 'main' into lucene_snapshot_10

2d31a70

Fix UOE in ContextIndexSearcher after last Lucene merge

ced0f7b

[Automated] Update Lucene snapshot to 10.0.0-snapshot-74e3c44063a

2430fff

[Automated] Update Lucene snapshot to 9.12.0-snapshot-ab262f917d4

d45dbc7

Fix compile issues after latest Lucene snapshot update

d24200e

spotless

8800107

Fixing more checkstyle and spotless issues after merging main

84bb2b4

Remove awaitsFix, as the issue was fixed

e916319

cbuescher approved these changes Oct 15, 2024

View reviewed changes

Restore SpecialPermission#check call in ExpressionScriptEngine

b6f646c

uschindler reviewed Oct 15, 2024

View reviewed changes

Merge branch 'main' into lucene_snapshot_10

1d5719d

cbuescher mentioned this pull request Oct 16, 2024

Rename methods in o.e.x.c.security.support.Automatons #114594

Merged

Merge branch 'main' into lucene_snapshot_10

d66c7e3

javanna added test-windows Trigger CI checks on Windows test-full-bwc Trigger full BWC version matrix tests test-arm Pull Requests that should be tested against arm agents labels Oct 17, 2024

javanna removed the request for review from benwtrent October 17, 2024 07:57

javanna added 4 commits October 17, 2024 10:35

Merge branch 'main' into lucene_snapshot_10

c2f698b

Merge branch 'main' into lucene_snapshot_10

cc85647

Merge branch 'main' into lucene_snapshot_10

d60179c

Merge branch 'main' into lucene_snapshot_10

a02ad20

javanna added 2 commits October 21, 2024 11:27

Merge branch 'main' into lucene_snapshot_10

83142c3

Merge branch 'main' into lucene_snapshot_10

ac85cdb

javanna merged commit 8efd08b into elastic:main Oct 21, 2024

javanna deleted the lucene_snapshot_10 branch October 21, 2024 11:38

arteam mentioned this pull request Oct 23, 2024

[test] Unmute FsDirectoryFactoryTests#testPreload #115438

Merged

javanna mentioned this pull request Feb 13, 2025

Remove usage of SortedSetDocValues#NO_MORE_ORDS #88004

Closed

pawankartik-elastic mentioned this pull request May 9, 2025

too_complex_to_determinize_exception is thrown in a few cases due to the patterns in index template #127972

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to Lucene 10#114741

Upgrade to Lucene 10#114741
javanna merged 580 commits intoelastic:mainfrom
javanna:lucene_snapshot_10

javanna commented Oct 14, 2024

cbuescher left a comment

uschindler left a comment

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 21, 2024

javanna commented Oct 21, 2024

Labels

12 participants

Conversation

javanna commented Oct 14, 2024

cbuescher left a comment

Choose a reason for hiding this comment

uschindler left a comment

Choose a reason for hiding this comment

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 20, 2024

javanna commented Oct 21, 2024

javanna commented Oct 21, 2024

Labels

12 participants