Optionally allow text similarity reranking to fail by thecoop · Pull Request #121784 · elastic/elasticsearch

thecoop · 2025-02-05T16:00:18Z

First part of #116796

Add a allow_reranker_failures option to reranker configuration, allowing errors to be ignored and the search to carry on. The option will be documented as part of the second stage PR

elasticsearchmachine · 2025-02-05T16:01:05Z

Hi @thecoop, I've created a changelog YAML for you.

thecoop · 2025-02-05T16:03:09Z

server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java

        RankFeaturePhaseRankCoordinatorContext rankFeaturePhaseRankCoordinatorContext,
        SearchPhaseController.ReducedQueryPhase reducedQueryPhase
    ) {
+        RankFeatureDoc[] docs = rankPhaseResults.getSuccessfulResults()


This changes the thread this happens in - not sure if this matters or not?

...rg/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankRetrieverBuilder.java

elasticsearchmachine · 2025-02-06T16:19:29Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

...test/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankTests.java

pmpailis

Thanks @thecoop , really nice :) Left some minor comments, with the error handling/logging pending, as well as how to propagate to the user that the results are not reranked (prob something in CompoundRetrieverBuilder ? )

pmpailis · 2025-02-10T10:30:42Z

...in/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankBuilder.java

        License.OperationMode.ENTERPRISE
    );

-    static final ConstructingObjectParser<TextSimilarityRankBuilder, Void> PARSER = new ConstructingObjectParser<>(NAME, args -> {


+1 on the cleanup :)

...test/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankTests.java

.../org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankMultiNodeTests.java

...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java

.../main/java/org/elasticsearch/search/rank/context/RankFeaturePhaseRankCoordinatorContext.java

pmpailis · 2025-02-10T11:21:38Z

server/src/main/java/org/elasticsearch/action/search/RankFeaturePhase.java

-                    context.onPhaseFailure(NAME, "Computing updated ranks for results failed", e);
+                    if (rankFeaturePhaseRankCoordinatorContext.failuresAllowed()) {
+                        // TODO: handle the exception somewhere
+                        // don't want to log the entire stack trace, it's not helpful here


...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml

pmpailis · 2025-02-12T13:08:43Z

Thanks for all the work and the iterations so far @thecoop ! Will documentation and propagating the non-reranked flag to the user be part of a second PR or will you append them on this one?

thecoop · 2025-02-12T13:22:15Z

Yes, they'll be in a second PR to add the user-visible sections

pmpailis

Thanks @thecoop ! Really nice ❤️ Left a couple of minor comments but no need for another review round :)

...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml

kderusso · 2025-03-18T19:59:14Z

...rc/yamlRestTest/resources/rest-api-spec/test/inference/70_text_similarity_rank_retriever.yml

          size: 10

+---
+"Text similarity reranking with allowed failure maintains custom sorting":


Don't we need cluster features here, for BWC?

I don't think so, the new option can only be used on new nodes, and new functionality is only usable when all nodes have been upgraded to support it. This is only adding functionality, there's no change to existing behaviour here.

Add an option to allow reranking to fail, and the docs to pass through as-is. Exposing the error to users and adding documentation is a later piece of work.

Backport #121784 to 8.19

Danouchka · 2025-12-11T23:07:04Z

Hello, when this will be released please ?

Add a lenient option to text similarity reranking

ad91d47

thecoop added >enhancement :Search Relevance/Search Catch all for Search Relevance labels Feb 5, 2025

thecoop requested a review from pmpailis February 5, 2025 16:00

elasticsearchmachine added the v9.1.0 label Feb 5, 2025

Update docs/changelog/121784.yaml

b447b33

thecoop commented Feb 5, 2025

View reviewed changes

thecoop added :Search Relevance/Ranking Scoring, rescoring, rank evaluation. and removed :Search Relevance/Search Catch all for Search Relevance labels Feb 5, 2025

propagate

1ac18fe

thecoop force-pushed the lenient-rerankers branch from 20c5735 to 1ac18fe Compare February 6, 2025 09:37

thecoop commented Feb 6, 2025

View reviewed changes

...rg/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankRetrieverBuilder.java Outdated Show resolved Hide resolved

thecoop added 2 commits February 6, 2025 10:58

leniency is only part of the text similarity builders

22bd2a0

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

505ae8b

thecoop force-pushed the lenient-rerankers branch from 65aac03 to 505ae8b Compare February 6, 2025 10:59

Remove superfluous xcontent serialization

c47b8a5

thecoop marked this pull request as ready for review February 6, 2025 16:19

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Feb 6, 2025

Add a test for lenient rerankers

bc81bd0

thecoop force-pushed the lenient-rerankers branch from f43701f to bc81bd0 Compare February 6, 2025 16:25

thecoop commented Feb 6, 2025

View reviewed changes

...test/java/org/elasticsearch/xpack/inference/rank/textsimilarity/TextSimilarityRankTests.java Outdated Show resolved Hide resolved

Rename lenient to failuresAllowed

2ef4e4e

thecoop changed the title ~~Add a lenient option to text similarity reranking~~ Feb 7, 2025

thecoop requested a review from benwtrent February 7, 2025 08:38

thecoop added 4 commits February 7, 2025 08:40

Use correct exception

9bb1d20

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

781e736

Update xcontent field name

fd396d9

Add another test

3ee20ed

pmpailis reviewed Feb 10, 2025

View reviewed changes

thecoop added 3 commits February 10, 2025 12:19

Actually check scores are passed through

108b89c

Remove clone

6cde906

Merge branch 'main' into lenient-rerankers

36a16e1

thecoop requested a review from pmpailis February 10, 2025 13:02

thecoop added 5 commits February 10, 2025 13:25

Fix test checks

bfbcca3

Set scores where there are none

f4f13c9

Stray comment

559aa32

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

cbd13b9

splotless

7b81006

thecoop force-pushed the lenient-rerankers branch from 339acc2 to 7b81006 Compare February 10, 2025 16:55

pmpailis reviewed Feb 11, 2025

View reviewed changes

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml Outdated Show resolved Hide resolved

thecoop added 2 commits February 12, 2025 09:29

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

3112cec

Change to constant_score for determinism

c3c08f6

pmpailis approved these changes Feb 13, 2025

View reviewed changes

...pack/inference/rank/textsimilarity/TextSimilarityRankFeaturePhaseRankCoordinatorContext.java Outdated Show resolved Hide resolved

...estTest/resources/rest-api-spec/test/rrf/800_rrf_with_text_similarity_reranker_retriever.yml Show resolved Hide resolved

thecoop added 4 commits February 17, 2025 10:32

Maintain existing sort order on failure

b0670a2

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

773d610

Merge remote-tracking branch 'upstream/main' into lenient-rerankers

7fa15e6

Fix transport version

01654d5

thecoop merged commit fc2f8fc into elastic:main Feb 17, 2025
17 checks passed

thecoop deleted the lenient-rerankers branch February 17, 2025 13:32

thecoop mentioned this pull request Feb 17, 2025

Add phase failures to search responses #122788

Open

thecoop mentioned this pull request Mar 11, 2025

Restore TextSimilarityRankBuilder XContent output #124564

Merged

kderusso reviewed Mar 18, 2025

View reviewed changes

thecoop mentioned this pull request May 21, 2025

Optionally allow text similarity reranking to fail (#121784) #128250

Merged

thecoop added a commit that referenced this pull request May 21, 2025

Optionally allow text similarity reranking to fail (#121784) (#128250)

f1cd6d7

Backport #121784 to 8.19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optionally allow text similarity reranking to fail#121784

Optionally allow text similarity reranking to fail#121784
thecoop merged 29 commits intoelastic:mainfrom
thecoop:lenient-rerankers

thecoop commented Feb 5, 2025 •

edited

Loading

elasticsearchmachine commented Feb 5, 2025

thecoop Feb 5, 2025

Uh oh!

elasticsearchmachine commented Feb 6, 2025

Uh oh!

pmpailis left a comment

pmpailis Feb 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmpailis Feb 10, 2025

Uh oh!

Uh oh!

pmpailis commented Feb 12, 2025

thecoop commented Feb 12, 2025

pmpailis left a comment

Uh oh!

Uh oh!

Uh oh!

kderusso Mar 18, 2025

thecoop Mar 24, 2025

Danouchka commented Dec 11, 2025

Labels

6 participants

Conversation

thecoop commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticsearchmachine commented Feb 5, 2025

thecoop Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Feb 6, 2025

Uh oh!

pmpailis left a comment

Choose a reason for hiding this comment

pmpailis Feb 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmpailis Feb 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pmpailis commented Feb 12, 2025

thecoop commented Feb 12, 2025

pmpailis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kderusso Mar 18, 2025

Choose a reason for hiding this comment

thecoop Mar 24, 2025

Choose a reason for hiding this comment

Danouchka commented Dec 11, 2025

Labels

6 participants

thecoop commented Feb 5, 2025 •

edited

Loading