Add recursive chunker

[dan-rubinstein]

Description

This change adds the recursive chunking strategy (see Javadocs) of the RecursiveChunker.java to learn how it works. The recursive chunker comes with a default separator set for plaintext along with one for markdown documents.

Example usage:

PUT _inference/sparse_embedding/my-endpoint
{
  "service": "<service>",
  "service_settings": {
     ...
  },
  "chunking_settings": {
    "strategy": "recursive",
    "separator_set": "markdown",
    "max_chunk_size": 250
  }
}

PUT _inference/sparse_embedding/my-endpoint
{
  "service": "<service>",
  "service_settings": {
     ...
  },
  "chunking_settings": {
    "strategy": "recursive",
    "separators": [<separator1>, <separator2>, ...],
    "max_chunk_size": 250
  }
}

Testing

• Unit testing
• Manually tested chunking the CONTRIBUTING.md document.

[elasticsearchmachine]

Hi @dan-rubinstein, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

Because of the small chunk size the generated chunks will never contain more than 1 sentence. Can you structure the test so that some chunks contain multiple heading sections.

For example if, if chunks size was 100 words and given the document

# heading1.1
## heading1.2.1 
TEST_SENTENCE * 3

## heading1.2.2 
TEST_SENTENCE * 2

# heading2.1
## heading2.2.1 
TEST_SENTENCE * 9

## heading2.2.2 
TEST_SENTENCE 

### heading2.3.1
TEST_SENTENCE 

### heading2.3.2
TEST_SENTENCE  

In this case, given an ordered list of separators, I would expect # heading1.1 -> # heading2.1 to be a single chunks. Then 2 more chunks for heading2.2.1 and heading2.2.2

Please add tests on longer documents that capture the hierarchical nature of the chunker

[dan-rubinstein]

Discussed offline with Dave. Adding this into the existing long document tests that randomly generate a document would require essentially re-writing the chunking logic into the testing file to generate the expected chunk limits. We've instead decided it makes sense to add a new test with a smaller fixed length document to cover this case.

[davidkyle]

LGTM

[dan-rubinstein]

@elasticmachine merge upstream

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.19