Elasticsearch Version

8.18, 8.19, 9.0, 9.1, 9.2 (and older versions that are no longer maintained)

Problem Description

Normally in a bulk request, if indexing for individual document fails (eg. due to invalid json), the rest of the documents in the bulk request will still succeed, and respond a 200 status, with error details for individual doc in the body. However, if the failing document has a pipeline configured, the entire request will fail with 400 status and none of the documents will be indexed.

This is reported by https://github.com/elastic/sdh-elasticsearch/issues/9486, and this comment pointed we may need to tighten up the error handling in IngestService.

Steps to Reproduce

Create a dummy pipeline

PUT /_ingest/pipeline/test-ds-pl-1
{
  "processors": [
    {
      "append": {
        "field": "name",
        "value": "value"
      }
    }
  ]
}

Bulk index invalid json doc entry without pipeline

PUT /_bulk
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z","@timestamp":"2024-07-01T00:00:00Z"}

• Responds 200, the first doc is indexed, the second doc is not. This is expected.

Bulk index invalid json doc entry with a pipeline

• note the duplicated @timestamp field

PUT /_bulk?pipeline=test-ds-pl-1
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z","@timestamp":"2024-07-01T00:00:00Z"}

• Responds 400 and no docs are indexed.

Bulk index invalid json doc entry with a pipeline

• note the missing closing bracket in the second doc

PUT /_bulk?pipeline=test-ds-pl-1
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"

• Responds 400 and no docs are indexed.