Metrics for incremental bulk splits#116765
Conversation
ankikuma
added
the
:Distributed/Distributed
elasticsearchmachine
added
the
Team:Distributed Indexing (obsolete)
|
Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing) |
|
Hi @ankikuma, I've created a changelog YAML for you. |
|
|
||
| metrics.add( | ||
| registry.registerLongAsyncCounter( | ||
| "es.indexing.high_watermark_splits.total", |
There was a problem hiding this comment.
Let's include coordinating in this name since it is at the coordinating level currently.
es.indexing.coordinating.low_watermark_splits.total
es.indexing.coordinating.high_watermark_splits.total
Tim-Brooks
left a comment
Tim-Brooks
left a comment
There was a problem hiding this comment.
A couple of smaller comments.
| } | ||
|
|
||
| if (out.getTransportVersion().onOrAfter(TransportVersions.INDEXING_PRESSURE_THROTTLING_STATS)) { | ||
| out.writeVLong(lowWaterMarkSplits); |
There was a problem hiding this comment.
These two branches can be joined.
| totalCoordinatingRequests = -1L; | ||
| } | ||
|
|
||
| if (in.getTransportVersion().onOrAfter(TransportVersions.INDEXING_PRESSURE_THROTTLING_STATS)) { |
There was a problem hiding this comment.
These two branches can be joined.
| return (currentUsage >= lowWatermark && size >= lowWatermarkSize) || (currentUsage >= highWatermark && size >= highWatermarkSize); | ||
| if (currentUsage >= highWatermark && size >= highWatermarkSize) { | ||
| highWaterMarkSplits.getAndIncrement(); | ||
| logger.trace(() -> Strings.format("Split bulk due to high watermark: currentUsage [%d] and size [%d]", currentUsage, size)); |
There was a problem hiding this comment.
Perhaps this should either be labelled "current bytes". Or the currentUsage variable should be printed in a bytes representation.
| .put(IndexingPressure.SPLIT_BULK_HIGH_WATERMARK.getKey(), "4KB") | ||
| .put(IndexingPressure.SPLIT_BULK_HIGH_WATERMARK_SIZE.getKey(), "1024B") | ||
| .build(); | ||
| } |
There was a problem hiding this comment.
There are tests in this class testIncrementalBulkLowWatermarkBackOff and testIncrementalBulkHighWatermarkBackOff. Can we modify them to ensure that IndexingPressure indexingPressure properly accounts for the splits? We check indexing pressure stats for other things in this IT so it should be a small addition.
Add metrics to track incremental bulk request splits due to indexing pressure. Resolves ES-9612
Tim-Brooks
added
auto-backport
Add metrics to track incremental bulk request splits due to indexing pressure. Resolves ES-9612
Add metrics to track incremental bulk request splits due to indexing pressure. Resolves ES-9612
3 participants
Adding metrics to track incremental bulk request splits due to indexing pressure. Also added tests to test the metrics.
While writing tests I found that the low and high watermark thresholds in IncrementalBulkIT.java need a small tweak (without this tweak all splits are caused by high watermark thresholds and the low watermark thresholds don't kick in).
Resolves ES-9612