[Azure Logs / Custom Azure Logs] Switch default processor to v2 #16618
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -57,6 +57,109 @@ streams: | |
| type: bool | ||
| multi: false | ||
| default: false | ||
| # | ||
| # Processor v2 only settings | ||
| # | ||
| - name: processor_version | ||
|
Contributor
There was a problem hiding this comment. are we ok that integration upgrades are going to automatically enable v2 without the user explicitly choosing?
Contributor
Author
There was a problem hiding this comment. We need to push users to the processor v2 while v1 is still available as fallback option. So if v2 falls short for any reason, users can still fallback to v1 while we address the problem. We need to drop v11 as soon as possible, since the v1's libraries are so old that sticking with them is becoming a risk. Footnotes
|
||
| type: select | ||
| title: Processor version | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: v2 | ||
| options: | ||
| - text: v1 (legacy) | ||
| value: v1 | ||
| - text: v2 | ||
| value: v2 | ||
| description: > | ||
| The processor version that the integration should use. Possible values are v1 (legacy) and v2 (recommended). | ||
| The v2 event hub processor is recommended for typical use cases. Defaults to v2. | ||
| - name: processor_update_interval | ||
| type: text | ||
| title: Processor update interval | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: 10s | ||
| description: >- | ||
| (Processor v2 only) How often the processor should attempt to claim partitions. | ||
|
|
||
| Default is `10` seconds. | ||
| - name: processor_start_position | ||
| type: select | ||
| title: Processor start position | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: earliest | ||
| options: | ||
| - text: earliest | ||
| value: earliest | ||
| - text: latest | ||
| value: latest | ||
| description: >- | ||
| (Processor v2 only) Controls from what position in the event hub the processor should start processing messages for all partitions. | ||
|
|
||
| Possible values are `earliest` and `latest`. | ||
|
|
||
| `earliest` starts processing messages from the last checkpoint, or the beginning of the event hub if no checkpoint is available. | ||
|
|
||
| `latest` starts processing messages from the the latest event in the event hub and continues to process new events as they arrive. | ||
|
|
||
| Default is `earliest`. | ||
| - name: partition_receive_timeout | ||
| type: text | ||
| title: Partition receive timeout | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: 5s | ||
| description: >- | ||
| (Processor v2 only) Maximum time to wait before processing the messages received from the event hub. | ||
|
|
||
| The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first. | ||
|
|
||
| Default is `5` seconds. | ||
| - name: partition_receive_count | ||
| type: text | ||
| title: Partition receive count | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: 100 | ||
| description: >- | ||
| (Processor v2 only) Maximum number of messages from the event hub to wait for before processing them. | ||
|
|
||
| The partition consumer waits up to a "receive count" or a "receive timeout", whichever comes first. | ||
|
|
||
| Default is `100` messages. | ||
| - name: migrate_checkpoint | ||
| type: bool | ||
| title: Migrate checkpoint information | ||
| multi: false | ||
| required: false | ||
| show_user: false | ||
| default: true | ||
| description: >- | ||
| (Processor v2 only) Controls processor behavior upon the initial transition from v1 to v2. If you are | ||
| starting directly with v2, you can disregard this setting. | ||
|
|
||
| Defaults to `true`, which means the processor will automatically migrate checkpoint information from the | ||
| v1 format to v2. | ||
|
|
||
| If set to `false`, the processor will bypass existing v1 checkpoints and replay all events in the | ||
| Event Hub from the start of the retention window. For example, if set to `false` on an Event Hub with | ||
| a 1-hour retention period, the processor will replay the last hour's worth of data. | ||
| - name: endpoint_suffix | ||
| type: text | ||
| default: core.windows.net | ||
| required: true | ||
| title: Storage account endpoint suffix | ||
| show_user: false | ||
| description: >- | ||
| (Processor v2 only) Override the default storage account endpoint suffix. | ||
|
|
||
| # Ensures agents have permissions to write data to `logs-*-*` | ||
| elasticsearch: | ||
| dynamic_dataset: true | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what does "the integration preview" mean?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We introduced the new
eventsdata stream as the entry point and router for the Azure Logs integration package.I plan to make the
eventsdata stream (aka Azure Logs v2) as the only data stream running an input.Unfortunately, years ago we updated the Azure Logs packages to allow multiple inputs to read from the same event hub, but this is a fatal flaw. We need to get rid of this.
The other data streams (activitylogs, firewall_logs, etc) are going to stay without an input.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We marked this
eventsdata stream named Azure Logs v2 as "preview" because we were not 100% sure this approach would work. But it seems working pretty well.