[Azure Logs] Set enabled: false as default for all data streams

[zmoog]

What does this PR do?

Set enabled: false as the default status for all Azure Logs data streams.

Why?

When users install the collective Azure Logs integration, all data streams start as enabled. With this default setting, we end up in a situation where Filebeat spawns a lot of "azureeventhub" behind the scenes: one for each existing data stream, and all receiving data from the same event hub:

                                   ┌────────────────┐                      
                                   │   azurelogs    │                      
                                   │ <<event hub>>  │                      
                                   └────────────────┘                      
                                            │                              
                                            │                              
                                            │                              
              ┌──────────────┬──────────────┼──────────────────────┐       
              │              │              │                      │       
          $Default       $Default       $Default               $Default    
     ┌────────┼──────────────┼──────────────┼──────────────────────┼──────┐
     │        ▼              ▼              ▼                      ▼      │
     │ ┌────────────┐ ┌────────────┐ ┌────────────┐           ┌─────────┐ │
     │ │   signin   │ │   audit    │ │  activity  │           │   ...   │ │
     │ │ <<input>>  │ │ <<input>>  │ │ <<input>>  │  .....    │<<input>>│ │
     │ └────────────┘ └────────────┘ └────────────┘           └─────────┘ │
     │                                                                    │
     └─Filebeat───────────────────────────────────────────────────────────┘

Many (data streams) to one (event hub) is not a good default.

In this case, each input receives a copy of each message published on the event hub. Some data streams have pipeline that can filter incoming logs by category (sign-in, audit) but other will try to ingest any log category (generic event hub and platform logs).

A better default is to have all input disabled.

In this case, Filebeat spawns a new input only if the user explicitly enables the data stream, with no unintended input running with no reason.

In addition, since the Azure Logs version 1.4.1 update, we started to strongly recommend using one event hub per log group (a set of related log categories).

Here's the recommended setup using multiple event hubs, where each input receives only the intended log categories:

            ┌────────────────┐             ┌───────────────┐       ┌────────────────┐
            │     adlogs     │             │ activitylogs  │       │  firewalllogs  │
            │ <<event hub>>  │             │ <<event hub>> │       │ <<event hub>>  │
            └────────────────┘             └───────────────┘       └────────────────┘
                     │                             │                        │        
                     │                             │                        │        
                     │                             │                        │        
              ┌──────┴───────┐                     │                        │        
              │              │                     │                        │        
          $Default       $Default              $Default                 $Default     
     ┌────────┼──────────────┼─────────────────────┼────────────────────────┼───────┐
     │        ▼              ▼                     ▼                        ▼       │
     │ ┌────────────┐ ┌────────────┐        ┌────────────┐           ┌────────────┐ │
     │ │   signin   │ │   audit    │        │  activity  │           │  firewall  │ │
     │ │ <<input>>  │ │ <<input>>  │        │ <<input>>  │           │ <<input>>  │ │
     │ └────────────┘ └────────────┘        └────────────┘           └────────────┘ │
     │                                                                              │
     └─Filebeat─────────────────────────────────────────────────────────────────────┘

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

[elasticmachine]

🚀 Benchmarks report

Package azure 👍(0) 💚(1) 💔(9)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
springcloudlogs	3816.79	2049.18	-1767.61 (-46.31%)	💔
activitylogs	1280.41	623.05	-657.36 (-51.34%)	💔
application_gateway	2074.69	1278.77	-795.92 (-38.36%)	💔
auditlogs	2207.51	1180.64	-1026.87 (-46.52%)	💔
firewall_logs	2004.01	1459.85	-544.16 (-27.15%)	💔
identity_protection	3267.97	1700.68	-1567.29 (-47.96%)	💔
platformlogs	4545.45	2132.2	-2413.25 (-53.09%)	💔
provisioning	2777.78	2008.03	-769.75 (-27.71%)	💔
signinlogs	1776.2	957.85	-818.35 (-46.07%)	💔

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-10-12T08:43:51.964+0000

• Duration: 15 min 17 sec

Test stats 🧪

Test	Results
Failed	0
Passed	123
Skipped	0
Total	123

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (10/10)	💚
Files	86.364% (19/22)	👎 -10.924
Classes	86.364% (19/22)	👎 -10.924
Methods	83.422% (156/187)	👎 -6.901
Lines	85.011% (2779/3269)	👎 -6.628
Conditionals	100.0% (0/0)	💚

[endorama]

Have we evaluated what will happen to users upgrading to this version? Will this alter significantly the behaviour of the current package?

[kaiyan-sheng]

+1 on @endorama's comment. This is a great change and let's make sure it's not breaking existing users.

[zmoog]

IIRC I tested this scenario, and the enabled setting is evaluated at install time only.

I will rerun the upgrade scenario and document it in the description.

[zmoog]

@endorama @kaiyan-sheng, I tested upgrading Azure Logs integration from version 1.3.0 to version 1.5.1.

In the following video, I:

1. Visit an existing integration version 1.3.0 to see that half of the data streams are enabled.
2. Run the upgrade
3. Check if the existing data streams are still enabled or disabled, as in version 1.3.0.
4. Check if the new Application Gateway data stream added in 1.5.0 is disabled (default setting).

CleanShot.2022-10-04.at.17.17.04.mp4

[kaiyan-sheng]

Looks good to me! Thank you for testing and the video! Could you add a changelog to this please?

[zmoog]

I have found only this downside.

When I try to install an "individual" integration (in the example below, Azure Active Directory) instead of the "collective" one (Azure Logs) the data stream is disabled.

This is both not great and okay.

• Not great: you need to enable something, or the integration will non collect data.
• Okay: Azure Active Directory has multiple data streams from different sub-services (Identity Protection and Provisioning service), so picking the used one can benefit efficiency.

Video:

CleanShot.2022-10-04.at.17.31.43.mp4

[kaiyan-sheng]

@zmoog I think it's Okay in this case because there are different data streams(audit, provisioning, sign-in...) under Azure Active Directory for users to choose from. If I understand correctly, we recommend users to use different event hubs for these log types under Azure Active Directory right?

[zmoog]

@zmoog I think it's Okay in this case because there are different data streams(audit, provisioning, sign-in...) under Azure Active Directory for users to choose from.

Yep, the idea is to invite users to enable only the data streams they use.

We also have other integrations with just one data stream. It would be disabled also for one data stream integration. Do you see this as a problem?

If I understand correctly, we recommend users to use different event hubs for these log types under Azure Active Directory right?

Each azureeventhub input connected to an event hub receives a copy of every message published.

Here's an example from Azure AD where:

1. Azure AD produces a new sign-in log with ID 42.
2. A diagnostic setting exports it to an event hub.
3. The two azureeventhub inputs connected to the event hub receive a copy of the sign-in log with ID 42 each.
4. The signinlogs data stream ingests it.
5. The auditlogs data stream drops it.

                                                                                             
                                                                         ┌──────────────────┐
                                                              signin:42  │    signinlogs    │
               signin:42                 signin:42                  ┌───▶│ <<data stream>>  │
  ┌─────────────┐        ┌──────────────┐        ┌──────────────┐   │    └──────────────────┘
  │  Azure AD   │        │  Diagnostic  │        │ logeventhub  │   │                        
  │ <<service>> │───────▶│   settings   │───────▶│<<event hub>> │───┤                        
  └─────────────┘        └──────────────┘        └──────────────┘   │                        
                                                                    │    ┌──────────────────┐
                                                                    │    │    auditlogs     │
                                                                    └───▶│ <<data stream>>  │
                                                             signin:42   └──────────────────┘
                                                                                             

The end goal is to avoid the scenario where users install the Azure Logs integration, they export log categories for only 2-3 of them. But, since the default setting is "enabled", they end up having one azureeventhub receiving data for each data stream.

Today we already have 10 data streams in the Azure Logs integration. We already have new ones planned, so this inefficiency and potential problems will probably worsen over time.

In the latest Azure Logs doc update, we expanded the event hub section advising about properly setting up the integration. High-volume deployment would probably benefit from a dedicated event hub. I plan to further expand this content.

Let me know what you think! 🙇

[kaiyan-sheng]

We also have other integrations with just one data stream. It would be disabled also for one data stream integration. Do you see this as a problem?

My opinion is having false as the default to prevent users from using one azureeventhub is worth the extra step of enabling individual data streams.

[zmoog]

My opinion is having false as the default to prevent users from using one azureeventhub is worth the extra step of enabling individual data streams.

Yeah, I think the trade-off is worth too.

Thank you, @kaiyan-sheng, for taking some time to think about this! 🙇