[watchguard_firebox] Initial release of the WatchGuard Firebox by brijesh-elastic · Pull Request #10110 · elastic/integrations

brijesh-elastic · 2024-06-08T18:58:06Z

Proposed commit message

Create New integration package watchguard_firebox.

Added a log data stream.
Added data collection logic for log data stream.
Added the ingest pipeline for log data stream.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboards and visualizations.
Added test for pipeline for log data stream.
Added system test cases for log data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/watchguard_firebox directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: watchguard_firebox - START ---
╭────────────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE            │ DATA STREAM │ TEST TYPE │ TEST NAME                                                                   │ RESULT │ TIME ELAPSED │
├────────────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ watchguard_firebox │             │ asset     │ dashboard watchguard_firebox-5fb58f2b-8720-4b57-89b6-0f727f0d260f is loaded │ PASS   │        882ns │
│ watchguard_firebox │             │ asset     │ search watchguard_firebox-0c03afcf-e39d-484a-8575-0630b7bb892a is loaded    │ PASS   │        308ns │
│ watchguard_firebox │             │ asset     │ search watchguard_firebox-19e90429-51c9-4f47-8889-baf56bdc7735 is loaded    │ PASS   │        237ns │
│ watchguard_firebox │             │ asset     │ search watchguard_firebox-5cbe635b-9d2e-4d14-b2d4-e7146c421eaf is loaded    │ PASS   │        195ns │
│ watchguard_firebox │             │ asset     │ search watchguard_firebox-9173ff94-4af2-4f6d-86bc-7123d0f335f5 is loaded    │ PASS   │        216ns │
│ watchguard_firebox │ log         │ asset     │ index_template logs-watchguard_firebox.log is loaded                        │ PASS   │        184ns │
│ watchguard_firebox │ log         │ asset     │ ingest_pipeline logs-watchguard_firebox.log-0.1.0 is loaded                 │ PASS   │        162ns │
╰────────────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: watchguard_firebox - END   ---
Done
--- Test results for package: watchguard_firebox - START ---
╭────────────────────┬─────────────┬───────────┬────────────────────────────┬────────┬──────────────╮
│ PACKAGE            │ DATA STREAM │ TEST TYPE │ TEST NAME                  │ RESULT │ TIME ELAPSED │
├────────────────────┼─────────────┼───────────┼────────────────────────────┼────────┼──────────────┤
│ watchguard_firebox │ log         │ pipeline  │ test-alarm.log             │ PASS   │  62.374853ms │
│ watchguard_firebox │ log         │ pipeline  │ test-diagnostic.log        │ PASS   │ 284.010886ms │
│ watchguard_firebox │ log         │ pipeline  │ test-event.log             │ PASS   │  153.26417ms │
│ watchguard_firebox │ log         │ pipeline  │ test-traffic.log           │ PASS   │ 230.823811ms │
│ watchguard_firebox │ log         │ pipeline  │ (ingest pipeline warnings) │ PASS   │ 264.588415ms │
╰────────────────────┴─────────────┴───────────┴────────────────────────────┴────────┴──────────────╯
--- Test results for package: watchguard_firebox - END   ---
Done
--- Test results for package: watchguard_firebox - START ---
╭────────────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE            │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ watchguard_firebox │ log         │ static    │ Verify sample_event.json │ PASS   │ 134.419749ms │
╰────────────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: watchguard_firebox - END   ---
Done
--- Test results for package: watchguard_firebox - START ---
╭────────────────────┬─────────────┬───────────┬───────────┬────────┬──────────────╮
│ PACKAGE            │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├────────────────────┼─────────────┼───────────┼───────────┼────────┼──────────────┤
│ watchguard_firebox │ log         │ system    │ udp       │ PASS   │ 39.83772453s │
╰────────────────────┴─────────────┴───────────┴───────────┴────────┴──────────────╯
--- Test results for package: watchguard_firebox - END   ---
Done

Related issues

Closes Watchguard Firebox #1203

Screenshots

elasticmachine · 2024-06-08T19:20:39Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2024-06-11T06:00:58Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2024-06-11T07:28:27Z

+                    "procid": "10"
+                }
+            },
+            "message": "Cannot relearn system MAC address, possible loop or MAC spoofing, ip=192.168.111.10, mac=00:50:da:c7:90:5d, interface=5",


For 3000012E, the doc describes the mac as observer mac address. We could copy this into observer.mac

kcreddy · 2024-06-11T07:32:05Z

+                    "procid": "10"
+                }
+            },
+            "message": "[eth0] Sending interface status event, logical=up link=up ip=10.0.0.1 mask=81.2.69.144",


Could add mask to related.ip

kcreddy · 2024-06-11T07:39:21Z

+            "dns": {
+                "question": {
+                    "name": "members.dyndns.org"
+                },
+                "resolved_ip": [
+                    "81.2.69.144"
+                ]


Can you also add dns.type:answer

kcreddy · 2024-06-11T07:44:25Z

+                    "procid": "10"
+                }
+            },
+            "message": "Received reply: HTTP/1.1 200 OK Date: Tue, 27 Nov 2012 17:14:57 GMT Server: Apache Content-Type: text/plain Connection: close good 192.168.53.88",


Can you parse them into http.* fields?

kcreddy · 2024-06-11T07:44:50Z

+                    "procid": "10"
+                }
+            },
+            "message": "Received reply: HTTP/1.1 200 OK Date: Tue, 27 Nov 2012 17:14:57 GMT Server: Apache Content-Type: text/plain Connection: close good 192.168.53.88",


Add IP address to related.ip

kcreddy · 2024-06-11T07:48:31Z

+                ]
+            },
+            "rule": {
+                "name": "\"FTP-00\""


Can you remove escape characters

elasticmachine · 2024-06-11T08:44:55Z

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

jamiehynds · 2024-06-11T08:45:31Z

 /packages/universal_profiling_symbolizer @elastic/obs-ds-intake-services
 /packages/vectra_detect @elastic/security-service-integrations
 /packages/vsphere @elastic/obs-infraobs-integrations
+/packages/watchguard_firebox @elastic/security-service-integrations


This integration will actually be owned by the @elastic/sec-deployment-and-devices team - can we ensure that's reflected in the codeowners please.

taylor-swanson

For performance and maintainability reasons, the large groks should be split out into individual processors for each log type.

Log types that extract back out to the same field need no additional processing (3000-0006 from pipeline_diagnostic, for example)
Use dissect first. It is faster than grok and in most cases can handle the patterns I see without issue
If dissect is not flexible enough, then grok can be used.

taylor-swanson

I'm still reviewing, but wanted to get some comments out in the mean time.

Also, please add the file kibana/tags.yml with the following contents:

- text: Security Solution
  asset_types:
    - dashboard
    - search

For example: https://github.com/elastic/integrations/blob/main/packages/cisco_asa/kibana/tags.yml

taylor-swanson · 2024-06-26T15:15:04Z

+      tag: rename_message_to_event_original
+      target_field: event.original
+      ignore_missing: true
+      description: Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document.


The problem with this is logstash may attach an event.original that is not formatted correctly. We've actually started overwriting event.original with message in other integrations.

One such PR that addressed this was #9438, however, it doesn't go into detail about what the problem is. Basically, there was a case where logstash was putting the entire beat event document into event.original, rather than the original message from the vendor appliance.

So basically, we want this:

- set: field: event.original copy_from: message - remove: field: message

~~This is still outstanding.~~

Edit: Disregard, there are some issues with package-spec validation at later versions. I'll have to think of a different way of handling this, but it'll come in the form of a wider change that'll affect other integrations as well.

kcreddy

LGTM (for my comments) 👍🏼

taylor-swanson

Just the message/event.original comment, otherwise everything else LGTM

taylor-swanson · 2024-07-01T14:32:59Z

+      tag: rename_message_to_event_original
+      target_field: event.original
+      ignore_missing: true
+      description: Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document.


~~This is still outstanding.~~

Edit: Disregard, there are some issues with package-spec validation at later versions. I'll have to think of a different way of handling this, but it'll come in the form of a wider change that'll affect other integrations as well.

taylor-swanson

LGTM

elasticmachine · 2024-07-10T10:43:18Z

💚 Build Succeeded

Buildkite Build
Commit: 4d6f52e

History

💚 Build #13062 succeeded ca515f9
💚 Build #12947 succeeded fd21f31
💚 Build #12927 succeeded f48fba7
💔 Build #12925 failed 813ea3a
💚 Build #12397 succeeded 288760e
💔 Build #12396 failed 99acff9

elastic-sonarqube · 2024-07-10T10:43:23Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-07-11T15:30:58Z

Package watchguard_firebox - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=watchguard_firebox

…ic#10110) - Created a new integration package for WatchGuard Firebox. - Added a log data stream. - Added data collection logic for log data stream. - Added the ingest pipeline for log data stream. - Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files. - Added dashboards and visualizations. - Added test for pipeline for log data stream. - Added system test cases for log data stream.

brijesh-elastic added 3 commits June 9, 2024 00:13

Initial release

fb06726

Update changelog entry

99acff9

Update github owner in manifest

288760e

kcreddy added New Integration Issue or pull request for creating a new integration package. Crest Contributions from Crest developement team. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Jun 11, 2024

kcreddy reviewed Jun 11, 2024

View reviewed changes

jamiehynds added the Team:Security-Deployment and Devices DEPRECATED Deployment and Devices Security team [elastic/sec-deployment-and-devices] label Jun 11, 2024

jamiehynds removed the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jun 11, 2024

jamiehynds reviewed Jun 11, 2024

View reviewed changes

taylor-swanson requested changes Jun 11, 2024

View reviewed changes

Comment thread packages/watchguard_firebox/data_stream/log/elasticsearch/ingest_pipeline/pipeline_alarm.yml

brijesh-elastic added 2 commits June 26, 2024 18:28

Resolve comments

813ea3a

Update CODEOWNERS file

f48fba7

brijesh-elastic requested review from kcreddy and taylor-swanson June 26, 2024 13:23

taylor-swanson reviewed Jun 26, 2024

View reviewed changes

Resolve comments

fd21f31

kcreddy approved these changes Jul 1, 2024

View reviewed changes

taylor-swanson reviewed Jul 1, 2024

View reviewed changes

Remove tag from beats.yml

ca515f9

taylor-swanson approved these changes Jul 1, 2024

View reviewed changes

taylor-swanson approved these changes Jul 3, 2024

View reviewed changes

Remove event.original remove processor

4d6f52e

taylor-swanson merged commit d6d9979 into elastic:main Jul 11, 2024

andrewkroh added the Integration:watchguard_firebox WatchGuard Firebox label Jul 22, 2024

brijesh-elastic mentioned this pull request Jul 31, 2026

[ti_eset] Add emailattachments data stream #19138

Open

Navigation Menu

Uh oh!

[watchguard_firebox] Initial release of the WatchGuard Firebox - #10110

[watchguard_firebox] Initial release of the WatchGuard Firebox#10110
taylor-swanson merged 8 commits into
elastic:mainfrom
brijesh-elastic:package-watchguard_firebox

brijesh-elastic commented Jun 8, 2024

elasticmachine commented Jun 8, 2024

elasticmachine commented Jun 11, 2024

Uh oh!

kcreddy Jun 11, 2024

kcreddy Jun 11, 2024

kcreddy Jun 11, 2024

kcreddy Jun 11, 2024

kcreddy Jun 11, 2024

kcreddy Jun 11, 2024

elasticmachine commented Jun 11, 2024

jamiehynds Jun 11, 2024

taylor-swanson left a comment

Uh oh!

taylor-swanson left a comment

Uh oh!

Uh oh!

Uh oh!

taylor-swanson Jun 26, 2024

taylor-swanson Jul 1, 2024 •

edited

Loading

Uh oh!

Uh oh!

kcreddy left a comment

taylor-swanson left a comment

taylor-swanson Jul 1, 2024 •

edited

Loading

taylor-swanson left a comment

elasticmachine commented Jul 10, 2024

elastic-sonarqube Bot commented Jul 10, 2024

elasticmachine commented Jul 11, 2024

Labels

6 participants

Uh oh!

Conversation

brijesh-elastic commented Jun 8, 2024

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jun 8, 2024

🚀 Benchmarks report

elasticmachine commented Jun 11, 2024

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jun 11, 2024

Choose a reason for hiding this comment

taylor-swanson left a comment

Choose a reason for hiding this comment

Uh oh!

taylor-swanson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

taylor-swanson Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

taylor-swanson left a comment

Choose a reason for hiding this comment

taylor-swanson Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

taylor-swanson left a comment

Choose a reason for hiding this comment

elasticmachine commented Jul 10, 2024

💚 Build Succeeded

History

elastic-sonarqube Bot commented Jul 10, 2024

Quality Gate passed

elasticmachine commented Jul 11, 2024

Labels

6 participants

taylor-swanson Jul 1, 2024 •

edited

Loading

taylor-swanson Jul 1, 2024 •

edited

Loading