Skip to content

[Sophos] UTM data stream: replace Javascript processing with ingest pipelines #6816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chemamartinez merged 14 commits into from
Jul 24, 2023
Merged

[Sophos] UTM data stream: replace Javascript processing with ingest pipelines #6816

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
9723bd7
Fix invalid event type for XG data stream
chemamartinez Jul 1, 2023
fa7243c
Update Sophos UTM data stream
chemamartinez Jul 1, 2023
b4b3c32
Add event outcome and log source to UTM events
chemamartinez Jul 2, 2023
92e53b1
Generate sample event and add it to package docs
chemamartinez Jul 4, 2023
c7b3e2a
Convert some fields from keyword to long
chemamartinez Jul 4, 2023
81369fa
Create dashboards for Sophos UTM
chemamartinez Jul 5, 2023
4408e57
Add screenshots of Sophos UTM dashboards
chemamartinez Jul 5, 2023
33e94b3
Update changelog
chemamartinez Jul 5, 2023
f92babb
Add requested changes
chemamartinez Jul 14, 2023
af30b24
Adapt dashboards to requested changes
chemamartinez Jul 14, 2023
39a94ad
Add service to event.action for some cases
chemamartinez Jul 18, 2023
bcddffb
Add requested changes in ingest pipelines
chemamartinez Jul 20, 2023
7127517
Remove extra conditions
chemamartinez Jul 24, 2023
577586a
Lowercase observer type
chemamartinez Jul 24, 2023
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions packages/sophos/_dev/build/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The Sophos integration collects and parses logs from Sophos Products.

Currently it accepts logs in syslog format or from a file for the following devices:
Currently, it accepts logs in syslog format or from a file for the following devices:

- `utm` dataset: supports [Unified Threat Management](https://www.sophos.com/en-us/support/documentation/sophos-utm) (formerly known as Astaro Security Gateway) logs.
- `xg` dataset: supports [Sophos XG SFOS logs](https://docs.sophos.com/nsg/sophos-firewall/17.5/Help/en-us/webhelp/onlinehelp/nsg/sfos/concepts/Logs.html).
Expand All @@ -18,9 +18,11 @@ Versions above this are expected to work but have not been tested.

## Logs

### Utm log
### UTM log

The `utm` dataset collects Unified Threat Management logs.
The `utm` dataset collects Unified Threat Management logs. Currently, it collects the following log categories: DNS, DHCP, HTTP and Packet Filter.

{{event "utm"}}

{{fields "utm"}}

Expand Down
4 changes: 2 additions & 2 deletions packages/sophos/_dev/deploy/docker/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@ services:
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9549 -p=udp /sample_logs/sophos-utm-*.log"
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9549 -p=udp /sample_logs/sophos-utm*.log"
sophos-utm-tcp:
image: docker.elastic.co/observability/stream:v0.7.0
volumes:
- ./sample_logs:/sample_logs:ro
entrypoint: /bin/bash
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9549 -p=tcp /sample_logs/sophos-utm-*.log"
command: -c "/stream log --start-signal=SIGHUP --delay=5s --addr elastic-agent:9549 -p=tcp /sample_logs/sophos-utm*.log"
sophos-xg-udp:
image: docker.elastic.co/observability/stream:v0.7.0
volumes:
Expand Down
102 changes: 0 additions & 102 deletions packages/sophos/_dev/deploy/docker/sample_logs/sophos-utm-generated.log
View file
Open in desktop

This file was deleted.

40 changes: 40 additions & 0 deletions packages/sophos/_dev/deploy/docker/sample_logs/sophos-utm.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
2023:04:28-00:00:04 sophos-test-vm1 dhcpd: DHCPDISCOVER from 04:7b:cb:a1:d2:93 via eth7: unknown client
2023:04:28-00:00:10 sophos-test-vm1 dhcpd: DHCPINFORM from 89.160.20.129 via eth7
2023:04:28-00:00:10 sophos-test-vm1 dhcpd: DHCPACK to 89.160.20.129 (04:7b:cb:a1:d2:93) via eth7
2023:04:28-06:50:45 sophos-test-vm1 dhcpd: DHCPACK on 89.160.20.129 to 04:7b:cb:a1:d2:93 (CLIENTHOST) via eth1
2023:04:28-09:21:39 sophos-test-vm1 dhcpd: DHCPREQUEST for 192.168.72.129 from 04:7b:cb:a1:d2:93 via eth7: wrong network.
2023:04:28-09:21:39 sophos-test-vm1 dhcpd: DHCPNAK on 192.168.72.129 to 04:7b:cb:a1:d2:93 via eth7
2023:04:28-09:22:03 sophos-test-vm1 dhcpd: DHCPOFFER on 10.50.148.12 to 04:7b:cb:a1:d2:93 via eth7
2023:04:28-00:04:36 sophos-test-vm1 dhcpd: Dynamic and static leases present for 89.160.20.129.
2023:04:28-08:15:27 sophos-test-vm2 dhcpd: Received DHCPv4 packet without client-id option and empty hlen field.
2023:04:28-08:15:27 sophos-test-vm2 dhcpd: WARNING: Host declarations are global. They are not limited to the scope you declared them in.
2023:04:28-03:26:51 sophos-test-vm1 dhcpd: Wrote 0 deleted host decls to leases file.
2023:04:28-03:26:51 sophos-test-vm1 dhcpd: Wrote 0 new dynamic host decls to leases file.
2023:04:28-03:26:51 sophos-test-vm1 dhcpd: Wrote 13 leases to leases file.
2023:04:28-03:30:27 sophos-test-vm2 dhcpd: Listening on LPF/eth0/04:7b:cb:a1:d2:93/89.160.20.0/24
2023:04:28-03:30:27 sophos-test-vm2 dhcpd: Sending on LPF/eth0/04:7b:cb:a1:d2:93/89.160.20.0/24
2023:04:28-03:30:27 sophos-test-vm2 dhcpd: Sending on Socket/fallback/fallback-net
2023:04:28-07:43:18 sophos-test-vm1 dhcpd: receive_packet failed on wlan1: Network is down
2023:04:28-08:11:33 sophos-test-vm1 dhcpd: 3 bad udp checksums in 5 packets
2023:04:28-09:22:03 sophos-test-vm1 dhcpd: 5 udp packets in 5 too long - dropped
2023:03:08-15:00:00 sophos-test-vm1 httpproxy[6267]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="HEAD" srcip="67.43.156.2" dstip="89.160.20.129" user="testuser" group="testgroup" ad_domain="example.com" statuscode="200" cached="0" profile="HTTP_Sophos_Profile_1" filteraction="REF_HTTP_ACTION" size="311" request="0x7fad9e44ac00" url="https://myurl.test.com/extension" referer="https://referer.test.com/" error="" authtime="0" dnstime="5" aptptime="0" cattime="200" avscantime="0" fullreqtime="32181" device="0" auth="0" ua="Microsoft BITS/7.8" exceptions="" category="178" reputation="trusted" categoryname="Internet Services" application="googplay" app-id="816" country="United States" content-type="application/octet-stream"
2023:03:08-15:00:00 sophos-test-vm1 httpproxy[6267]: id="0002" severity="info" sys="SecureWeb" sub="http" name="web request blocked" action="block" method="GET" srcip="67.43.156.2" dstip="89.160.20.129" user="testuser" group="testgroup" ad_domain="example.com" statuscode="416" cached="0" profile="HTTP_Sophos_Profile_1" filteraction="REF_HTTP_ACTION" size="311" request="0x7fad9e44ac00" url="https://myurl.test.com/extension" referer="https://referer.test.com/" error="" authtime="0" dnstime="0" aptptime="0" cattime="191" avscantime="0" fullreqtime="50667" device="0" auth="0" ua="Microsoft BITS/7.8" exceptions="" category="178" reputation="trusted" categoryname="Internet Services" application="googplay" app-id="816" content-type="application/octet-stream" reason="range"
2023:03:08-15:00:06 sophos-test-vm1 httpproxy[6267]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0x7fad90f7ed00" function="fileextension_scan" file="fileextensionscanner.c" line="154" message="error converting file name to utf-8 from x-user-defined: Conversion from character set 'x-user-defined' to 'UTF8' is not supported"
2023:03:08-15:00:50 sophos-test-vm1 httpproxy[6267]: id="0003" severity="info" sys="SecureWeb" sub="http" request="0x7fadb4088d00" function="ssl_raw_read" file="ssl.c" line="1012" message="SSL_ERROR_SYSCALL: ret=-1 error=Broken pipe"
2023:03:08-15:00:39 sophos-test-vm1 httpproxy[6267]: id="0060" severity="info" sys="SecureWeb" sub="http" name="web request blocked, forbidden category detected" action="block" method="GET" srcip="67.43.156.2" dstip="89.160.20.129" user="testuser" group="testgroup" ad_domain="example.com" statuscode="403" cached="0" profile="HTTP_Sophos_Profile_1" filteraction="REF_HTTP_ACTION" size="4764" request="0x7fad933c8b00" url="https://myurl.test.com/extension" referer="https://referer.test.com/" error="" authtime="0" dnstime="249" aptptime="0" cattime="206" avscantime="0" fullreqtime="1499" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" exceptions="" country="Germany" category="141" reputation="trusted" categoryname="Portal Sites" application="yahoo" app-id="553" reason="category"
2023:03:08-15:00:50 sophos-test-vm1 httpproxy[6267]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="POST" srcip="67.43.156.2" dstip="89.160.20.129" user="testuser" group="testgroup" ad_domain="example.com" statuscode="200" cached="0" profile="HTTP_Sophos_Profile_1" filteraction="REF_HTTP_ACTION" size="92" request="0x7fad8b180000" url="https://myurl.test.com/extension" referer="https://referer.test.com/" error="" authtime="0" dnstime="0" aptptime="0" cattime="137" avscantime="3221" fullreqtime="407968" device="0" auth="0" ua="CloudKitWin/57.0.0.0 (Windows/10.0.0.0)" exceptions="" category="170" reputation="trusted" categoryname="Personal Network Storage" application="icloud" app-id="666" content-type="application/x-protobuf" sandbox="-"
2023:03:08-15:05:47 sophos-test-vm1 httpproxy[6267]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" srcip="67.43.156.2" dstip="89.160.20.129" user="testuser" group="testgroup" ad_domain="example.com" statuscode="200" cached="0" profile="HTTP_Sophos_Profile_1" filteraction="REF_HTTP_ACTION" size="23229" request="0x7fad93120200" url="https://myurl.test.com/extension" referer="https://referer.test.com/" error="" authtime="0" dnstime="257" aptptime="0" cattime="170" avscantime="0" fullreqtime="8408149" device="0" auth="0" ua="" exceptions="ssl,certcheck,certdate" category="105" reputation="trusted" categoryname="Business" country="Germany"
2023:03:08-15:08:20 sophos-test-vm1 httpproxy[6267]: id="0071" severity="info" sys="SecureWeb" sub="http" name="web request warned, forbidden category detected" action="warn" method="OPTIONS" srcip="67.43.156.2" dstip="89.160.20.129" user="testuser" group="testgroup" ad_domain="example.com" statuscode="403" cached="0" profile="HTTP_Sophos_Profile_1" filteraction="REF_HTTP_ACTION" size="3783" request="0x7fad9be42000" url="https://myurl.test.com/extension" referer="https://referer.test.com/" error="" authtime="0" dnstime="211" aptptime="0" cattime="149" avscantime="0" fullreqtime="1265" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" exceptions="" country="Germany" reason="category" category="9998" reputation="unverified" categoryname="Uncategorized"
2023:03:08-15:19:36 sophos-test-vm1 httpproxy[6267]: id="0072" severity="info" sys="SecureWeb" sub="http" name="Contentfilter warned and proceeded" url="https://myurl.test.com/extension" srcip="67.43.156.2" category="9998" categoryname="Uncategorized"
2023:03:08-15:20:40 sophos-test-vm1 httpproxy[6267]: id="0064" severity="info" sys="SecureWeb" sub="http" name="web request blocked, forbidden file extension detected" action="block" method="GET" srcip="67.43.156.2" dstip="89.160.20.129" user="testuser" group="testgroup" ad_domain="example.com" statuscode="403" cached="0" profile="HTTP_Sophos_Profile_1" filteraction="REF_HTTP_ACTION" size="2814" request="0x7fad92a81700" url="https://myurl.test.com/extension" referer="https://referer.test.com/" error="" authtime="0" dnstime="0" aptptime="0" cattime="0" avscantime="0" fullreqtime="638695" device="0" auth="0" ua="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" exceptions="" overridecategory="1" overridereputation="1" category="105,105,105,105,105" reputation="neutral" categoryname="Business,Business,Business,Business,Business" application="google" app-id="182" reason="extension" extension="com" filename="file1.zw-net.com"
2023:04:28-00:00:01 sophos-test-vm1 named[7442]: trust-anchor-telemetry '_test-1/IN' from 89.160.20.129
2023:04:28-02:03:58 sophos-test-vm1 named[7442]: clients-per-query increased to 15
2023:04:28-07:43:18 sophos-test-vm1 named[7442]: no longer listening on 192.168.1.123#53
2023:04:28-07:43:18 sophos-test-vm1 named[7442]: listening on IPv4 interface wlan1, 192.168.1.123#53
2023:04:28-09:12:31 sophos-test-vm1 named[7442]: clients-per-query decreased to 14
2023:04:28-00:00:01 sophos-test-vm1 ulogd[12511]: id="2021" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped (GEOIP)" action="drop" fwrule="60019" initf="eth6" srcmac="1a:fc:2b:33:4d:5f" dstmac="00:11:22:33:44:55" srcip="67.43.156.2" dstip="89.160.20.129" proto="6" length="44" tos="0x00" prec="0x00" ttl="238" srcport="36584" dstport="8888" tcpflags="SYN"
2023:04:28-00:18:05 sophos-test-vm1 ulogd[12511]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="150" initf="eth7" outitf="eth6" srcmac="1a:fc:2b:33:4d:5f" dstmac="00:11:22:33:44:55" srcip="67.43.156.2" dstip="89.160.20.129" proto="1" length="30" tos="0x00" prec="0x00" ttl="5" type="8" code="0"
2023:04:28-00:18:06 sophos-test-vm1 ulogd[12511]: id="2001" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped" action="drop" fwrule="60001" initf="eth6" mark="0x207c" app="124" srcmac="1a:fc:2b:33:4d:5f" dstmac="00:11:22:33:44:55" srcip="67.43.156.2" dstip="89.160.20.129" proto="17" length="87" tos="0x00" prec="0x00" ttl="59" srcport="53" dstport="30799"
2023:04:28-00:00:01 sophos-test-vm1 ulogd[12511]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="174" initf="eth6" outitf="eth7" srcmac="1a:fc:2b:33:4d:5f" dstmac="00:11:22:33:44:55" srcip="67.43.156.2" dstip="89.160.20.129" proto="6" length="52" tos="0x02" prec="0x00" ttl="126" srcport="55267" dstport="445" tcpflags="SYN"
2023:04:28-00:50:54 sophos-test-vm1 ulogd[12511]: id="2021" severity="info" sys="SecureNet" sub="packetfilter" name="Packet dropped (GEOIP)" action="drop" fwrule="60019" initf="eth6" srcmac="1a:fc:2b:33:4d:5f" dstmac="00:11:22:33:44:55" srcip="67.43.156.2" dstip="89.160.20.129" proto="6" length="88" tos="0x00" prec="0x00" ttl="227" srcport="443" dstport="42181" tcpflags="ACK PSH"
2023:04:28-11:28:41 sophos-test-vm1 ulogd[12511]: id="2014" severity="info" sys="SecureNet" sub="packetfilter" name="DNS request" action="DNS request" fwrule="60011" initf="eth7" srcmac="00:50:56:b6:27:aa" dstmac="00:11:22:33:44:55" srcip="67.43.156.2" dstip="89.160.20.129" proto="17" length="75" tos="0x00" prec="0x00" ttl="64" srcport="34200" dstport="53"
17 changes: 17 additions & 0 deletions packages/sophos/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,21 @@
# newer versions go on top
- version: "3.0.0"
changes:
- description: Add support to UTM Packetfilter logs.
type: enhancement
link: https://github.com/elastic/integrations/pull/6816
- description: Add support to UTM HTTP logs.
type: enhancement
link: https://github.com/elastic/integrations/pull/6816
- description: Add support to UTM DHCP logs.
type: enhancement
link: https://github.com/elastic/integrations/pull/6816
- description: Add support to UTM DNS logs.
type: enhancement
link: https://github.com/elastic/integrations/pull/6816
- description: Deprecate JavaScript processing in favor of ingest pipelines for UTM logs.
type: enhancement
link: https://github.com/elastic/integrations/pull/6816
- version: "2.11.1"
changes:
- description: Fix indentation in log agent config.
Expand Down
Loading