[Prisma Access] Add S3 ingestion path for Strata Logging Service JSON logs #18054

Integration Name

Palo Alto Prisma Access [prisma_access]

Dataset Name

prisma_access.event (new input type for existing data stream)

Integration Version

1.7.0

Agent Version

9.3.1

OS Version and Architecture

N/A — architecture-level feature request

User Goal

Add an alternative ingestion path to the Prisma Access integration that reads JSON logs from an S3 bucket, using the aws-s3 input. Palo Alto's Strata Logging Service supports forwarding logs to AWS S3 as batched, Snappy-compressed JSON files.

This S3-based path simplifies the ingestion architecture. The current TCP/CEF syslog path requires deploying multiple Elastic Agents behind a load balancer to handle high event rates, adding operational complexity. S3 provides natural buffering that decouples the ingestion rate from the processing rate and eliminates the need for load-balanced agent fleets.

Existing Features

The Prisma Access integration currently only supports ingestion via TCP syslog (CEF format). While this path can be scaled to high throughput by adding agents and increasing worker counts, doing so requires a complex architecture: multiple Elastic Agents behind a load balancer, careful tuning of worker concurrency, and significant CPU resources for Filebeat's decode_cef processing.

An S3-based ingestion path would be significantly simpler to operate and scale.

What did you see?

The TCP/CEF syslog path works but requires considerable architectural effort to scale — multiple agents, load balancing, and tuning. The S3 path is a cleaner solution because it eliminates the need for load-balanced agent fleets entirely and leverages S3's built-in buffering and durability.

Implementation Details

Prerequisites

1. Snappy decompression support in the aws-s3 input (Beats).
   Strata Logging Service compresses S3 objects using Snappy. The Elastic aws-s3 input currently only supports gzip decompression. Snappy support must be added.

   • A proof-of-concept branch exists: andrewkroh/beats@add-snappy-decompression-support
   • This should be contributed upstream to elastic/beats as a standalone PR.

2. Access to Palo Alto Strata Logging Service.
   We need to verify the exact JSON field schema produced by the S3 forwarding destination. The Palo Alto documentation confirms the output is JSON but does not publish a complete field schema. Sample data from Strata is required to:

   • Confirm field names (are they the same as CEF extension key names, or different?).
   • Determine whether the existing ingest pipeline can be adapted or a new pipeline is needed.
   • Build pipeline test fixtures.

Integration Changes

• Add a new input variant to the Prisma Access integration using the aws-s3 input type, alongside the existing TCP input.
• Write or adapt an ingest pipeline that handles the JSON field format from Strata Logging Service (the current pipeline expects CEF-decoded fields under cef.extensions.*).
• Add pipeline tests with representative sample data from Strata.

Anything else?

Palo Alto documentation reference:

• Forward Logs to AWS S3 — confirms Snappy compression and JSON format: "Strata Logging Service batch logs based on either a size limit of 1000 lines or a time limit of 10 seconds, whichever is reached first. Strata Logging Service compresses the log file using Snappy and forwards it to S3 bucket in JSON format."

Related alternatives considered:

• Strata → Security Lake (S3 Parquet/OCSF): OCSF is a different schema from CEF, requiring a full pipeline rewrite. Risk of lossy translation for vendor-specific PanOS fields. Higher complexity.
• Optimize existing TCP/CEF path: Addresses symptoms but the architecture remains complex to operate at scale.