[Filebeat/Journald] Enable reading from all journals

[belimawr]

Proposed commit message

This commit adds the `merge` option to the Journald configuration,
when enabled, it passes the `--merge` flag to `journalctl`, which
enables reading from all journals, including remote ones.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Disruptive User Impact
## Author's Checklist

How to test this PR locally

Option 1: ensure --merge is added when calling journalctl

That is the easiest way to test, run Filebeat with the following configuration:

filebeat.yaml

filebeat.inputs:
  - type: journald
    id: foo
    merge: true

output.discard:
  enabled: true

logging:
  to_stderr: true

The following command will print only a few keys from the logs, making it easier to read them

go run . --path.home=$PWD 2>&1 | jq '{"ts": ."@timestamp", "lvl": ."log.level", "logger": ."log.logger", "m": .message}' -c

Ensure --merge is in the journalctl command, you should see the following log message

Journalctl command: journalctl --utc --output=json --no-pager --all --merge --no-tail

Option 2: Setup systemd-journal-remote and systemd-journal-upload using VMs

The easiest way to test this is follow this tutorial to easily setup one VM sending journal logs to another. You'll need at least 2 VMs:

• VM1: This one will receive all the logs and run Filebeat
• VM2: This one will just send logs to the first one

Once you have setup both (or more) VMs follow those steps to test Filebeat. All commands must be executed as root.

1. Deploy the Elastic Stack, make sure it will be accessible from inside the Kind cluster
   I use my own fork of elastic-package to bind on all interfaces.

2. Ensure the logs from VM2 are being shipped to VM1:
   In VM1 run (it will block):

   journalctl -f  --merge
   

   Then in VM2 run:

   echo "foo from VM2"| systemd-cat
   

   This will log to the journal, you should see the message appearing in VM1

3. Deploy Filebeat
   Run Filebeat with the following configuration, adjust the output settings to your environment

   Details

   filebeat.inputs:
     - type: journald
       id: foo
       seek: tail # Only ingest new journal entries
       merge: true
   
   output.elasticsearch:
     hosts:
       - https://10.0.0.42
     username: elastic
     password: changeme
     ssl.verification_mode: none
   
   logging:
     to_stderr: true

4. Generate some logs in both VMs:

   • On each VM run echo "bla bla bla" | systemd-cat. Or use any
     other message. Anything you send to the standard input of
     systemd-cat will end up in journal logs

5. Ensure the logs are coming in in Kibana

   1. Create a dataview to show logs from filebeat-9.2.0
   2. Go to Discover and check for the logs. Look at host.hostname,
      you'll see the different machine's hostname there.

Related issues

• Closes [Journald] Enable reading from remote hosts #46656

## Use cases
## Screenshots
## Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-journald.md

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[vishaangelova]

Docs LGTM

[rdner]

We should document and implement version constraints from the official journalctl documentation.