Add support for bbolt-based storage in Filebeat by rdner · Pull Request #48879 · elastic/beats

rdner · 2026-02-16T15:50:44Z

Proposed commit message

Add a new bbolt (BoltDB) storage backend for the Filebeat registry,
configurable via `filebeat.registry.backend: bbolt`. The bbolt backend
provides persistent on-disk storage with support for database compaction
and TTL-based entry cleanup.
This feature is experimental and may change or be removed in future releases.

Assisted by Cursor.

To see the registry content (debugging):

Make a snapshot (copy) of the running file
Inspect the snapshot with this tool https://github.com/boltdb/boltd (runs a web interface).

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
I have added an entry in ./changelog/fragments using the changelog tool.

Based on #48948 which needs to be merged first.

github-actions · 2026-02-16T15:50:55Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

github-actions · 2026-02-20T15:35:17Z

Vale Linting Results

Summary: 1 warning, 5 suggestions found

⚠️ Warnings (1)

File	Line	Rule	Message
docs/reference/filebeat/configuration-general-options.md	114	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'and so on' instead of 'etc'.

💡 Suggestions (5)

File	Line	Rule	Message
docs/reference/filebeat/configuration-general-options.md	114	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.
docs/reference/filebeat/configuration-general-options.md	154	Elastic.WordChoice	Consider using 'deactivates, deselects, hides, turns off, makes unavailable' instead of 'disables', unless the term is in the UI.
docs/reference/filebeat/configuration-general-options.md	154	Elastic.WordChoice	Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
docs/reference/filebeat/configuration-general-options.md	163	Elastic.WordChoice	Consider using 'deactivates, deselects, hides, turns off, makes unavailable' instead of 'disables', unless the term is in the UI.
docs/reference/filebeat/configuration-general-options.md	163	Elastic.WordChoice	Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

github-actions · 2026-02-20T15:35:25Z

🔍 Preview links for changed docs

rdner

Added some clarifications on choice of constants.

As an option, now Filebeat can be configured to store file offset tracking in a bbolt storage instead of the WAL memlog implementation. Also, improved the integration testing framework to accommodate required checks for these changes. Assisted by Cursor.

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

coderabbitai · 2026-02-27T11:48:12Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a3aa50d and 8fe32d2.

📒 Files selected for processing (2)

libbeat/statestore/backend/bbolt/compaction.go
libbeat/statestore/backend/bbolt/store.go

🚧 Files skipped from review as they are similar to previous changes (1)

libbeat/statestore/backend/bbolt/compaction.go

📝 Walkthrough

Walkthrough

Adds support for a new bbolt-backed registry backend for Filebeat selectable via filebeat.registry.backend (default memlog). Implements a bbolt Registry and per-store DBs, config structs for timeout, fsync, compaction, and TTL-based retention, and backend selection wiring in the store initialization. Includes compaction and retention logic, extensive docs and reference YAML updates, and integration and unit tests covering bbolt behavior.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Tip

Try Coding Plans. Let us write the prompt for your AI agent so you can ship faster (with fewer bugs).
Share your feedback on Discord.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@libbeat/statestore/backend/bbolt/compaction.go`:
- Line 35: The cleanup routine currently deletes every file matching the glob
tempDbPrefix + "*" (tempdb*), which can remove valid store DBs whose names begin
with "tempdb"; update the deletion logic in
libbeat/statestore/backend/bbolt/compaction.go to only remove true temporary DB
files by matching the exact naming convention used when creating temps (use a
stricter pattern such as tempDbPrefix followed by a fixed separator or
extension, e.g. tempdb_ or tempdb.<unique>.tmp, rather than tempdb*), and change
the code in the cleanup function (the block around where tempDbPrefix is used
for removal) to check for that exact separator/extension before unlinking files
so real store files like "<store>.db" are never deleted.

ℹ️ Review info

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6bfbe86 and a3aa50d.

📒 Files selected for processing (14)

changelog/fragments/1771601358-add-bbolt-registry-backend.yaml
docs/reference/filebeat/configuration-general-options.md
docs/reference/filebeat/filebeat-reference-yml.md
filebeat/_meta/config/filebeat.global.reference.yml.tmpl
filebeat/beater/store.go
filebeat/config/config.go
filebeat/filebeat.reference.yml
filebeat/testing/integration/filestream_bbolt_test.go
libbeat/statestore/backend/bbolt/bbolt.go
libbeat/statestore/backend/bbolt/bbolt_test.go
libbeat/statestore/backend/bbolt/compaction.go
libbeat/statestore/backend/bbolt/config.go
libbeat/statestore/backend/bbolt/store.go
x-pack/filebeat/filebeat.reference.yml

rdner · 2026-02-27T12:54:54Z

@orestisfl @belimawr @VihasMakwana @AndersonQ please focus on critical issues in your final review.

Just a reminder that this is an experimental feature which is built mainly for assessment and testing on our side. Any minor issues, refactoring, etc. can be contributed in later follow ups.

orestisfl

Just a reminder that this is an experimental feature which is built mainly for assessment and testing on our side. Any minor issues, refactoring, etc. can be contributed in later follow ups.

Fair. Code LGTM. Haven't focused much on config so an extra approval would be good here.

AndersonQ

I have a question about the RW mutex. I added the comments there

orestisfl

Quick note, should we feature-guard it with an environmental variable?

rdner · 2026-02-27T14:32:32Z

@orestisfl I think the configuration flag we introduced here is sufficient.

AndersonQ

It looks good. I left a comment, I think we need a default TTL, but it does not need to be addressed now.

Did you test it "under agent"? Is the bbolt db being copied to the agent diagnostics?

rdner · 2026-02-27T15:05:59Z

Did you test it "under agent"? Is the bbolt db being copied to the agent diagnostics?

No, I have not.

The bbolt storage is not included into the diagnostics yet and I don't think it should be in the scope of this PR. Only when we're committed to this feature at least at the technical preview level we can change this:

beats/filebeat/beater/diagnostics.go

Lines 41 to 47 in c6f22b0

    
           preFilesList := [][]string{ 
        
           	{"^registry$"}, 
        
           	{"^registry", "filebeat$"}, 
        
           	{"^registry", "filebeat", "meta\\.json$"}, 
        
           	{"^registry", "filebeat", "log\\.json$"}, 
        
           	{"^registry", "filebeat", "active\\.dat$"}, 
        
           	{"^registry", "filebeat", "[[:digit:]]*\\.json$"},

cmacknz · 2026-02-27T16:01:34Z

Before we fully integrate this into Elastic Agent we need to validate the effect it has. I am particularly interested in if our current schema and access patterns will work well with bbolt or if we have to change them - filelog does not use bbolt the same way we do.

So I am in agreement that we should get this merge and start the evaluation, if I have any comment before merge it's that we probably don't need to formally document it until we have learned how to use (yes it's experimental but we really don't intend for anyone to use it so why tell them about it now). That is not blocking though.

I don't think we need to be perfect in this PR, in fact widely using bbolt is impossible anyway until we have a way to migrate from registry to bbolt without breaking anything for users and can articulate the benefits of the change.

rdner · 2026-02-27T16:06:59Z

if I have any comment before merge it's that we probably don't need to formally document it until we have learned how to use (yes it's experimental but we really don't intend for anyone to use it so why tell them about it now).

The docs are already there (removing them is work, additional CI time) and they're to tell us developers about the currently available options. Judging by the amount of questions I got here about the available parameters, I'd rather keep it.

belimawr · 2026-02-27T21:01:58Z

Did you test it "under agent"? Is the bbolt db being copied to the agent diagnostics?

No, I have not.

The bbolt storage is not included into the diagnostics yet and I don't think it should be in the scope of this PR. Only when we're committed to this feature at least at the technical preview level we can change this:

beats/filebeat/beater/diagnostics.go

Lines 41 to 47 in c6f22b0

preFilesList := [][]string{

{"^registry$"},

{"^registry", "filebeat$"},

{"^registry", "filebeat", "meta\\.json$"},

{"^registry", "filebeat", "log\\.json$"},

{"^registry", "filebeat", "active\\.dat$"},

{"^registry", "filebeat", "[[:digit:]]*\\.json$"},

I think for testing and experimenting it is valuable to have the bbolt store in the diagnostics from the beginning, so if things go wrong and we collect a diagnostics, we have the bbolt db there.

However, I don't mind merging this PR without it.

rdner self-assigned this Feb 16, 2026

rdner added Filebeat Filebeat backport-skip Skip notification from the automated backport with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team skip-changelog labels Feb 16, 2026

botelastic Bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 16, 2026

rdner force-pushed the filestream-bbolt-registry branch 7 times, most recently from d36e0e3 to 3acb505 Compare February 18, 2026 16:29

rdner mentioned this pull request Feb 19, 2026

Extend the integration testing framework #48948

Merged

3 tasks

rdner force-pushed the filestream-bbolt-registry branch from 3acb505 to 8a2ebb6 Compare February 19, 2026 10:15

This was referenced Feb 19, 2026

[8.19](backport #48948) Extend the integration testing framework #48958

Merged

[9.2](backport #48948) Extend the integration testing framework #48959

Merged

[9.3](backport #48948) Extend the integration testing framework #48960

Merged

rdner force-pushed the filestream-bbolt-registry branch 2 times, most recently from b703ae1 to c922e3b Compare February 20, 2026 15:33

rdner force-pushed the filestream-bbolt-registry branch from c922e3b to 538a845 Compare February 20, 2026 15:46

rdner commented Feb 20, 2026

View reviewed changes

Comment thread libbeat/statestore/backend/bbolt/bbolt.go

Comment thread libbeat/statestore/backend/bbolt/bbolt.go

Comment thread libbeat/statestore/backend/bbolt/bbolt.go

rdner removed the skip-changelog label Feb 20, 2026

rdner force-pushed the filestream-bbolt-registry branch from 538a845 to 34786d8 Compare February 20, 2026 16:29

rdner requested a review from AndersonQ February 20, 2026 16:44

rdner added 2 commits February 27, 2026 12:32

Add error check for db.Close

81aa26f

Add more error handling

a3aa50d

rdner requested review from Copilot and orestisfl February 27, 2026 11:43

Copilot started reviewing on behalf of rdner February 27, 2026 11:44 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

Comment thread libbeat/statestore/backend/bbolt/bbolt.go

coderabbitai Bot reviewed Feb 27, 2026

View reviewed changes

Comment thread libbeat/statestore/backend/bbolt/compaction.go

rdner requested a review from belimawr February 27, 2026 12:29

orestisfl reviewed Feb 27, 2026

View reviewed changes

Comment thread libbeat/statestore/backend/bbolt/store.go

Comment thread libbeat/statestore/backend/bbolt/compaction.go Outdated

Comment thread libbeat/statestore/backend/bbolt/store.go

rdner added 2 commits February 27, 2026 13:42

Refactor the retention loop

f8e7728

Add clarification comment

8fe32d2

rdner requested a review from orestisfl February 27, 2026 12:48

orestisfl approved these changes Feb 27, 2026

View reviewed changes

rdner requested a review from cmacknz February 27, 2026 13:03

AndersonQ reviewed Feb 27, 2026

View reviewed changes

Comment thread libbeat/statestore/backend/bbolt/store.go

Comment thread libbeat/statestore/backend/bbolt/store.go

rdner requested a review from AndersonQ February 27, 2026 13:48

orestisfl approved these changes Feb 27, 2026

View reviewed changes

AndersonQ approved these changes Feb 27, 2026

View reviewed changes

Comment thread docs/reference/filebeat/configuration-general-options.md

belimawr approved these changes Feb 27, 2026

View reviewed changes

rdner merged commit a7ff790 into elastic:main Mar 2, 2026
214 checks passed

rdner deleted the filestream-bbolt-registry branch March 2, 2026 08:15

coderabbitai Bot mentioned this pull request Mar 5, 2026

[beatreceiver] add remote state store extension and integrate it into fbreceiver #49191

Merged

6 tasks

coderabbitai Bot mentioned this pull request May 26, 2026

Integrate the OTel filestorage extension #50130

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add support for bbolt-based storage in Filebeat#48879

Add support for bbolt-based storage in Filebeat#48879
rdner merged 34 commits into
elastic:mainfrom
rdner:filestream-bbolt-registry

rdner commented Feb 16, 2026 •

edited

Loading

github-actions Bot commented Feb 16, 2026

github-actions Bot commented Feb 20, 2026 •

edited

Loading

github-actions Bot commented Feb 20, 2026 •

edited

Loading

rdner left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

coderabbitai Bot commented Feb 27, 2026 •

edited

Loading

Walkthrough

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdner commented Feb 27, 2026

orestisfl left a comment

AndersonQ left a comment

Uh oh!

Uh oh!

orestisfl left a comment

rdner commented Feb 27, 2026

AndersonQ left a comment

Uh oh!

rdner commented Feb 27, 2026

cmacknz commented Feb 27, 2026

rdner commented Feb 27, 2026

belimawr commented Feb 27, 2026

Uh oh!

Labels

9 participants

Uh oh!

Conversation

rdner commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

github-actions Bot commented Feb 16, 2026

🤖 GitHub comments

github-actions Bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vale Linting Results

github-actions Bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

rdner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

coderabbitai Bot commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdner commented Feb 27, 2026

orestisfl left a comment

Choose a reason for hiding this comment

AndersonQ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

orestisfl left a comment

Choose a reason for hiding this comment

rdner commented Feb 27, 2026

AndersonQ left a comment

Choose a reason for hiding this comment

Uh oh!

rdner commented Feb 27, 2026

cmacknz commented Feb 27, 2026

rdner commented Feb 27, 2026

belimawr commented Feb 27, 2026

Uh oh!

Labels

9 participants

rdner commented Feb 16, 2026 •

edited

Loading

github-actions Bot commented Feb 20, 2026 •

edited

Loading

github-actions Bot commented Feb 20, 2026 •

edited

Loading

coderabbitai Bot commented Feb 27, 2026 •

edited

Loading