[Osquerybeat] Add filtering functionality for osquery extension

[brian-mckinney]

Proposed commit message

This PR adds functionality for filtering go structs based on the query context passed by osquery to our extension.

It introduces a Filter object, and a way to build Filter objects from an osquery QueryContext. The QueryContext is what tells the extension how to scope the request. With the list of filters, you can feed a go struct to the Matches function and it will return true or false.

The Filter object requires that the struct have osquery tags, with values corresponding to the osquery column name. For example:

type MockEntry struct {
	StringValue string `osquery:"string_value"`
	IntValue int `osquery:"int_value"`
	FloatValue float64 `osquery:"float_value"`
	BoolValue bool `osquery:"bool_value"`
}

Here is an example of a generate function that uses filters (see my amcache PR)

func GenerateFunc(log *logger.Logger) table.GenerateFunc {
    return func(ctx context.Context, queryContext table.QueryContext) ([]map[string]string, error) {
        filters := filters.GetConstraintFilters(queryContext) // <-- Filters are extracted from the query context

        result := make([]MockEntry, 0)
        for _, entry := range GetMockEntries() {
            for _, filter := range filters {
                if filter.Matches(entry) {
                    result = append(result, entry)
                }
            }
        }

       marshalled, err := MarshalMockEntries(result)// convert the entries to a map[string]string
       if err != nil {
           return nil, err
       }
       return marshalled, nil
    }
}

This matrix shows what operators osquery officially supports, and which ones this PR implements

Operator	Osquery	Our Extension
Equals	✅	✅
GreaterThan	✅	✅
LessThanOrEquals	✅	✅
LessThan	✅	✅
GreaterThanOrEquals	✅	✅
Match	✅	❌
Like	✅	✅
Glob	✅	✅
Regexp	✅	✅
Unique	✅	❌

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @brian-mckinney? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

[marc-gr]

This looks nice! Left a couple comments.

If I am understanding correctly though, it would be used to filter information after it's been already processed by the table generation logic, and used to filter what is sent back to osqueryd. Would be nice to have some helpers to use for cases where we can filter before (like in browser_history, where we can scope the queries we build using timestamp filters etc). But I think this is probably not easy to generalize since each case would be different. Happy to let it as is and if we see any more patterns we can add them later on

[brian-mckinney]

Would be nice to have some helpers to use for cases where we can filter before (like in browser_history, where we can scope the queries we build using timestamp filters etc). But I think this is probably not easy to generalize since each case would be different. Happy to let it as is and if we see any more patterns we can add them later on

This is definitely doable, because the Filter objects are built at query time, so you would still have access to all of the data you would need for pre-processing. You could add a helper function to the filter object to get what you need

[marc-gr]

LGTM waiting to address the linter and formatting issues