[osquerybeat] Amcache osquery extension

[brian-mckinney]

Proposed commit message

Adds the amcache table to the osquery extension packaged with osquerybeat

See further discussion: https://github.com/elastic/endpoint-dev/issues/17096

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None

Author's Checklist

• [ ]

How to test this PR locally

# Build the extension
cd x-pack/osquerybeat
mage buildext

# Run tests
cd ext/osquery-extension/pkg/browserhistory
go test -v

# Test with osquery
osqueryi --extension /path/to/osquery-extension --allow-unsafe

# Query examples
osqueryi> SELECT * FROM amcache_application LIMIT 10;
osqueryi> SELECT * FROM amcache_application_file LIMIT 10;
osqueryi> SELECT * FROM amcache_application_shortcut LIMIT 10;
osqueryi> SELECT * FROM amcache_driver_binary LIMIT 10;
osqueryi> SELECT * FROM amcache_device_pnp LIMIT 10;

Related issues

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @brian-mckinney? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b amcache upstream/amcache
git merge upstream/main
git push upstream amcache

[brian-mckinney]

I created a branch with a combined table amcache_application_unified for this table, we do the extra processing in our extension binary to join the entries.

I also created a VIEW that joins the two application tables and presents a single table.

CREATE VIEW amcache_applications as SELECT * FROM amcache_application_file JOIN amcache_application using (program_id);

I then ran a few queries with the performance timer turned on to see how the three scenarios compared (prebuilt combined table, view, manual joins). Here are the results

Scenario: Select 10 entries

Table Type	Query	Round 1	Round 2	Round 3	Average
combined table	SELECT * FROM amcache_application_unified LIMIT 10;	16.116	16.236	17.01	16.454
joined table as view	SELECT * FROM amcache_applications LIMIT 10;	13.013	12.408	11.237	12.219
joined table	SELECT * from amcache_application_file JOIN amcache_application using(program_id) LIMIT 10;	14.599	11.416	13.5	13.172

Scenario: Select all entries with a WHERE clause

Table Type	Query	Round 1	Round 2	Round 3	Average
combined table	SELECT * FROM amcache_application_unified where program_id = '0000ec3897651aee4b488a85c4b60316a43200000904';	1.056	1.79	1.059	1.302
joined table as view	SELECT * FROM amcache_applications where program_id = '0000ec3897651aee4b488a85c4b60316a43200000904';	1.244	1.186	1.167	1.199
joined table	SELECT * from amcache_application_file JOIN amcache_application using(program_id) WHERE program_id = '0000ec3897651aee4b488a85c4b60316a43200000904';	1.312	1.24	1.248	1.267

In each case, the best overall performer was the VIEW table. I think this is the best solution as well because it means less processing and overhead on our side, but also a more convenient interface for the customer.

I have added a view creator to the extension in this PR.

[brian-mckinney]

@andrewkroh I have responded to the feedback and made requested changes. For the ones I didn't address, I responded with questions

I came in to start analyzing the new dependencies to see if they are of good quality and necessary (as opposed to A little copying is better than a little dependency.), but didn't get to that analysis yet.

I want to provide a bit more information about the added dependencies

Background Info

• Data for the amcache tables is derived from the amcache hive located at C:\Windows\appcompat\Programs\Amcache.hve. This file is in the format of a windows registry file, but is not accessible via the traditional windows registry apis.
• We have to parse the file with an "offline" registry parser
• In order to parse the file, we have to read it first. This is tricky because Windows keeps the file perpetually locked and in use, in such a way that even the system account cannot read it.

---

github.com/forensicanalysis/fslib

License

MIT

Description

This library allows for low level reads of files, by parsing the mft and reading the raw bytes. This allows us to bypass the security around the amcache hive and read it directly from disk. This is the standard tactic for other amcache tools as well.

Reasoning for inclusion

The functionality that it provides is non-trivial and would be a significant effort to reproduce

www.velocidex.com/golang/regparser

License

Apache 2.0

Description

This library is the only offline registry parsing library I was able to find. Due to the unique way we have to get at the amcache hive, this is necessary to parse the file after we acquire it

Reasoning for inclusion

Again, the functionality that it provides is non-trivial and would be a significant effort to reproduce

[brian-mckinney]

Failed test is unrelated to this PR, occurring in unrelated code. I made a flaky test issue for it: #47724

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b amcache upstream/amcache
git merge upstream/main
git push upstream amcache