FIPS complaint agent file vault

[michel-laterman]

What does this PR do?

Add elastic file vault seed implementation to allow variable length salt sizes.
FIPS distributions will strictly use the new .seedV2 format with the salt size set to 16 for FIPS compliance.
When not in FIPS only the (V1) .seed file is used.

Seed file version is as follows:

• .seed files remain unchanged from the current released implementation. Used in non-FIPS distribution.
• .seedV2 files are new; they contain the seed followed by the salt size (as a little endian uint32 value). This allows specify the salt size in use by the agent vault. Used by the FIPS distribution.

Why is it important?

Current file vault does not meet FIPS compliance.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

No replacement path from a non-FIPS to a FIPS-agent (or vice-versa).

[mergify]

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[michel-laterman]

This is currently a draft as I need to do testing.
I may need to have a different default salt size for fips/nonfips distributions to start.

My thought is that this approach will give us the ability to migrate from saltSize 8 to 16 which will be useful if customers want to migrate agents (they will need to go from agent version N-1 -> N -> N-fips).

Migration is not part of this PR.

[blakerouse]

This is currently a draft as I need to do testing. I may need to have a different default salt size for fips/nonfips distributions to start.

My thought is that this approach will give us the ability to migrate from saltSize 8 to 16 which will be useful if customers want to migrate agents (they will need to go from agent version N-1 -> N -> N-fips).

Can you explain this more? Why not just N-1 -> N-fips? Could it be made that way?

[michel-laterman]

This is currently a draft as I need to do testing. I may need to have a different default salt size for fips/nonfips distributions to start.
My thought is that this approach will give us the ability to migrate from saltSize 8 to 16 which will be useful if customers want to migrate agents (they will need to go from agent version N-1 -> N -> N-fips).

Can you explain this more? Why not just N-1 -> N-fips? Could it be made that way?

I'll admit that i'm not 100% sure on this; I'm assuming that a FIPS agent can not run the migration as it would need to use non-compliant crypto settings in order to do so

[michel-laterman]

Right now the integration tests fail as one of the upgrade tests tries to go from an artifact that is built from this PR to the latest snapshot.
In terms of changes from this PR it goes from a vault with the V2 seed file + vault, to no seed file detected.
After we "upgrade" the vault contains:

drwxr-x--- 2 root root 4096 Mar 12 17:14 .
drwxrwx--- 5 root root 4096 Mar 12 17:14 ..
-rw------- 1 root root    0 Mar 12 17:11 .lock
-rw------- 1 root root   32 Mar 12 17:14 .seed
-rw------- 1 root root   36 Mar 12 17:11 .seedV2
-rw------- 1 root root  125 Mar 12 17:14 8c6b8035489e04adf7302525047eb100b54b791916b09a3cc5c73b8b59bb09ee # V1 vault
-rw------- 1 root root  133 Mar 12 17:11 8fee45c194974431fa8a0cfa3355ed15e6c0e742fdfb4992767198a77dd16bd3 # V2 vault

The upgraded agent fails to start with could not load agent info: could not get agent info from store: fail to read configuration /opt/Elastic/Agent/fleet.enc for the agent: fail to decode bytes: cipher: message authentication failed

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[blakerouse]

I really want to just understand what happens here so

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fips-agent-vault upstream/fips-agent-vault
git merge upstream/main
git push upstream fips-agent-vault

[blakerouse]

With product saying that we don't need an upgrade path then I am good with this PR. I think this decision will be re-visited in the future. I could see an organization with 100+ agents having to re-deploy to get FIPS as being an issue.

I would think we could support upgrade to non-FIPS that adjusts the size of the salt to FIPS compliant, then upgrade to a FIPS compliant binary that is now using the salt. It is interesting topic if a FIPS binary upgrade using non-FIPS compliant path to support an upgrade is not non-FIPS if then it uses FIPS from that upgrade. To me this is an interesting discussion.

[pkoutsovasilis]

interestingly enough I do see a CI failure with this error

POST git-upload-pack (305 bytes)
--
  | From https://github.com/elastic/elastic-agent
  | * branch                  main       -> FETCH_HEAD
  | = [up to date]            main       -> origin/main
  | Warning: you are leaving 9 commits behind, not connected to
  | any of your branches:
  |  
  | c1beeb9557 Remove V2 seed functions from non-FIPS agents
  | 83a2014771 Provide v1 seedfile when v2 size matches
  | 18493241fa Use saltSize of 8 for non-fips
  | e2440300fe Fix unit tests
  | ... and 5 more.
  |  
  | If you want to keep them by creating a new branch, this may be a good time
  | to do so with:
  |  
  | git branch <new-branch-name> c1beeb9557
  |  
  | HEAD is now at 2336fb88bf [CI] Grant build and read permission for serverless beats pipeline (#7449)
  | Current branch: HEAD
  | Switched to a new branch 'pr_merge_7360'
  | New branch created: pr_merge_7360
  | Auto-merging internal/pkg/agent/vault/vault_file_notwindows.go
  | CONFLICT (content): Merge conflict in internal/pkg/agent/vault/vault_file_notwindows.go
  | Automatic merge failed; fix conflicts and then commit the result.
  | Merge failed: 1

but GH reports no conflicts 🤔

[simitt]

Add elastic file vault seed implementation to allow variable length salt sizes.
.seedV2 files are new; they contain the seed followed by the salt size (as a little endian uint32 value). This allows specify the salt size in use by the agent vault. Used by the FIPS distribution.

I don't see how the vault salt size can be set, it always uses the default of size 16 for fips. It's also not clear to me why it would need to be able to deal with variable salt size?

Looking at the current code, are all these changes really necessary and relevant?

• now that non-fips is unchanged, what is the reasoning behind introducing a .seedV2 file for fips rather than also using the .seed file?
• The only place where I see a salt size before the changes is in vault_file_notwindows.go. It can easily be that I am overlooking something, so please push back if that is the case, but wouldn't the simplest required change be to only switch the saltSize=16 in this file for the fips case and keep everything else untouched? Can you elaborate why this wouldn't be sufficient or desired?

[michel-laterman]

Going for least lines of (code) change in this case really limits the agent in the future if we ever need to re-visit any of our agent state encryption.

Additionally, having an explicit .seedV2 makes it really clear that it's not interoperable with the existing .seed files. This can also be a clear signal to us if we need to debug instances where the agent has been replaced (for FIPS) but for some reason the vault was left intact.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
51.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: a8bac73

History

• 💔 Build #18983 failed 287da67
• 💔 Build #18965 failed b15701b
• 💔 Build #18931 failed c1beeb9
• 💚 Build #18714 succeeded 83a2014
• 💔 Build #18657 failed 1849324
• 💔 Build #18647 failed e244030

cc @michel-laterman