Ensure Agent configuration and state persist across restarts in Fleet mode

[naemono]

Resolves #8819
Related: elastic/elastic-agent#5185

Testing procedure

Bring es/kibana/agents online. Verified fleet page in kibana in fleet-mode, verified logs of agents themselves in non-fleet-mode. Restarted/killed agent pods, and ensured that the ones in the fleet-ui in Kibana didn't change names, or additional new ones weren't added.

For advanced configuration:
Add the following to the Agent manifest:

  config:
    fleet:
      enabled: true
    providers.kubernetes:
      add_resource_metadata:
        deployment: true

Exec into running agent pod and execute elastic-agent inspect verify that the following is present in the output:

providers:
  kubernetes:
    add_resource_metadata:
      deployment: true

Tested

• 9.2.0-snapshot (fleet mode)
• 9.2.0-snapshot (standalone mode)
• 9.2.0-snapshot (fleet mode + advanced configuration)
• 8.19 (fleet mode)
• 8.19 (standalone mode)
• 8.19 (fleet mode + advanced configuration)
• 8.13 (fleet mode)
• 8.13 (standalone mode)
• 8.13 (fleet mode + advanced configuration)
• 7.17 (fleet mode)
• 7.17 (standalone mode)

[prodsecmachine]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Licenses	0	0	0	0	0 issues
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[naemono]

buildkite test this

[pebrc]

On 8.13.0 I ran into:

 Error: fail to read action store '/usr/share/elastic-agent/state/data/action_store.yml': yaml: input error: fail to decode bytes: cipher: message authentication failed         

Also, we need to verify that advanced configuration still works.

[pebrc]

Another issue 9.1.2 when upgrading from ECK 3.1. to this PR:

│ Error: fail to read state store '/usr/share/elastic-agent/state/data/state.enc': failed migrating YAML store JSON store: could not parse YAML                                                                                                                                                                             │
│ fail to decode bytes: cipher: message authentication failed    

[pebrc]

The errors I hightlighted above are all coming from the migration from the status quo in 3.1 where CONFIG_PATH is /etc/agent to the new CONFIG_PATH. Because agent stores information needed to decrypt its state in the CONFIG_PATH this is lost on the migration (or on any container restart I guess)

The guidance from @pkoutsovasilis is to force re-enroll with FLEET_FORCE=true develop a migration process that on the first attempt deletes fleet.enc and fleet.enc.lock from the state path. We need tracking for the migration (it should only be applied once when migrating from 3.1. to 3.2 or greater). I am wondering if we could use a marker file in the state path to track this and rely on a shell script. An alternative would be an annotation on the pod itself.

if [[ ! -f "/usr/share/elastic-agent/state/eck.config_migrated" ]]; then
  echo "Attempting to remove fleet.enc and fleet.enc.lock from state path (ignore if not present)"
  rm -f "/usr/share/elastic-agent/state/fleet.enc" "/usr/share/elastic-agent/state/fleet.enc.lock" 2>/dev/null || true
  echo "Creating eck.config_migrated marker"
  touch "/usr/share/elastic-agent/state/eck.config_migrated"
fi

What I don't know yet is how we detect the condition, that we actually need to delete these files. Ideally this should only happen when migrating from 3.1. to >=3.2

Update: I am suspecting the migration is only necessary if a user went back and forth between CONFIG_PATH=/etc/agent and CONFIG_PATH=STATE_PATH as I did during testing. So I think we could maybe provide this proactively to support in case uses switch back and forth between ECK versions and run into this. But we do not need to add this to the ECK orchestration logic.

[naemono]

8.13 fleet-mode is throwing errors with this setup, specifically with the fleet-server agent:

Error: preparing STATE_PATH(/usr/share/elastic-agent/state) failed: mkdir /usr/share/elastic-agent/state/data: permission denied

Didn't investigate the issue as of yet...

7.17, 8.18, 9.x fleet-mode doesn't have the issue...

[naemono]

Testing advanced (9.2.0-snap) doesn't seem to work:

[root@gke-mmontgomery-dev-clus-default-pool-dbf006f3-xyov elastic-agent]# elastic-agent inspect | grep add_resource_metadata
[root@gke-mmontgomery-dev-clus-default-pool-dbf006f3-xyov elastic-agent]#

is this what you wanted applied @pebrc ? I'm not terribly familiar with this advanced config...

❯ kc get agent -n elastic-stack  eck-stack-eck-agent -o yaml | yq '.spec.config'
providers.kubernetes:
  add_resource_metadata:
    deployment: true

[pebrc]

Testing advanced (9.2.0-snap) doesn't seem to work:

I think you are right. The implementation from this PR seems to break advanced config mode.
Only if I manually delete rm state/elastic-agent.yml and restart the agent will it pick up the advanced config specified with the -c flag. This sounds like a bug to me. cc @pkoutsovasilis

The previous implementation shipped with 3.1 declared /etc/agent to be the config directory, so anything we put there was automatically picked up.

[pebrc]

After speaking with @pkoutsovasilis the relevant code on agent is https://github.com/elastic/elastic-agent/blob/6186951dfbad2a7e0e1a37c26097d5b4d9d38dba/internal/pkg/agent/cmd/container.go#L853-L858 It seems there is a bug in that this code only checks if the file is there but does not actually take contents into account.

We could look into an init container that removes the file if it exists to force agent to copy it again.

[pebrc]

We could look into an init container that removes the file if it exists to force agent to copy it again.

This might not be necessary after all. It seems that elastic-agent inspect is not a reliable way to check the configured providers. We need to probably test against the actually ingested data to see if it works.

[pebrc]

LGTM I did extensive testing on this one. @rhr323 we need a known issue entry for this one. If users downgrade to 3.1 or before and upgrade again to 3.2 they might run into errors like

│ Error: fail to read state store '/usr/share/elastic-agent/state/data/state.enc': failed migrating YAML store JSON store: could not parse YAML                                                                                                                                                                             │
│ fail to decode bytes: cipher: message authentication failed    

or

 Error: fail to read action store '/usr/share/elastic-agent/state/data/action_store.yml': yaml: input error: fail to decode bytes: cipher: message authentication failed         

in these cases they should add the FLEET_FORCE=true environment variable to their manifest to force agent to enrol anew (it can be removed once the agent has re-enroled)

[rhr323]

💚 All backports created successfully

Status	Branch	Result
✅	3.2

Questions ?

Please refer to the Backport tool documentation