Adds certificate identity field to cross-cluster API keys by gmjehovich · Pull Request #134604 · elastic/elasticsearch

gmjehovich · 2025-09-12T02:12:14Z

This PR adds support for a certificate_identity field in cross-cluster API keys in order to enable linking these API keys to a certificate-based identity.

Overview of changes:

Add certificate_identity field to API key documents and requests
Support creation/updates of cross-cluster API keys with cert identity
Update serialization and response classes
Add mixed-cluster safety checks and Security Features

elasticsearchmachine · 2025-09-12T02:12:40Z

Hi @gmjehovich, I've created a changelog YAML for you.

jfreden

Good job! I took an initial look and left some comments related to the mixed cluster/upgrade scenarios.

Looks like there are some compilation issues now and you need the new field in TransportUpdateCrossClusterApiKeyAction and BulkUpdateApiKeyRequest (pass null since adding the same identity to several api keys is not a use case yet).

An upgrade test in ApiKeyBackwardsCompatibilityIT would be nice, to make sure that the upgrade path works as expected. QueryableBuiltInRolesUpgradeIT uses node features here you might be able to copy/break out some of that logic for reuse.

Since we're touching the API Key interfaces it would be good to add the test-update-serverless label to make sure the changes to the constructors are compatible with serverless.

In the PKI Realm when the username_pattern is configured, we compile the pattern as part of the setting validation. I wonder if we should do that here too? See PKIRealm here.

...ck/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/ApiKey.java

jfreden · 2025-09-12T12:17:37Z

...in/java/org/elasticsearch/xpack/core/security/action/apikey/BaseBulkUpdateApiKeyRequest.java

        @Nullable final Map<String, Object> metadata,
-        @Nullable final TimeValue expiration
+        @Nullable final TimeValue expiration,
+        @Nullable final String certificateIdentity


Looks like there is no other way to add certificateIndentity to the update request because of how the ApiKeyService is coupled with BaseBulkUpdateApiKeyRequest. It's out of scope for this PR to fix that coupling I think, but it gets a little awkward to have to include the new field in bulk just to be able to use the update path.

jfreden · 2025-09-12T12:52:50Z

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/ApiKeyService.java

                            }, listener::onFailure))
                        )
                    );
+            } catch (MapperParsingException e) {


When creating or updating an API Key document in the security index, the prepareIndexIfNeededThenExecute is used. That method makes sure that the minimum mapping version that's supported by the cluster is applied to the security index, see this.

If node features are used to guard against writing the new field, that means that we can guarantee that all nodes have a mapping that supports that node feature. So in other words, if the cluster has the new feature, you can assume the mapping is up to date so this exception handling is not needed.

jfreden · 2025-09-12T12:59:41Z

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/authc/ApiKeyService.java


    private static final Logger logger = LogManager.getLogger(ApiKeyService.class);
    private static final DeprecationLogger deprecationLogger = DeprecationLogger.getLogger(ApiKeyService.class);
+    private final SecurityFeatures securityFeatures = new SecurityFeatures();


SecurityFeatures is just holding the NodeFeature records that the security plugin manages. To check if a node has a feature, an instance of the FeatureService is needed.

Before writing the new field, I think all that's needed is a check using clusterHasFeature(CERTIFICATE_IDENTITY_FIELD_FEATURE) and throw an error if the feature is not supported yet.

jfreden · 2025-09-12T14:22:54Z

As a follow up to this PR, the specification for create/update/query/get should be updated here. Here is an example PR

cla-checker-service · 2025-09-18T15:21:47Z

💚 CLA has been signed

github-actions · 2025-09-18T15:23:31Z

🔍 Preview links for changed docs

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

… field

elasticsearchmachine · 2025-09-18T18:26:31Z

Hi @gmjehovich, I've created a changelog YAML for you.

…ssClusterAPIKey methods to account for new field

gmjehovich · 2025-09-26T10:13:54Z

...rolling-upgrade/src/test/java/org/elasticsearch/upgrades/ApiKeyBackwardsCompatibilityIT.java

+    public void testCertificateIdentityBackwardsCompatibility() throws Exception {
+        assumeTrue(
+            "certificate identity backwards compatibility only relevant when upgrading from pre-9.2.0",
+            UPGRADE_FROM_VERSION.before(Version.V_9_2_0)


Is this the correct version, or is this feature going to be added in 9.3.0?

The PR is labeled for 9.2 that is feature freeze on September 30th so depending on when it's merged it will be 9.2 or 9.3.

jfreden

LGTM!

Great job on this! I only have a couple of non-blocking comments. Let me know if you want to discuss more in detail.

jfreden · 2025-09-29T11:27:05Z

...e/src/main/java/org/elasticsearch/xpack/core/security/action/apikey/CertificateIdentity.java

+                Pattern.compile(value);
+            } catch (PatternSyntaxException e) {
+                throw new IllegalArgumentException(
+                    "Invalid certificate_identity format: [" + value + "]. Must be a valid regex name pattern.",


Suggested change

"Invalid certificate_identity format: [" + value + "]. Must be a valid regex name pattern.",

"Invalid certificate_identity format: [" + value + "]. Must be a valid regex.",

jfreden · 2025-09-29T11:28:28Z

...ava/org/elasticsearch/xpack/core/security/action/apikey/UpdateCrossClusterApiKeyRequest.java

+        if (roleDescriptors == null && metadata == null && certificateIdentity == null) {
            validationException = addValidationError(
-                "must update either [access] or [metadata] for cross-cluster API keys",
+                "must update [access] or [metadata] or [certificate_identity] for cross-cluster API keys",


nit: this is still unchanged.

jfreden · 2025-09-29T11:39:10Z

.../org/elasticsearch/xpack/security/rest/action/apikey/RestCreateCrossClusterApiKeyAction.java

        PARSER.declareObject(constructorArg(), CrossClusterApiKeyRoleDescriptorBuilder.PARSER, new ParseField("access"));
        PARSER.declareString(optionalConstructorArg(), new ParseField("expiration"));
        PARSER.declareObject(optionalConstructorArg(), (p, c) -> p.map(), new ParseField("metadata"));
+        PARSER.declareField(


In the create flow there is no use case for explicit null vs implicit. I wonder if just using String for create would simplify getCertificateIdentityFromCreateRequest slightly. You still want to do the validation so you might have to extract to a static method if you decide to implement this. I'll leave it up to you.

jfreden · 2025-09-29T11:40:22Z

...plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/ApiKeyServiceTests.java

        assertThat(e2.getMessage(), containsString("owner user role descriptors must not include restriction"));
    }

+    public void testMaybeBuildUpdatedDocumentCertificateIdentityHandling() throws Exception {


jfreden · 2025-09-29T11:46:20Z

...rolling-upgrade/src/test/java/org/elasticsearch/upgrades/ApiKeyBackwardsCompatibilityIT.java

+    }
+
+    @SuppressWarnings("unchecked")
+    private Map<Boolean, RestClient> getRestClientByCertificateIdentityCapability() throws IOException {


I don't see any updates to RolesBackwardsCompatibilityIT or AbstractUpgradeTestCase was it not possible to put that in shared code? No worries if it doesn't make sense :)

This PR is a follow up to #134137 and #134893. It adds serialization and verification of the new header: - The signing configurations are now used to generate a signature that's passed as a header for cross cluster api keys. - The signature headers are deserializaed and validated on the server side and auth fails if the validation fails. This PR does not use the certificate identity that was added in #134604 to verify that the identity in the passed leaf certificate belongs to the signed cross cluster API key by matching it against the API key certificate identity pattern. That will be done in a follow up PR to keep the scope of this PR manageable.

This adds documentation for the RCS Strong Verification feature added in elastic/elasticsearch#136299, elastic/elasticsearch#134137, elastic/elasticsearch#134893, elastic/elasticsearch#135674 and elastic/elasticsearch#134604. Related settings docs PR: elastic/elasticsearch#137822 --------- Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> Co-authored-by: Edu González de la Herrán <25320357+eedugon@users.noreply.github.com>

gmjehovich requested a review from jfreden September 12, 2025 02:12

gmjehovich self-assigned this Sep 12, 2025

gmjehovich added >enhancement :Security/Security Security issues without another label Team:Security Meta label for security team v9.2.0 labels Sep 12, 2025

jfreden reviewed Sep 12, 2025

View reviewed changes

gmjehovich closed this Sep 18, 2025

gmjehovich force-pushed the rcs_improvements branch from 7af055d to 3523c09 Compare September 18, 2025 15:24

Add integration testing for CrossCluster API Key certificate_identity…

c919d6c

… field

gmjehovich reopened this Sep 18, 2025

[CI] Auto commit changes from spotless

7458dd3

gmjehovich added the test-update-serverless label Sep 18, 2025

gmjehovich and others added 11 commits September 18, 2025 13:26

Update docs/changelog/134604.yaml

f181837

Add 'certificate_identity' field to API Key, modify create/update Cro…

0a639c6

…ssClusterAPIKey methods to account for new field

fix merge conflicts

3a25dfa

[CI] Auto commit changes from spotless

2d32743

Merge branch 'main' into rcs_improvements

cf45055

Fix constructors, spotlessApply

0f04676

Fix createCrossClusterApiKey unit test

ef5a9e5

[CI] Auto commit changes from spotless

8883e07

Fix updateCrossClusterApiKeyRequestTest

539302b

[CI] Auto commit changes from spotless

13a9a69

Merge branch 'main' into rcs_improvements

fa1dff4

Delete redundant integ tests

4ef30ed

gmjehovich commented Sep 26, 2025

View reviewed changes

Consolidate redundant code in ApiKeyBackwardsCompatibilityIT

6de1798

jfreden approved these changes Sep 29, 2025

View reviewed changes

elasticsearchmachine added v9.3.0 and removed v9.2.0 labels Oct 2, 2025

gmjehovich and others added 12 commits October 2, 2025 09:32

Merge branch 'main' into rcs_improvements

05ffd83

Merge branch 'main' into rcs_improvements

95233d1

Merge branch 'main' into rcs_improvements

01f4c89

Change cert_identity validation failure message

af01316

Update cert_identity version to 9.3.0 in APIKey BWC Test

9706f85

[CI] Auto commit changes from spotless

7049572

CertID parser no longer differentiates between explicit vs implicit null

e0562b9

[CI] Auto commit changes from spotless

2e440b5

Move shared bwc test logic to AbstractUpgradeTestCase

98c4896

[CI] Auto commit changes from spotless

e8af6c3

Rename cleanUp method in TokenBackwwardsCompatbilityIT

13e010a

Merge branch 'main' into rcs_improvements

9c008eb

gmjehovich merged commit ce18c5b into elastic:main Oct 3, 2025
40 checks passed

This was referenced Oct 3, 2025

Send cross cluster api key signature as header #135674

Merged

Add specification for certificate_identity field for cross-cluster ap… elastic/elasticsearch-specification#5424

Closed

gmjehovich mentioned this pull request Oct 8, 2025

Add certificate identity field to Cross-Cluster API Key Specs elastic/elasticsearch-specification#5446

Merged

jfreden mentioned this pull request Oct 9, 2025

Remove ES version dependency in bwc tests #136255

Merged

jfreden mentioned this pull request Oct 14, 2025

Validate Certificate Identity provided in Cross Cluster API Key Certificate #136299

Merged

legrego mentioned this pull request Oct 16, 2025

Support for cluster identities in cross-cluster API keys elastic/kibana#239428

Open

This was referenced Nov 10, 2025

Add RCS Strong Verification Documentation #137822

Merged

Add RCS Strong Verification Documentation elastic/docs-content#3864

Closed

Add RCS Strong Verification Documentation elastic/docs-content#3865

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds certificate identity field to cross-cluster API keys#134604

Adds certificate identity field to cross-cluster API keys#134604
gmjehovich merged 49 commits intoelastic:mainfrom
gmjehovich:rcs_improvements

gmjehovich commented Sep 12, 2025 •

edited

Loading

elasticsearchmachine commented Sep 12, 2025

jfreden left a comment •

edited

Loading

Uh oh!

jfreden Sep 12, 2025

jfreden Sep 12, 2025

jfreden Sep 12, 2025

jfreden commented Sep 12, 2025

cla-checker-service bot commented Sep 18, 2025 •

edited

Loading

github-actions bot commented Sep 18, 2025 •

edited

Loading

github-actions bot commented Sep 18, 2025

When to use applies_to tags:

What NOT to do:

elasticsearchmachine commented Sep 18, 2025

gmjehovich Sep 26, 2025

jfreden Sep 29, 2025

jfreden left a comment

jfreden Sep 29, 2025

jfreden Sep 29, 2025

jfreden Sep 29, 2025

jfreden Sep 29, 2025

jfreden Sep 29, 2025

Uh oh!

Labels

3 participants

	"Invalid certificate_identity format: [" + value + "]. Must be a valid regex name pattern.",
	"Invalid certificate_identity format: [" + value + "]. Must be a valid regex.",

Conversation

gmjehovich commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

elasticsearchmachine commented Sep 12, 2025

jfreden left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jfreden commented Sep 12, 2025

cla-checker-service bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

github-actions bot commented Sep 18, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

elasticsearchmachine commented Sep 18, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jfreden left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Labels

3 participants

gmjehovich commented Sep 12, 2025 •

edited

Loading

jfreden left a comment •

edited

Loading

cla-checker-service bot commented Sep 18, 2025 •

edited

Loading

github-actions bot commented Sep 18, 2025 •

edited

Loading