Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Configure SSL client authentication with keystore and custom truststore for MSK Connect

0

Hello,

I'm trying to integrate an external Kafka cluster using MSK Connect. This external cluster needs Kafka clients to support mTLS, so providing a keystore is mandatory for a successful SSL handshake. The Kafka clients also need to trust a certificate, which ist not inside the default JVM cacerts. Since the MSK Connect documentation states, that "ssl." properties are not supported inside the worker configuration (https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-supported-worker-config-properties.html), I am wondering:

  • Is it even possible to provide keystores for Kafka connection/authentication with MSK Connect?
  • Is it possible to connect to a Kafka cluster requiring mTLS/SSL authentication?
  • Can I override the truststore location for MSK Connect?

I tried this connector configuration, but it resulted in a SSL handshake exception inside the worker:

{
  "connector.class": "io.confluent.connect.s3.S3SinkConnector",
  "s3.region": "<region>",
  "flush.size": "1",
  "tasks.max": "1",
  "format.class": "io.confluent.connect.s3.format.json.JsonFormat",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "s3.bucket.name": "<bucketName>",
  "store.kafka.keys": "true",
  "s3.ssea.name": "aws:kms",
  "schema.compatibility": "NONE",
  "topics": "test",
  "store.kafka.headers": "true",
  "s3.sse.kms.key.id": "<s3KeyId>",
  "headers.format.class": "io.confluent.connect.s3.format.json.JsonFormat",
  "partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
  "keys.format.class": "io.confluent.connect.s3.format.json.JsonFormat",
  "storage.class": "io.confluent.connect.s3.storage.S3Storage",
  "value.converter.schemas.enable": "false",
  "security.protocol": "SSL",
  "ssl.keystore.location": "${s3import:<region>:<bucketName>/keystore.jks}",
  "ssl.truststore.location": "${s3import:<region>:<bucketName>/truststore.jks}",
  "ssl.keystore.password": "<password>",
  "ssl.truststore.password": "<password>",
  "ssl.key.password": "<password>"
}
Topics
Analytics
Tags
Amazon Managed Streaming for Apache Kafka (Amazon MSK)
Language
English
asked 25 days ago14 views
1 Answer
0

To address your questions about configuring SSL client authentication with keystores and custom truststores for MSK Connect:

Unfortunately, MSK Connect does not currently support providing custom keystores or truststores for Kafka connections that require mTLS/SSL authentication. The "ssl." properties that you're trying to use in your connector configuration are not supported within the MSK Connect worker configuration.

Amazon MSK itself uses AWS Certificate Manager (ACM) certificates for its broker authentication, and any truststore that trusts Amazon Trust Services should also trust MSK brokers. However, when connecting to external Kafka clusters that require client certificate authentication, MSK Connect has limitations.

For Amazon MSK clusters, client authentication can be enabled using mutual TLS, which requires an AWS Private CA. The client would need to present a certificate signed by this CA during the TLS handshake. However, this is for clients connecting to MSK clusters, not for MSK Connect acting as a client to external clusters.

The error you're experiencing with the SSL handshake exception is likely because MSK Connect cannot use the custom keystore and truststore configurations you're trying to provide.

As an alternative approach, you might need to:

  1. Consider using a different solution for connecting to your external Kafka cluster that requires mTLS, such as setting up a self-managed Kafka Connect cluster on EC2 where you have full control over the SSL configuration.
  2. If possible, modify the authentication requirements of your external Kafka cluster to support other authentication methods that are compatible with MSK Connect.

Unfortunately, there isn't currently a way to override the truststore location or provide custom keystores for MSK Connect when connecting to external Kafka clusters requiring mTLS.
Sources
Mutual TLS client authentication for Amazon MSK - Amazon Managed Streaming for Apache Kafka
Configuring MSK Connect worker to pass SSL certs | AWS re:Post

answered 25 days ago

Relevant content