Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

IPv6-only EC2 instance in AL2023 cannot resolve ECR public dual-stack endpoints (DNS resolution failure)

0

I am attempting to deploy a Rust service on an IPv6-only EC2 Spot Instance (Amazon Linux 2023) within a public subnet. The instance needs to pull Docker images from ECR's public dual-stack endpoints. I am consistently encountering 504 Gateway Timeout errors from CloudFront, which upon investigation of EC2 UserData logs, points to a Connect timeout during docker login and docker pull operations. This indicates a network connectivity issue, specifically with DNS resolution for IPv6.

Current Setup:

  • EC2 Instance: Amazon Linux 2023, t3.micro Spot Instance.
  • VPC: Configured with IPv6 CIDR, public subnets with AssignIpv6AddressOnCreation: true.
  • Routing: Internet Gateway configured with routes for both IPv4 and IPv6 (::/0).
  • Security Group: Allows outbound IPv6 traffic (::/0). Inbound rules are configured for IPv6 on the application port (8080) and SSH (22). Redundant IPv4 egress rules have been removed.
  • IAM Role: Attached to the EC2 instance with AmazonEC2ContainerRegistryReadOnly, CloudWatchAgentServerPolicy, and AmazonSSMManagedInstanceCore policies.

Steps Taken in UserData Script:

  1. Installed aws-cfn-bootstrap, docker, bind-utils, and curl using dnf install --allowerasing.
  2. Started and enabled Docker.
  3. Explicitly configured Google's public IPv6 DNS server (2001:4860:4860::8888) in /etc/resolv.conf.

Observed Behavior in UserData Logs:

  • cat /etc/resolv.conf confirms nameserver 2001:4860:4860::8888 is present.
  • dig AAAA ecr.us-west-2.amazonaws.com returns ANSWER: 0, indicating no IPv6 addresses are resolved for ECR.
  • curl -6 -v https://ecr.us-west-2.amazonaws.com fails with curl: (6) Could not resolve host: ecr.us-west-2.amazonaws.com.
  • curl -6 -v https://ipv6.google.com also fails with "Could not resolve host".
  • docker login and docker pull commands subsequently fail due to host resolution issues.

Despite configuring an IPv6 DNS resolver and having an IPv6 route to the Internet Gateway, the instance is unable to resolve public IPv6 hostnames.

Question: What further steps can I take to troubleshoot and resolve the IPv6 DNS resolution issue for public endpoints (like ECR and S3) on an IPv6-only Amazon Linux 2023 EC2 instance in a public subnet? Are there specific VPC or EC2 configurations (e.g., DHCP options, NACLs, or other network settings) that need to be explicitly set for IPv6 DNS resolution to work correctly in this scenario?

Topics
ComputeNetworking & Content Delivery
Tags
Amazon EC2Amazon VPCAmazon LinuxIPv6
Language
English
asked a month ago37 views
2 Answers
1

ECR has IPv6 support as of May 2025.

curl -6 -v https://ipv6.google.com also fails with "Could not resolve host".

From your output, name resolution is not working. You will need to resolve that first. The default AWS DNS server supports IPv6 DNS resolution.

For ECR over IPv6, use the following

  • OCI and Docker client requests: <registry-id>.dkr-ecr.<aws-region>.on.aws
  • AWS API requests: ecr.<aws-region>.api.aws

More information from documentation

Below output from dual stack AL2023 EC2 instance.

[ec2-user@ip-172-31-16-179 ~]$ cat /etc/os-release | grep PRETTY
PRETTY_NAME="Amazon Linux 2023.8.20250908"
[ec2-user@ip-172-31-16-179 ~]$ cat /etc/resolv.conf | grep name
nameserver 172.31.0.2
[ec2-user@ip-172-31-16-179 ~]$ curl -I https://ipv6.google.com
HTTP/2 200
content-type: text/html; charset=ISO-8859-1
content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-4YnwydjAcL7xHuGsNA8cCg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
accept-ch: Sec-CH-Prefers-Color-Scheme
p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
date: Wed, 10 Sep 2025 14:32:49 GMT
server: gws
x-xss-protection: 0
x-frame-options: SAMEORIGIN
expires: Wed, 10 Sep 2025 14:32:49 GMT
cache-control: private
set-cookie: AEC=AVh_V2jLeq9WZ18wx64G9yXlWzFqO7w0uzOT5gk8iQ-S2EbUqAthx43CoOY; expires=Mon, 09-Mar-2026 14:32:49 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
set-cookie: NID=525=cOGnt7ArGycASnI2ZUIKN1uUnE_RhcgjB43t6RFVbkgdbeICX82g9seCEVgE6YT8NOHnh6oFbuuu1Z9bzIMZLb2ug32qG6bglXwyYCvTmgmxHlxjy34Ktsa1MiInTf7rZOxxD2oAxGeSTROjcqBY62HJ7nnzB0hU3C-juX6tejOfbaXy_7AbWQTmFeLDL0yMOqBfrLxkuM9HUdo; expires=Thu, 12-Mar-2026 14:32:49 GMT; path=/; domain=.google.com; HttpOnly
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

[ec2-user@ip-172-31-16-179 ~]$ curl -6 -I https://ecr.us-west-1.amazonaws.com
curl: (6) Could not resolve host: ecr.us-west-1.amazonaws.com
[ec2-user@ip-172-31-16-179 ~]$ curl -6 -I https://ecr.us-west-1.api.aws
HTTP/1.1 404 Not Found
x-amzn-RequestId: f2bca94a-3a26-4561-9cc0-d057ddb0dc8f
Date: Wed, 10 Sep 2025 14:36:30 GMT
connection: keep-alive

[ec2-user@ip-172-31-16-179 ~]$ dig +short AAAA ecr.us-west-1.api.aws
2600:1f70:c000:c0:ee3e:c809:ca61:e30
AWS
EXPERT
answered 25 days ago
0

To resolve the IPv6 DNS resolution issue for your IPv6-only EC2 instance, you need to implement DNS64 and NAT64 to enable communication with IPv4-only services. Here's how to address your specific problem:

  1. Enable DNS64 on your subnet:
  • Use the AWS CLI command modify-subnet-attribute or the VPC console (select your subnet and choose Actions > Edit subnet settings)
  • This enables the Amazon Route 53 Resolver to synthesize IPv6 addresses for IPv4-only services
  1. Configure a NAT Gateway:
  • Deploy a NAT Gateway in your public subnet
  • Update your route tables to direct traffic to the NAT Gateway for the synthesized IPv6 addresses (64:ff9b::/96)
  • The NAT Gateway will perform the necessary translation between IPv6 and IPv4
  1. Use the correct ECR dual-stack endpoints:
  • For Docker client requests: <registry-id>.dkr-ecr.<aws-region>.on.aws (not amazonaws.com)
  • For AWS API requests: ecr.<aws-region>.api.aws
  • For ECR Public: ecr-public.aws.com (for Docker client) or ecr-public.us-east-1.api.aws (for API)
  1. Verify network connectivity:
  • Check that your security groups and NACLs allow outbound IPv6 traffic (which you've already done)
  • Ensure your VPC and subnet have the proper IPv6 CIDR blocks assigned
  1. Troubleshoot DNS resolution:
  • After enabling DNS64, test resolution again with dig or nslookup
  • With DNS64 enabled, the Route 53 Resolver should return synthesized IPv6 addresses for IPv4-only services

The key insight is that your IPv6-only instance cannot directly communicate with IPv4-only services without DNS64 and NAT64 working together. DNS64 provides synthesized IPv6 addresses, and NAT64 (via NAT Gateway) handles the actual protocol translation.
Sources
Designing DNS for IPv6 - IPv6 on AWS
DNS64 and NAT64 - Amazon Virtual Private Cloud
Troubleshoot connection to AWS service endpoints from Linux | AWS re:Post
Making requests to Amazon ECR registries - Amazon ECR

answered a month ago

Relevant content