Problem

We noticed in some deployments, even though a instance has been terminated, the ENIs allocated by ipamD are NOT released back to EC2. In addition, the secondary IP addresses allocated on these ENI are also NOT released back to EC2.

When there are too many of these leaked ENIs and Secondary IP addresses, subnet available IP pool can be depleted. And node in cluster will failed to allocate secondary IP addresses. When this happens, Pod may not able to get an IP and get stucked in ContainerCreating

You can verify if you are running into this issue in console:

and in description aws-K8S-i-02cf6e80932099598, the instance i-02cf6e80932099598 has already been terminated

Workaround

Manually delete these ENI after confirming the instance has already been terminated.

Be careful, when ipamD is in the middle of create/attach ENIs, the eni will show up as available. but the instance-id should be valid