Why don't my Amazon S3 objects replicate when I set up replication between my buckets?

Short description

Note: You can use Amazon S3 replication only for general purpose buckets. You can't use replication for directory buckets and table buckets.

To troubleshoot Amazon S3 objects that don't replicate for cross-Region replication (CRR) or same-Region replication (SRR), check your destination bucket permissions. Also, check the public access settings and bucket ownership settings.

After you resolve the issues that caused the replication to fail, there might be objects in the source bucket that still don't replicate. By default, Amazon S3 replication doesn't replicate existing objects or objects with a FAILED or REPLICA replication status. To check the replication status of objects, see How do I view objects that failed replication from one Amazon S3 bucket to another? For these objects, use S3 Batch Replication.

Resolution

Identify replication setup issues

Upload an object to the source bucket to test the replication after each configuration change. It's a best practice to change one configuration at a time to identify any replication setup issues.

Also, activate the s3:Replication:OperationFailedReplication event type notification to determine the cause of the failure.

Grant the minimum Amazon S3 permissions

Confirm that the AWS Identity Access Management (IAM) role that you used in the replication rule has the correct permissions. If the source and destination buckets are in different AWS accounts, then confirm that the destination account's bucket policy grants permissions to the replication role. The following example IAM policy has the minimum required permissions for replication:

{    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::SourceBucket"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": [
                "arn:aws:s3:::SourceBucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateTags"
            ],
            "Resource": "arn:aws:s3:::DestinationBucket/*"
        }
    ]
}

Note: Replace SourceBucket with your source bucket and DestinationBucket with your destination bucket.

Based on the replication rule options, you might need to grant additional permissions.

The IAM role must have a trust policy that allows Amazon S3 to assume the role to replicate objects. Example trust policy:

{    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Grant additional Amazon S3 permissions

If you set the replication rule to Change object ownership to the destination bucket owner, then you must configure additional permissions.

Note: If the destination bucket's object ownership is Bucket owner enforced, then you don't need Change object ownership to the destination bucket owner in the replication rule. The change occurs by default.

To grant the IAM role the s3:ObjectOwnerOverrideToBucketOwner permissions, add the following permission to the Amazon S3 object policy:

{    "Effect": "Allow",
    "Action": [
        "s3:ObjectOwnerOverrideToBucketOwner"
    ],
    "Resource": "arn:aws:s3:::DestinationBucket/*"
}

Note: Replace DestinationBucket with your destination bucket.

Also, add the following s3:ObjectOwnerOverrideToBucketOwner permission in the bucket policy for the destination account:

{    "Sid": "1",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
    },
    "Action": [
        "s3:ObjectOwnerOverrideToBucketOwner"
    ],
    "Resource": "arn:aws:s3:::DestinationBucket/*"
}

Note: Replace SourceBucket-account-ID with the source bucket account, source-account-IAM-role with the source account IAM role, and DestinationBucket with the destination bucket.

If you activated delete marker replication on the replication rule, then the IAM role must have the following s3:ReplicateDelete permissions:

{    "Effect": "Allow",
    "Action": [
        "s3:ReplicateDelete"
    ],
    "Resource": "arn:aws:s3:::DestinationBucket/*"
}

Note: Replace DestinationBucket with your destination bucket.

If the destination bucket is in another account, then the destination bucket owner must also add the following permission in the bucket policy:

{    "Version": "2012-10-17",
    "Id": "PolicyForDestinationBucket",
    "Statement": [
        {
            "Sid": "Permissions on objects",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
            },
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateTags",
                "s3:ObjectOwnerOverrideToBucketOwner",
                "s3:ReplicateDelete"
            ],
            "Resource": "arn:aws:s3:::DestinationBucket/*"
        },
        {
            "Sid": "Permissions on bucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
            },
            "Action": [
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning"
            ],
            "Resource": "arn:aws:s3:::DestinationBucket"
        }
        
    ]
}

Note: Replace arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role with the Amazon Resource Name (ARN) of your replication role and DestinationBucket with your destination bucket.

Grant AWS KMS permissions

If you encrypted the bucket's source objects with an AWS Key Management Service (AWS KMS) key, then the replication rule must include AWS KMS encryption.

To configure the required permissions, complete the following steps:

1. Open the Amazon S3 console.
2. Choose the source bucket.
3. Choose the Management tab, and then under Replication rules choose the replication rule.
4. Choose Edit.
5. Under Encryption, choose Replicate objects encrypted with AWS KMS.
6. Under AWS KMS key for encrypting destination objects, select an AWS KMS key. The default option is to use the AWS KMS key (aws/S3).

For example replication policies, see Example policies - Using SSE-S3 and SSE-KMS with replication.

Note: If the destination bucket is in a different account, then specify an AWS KMS customer managed key that the destination account owns. The default aws/S3 key encrypts the objects with the AWS managed key that the source account owns. However, you can't share the AWS managed key with another account.

Grant additional AWS KMS permissions for cross-account scenarios

To use the destination account's AWS KMS key to encrypt the destination objects, the destination account must allow the replication role in the key policy. Example policy:

{    "Sid": "AllowS3ReplicationSourceRoleToUseTheKey",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role"
    },
    "Action": [
        "kms:GenerateDataKey",
        "kms:Encrypt"
    ],
    "Resource": "*"
}

Note: Replace arn:aws:iam::SourceBucket-account-ID:role/service-role/source-account-IAM-role with the replication role ARN. If you use an asterisk (*) for Resource, then the policy grants permission only to the replication role. However, the policy doesn't allow the replication role to expand its permissions.

Also, you must add the following minimum permissions to the replication role's IAM policy from the source account:

[      {  
        "Effect": "Allow",  
        "Action": [  
            "kms:Decrypt",  
            "kms:GenerateDataKey"  
        ],  
        "Resource": [  
            "SourceKmsKeyArn"  
        ]  
    },  
    {  
        "Effect": "Allow",  
        "Action": [  
            "kms:GenerateDataKey",  
            "kms:Encrypt"  
        ],  
        "Resource": [  
            "DestinationKmsKeyArn"  
        ]  
    }  
]

By default, the AWS KMS key policy grants the root user full permissions to the key. You can delegate these permissions to other users in the same account. You can also use an IAM policy to grant the replication role permissions to the source KMS key. This configuration allows the required access unless there are Deny statements in the source AWS KMS key policy.

Check for explicit deny and conditional allow statements

Important: Before you remove explicit Deny statements, confirm the reason they exist and whether the statements affect data security.

If your objects still don't replicate after you validated the permissions, then check for explicit Deny statements that can cause replication to fail.

Remove Deny statements in the destination bucket policy or AWS KMS key policy that restrict access to the following resources:

• Specific CIDR ranges
• Virtual private cloud (VPC) endpoints
• Amazon S3 access points

Also, remove Deny statements or permissions boundaries from the IAM role policy and AWS Organizations service control policies (SCPs) in both the source and destination accounts.

Check for S3 bucket keys

If the source or destination AWS KMS keys grant permissions based on the encryption context, then check whether you use S3 Bucket Keys. If the buckets use S3 Bucket Keys, then the encryption context must be for the bucket level resource. Example encryption context:

"kms:EncryptionContext:aws:s3:arn": ["arn:aws:s3:::SOURCE_BUCKET_NAME"]
"kms:EncryptionContext:aws:s3:arn": ["arn:aws:s3:::DESTINATION_BUCKET_NAME"]

Note: Replace SOURCE_BUCKET_NAME with the source bucket and DESTINATION_BUCKET_NAME with the destination bucket.

If you don't use S3 Bucket Keys for the source or destination buckets, then the encryption context must be the object level resource. Example encryption context:

"kms:EncryptionContext:aws:s3:arn": ["arn:aws:s3:::SOURCE_BUCKET_NAME/*"]
"kms:EncryptionContext:aws:s3:arn": ["arn:aws:s3:::DESTINATION_BUCKET_NAME/*"]

Note: Replace SOURCE_BUCKET_NAME with the source bucket and DESTINATION_BUCKET_NAME with the destination bucket.

Check your object ACLs

Check whether the source and destination buckets use access control lists (ACLs) that block access. If the object has an ACL that allows public access but the destination bucket uses S3 Block Public Access, then replication fails.

Confirm source object ownership

If another account uploaded the objects in the source bucket, then the source account might not have access to those objects. Check the source bucket has ACLs activated.

If the source bucket has ACLs deactivated, then the source account is the owner of all objects in the bucket. If the source bucket has ACLs activated, then check whether object ownership is set to Object owner preferred or Bucket owner preferred. If ownership is set to Bucket owner preferred, then the source bucket objects must have the bucket-owner-full-control ACL permission.

The source account can deactivate ACLs to take ownership of all objects in their bucket. Most use cases don't require ACLs to manage access. Instead, it's a best practice to use IAM and bucket policies to manage access to Amazon S3 resources. Before you deactivate ACLs, make sure that your bucket and IAM policies grant the required permissions to keep existing Amazon S3 access.

Specify the correct replication rule filter

Make sure that you correctly specified the replication rule filter.

If you specify a rule filter with a combination of a key prefix and object tags, then Amazon S3 performs a logical AND operation. The rule applies to a subset of objects with a specific key prefix and specific tags.

Related information

Examples for configuring live replication