Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Aurora RDS Auto-Upgrade stuck in a loop, unable to cancel/delete

0

So overnight, one of our Aurora databases decided to auto-upgrade to MySQL 8.0 but is seemingly stuck in a loop. It keeps making snapshots and "performing upgrades" for a day. Meanwhile the database is not accessible. No actions are available to us here.

We've already replaced that database with a new cluster in the meantime. Is there no way to cancel or delete the stuck DB upgrade? We've even created a support case but nobody's looked at it yet. There ought to be an option to cancel/delete/ask for support option in the console after a long period of non-completion.

The general event loop for the upgrade goes like this in a loop:

  • Database cluster engine major version upgrade started. Cluster remains online.
  • Upgrade preparation in progress: Starting online upgrade prechecks.
  • Upgrade preparation in progress: Completed online upgrade prechecks.
  • Taking database cluster offline while the primary instance completes the patch/upgrade process.
  • Upgrade preparation in progress: Starting offline upgrade prechecks.
  • Upgrade preparation in progress: Completed offline upgrade prechecks.
  • Upgrade in progress: Creating pre-upgrade snapshot [snapshot-name].
  • Upgrade in progress: Cloning volume.
  • Database cluster engine major version upgrade started. Cluster remains online. (Repeats)
Topics
AnalyticsMigration & ModernizationDatabase
Tags
Support CaseMySQLAmazon Relational Database ServiceAmazon Aurora
Language
English
asked 18 days ago61 views
3 Answers
0

I'm so sorry for the concern,

I've located your support case and flagged it internally. Please monitor your Support Center for updates: http://go.aws/support-center.

- Randi S.

AWS
EXPERT
answered 18 days ago
  • Leo
    18 days ago

    It's been hours and still no response/indications of the case being followed up on (Case ID ****************), can this be expedited?

0

What you're experiencing is a known issue with Aurora MySQL major version upgrades that can get stuck in a loop. Based on the event log you shared, the upgrade process is repeatedly running through the precheck phases and creating snapshots, but not completing the actual upgrade.

Several factors can cause an upgrade to stall or loop:

  1. Long-running or idle transactions that might be holding table locks
  2. XA transactions in the prepared state that need to be committed or rolled back
  3. Ongoing DDL (Data Definition Language) statements
  4. Large number of uncommitted changes across many rows
  5. High number of undo records
  6. Cross-Region replicas that haven't been patched yet

Since you've already created a support case, that's the right approach for getting this resolved. AWS Support will need to intervene to either complete or cancel the stuck upgrade process. Unfortunately, there isn't a self-service option in the console to cancel an in-progress major version upgrade once it has started.

While waiting for support to respond, you've made the right decision by creating a new cluster to maintain your operations. This is often the fastest way to restore service when an upgrade gets stuck.

When AWS Support responds to your case, they'll likely need to investigate the specific cause of the loop and may need to take manual actions on the backend to terminate the stuck upgrade process.
Sources
Troubleshooting for Aurora MySQL in-place upgrade - Amazon Aurora
Aurora DB Upgrade Stuck During Migration from 5.7.mysql_aurora.2.11.5 to Latest Version | AWS re:Post
Troubleshoot Aurora MySQL upgrade issues | AWS re:Post

answered 18 days ago
AWS
EXPERT
reviewed 18 days ago
0

I've looked into your case and I've had it forwarded for further review. Please keep an eye on your Support Center for the updates and further discussion on this topic: go.aws/support-center

Thank you for your patience while this is reviewed.

— Kraig E.

AWS
EXPERT
answered 18 days ago

Relevant content