GitHub Community
[2025-03-03] Incident Thread #152883
-
|
❗ An incident has been declared:Incident with Issues, Git Operations and API Requests Subscribe to this Discussion for updates on this incident. Please upvote or emoji react instead of commenting +1 on the Discussion to avoid overwhelming the thread. Any account guidance specific to this incident will be shared in thread and on the Incident Status Page. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments
-
UpdateWe are seeing intermittent timeouts across our various services. We are currently investigating and will provide updates. |
Beta Was this translation helpful? Give feedback.
-
UpdateWe are investigating intermittent connectivity issues between our backend and databases and will provide further updates as we have them. The current impact is you may see elevated latency while using our services. |
Beta Was this translation helpful? Give feedback.
-
UpdateWe have seen recovery across our services and impact is mitigated. |
Beta Was this translation helpful? Give feedback.
-
Incident ResolvedThis incident has been resolved. |
Beta Was this translation helpful? Give feedback.
-
Incident SummaryOn March 3rd 2025 between 04:07 UTC and 09:36 UTC various GitHub services were degraded with an average error rate of 0.03% and peak error rate of 9%. This issue impacted web requests, API requests, and git operations. This incident was triggered because a network node in one of GitHub's datacenter sites partially failed, resulting in silent packet drops for traffic served by that site. At 09:22 UTC, we identified the failing network node, and at 09:36 UTC we addressed the issue by removing the faulty network node from production. In response to this incident, we are improving our monitoring capabilities to identify and respond to similar silent errors more effectively in the future. |
Beta Was this translation helpful? Give feedback.
Incident Summary
On March 3rd 2025 between 04:07 UTC and 09:36 UTC various GitHub services were degraded with an average error rate of 0.03% and peak error rate of 9%. This issue impacted web requests, API requests, and git operations.
This incident was triggered because a network node in one of GitHub's datacenter sites partially failed, resulting in silent packet drops for traffic served by that site. At 09:22 UTC, we identified the failing network node, and at 09:36 UTC we addressed the issue by removing the faulty network node from production.
In response to this incident, we are improving our monitoring capabilities to identify and respond to similar silent errors more effectively in…