Google engineers were alerted to the issue on 22 September 2021 15:28 US/Pacific and immediately started an investigation. Addressing this is a key part of our corrective actions. Google's standard change control policy is to roll out all production updates progressively however, in this specific case, errors in the rollout process meant that the change rolled out much more rapidly. region and resulted in payment failure notifications being sent out to affected customers.
This update unintentionally caused a situation in which no credit card payments were able to be processed in the U.S. The trigger for this service disruption was an update to the payment configuration intended to support virtual credit cards (one-time credit card numbers from Visa and MasterCard). We are taking additional steps to prevent similar problems in the future, as outlined below.
Our engineering team has deployed a fix for this and has ensured that account settings are restored. We sincerely apologize for the inconvenience caused by the incorrect notification to our customers. After customers attempted to update their payment information in the admin console, they received a message that the payment could not be processed. Visit the payment overview page to make sure your payment information is up to date and to pay any outstanding charges.” Customers also received a notification alert in the Google Cloud Console to update their primary payment method. The issue persisted for 11 hours, 20 minutes.Īll customers in the United States who use credit card billing were inadvertently notified via email “Your payment information could not be processed. On 22 September 2021 14:10 US/Pacific, Google Cloud, Workspace, and Geo (Maps) customers in the United States (U.S.) incorrectly received alert notifications on the Google Cloud Console user interface and an email regarding an issue with processing automatic payments. Incident began at 21:10 and ended at 08:30 (times are in Coordinated Universal Time (UTC)). For incidents related to Google Analytics, visit the Google Ads Status Dashboard. For additional information on these services, please visit. Learn more about what's posted on the dashboard in this FAQ. If you are experiencing an issue not listed here, please contact Support. Check back here to view the current status of the services listed below. This page provides status information on the services that are part of Google Workspace.