False-positive “DB Node Down Severe Alert” alerts
Incident Report for YugabyteDB Aeon
Resolved
A new component was added during the regular deployment process for YugabyteDB Managed to improve monitoring. This new component’s metrics were included in the monitoring for customer alerts before all the required infrastructure changes were made. The preemptive addition of this component to customer-facing alerts resulted in some customers receiving false-positive email alerts for their DB Nodes and Clusters. To rectify this issue, the customer-facing alerts have since been redeployed, excluding the metrics from the new component management tool. Yugabyte apologizes for any confusion these false-positive alerts may have caused our customers; if you are experiencing any issues that you believe are related to this matter, don’t hesitate to get in touch with Yugabyte Support via your normal channels.
Posted Jul 20, 2023 - 17:30 UTC
Update
We are continuing to monitor for any further issues.
Posted Jul 20, 2023 - 15:55 UTC
Monitoring
A fix has been implemented and Yugabyte are monitoring results.
Posted Jul 20, 2023 - 13:00 UTC
Identified
During a routine monitoring upgrade that occurred between 10:00 - 11:00 UTC, several customers received emails alerting them that one or more of their DB Nodes were down. This alert was triggered erroneously and should be ignored, Yugabyte are correcting the issue now and will update this status once complete. Until then, please disregard any “DB Node Down Sever Alert” messages you may receive. If you are experiencing connectivity issues with your cluster or any of your nodes, please log a support ticket through the support portal. Thank you.
Posted Jul 20, 2023 - 12:33 UTC
This incident affected: YugabyteDB Aeon Management Services.