Incident Report: 04/04/2023 MySql Database Connection Loss

The following is the incident report for the MySql Database connection loss that occurred 04 April, 2023. We understand this issue has impacted our valued customers and their employees, and we apologize to everyone who was affected.

We understand it’s an ordeal and has been a very frustrating and tiring time for you. But now we have rectified the issue and everything is working fine.

 

Incident Details
Parent Incident #INC_DB_00001
# of Child Incidents 0
Date(s) 04/04/2023
Start Time 12.PM IST
End Time 7.00 PM IST

Description of Incident

MySql database connection on few of the HRMS instances were lost on Tuesday, 04 April, from 12:00 AM IST. This was brought to our notice as employees from this impacted HRMS instances were not able to access the application and login into the system.

Root Cause

Few applications could not connect to MySql database because of corrupt ibdata1 file. ibdata1 is a single system tablespace data file. The file is used to store data and indexes of all the tables of a MySQL database running on an InnoDB storage engine. Due to hardware failure of host server, this ibdata1 file got corrupted.

Resolution & Recovery

As soon as server administration team became aware of this issue, the team attempted restarting the server. But due to corrupt ibdata1 file, the best possible approach was to restore the database backups. As part of our data policy, we take daily backups. We used the backups taken on Monday, 03 April, 2023 at midnight and restored on new database server. Hence the service was restored.

Data Loss

There could be transactional data loss that might have occurred between 12:00 AM till 12.00 PM IST on Tuesday, 04 April as latest backup was taken on Monday, 03 April Midnight.

Actions and Recommendations

An investigation is underway to develop better mechanism for identifying hardware issues and notification alerts if any file corruptions.

When we are able to determine the best approach, a follow-up message will be issued and we will propose configuration changes to make service restoration immediate and transparent in the future.

uKnowva HRMS is committed to continually and quickly improving our technology and operational processes to prevent such incidents. We appreciate your patience and again apologize for the impact to you, your users, and your organization. We thank you for your business and continued support.

If you are still facing issues, please reach our Customer Support Team at This email address is being protected from spambots. You need JavaScript enabled to view it.

Was this Article helpful?