The following is the incident report for the MySql Database connection loss that occurred 04 April, 2023. We understand this issue has impacted our valued customers and their employees, and we apologize to everyone who was affected.
We understand it’s an ordeal and has been a very frustrating and tiring time for you. But now we have rectified the issue and everything is working fine.
| Incident Details | |
|---|---|
| Parent Incident | #INC_DB_00001 |
| # of Child Incidents | 0 |
| Date(s) | 04/04/2023 |
| Start Time | 12.PM IST |
| End Time | 7.00 PM IST |
Description of Incident
MySql database connection on few of the HRMS instances were lost on Tuesday, 04 April, from 12:00 AM IST. This was brought to our notice as employees from this impacted HRMS instances were not able to access the application and login into the system.
Root Cause
Few applications could not connect to MySql database because of corrupt ibdata1 file. ibdata1 is a single system tablespace data file. The file is used to store data and indexes of all the tables of a MySQL database running on an InnoDB storage engine. Due to hardware failure of host server, this ibdata1 file got corrupted.
Resolution & Recovery
As soon as server administration team became aware of this issue, the team attempted restarting the server. But due to corrupt ibdata1 file, the best possible approach was to restore the database backups. As part of our data policy, we take daily backups. We used the backups taken on Monday, 03 April, 2023 at midnight and restored on new database server. Hence the service was restored.
Data Loss
There could be transactional data loss that might have occurred between 12:00 AM till 12.00 PM IST on Tuesday, 04 April as latest backup was taken on Monday, 03 April Midnight.
Actions and Recommendations
An investigation is underway to develop better mechanism for identifying hardware issues and notification alerts if any file corruptions.
When we are able to determine the best approach, a follow-up message will be issued and we will propose configuration changes to make service restoration immediate and transparent in the future.
uKnowva HRMS is committed to continually and quickly improving our technology and operational processes to prevent such incidents. We appreciate your patience and again apologize for the impact to you, your users, and your organization. We thank you for your business and continued support.
If you are still facing issues, please reach our Customer Support Team at This email address is being protected from spambots. You need JavaScript enabled to view it.