NS17/NS23 linux server OFFLINE for FSCK
13-04-2013
12:00PM - The Linux server NS17/23 has had an unexpected file system issue and was having some errors due to the same. The server had to be taken offline for an unscheduled emergency File System Check. This process can take 30-45 minutes and can sometimes take longer if there is a larger number of file system errors. The FSCK is presently in progress and we will update you once it is completed and server is brought back online.
This process will not affect any website data and there would be no data loss. Only file system related errors get corrected after which the server will automatically come back online.
1:00PM - FSCK is still running. Server appears to be fixing a large number of errors and is still offline. Unfortunately, the process does not show a progress bar or time indication to provide an exact time frame but we expect that it could be completed any time now and on the higher side should be done within another hour.
1:20PM - FSCK still in progress. We have to allow the process to complete as stopping it halfway could lead to file system corruption or other issues.
2:00PM - FSCK is still running at this time and is taking much longer than estimated. We will update once it completes and server is back online. It is best to allow it to complete so that it does not lead to any other issues.
2:30PM - A screenshot of what a typical FSCK (at boot) screen looks like is below. No progress is shown. The process is still running.
3:00PM - FSCK is still running. We have verified with the data center team as well who have physically connected to the server and checked that the FSCK is still in progress.
4:00PM - Since this has taken an abnormally long time, we have gone ahead and rebooted the server and it has come back normally now without any errors. We will be monitoring it now and if required will schedule another FSCK run.
4:30PM - The server has gone into read-only mode again. We will have to unfortunately run the FSCK again now. Rebooting server into maintenance mode.
5:45PM - We had to finally boot the server into rescue mode and the FSCK is presently running. Since this has been run manually and not at boot time, we can see that it is progressing. A lot of orphan inodes have been located and fixed. We hope that it should get done within an hour now but it depends on how many errors it finds.
6:00PM - FSCK has completed. Server has been restarted and all services are responding normally now.
