On initial boot of a freshly deployed (Ubuntu) Linux appliance on VMware vSphere 5 the machine stopped working because of an inconsistent disk. According to the machine 255 days where passed without the disk being checked. After a file system check is forced the machine stops with an error UNEXPECTED INCONSISTENCY followed by fsck / [5001] terminated with status 4.
/dev/sda1 has gone x days without being checked
The first message was very strange since the machine was freshly deployed from an OVF. According to the vendor this issue occurs when the date and time on the Hypervisor Host is wrong. After correcting the date and time of the hypervisor to the current date and time this problem should be resolved.
I verified the date and time of the ESX host, virtual machine and my laptop; they all matched (although the ESX host and virtual machine show the UTC time and I run in DST (+1). Since this problem occurred on October 29th 2013, I tried changing the time to the past hoping to trick the machine. During the attempts I found the Superblock last mount time of the devices:
- /dev/sda1 – Fri Feb 15 10:53:23 2013
- /dev/sda6 – Fri Aug 16 18:32:41 2013
The results of my test where unsuccessful, but it’s worth sharing anyway
Attempt #1 – January 1st 2013
/dev/sda1 – Superblock last mount time (Fri Feb 15 10:53:34 2013, now = Tue Jan 1 14:49:08 2013) is in the future
/dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Tue Jan 1 14:49:08 2013) is in the future
Attempt #2 – May 15th 2013
/dev/sda1 – Clean
/dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Tue Jan 1 14:49:08 2013) is in the future
Attempt #3 – August 8th 2013
/dev/sda1 – Clean
/dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Thu Aug 18 15:07:31 2013) is in the future
Attempt #4 – August 15th 2013
/dev/sda1 – /dev/sda1 gas gone 180 days without being checked, check forced
/dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Thu Aug 15 18:33:05 2013) is in the future
Attempt #5 – August 16th 2013
/dev/sda1 – /dev/sda1 gas gone 182 days without being checked, check forced
/dev/sda6 – /dev/sda6 gas gone 182 days without being checked, check forced
Attempt #6 – December 12th 2013
/dev/sda1 – /dev/sda1 gas gone 299 days without being checked, check forced
/dev/sda6 – /dev/sda6 gas gone 299 days without being checked, check forced
As you can see I tried a lot (while not all of them made sense, I agree) but I could not find the sweet spot. Time for the next approach!
fsck – file system check
The machine clearly indicated two problems:
- The last file system check was executed 255 days ago
- There are errors with the consistency of the disk (filesystem has errors)
Since I found no other solution I addressed both problems at the same time: run a file system check.
Since the machine halted I was unable to run any command, including fsck,. So I mounted a Linux Live CD (FedoraLiveCD in my case, but you could also use the Ubuntu LiveCD) and ensured it booted from CD (change the boot order). After starting a shell (Terminal) I ran the su command to elevate my permissions.
su
Next I ran the fsck command on the /dev/sda1 device with the –f argument to force checking of the file system and the –y argument to answer Yes on all questions.
fsck /dev/sda1 -f –y
And the same command but for the the /dev/sda6device,.
fsck /dev/sda6 -f -y
Success
Sure enough (after unmounting the ISO) the machine now boots succesfull.
.
English 
Nederlands