{"id":3787,"date":"2012-06-11T15:18:54","date_gmt":"2012-06-11T13:18:54","guid":{"rendered":"https:\/\/ingmarverheij.com\/?p=3787"},"modified":"2012-06-11T15:27:23","modified_gmt":"2012-06-11T13:27:23","slug":"virtual-sqlserver-impaired-by-residual-snapshot-after-veeam-backup","status":"publish","type":"post","link":"https:\/\/ingmarverheij.com\/en\/virtual-sqlserver-impaired-by-residual-snapshot-after-veeam-backup\/","title":{"rendered":"Virtual (SQL)server impaired by residual snapshot after Veeam backup"},"content":{"rendered":"

\"\"<\/a><\/p>\n

Recently I had to troubleshoot a SQL server that performed nightly batch jobs for a management information system. Under normal conditions this required 6.5  hours but this was suddenly increased to 11.5 hours. An increase of 75%!<\/p>\n

Because of this delay the information wasn\u2019t presented on time with a lot of implications.  Several departments where asked what has changed in the past days, of course the answer was \u201cnothing\u201d. <\/p>\n

<\/p>\n

Point in time<\/h2>\n

The delay of the batch was introduced since the 6th of June (increasing from 400 to 600+ minutes):<\/p>\n

\"\"<\/a><\/p>\n

 <\/p>\n

VMware vSphere Client<\/h2>\n

Performance<\/h3>\n

The performance metrics of the virtual machines showed a decrease in both processor and disk performance while the network was hardly affected. <\/p>\n

This is unexpected since the content of the batch job is unchanged, and the same applies for the infrastructure. No (major) changes are executed that justify the decrease in performance<\/p>\n

\"\"<\/a>\"\"<\/a>\"\"<\/a><\/p>\n

Storage<\/h3>\n

There was a sudden increase (of ~ 600GB) in allocated disk space, with a substantial amount for snapshots. Aha!<\/p>\n

\"\"<\/a><\/p>\n

Snapshot<\/h3>\n

Unless a change is performed (and a rollback is required) no snapshot should be present. However there was a snapshot called \u201cConsolidate Helper- 0\u201d .<\/p>\n

\"\"<\/a><\/p>\n

This snapshot was residual after a failed Veeam backup (as described Jim Jones in this article<\/a>).<\/p>\n

 <\/p>\n

Veeam Backup & Replication<\/h2>\n

To verify that the snapshot indeed was a leftover of a failed backup I verified the backup log. And indeed, after performing a successful backup on the 4th the backup of 5th of june ended with a warning:<\/em><\/p>\n

Removing snapshot\nUnable to connect to the remote server \nNo connection could be made because the target machine actively refused it xxx.xxx.xxx.xxx:443 \nVeeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. \nPossible causes for snapshot removal failure: \n- Network connectivity issue, or vCenter Server is too busy to serve the request \n- ESX host was unable to process snapshot removal request in a timely manner \n- Snapshot was already removed by another application<\/pre>\n
The backup on the 6th of june could not be completed at all and ended with an error:<\/em><\/p>\n
Initializing target session\nRemoveSnapshot failed, snapshotRef "snapshot-35436", timeout "3600000" \nUnable to access file  since it is locked<\/pre>\n
\"\"<\/a>\"\"<\/a>\"\"<\/a><\/p>\n
 <\/p>\n
Result<\/h2>\n
After removing the snapshot the storage space was reclaimed<\/p>\n
\"\"<\/a><\/p>\n
  
and the time required to perform the batch job was back to normal<\/p>\n
\"\"<\/a><\/p>\n
 <\/p>\n
Moral of the story<\/h2>\n
Be careful with snapshots of virtual machines. The impact on the performance can be dramatic and the time-to-fix can be quite a while if you\u2019re unaware of this. <\/p>\n
 <\/p>\n
More information :<\/p>\n