I am trying to backup a VM which before it had Oracle installed backed up with no problems. It is a Windows 2008 R2 server which sites on an 8 node Hyper-V 2012 R2 cluster (with CSVs). I am using DPM 2012 R2 to run the backups which have been
successful for the last few weeks but then Oracle was installed and the backups have failed since then.
The job fails and a large number of VSS writers go into a 'Failed' state with the 'Last Error' showing as 'Timed out'. I then get 4 popups appear each referring to a 'temporary' drive which appears briefly in Disk Management with a RAW file system. These pops say "You need to format the disk in drive X: before you can use it. Do you want to format it?" (where X: is replaced by the drive letter assigned to each 'temporary' drive).
The System event log is populated with a large number of warnings with Event 51, Disk, stating "An error was detected on device \Device\Harddisk<number>\DR<number> during a paging operation". There are also a few warnings for Ntfs (eventid 57) stating that "The system failed to flush data to the transaction log. Corruption may occur."
Prior to these warnings there are 5 other warnings for partmgr (eventid 58) stating "The disk signature of disk <number x> is equal to the disk signatuire of disk <number y>" and 4 errors (eventid 1, VDS Basic Provider), "Unexpected failure. Error code: 490@01010004".
There is a script which is run to stop the Oracle application on the server and if this is run then the backups will complete successfully. We have been troubleshooting this by running a certain amount of the script and seeing which part affects the backup and it seems that if the Weblogic (wls_reports) service is stopped then the backup will succeed but if it is running then the backup will fail and the above symptoms occur.
Another point which may help is that there is a pre-production server which resides on a Windows 2008 R2 Hyper-V standalone server, has the same scripts and installation of Oracle but backs up without any issues.
I have experienced VSS writer failures before with VM backups but I have not seen this before. It is not intermittent and I can find no work around to alleviate the problem of having no backup (except stopping this service or shutting the server down, but as it is a production server this is not practical).
I have also posted this into the WSB forum under the following which may have further useful info:
VSS Writer failures during VM backup of server with Oracle application installed
If anyone has experienced this before or have any suggestions / advice it would be much appreciated.
Thanks
Chris
The job fails and a large number of VSS writers go into a 'Failed' state with the 'Last Error' showing as 'Timed out'. I then get 4 popups appear each referring to a 'temporary' drive which appears briefly in Disk Management with a RAW file system. These pops say "You need to format the disk in drive X: before you can use it. Do you want to format it?" (where X: is replaced by the drive letter assigned to each 'temporary' drive).
The System event log is populated with a large number of warnings with Event 51, Disk, stating "An error was detected on device \Device\Harddisk<number>\DR<number> during a paging operation". There are also a few warnings for Ntfs (eventid 57) stating that "The system failed to flush data to the transaction log. Corruption may occur."
Prior to these warnings there are 5 other warnings for partmgr (eventid 58) stating "The disk signature of disk <number x> is equal to the disk signatuire of disk <number y>" and 4 errors (eventid 1, VDS Basic Provider), "Unexpected failure. Error code: 490@01010004".
There is a script which is run to stop the Oracle application on the server and if this is run then the backups will complete successfully. We have been troubleshooting this by running a certain amount of the script and seeing which part affects the backup and it seems that if the Weblogic (wls_reports) service is stopped then the backup will succeed but if it is running then the backup will fail and the above symptoms occur.
Another point which may help is that there is a pre-production server which resides on a Windows 2008 R2 Hyper-V standalone server, has the same scripts and installation of Oracle but backs up without any issues.
I have experienced VSS writer failures before with VM backups but I have not seen this before. It is not intermittent and I can find no work around to alleviate the problem of having no backup (except stopping this service or shutting the server down, but as it is a production server this is not practical).
I have also posted this into the WSB forum under the following which may have further useful info:
VSS Writer failures during VM backup of server with Oracle application installed
If anyone has experienced this before or have any suggestions / advice it would be much appreciated.
Thanks
Chris