Hi everyone,
I've a strange issue on our AlwaysOn Cluster Sql Server 2012 Enterprise multi-instance, I'm evaluating to open a support case to MS but I would share the issue with the community before going on.
Sql server are dedicated to host Sharepoint 2010 farm content databases.
Scenario
n.2 VM (hypervisor ESX 5.1) Windows Server 2008 R2; each vm has 2 virtual disk, and 4 raw-mapped disk
each VM has MSFC role installed; quorum model is "file share majority"; no shared disk are mapped to clustered services.
each vm host a Sql Server 2012 enterprise installation; each VM host n.3 SQL named instance (Development, Testing, Production)
there are 3 Availability Group configured, one for each named-instance.
Raw-Mapped disk are configured in this way: raw1(500GB) for sql data, raw2(500GB) for t-log; raw3 and raw4 are dynamic disk configured in spanning mode (500+500GB) as sql backup repository.
Issue
note: This issue came out recently (while entire enviroment is on production since 1 year)
Our full backup job include a backup checksum and VERIFY.
Since a couple of weeks, this verify on a particular contentdb (CrawlStoreDB, around 80GB) cause the availability group to go in an unhealty state (RESOLVING).
From CLUSTER DIAGNOSTIC EXTENDED EVENT LOG I see this entries:
- info_message 2013-05-27 15:44:42.4845706 [hadrag] SQLFetch() returns -1 with following information
- info_message 2013-05-27 15:44:42.4845706 [hadrag] ODBC Error: [42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Could not serialize the data for node 'filePath' because it contains a character (0x0000) which is not allowed in XML. To retrieve this data convert it to binary, varbinary or image data type (6842)
- info_message 2013-05-27 15:44:42.4845706 [hadrag] No more diagnostics results
- info_message 2013-05-27 15:44:42.4845706 [hadrag] Discard the pending result sets
- info_message 2013-05-27 15:44:42.4845706 [hadrag] ODBC Error: [24000] [Microsoft][SQL Server Native Client 11.0]Invalid cursor state (0)
From AlwaysOn health event file
availability_replica_state_change 2013-05-27 15:45:02.6615881 PRIMARY_NORMAL RESOLVING_NORMAL CABD99D4-8591-4B37-9160-6738DAE9C851
From Failover Cluster Manager Events
- Cluster resource 'AGPROD' in clustered service or application 'AGPROD' failed.
- The Cluster service failed to bring clustered service or application 'AGPROD' completely online or offline. One or more resources may be in a failed state. This may impact the availability of the clustered service or application.
this situation remains "looping" for 1 hour, unless forcing a manual failover in SMSS; during this hour, the availability group is unreachble (RESOLVING)
If I deactivate the VERIFY on backup job, the issue doesn't happen.
Any idea to point out the problem?