I have AlwaysOn SQL 2012 Enterprise set-up using Windows Failover Clustering Services (not FCI), and have 1 Primary node (P), 1 Synchronous Commit Auto Failover (SC), and 1 Asynchronous Commit Manual Failover (AC) node. It is set up to prefer secondary, with the highest priority given to AC node.
I am using Ola Hollengren's scripts for Database Maintenance jobs, including a native BACKUP LOG job for the transaction logs of all user databases on a 1 minute schedule. His scripts already consider AlwaysOn, and although the job is set-up on all 3 nodes, only ever runs on AC node.
The job has been running successfully since initial set-up almost 1 year ago, but suddenly yesterday morning started to fail with the following error, only on 1 of the 13 databases in my availability group:
Date and time: 2014-06-08 09:36:11
Command: BACKUP LOG [my_db] TO DISK = N'E:\MSSQL\\Transaction Dumps\my_db\MySQLCL$MySQLAG_my_db_20140608_093610_U_LOG.trn' WITH CHECKSUM, COMPRESSION
Msg 35250, Level 16, State 11, Server AC, Line 1
The connection to the primary replica is not active. The command cannot be processed.
Msg 3013, Level 16, State 1, Server AC, Line 1
BACKUP LOG is terminating abnormally.
Outcome: Failed
Duration: 00:01:00
The other 12 databases continued to backup successfully.
Checking the Availability Group dashboard, windows event logs, and SQL Server error logs, including Failover Cluster events showed no issues.
However, monitoring software (Idera SQLdm) showed blocked sessions on P node. When I ran sp_who2, it showed that a background process was being blocked by another background process with an HADR BACKUP LOCK.
Since both processes were background processes, I was unable to kill either process. I temporarily disabled the transaction log backup job, but the blocked process was still active.
I ran DBCC CHECKDB (my_db) WITH all_errormsgs, no_infomsgs, data_purity on both P and AC nodes, with no errors. However, on AC node, it also showed 1 transaction rolled forward and 0 transactions rolled back. This also had the effect of releasing the blocked background process, but another background process was now blocking with the same HADR BACKUP LOCK.
I tried to restart SQL Server Agent on AC node, which did not immediately seem to work. However, after a few minutes, I noticed that the block had disappeared. I re-enabled the transaction log backup job on AC and it started working normally again. The error has not occurred again, but I am at a loss as to what happened, and how to prevent it from happening again.
Any help would be greatly appreciated.
Diane