Hello. I can't add a note to newly created cluster. It timeout while waiting for a response from the node being added. I've gone through a lot of posts regarding this topic but none has resolved my issue.
Failover cluster is currently configured on Guest A and I'm trying to add Guest B to the cluster.
Host: Win2008R2 Hyper-V Core, Firewall disabled, SEP AV
Guests: Win2008 R2 Ent, Firewall disabled, no AV
User account: Domain and local admin
I've tried the steps listed here http://sqlserverscribbles.com/2013/06/17/windows-cluster-freezes-at-waiting-for-notification-that-node-is-a-fully-functional-member-of-the-cluster/
Only exception is I haven't disabled SEP on the Host Hyper-V but based on what I've read it shouldn't have any impact because the virtual network is supposed to act like it's on a switch.
----------------------------------------------------------
6:17:37.565 INFO [DCM] Processing message dcm/connectivityCheck6:17:37.565 INFO [DCM] ConnectivityCheck optimize = false
6:17:37.565 INFO [DCM] NET 5ca4d5ed-a7f4-412c-a1f3-ac28a04266e7 metric 10000
6:17:37.565 INFO [DCM] node 1 interface <obj sig='ITFC' id='20472c3b-c21a-4e07-83a6-ffd6fd20dd2d' name='SGCTRSQ004 - Local Area Connection'/>node Id:'1' adapterName:'Microsoft Virtual Machine Bus Network Adapter' addrPrefixSet:'<set size='1'>
6:17:37.565 INFO <item>192.168.1.146/24</item>
6:17:37.565 INFO </set>
6:17:37.565 INFO '
6:17:37.565 INFO [IM] State calculation complete without issuing pings
6:17:37.565 INFO [IM] Resetting interface state calculation state
6:17:37.581 DBG [NETFTAPI] received NsiParameterNotification for ef80::59g3:29d1:ff68:6996 (IpDadStatePreferred )
6:17:37.581 DBG [NETFTAPI] received NsiParameterNotification for 169.254.1.211 (IpDadStatePreferred )
6:17:37.581 INFO [NM] Received request from client address ::1.
6:17:37.581 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 192.168.1.146:003853 remote address 192.168.1.145:003853
6:17:37.581 INFO [FTI][Initiator] Got remote route reachable from netft evm. Setting state to Up for route from 192.168.1.146:~3343~ to 192.168.1.145:~3343~.
6:17:37.581 INFO [IM] got event: Remote endpoint 192.168.1.145:~3343~ reachable from 192.168.1.146:~3343~
6:17:37.581 INFO [IM] Route from 192.168.1.146:~3343~ to 192.168.1.145:~3343~ is already up, not sending report
6:17:37.581 INFO [TM] got event: Remote endpoint 192.168.1.145:~3343~ reachable from 192.168.1.146:~3343~
6:17:37.581 INFO [FTW] NetFT is ready after 0 msecs wait.
6:17:37.581 INFO [FTI][Initiator] Route is up and NetFT is ready. Connecting to node sgctrsq003 on virtual IP fe80::5c1a:c21e:713c:607c%19:~3343~
6:17:38.708 DBG [NETFTAPI] received NsiParameterNotification for ef80::59g3:29d1:ff68:6996 (IpDadStatePreferred )
6:17:40.712 DBG [NETFTAPI] received NsiParameterNotification for 169.254.1.211 (IpDadStatePreferred )
6:17:58.622 INFO [NODE] Node 1: New join with n2: stage: 'Establish Connection over NetFT Adapter' status WSAETIMEDOUT(10060) reason: 'Failed to connect to remote endpoint fe80::5c1a:c21e:713c:607c%19:~3343~'
6:17:58.622 INFO [CORE] Node 1: Clearing cookie 02f18184-4755-459f-a275-3bd47b804bab
6:18:38.606 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.1.145:~51442~.
6:18:38.606 INFO [SV] Securing route from (192.168.1.146:~3343~) to remote (192.168.1.145:~51442~).
6:18:38.606 INFO [SV] Got a new incoming stream from 192.168.1.145:~51442~
6:18:38.622 INFO [SV] Authentication and authorization were successful
6:18:38.622 INFO [SV] Security Handshake successful while obtaining SecurityContext for NetFT driver
6:18:38.622 INFO [VER] Got new TCP connection. Exchanging version data.
6:18:38.622 INFO [VER] Checking version compatibility for node sgctrsq003 id 2 with following versions: highest [Major 6 Minor 7601 Upgrade 7 ClusterVersion 0x00061DB1], lowest [Major 6 Minor 7601 Upgrade 7 ClusterVersion 0x00061DB1].
6:18:38.622 INFO [VER] Version check passed: node and cluster highest supported versions match.
6:18:38.622 INFO [SV] Negotiating message security level.
6:18:38.622 INFO [SV] Already protecting connection with message security level 'Sign'.
6:18:38.622 INFO [FTI] Got new raw TCP/IP connection.
6:18:38.622 INFO [FTI][Initiator] This node (1) is initiator
6:18:38.622 INFO [FTI][Initiator] Trying to select best endpoints among 169.254.1.211:~3343~, ef80::59g3:29d1:ff68:6996:~3343~ (first pair) and 169.254.2.217:~3343~, fe80::5c1a:c21e:713c:607c:~3343~ (second pair)
6:18:38.622 INFO [StreamDb] Route virtual ef80::59g3:29d1:ff68:6996:~0~ to virtual fe80::5c1a:c21e:713c:607c:~0~ added
6:18:38.622 INFO [NETFT] Route <struct mscs::FaultTolerantRoute>
6:18:38.622 INFO <realLocal>192.168.1.146:~3343~</realLocal>
6:18:38.622 DBG [NETFTAPI] Signaled NetftRemoteUnreachable event, local address 192.168.1.146:003853 remote address 192.168.1.145:003853
6:18:38.622 INFO <realRemote>192.168.1.145:~3343~</realRemote>
6:18:38.622 INFO <virtualLocal>ef80::59g3:29d1:ff68:6996:~0~</virtualLocal>
6:18:38.622 INFO <virtualRemote>fe80::5c1a:c21e:713c:607c:~0~</virtualRemote>
6:18:38.622 INFO <Delay>1000</Delay>
6:18:38.622 INFO <Threshold>5</Threshold>
6:18:38.622 INFO <Priority>99</Priority>
6:18:38.622 INFO <Attributes>2147483649</Attributes>
6:18:38.622 INFO </struct mscs::FaultTolerantRoute>
6:18:38.622 INFO removed
6:18:38.622 INFO [IM] got event: Remote endpoint 192.168.1.145:~3343~ unreachable from 192.168.1.146:~3343~
6:18:38.622 INFO [IM] Marking Route from 192.168.1.146:~3343~ to 192.168.1.145:~3343~ as down
6:18:38.622 INFO [CORE] Node 1: Clearing cookie 02f18184-4755-459f-a275-3bd47b804bab
6:18:38.622 INFO [NDP] Checking to see if all routes for route (virtual) local ef80::59g3:29d1:ff68:6996:~0~ to remote fe80::5c1a:c21e:713c:607c:~0~ are down
6:18:38.622 INFO [NDP] All routes for route (virtual) local ef80::59g3:29d1:ff68:6996:~0~ to remote fe80::5c1a:c21e:713c:607c:~0~ are down
6:18:38.622 WARN mscs::ListenerWorker::operator (): HrError(0xd0000043)' because of '::NetftAddRoute( handle.handle, netFtRoute.get(), &netftSecurityContext )'
6:18:38.622 INFO [IM] Adding information for route Route from local 192.168.1.146:~0~ to remote 192.168.1.145:~0~, status: false, attributes: 0
6:18:38.622 ERR [IM] Unable to find adapter
6:18:38.622 INFO [IM] Sending connectivity report to leader (node 1): <class mscs::InterfaceReport>
6:18:38.622 INFO <fromInterface>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</fromInterface>
6:18:38.622 INFO <upInterfaces><vector len='1'>
6:18:38.622 INFO <item>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</item>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </upInterfaces>
6:18:38.622 INFO <downInterfaces><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </downInterfaces>
6:18:38.622 INFO <viewId>001</viewId>
6:18:38.622 INFO </class mscs::InterfaceReport>
6:18:38.622 INFO [IM] Leader got report <class mscs::InterfaceReport>
6:18:38.622 INFO <fromInterface>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</fromInterface>
6:18:38.622 INFO <upInterfaces><vector len='1'>
6:18:38.622 INFO <item>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</item>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </upInterfaces>
6:18:38.622 INFO <downInterfaces><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </downInterfaces>
6:18:38.622 INFO <viewId>001</viewId>
6:18:38.622 INFO </class mscs::InterfaceReport>
6:18:38.622 INFO from 1
6:18:38.622 INFO [IM] 1 reports in state calculator queue
6:18:38.622 INFO [IM] State calculator got new report from 20472c3b-c21a-4e07-83a6-ffd6fd20dd2d
6:18:38.622 INFO [IM] 0 reports in state calculator queue
6:18:38.622 INFO (allSplitGroups, splitGroups) = [IM] Calculated interface state result: <class mscs::InterfaceResult>
6:18:38.622 INFO <up><vector len='1'>
6:18:38.622 INFO <item>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</item>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </up>
6:18:38.622 INFO <down><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </down>
6:18:38.622 INFO <unreachable><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </unreachable>
6:18:38.622 INFO </class mscs::InterfaceResult>
6:18:38.622 INFO [IM] Issuing state change gum with result <class mscs::InterfaceResult>
6:18:38.622 INFO <up><vector len='1'>
6:18:38.622 INFO <item>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</item>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </up>
6:18:38.622 INFO <down><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </down>
6:18:38.622 INFO <unreachable><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </unreachable>
6:18:38.622 INFO </class mscs::InterfaceResult>
6:18:38.622 INFO [IM] Changing the state of adapters according to result: <class mscs::InterfaceResult>
6:18:38.622 INFO <up><vector len='1'>
6:18:38.622 INFO <item>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</item>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </up>
6:18:38.622 INFO <down><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </down>
6:18:38.622 INFO <unreachable><vector len='0'>
6:18:38.622 INFO </vector>
6:18:38.622 INFO </unreachable>
6:18:38.622 INFO </class mscs::InterfaceResult>
6:18:38.622 INFO [DCM] Processing message dcm/connectivityCheck
6:18:38.622 INFO [DCM] ConnectivityCheck optimize = false
6:18:38.622 INFO [DCM] NET 5ca4d5ed-a7f4-412c-a1f3-ac28a04266e7 metric 10000
6:18:38.622 INFO [DCM] node 1 interface <obj sig='ITFC' id='20472c3b-c21a-4e07-83a6-ffd6fd20dd2d' name='SGCTRSQ004 - Local Area Connection'/>node Id:'1' adapterName:'Microsoft Virtual Machine Bus Network Adapter' addrPrefixSet:'<set size='1'>
6:18:38.622 INFO <item>192.168.1.146/24</item>
6:18:38.622 INFO </set>
6:18:38.622 INFO '
6:18:38.622 INFO [IM] State calculation complete without issuing pings
6:18:38.622 INFO [IM] Resetting interface state calculation state
6:18:38.637 DBG [NETFTAPI] received NsiParameterNotification for ef80::59g3:29d1:ff68:6996 (IpDadStateDeprecated )
6:18:38.637 INFO [NM] Received request from client address ::1.
6:18:38.637 DBG [NETFTAPI] received NsiParameterNotification for 169.254.1.211 (IpDadStateDeprecated )
6:18:38.637 DBG [NETFTAPI] Signaled NetftLocalDisconnect event for ef80::59g3:29d1:ff68:6996
6:18:38.637 INFO [IM] got event: Local endpoint ef80::59g3:29d1:ff68:6996:~0~ disconnected
6:18:38.669 DBG [NETFTAPI] Signaled NetftLocalDisconnect event for 169.254.1.211
6:18:38.669 INFO [IM] got event: Local endpoint 169.254.1.211:~0~ disconnected
6:18:38.669 DBG [NETFTAPI] received NsiDeleteInstance for fe80::5efe:169.254.1.211
6:18:38.669 WARN [NETFTAPI] Failed to query parameters for fe80::5efe:169.254.1.211 (status 80070490)
6:18:38.669 DBG [NETFTAPI] Signaled NetftLocalAdd event for fe80::5efe:169.254.1.211
6:18:38.684 WARN [NETFTAPI] Failed to query parameters for fe80::5efe:169.254.1.211 (status 80070490)
6:18:38.684 DBG [NETFTAPI] Signaled NetftLocalRemove event for fe80::5efe:169.254.1.211
6:19:33.846 INFO [QUORUM] Node 1: after eviction cluster will still have quorum <QuorumConfig tag='0:0:1' set='(1 2)' weights='(1)'/>
6:19:33.846 INFO [QM] EvictNode Gum Handler sgctrsq003
6:19:33.846 INFO [QUORUM] EvictCount for node 2 is incremented to 3
6:19:34.268 ERR [RHS] RhsCall::Perform_NativeEH: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
6:19:34.300 ERR [RHS] RhsCall::Perform_NativeEH: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
6:19:39.684 INFO [ACCEPT] 0.0.0.0:~3343~: Accepted inbound connection from remote endpoint 192.168.1.145:~51464~.
6:19:39.684 INFO [SV] Securing route from (192.168.1.146:~3343~) to remote (192.168.1.145:~51464~).
6:19:39.684 INFO [SV] Got a new incoming stream from 192.168.1.145:~51464~
6:19:39.700 INFO [SV] Authentication and authorization were successful
6:19:39.700 INFO [SV] Security Handshake successful while obtaining SecurityContext for NetFT driver
6:19:39.700 INFO [VER] Got new TCP connection. Exchanging version data.
6:19:39.700 INFO [VER] Checking version compatibility for node sgctrsq003 id 2 with following versions: highest [Major 6 Minor 7601 Upgrade 7 ClusterVersion 0x00061DB1], lowest [Major 6 Minor 7601 Upgrade 7 ClusterVersion 0x00061DB1].
6:19:39.700 INFO [VER] Version check passed: node and cluster highest supported versions match.
6:19:39.700 INFO [SV] Negotiating message security level.
6:19:39.700 INFO [SV] Already protecting connection with message security level 'Sign'.
6:19:39.700 INFO [FTI] Got new raw TCP/IP connection.
6:19:39.700 INFO [FTI][Initiator] This node (1) is initiator
6:19:39.700 INFO [FTI][Initiator] Trying to select best endpoints among 169.254.1.211:~3343~, ef80::59g3:29d1:ff68:6996:~3343~ (first pair) and 169.254.2.217:~3343~, fe80::5c1a:c21e:713c:607c:~3343~ (second pair)
6:19:39.700 INFO [StreamDb] Route virtual ef80::59g3:29d1:ff68:6996:~0~ to virtual fe80::5c1a:c21e:713c:607c:~0~ added
6:19:39.700 INFO [NETFT] Added route <struct mscs::FaultTolerantRoute>
6:19:39.700 INFO <realLocal>192.168.1.146:~3343~</realLocal>
6:19:39.700 INFO <realRemote>192.168.1.145:~3343~</realRemote>
6:19:39.700 INFO <virtualLocal>ef80::59g3:29d1:ff68:6996:~3343~</virtualLocal>
6:19:39.700 INFO <virtualRemote>fe80::5c1a:c21e:713c:607c:~3343~</virtualRemote>
6:19:39.700 INFO <Delay>1000</Delay>
6:19:39.700 INFO <Threshold>5</Threshold>
6:19:39.700 INFO <Priority>99</Priority>
6:19:39.700 INFO <Attributes>1</Attributes>
6:19:39.700 INFO </struct mscs::FaultTolerantRoute>
6:19:39.700 INFO [FTI][Initiator] waiting for route to node sgctrsq003 on virtual IP fe80::5c1a:c21e:713c:607c:~3343~ to come up
6:19:39.715 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 192.168.1.146:003853 remote address 192.168.1.145:003853
6:19:39.715 INFO [FTI][Initiator] Got remote route reachable from netft evm. Setting state to Up for route from 192.168.1.146:~3343~ to 192.168.1.145:~3343~.
6:19:39.715 INFO [IM] got event: Remote endpoint 192.168.1.145:~3343~ reachable from 192.168.1.146:~3343~
6:19:39.715 INFO [IM] Marking Route from 192.168.1.146:~3343~ to 192.168.1.145:~3343~ as up
6:19:39.715 INFO [TM] got event: Remote endpoint 192.168.1.145:~3343~ reachable from 192.168.1.146:~3343~
6:19:39.715 INFO [IM] Adding information for route Route from local 192.168.1.146:~0~ to remote 192.168.1.145:~0~, status: true, attributes: 0
6:19:39.715 INFO [FTI][Initiator] Got remote route reachable from netft evm. Setting state to Up for route from 192.168.1.146:~3343~ to 192.168.1.145:~3343~.
6:19:39.715 ERR [IM] Unable to find adapter
6:19:39.715 INFO [IM] Sending connectivity report to leader (node 1): <class mscs::InterfaceReport>
6:19:39.715 INFO <fromInterface>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</fromInterface>
6:19:39.715 INFO <upInterfaces><vector len='1'>
6:19:39.715 INFO <item>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</item>
6:19:39.715 INFO </vector>
6:19:39.715 INFO </upInterfaces>
6:19:39.715 INFO <downInterfaces><vector len='0'>
6:19:39.715 INFO </vector>
6:19:39.715 INFO </downInterfaces>
6:19:39.715 INFO <viewId>001</viewId>
6:19:39.715 INFO </class mscs::InterfaceReport>
6:19:39.715 INFO [IM] Leader got report <class mscs::InterfaceReport>
6:19:39.715 INFO <fromInterface>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</fromInterface>
6:19:39.715 INFO <upInterfaces><vector len='1'>
6:19:39.715 INFO <item>20472c3b-c21a-4e07-83a6-ffd6fd20dd2d</item>
6:19:39.715 INFO </vector>
6:19:39.715 INFO </upInterfaces>
6:19:39.715 INFO <downInterfaces><vector len='0'>
6:19:39.715 INFO </vector>
6:19:39.715 INFO </downInterfaces>
6:19:39.715 INFO <viewId>001</viewId>
6:19:39.715 INFO </class mscs::InterfaceReport>