Does restarting the FSW has impact on the SQL AAG?

I guys,

There is some maintenance comming up, and i have to restart a server holding the FSW for our AAG (2012) cluster. Does dis has direct impact to the availability of the AAG? I can imagine that at the time the FSW is gone, i cannot  loose a node because then the cluster will stop working due to lack of majority, but other than that?

Thanks!