Multiple Instances and AlwaysOn Availability Groups

Hi,

I have 2 SQL Servers: A & B
Each has 2 instances and each instance is part of an database AlwaysOn Availability Group

Server A - instance 1, AlwaysOn Group1 & instance2, AlwaysOn Grroup2
Server B- instance 1, AlwaysOn Group 1 & instance2, AlwaysOn Grroup2

I configured 2 AlwaysOn groups, which worked for a brief amount of time:

Server A - had instance 1 (primary) and instance 2 (secondary)
Server B - had instance 1 (secondary) and instance 2 (primary)

However after a few minutes, Server A instance 1 (secondary) changed to the not synchronized state and I'm struggling to add Server A instance 1 as a replica in AlwaysOnGroup1

Can someone tell me if I'm doing something wrong? Can I have 2 instances participating in separate AlwaysOn Database Availability Groups on the same server?

I'm wondering maybe the configuration is failing because one server holds a primary and the other a secondary - perhaps each server should hold only a primary or secondary?

Thanks