2 Availability Groups and Listeners on one server/instance.

I am looking at setting up the following

Server 1

Database(s) existing in an availability group that will be Primarily on Server 1 but have a secondary non readable replica on Server 2

Server 2

Database(s) existing in an availability group that will be Primarily on Server 2 but have a secondary non readable replica on Server 1

So the plan is to run normally in an "active/active" configuration with the 2 Availability Groups on 2 Separate Servers but with the possibility of both Availability Groups running on the same server for a limited time.

I would have a listener for each Availability Group.

Are there any issues or "gotchas" I would need to be aware of with this configuration?

DB Version will be SQL Server 2012 Enterprise Edition.