Unless they changed it, the last time I tried to go from Durmond Priory or Vigil's Keep to Fort Trinity with my Order of Whispers ele, the guards wouldn't let me through to the gate (I've finished the story with her too...).
I once tried it on my Orders of Whispers necromancer. Could of course use the gate at Fort Trinity to get to any of the other order's keeps. Could also get out of the gate area in all those keeps. However could only get back into the gate areas in the Whispers and Vigil keeps. Priory would not let me through. This must be a bug because at that point in the story I was supposed to be a "commander" of all the orders.