Temple - closing - inconsistent

My Temple sometimes closes prematurely.
i.e. I will teleport out of the temple and most of the time i'm able to re-enter the temple, interact with the console etc..

However, every once in a while, maybe 10-20% of the time, I teleport out of the temple and it closes on me and rooms destabalize before I can put Juatolotlis Medallion on them or finish the temple.

Just lost a tier 3 sacrifical chamber, and several other rooms...

Kind of annoying, please fix.

I play a lot of co-op, so maybe that has something to do with it? I think perhaps it happens more often when I'm playing with a friend.