Time of need bugged with weapon swap

The time of need spirit gem is not working if you have a weapon swap with it active on both weapon slots. Every time you weapon swap it seems to reset its cooldown and increase its cooldown massively. This means if you weapon swap it never activates. I have tested this on a few different classes and weapon types and its does this on all of the ones I tried