Lord of the Wilds potentially bricking Elemental Conflux

I got a situation here were swaping from Lord of the Wilds bricked my Elemental Conflux gem individually.
Meaning, it only broke one gem, making it show a error message like "you can't use this gem with this weapon configuration".

Changing to a new of the same gem fix it, but it seems that the first gem is now permanently bricked.
Being the obvious problem the fact that I lost a gem and I got close to not being able to replace it. I was lucky I had a second one in stash.