Weapon set Skill point - Attribute

One of my Quivers has +Dex which allows me to use my Rare bow (It requires 212 Dex) in Weapon set II; One of my Quivers doesn't have +Dex which allows me to use Widowhail since it doesn't require so much Dex.

Now, i want to use these Weapon set Skill points to change from Dex to Int in Weapon set II on the same "passive skills" for more DPS, but it doesn't work, i can only pick "new" passive skills. We should be able to use those points to change attribute stats on the same "Path".