Limiting Skill to Weapon Swap also affect buffs

I have weapon set II with duration passive tree, normally if I manually weapon swap and use buff skill, then swap back, I still keep the buff and bonuses from weapon set II passive tree. However when I limit my buff skill to weapon swap II, the buff would disappear when swapping to weapon set I.

I think this is a bug and should be fixed, since its behavior is different from manual swapping. It should be a major QoL, so I hope it be fixed ASAP, thanks!