Can't allocate minions with empty Spellslinger

Was in the process of writing a long post about my character getting bricked by not being to allocate minions on it anymore. Turns out it only happens when I have a Dueling Wand equipped with no Skill Gem in Spellslinger - minion gems say "Not enough Spirit to summon desired amount". When I fill it with Skill Gem, it's no longer an issue.