Force Interaction with Chests & others while Minions are close

Please provide controller players with the ability to open chests, pick up loot, use trial altars, deploy ritual vessel in ultimatum even if minions are close.
this is so annoying, is nobody playing this game with controller in your dev team?