Draiocht wisp on mansion

I was mapping and moving the wisps closer to the mansion so i could get my last 2 empowered boss skill points and on completing a map and interacting with the wisps went to the mansion itself and locking me out of ever doing the map

this has also happened to me