Bug/Issue Dark Shrine of Selfishness not working

So far I have used 2 prismatic shards on the shrine to try to get rid of my kids after my character got divorced but for some reason they remain in my house. I am not sure why this is happening and I don't particularly want to try again because after two times of it not working and with the difficulty of getting the prismatic shard, its not really something desirable to do.