Managed to remove them by placing other plants and shells on them. The remove tool worked on some after the procedure and others were replaced by the shells. Took multiple tries on some. One (The anemone to the right) Needed to be mowed over with rock. I had displaced the anemone earlier. So might be because of that.
Again. When something grows on earlier edited soil it can't be removed. And seemingly not eaten either, can't confirm though because it keeps seeding. So it's not only visual.