The ants get stuck in the path leading up to the surface somehow, this happens when I had already placed markers outside the nest. I alternate between sending them underground and then to the surface, this seems to fix it.