Just give the minors a status effect that grants 10 seconds of invulnerability to the refuse chamber damage if they start upgrading a refuse chamber tile. That should do the job.
About idling ants - I think there should be less movement from idle ants in general. Idle ants usually don't move that much IRL and it just puts ton of unnecessary strain on the system when it tries to do pathfinding for a blob of 200 idle ants. You can add some idle animations instead (like ants cleaning their antennae, legs, butts, etc.). That saves a ton of processing power makes it look more realistic and adorable at the same time.