sit idly by

sit idly by

sit (idly) by (and (do something))

to take no action.

I felt we simply couldn't sit by and see the peace process threatened.

We can't afford to sit idly by while dangerous people threaten our way of life.

Cambridge English Idioms Dictionary