black mark
if you get a black mark, people think that something you have done is bad and they will remember it in future.
This administrative error will be a black mark on his record.
If I'm late for work it'll be another black mark against me. (often + against)