According to the OP's comments, later prisoners can observe the fate of earlier prisoners. But now it's easy to ensure that at least eight prisoners survive:
$A,B,C,$ and $D$ assume that the sum of all the hats modulo $5$ is equal to $0,1,2,$ and $3$ respectively, and base their answer on this assumption. As soon as one of them goes free, everybody else knows this sum, so they can answer correctly.
If none of them go free, the sum must be $4$. And of course if $A,B,$ or $C$ goes free, then $B,C,$ and/or $D$ are freed from their obligations, and will go free too. So the number of survivors can be anything from $8$ to $12$.