The probability of scoring a penalty is twice the probability of missing it. Total 5 penalties are taken, then the probability of scoring 3 out of 5? What I Did: 3 scoring penalties among 5 can be chosen in `5c3` ways ! After this I'm confused on how to proceed !!

$P(\text{scoring})=2P(\text{missing})$ and $P(\text{scoring})+P(\text{missing})=1$, therefore $P(\text{scoring})=2/3$ and $P(\text{missing})=1/3$.

It's Binomial Probability. Your answer is $\binom{5}{3}(2/3)^3(1/3)^2$, or in your notation $(5c3)(2/3)^3(1/3)^2$.