At a high level the max turnover rate is 100% if everyone leaves. It can exceed 100% if you hire new people during the month and they are also gone by the end of the month.
The formula you are using is good for typical (low) turnover rates, not for extreme situations like this. In your example simply taking the headcount at the beginning of the month and end of the month and averaging that won't work. You should do a more granular average - by number of weeks or even days.
Let us say the five employees stayed for 12 days each. Your average employees during the month would be = 5x12/30 = 2
Monthly turnover rate = (5/2) x 100 = 250%