A major reason why you cannot use this argument is that you seem to saying that because you know every finite field must have $p^{m}$ elements you get a contradiction.
However how do you know that you don't need this result in order to prove that finite field must have $p^{m}$ elements.
So does the proof of finite field must have $p^{m}$ elements, require the knowledge of the prime subfield and does this in turn require knowing that $\mathbb{Z}_{n}$ is a field iff $n$ is prime...