The formula is Burnside's Lemma. You take a group, in this case the group of rotations of a square: $0, 90, 180, 270$ degree rotations, and look at which colourings are fixed by each rotation. The formula is
> $$ \frac{1}{\text{size of group}}\left( \text{colourings fixed by the first group element} + \text{colourings fixed by the second group element} + \cdots \right) $$
In this case
* $3^9$ colourings are fixed by a $0$ degree rotation
* $3^3$ colourings are fixed by a $90$ or $270$ degree rotation (such colourings are determined by how the top-left, top-centre and centre squares are coloured)
* $3^5$ colourings are fixed by a $180$ degree rotation (determined by the top three squares and the centre-left and centre squares)
Applying Burnside's Lemma, there are
$$ \frac14 \left( 3^9 + 2 \cdot 3^3 + 3^5 \right) = 4995 $$
distinct colourings.