From the context (a change of scale using discrete units), this should certainly mean **floor** as on page 5 of Gauss's _Werke_ 2
> per signum $[x]$ exprimemus integrum ipsa $x$ proxime minorem, ita ut $x-[x]$ semper fiat quantitas positiva intra limites $0$ et $1$ sita
i.e. the next lower integer.