$f(x)=O(x)$ as e.g. $x\to \infty$ means that there exists a constant $M>0$ such that $|f(x)|\leq Mx $ for sufficiently large $x$.
Lets analyze your example. First note that the powerseries of cotangent is $\cot x= \frac{1}{\tan x}= \frac{1}{x}-\frac{1}{3}x -\dots$ where the rest of the terms involve powers $x^m, m\geq 3$. Therefore, for $x$ close to zero these terms are very small comparing to $x$, so you can write $\cot x= \frac{1}{x}- \frac{1}{3}x + O(x^3)$. This means that there exists a constant $M>0$ such that $$|\cot x-(\frac{1}{x}- \frac{1}{3}x)| \leq M|x|^3$$ for sufficiently small $x$.
In your case $f(n)=\frac{1}{4n\tan(\pi/n)}=\frac{1}{4n}\cot(\pi/n)= \frac{1}{4\pi}- \frac{\pi}{12n^2}+O(1/n^4)$ where we have an extra power in the $O$ coming from the term $1/4n$.