Is there a way to use a Hill cypher where the ciphering matrix is not square? I've studied Hill's theory where, in order to cypher a vector $p$ of dimension $n\times 1$, we use a squared matrix $A$. We would multiply $Ap$ and get the $n\times 1$ vector $c$, we would then make a modulo $m$ operation, where $m$ is the number of characters we're using to encipher, so this vector $c$ has now values $\in \\{1,2,...,m\\}^n$. In order to get the original vector we would do the inverse process, multiply ${A^{-1}}c$ (we would have to make sure $A$ has an inverse matrix first), the result modulo $m$ is the original vector $p$. My question is: Is there a Hill cypher where $A$ may not be square? if so, how would it work? if not, is there a proof?

If $A$ is a $p \times q$-matrix then it transforms a $q \times 1$ vector to a $p \times 1$ vector (over $\mathbb{Z}_m$ in your case). This can only be bijective (invertible) iff $p=q$ because we map $\mathbb{Z}_m^q$ to $\mathbb{Z}_m^p$ and these sets have the same size iff $p=q$. So we need a square invertible matrix to get a valid encryption system with unique decryption.