Use row reduction for the augmented matrix: \begin{align} \begin{bmatrix}1&3&1&8\\\0&1&1&k\\\2&-2&-6&8\end{bmatrix}\rightsquigarrow\begin{bmatrix}1&3&1&8\\\0&1&1&k\\\0&-8&-8&-8\end{bmatrix}\rightsquigarrow \begin{bmatrix}1&3&1&8\\\0&1&1&k\\\0&0&0&(k-1)8\end{bmatrix}\end{align} Thus $\operatorname{rank}A'=2$ if and only if $k=1$.
Moreover, the solutions are an affine subspace of dimension $1$: if we continue row reduction to obtain the reduced row echelon form, dropping the last (zero) row,we have $$\begin{bmatrix}1&3&1&8\\\0&1&1&1\end{bmatrix}\rightsquigarrow\begin{bmatrix}1&0&-2&5\\\0&1&1&1\end{bmatrix}$$ whence the solutions: \begin{align} \begin{cases}x&=2z-5,\\\ y&=-z-1,\end{cases}\quad\text{or, in vector form:}\quad \begin{bmatrix}x\\\y\\\z\end{bmatrix}=z\begin{bmatrix}2\\\\-1\\\1\end{bmatrix}-\begin{bmatrix}5\\\1\\\0\end{bmatrix}. \end{align}