You need to do it in this way . Let $x$ be the number of tries required to get the first $5$ then $p_X(x) = (\frac{5}{6})^{x-1}\frac{1}{6}$ After this you need $3$ more $5$ . You can treat this as a completely new event , because it doesn't depend on what happened earlier .
Now let $y$ be the total number of trials needed to get four $5$ , then you have already wasted $x$ tries . You are left with only $y-x$ tries now . Now the ways of getting two $5$s will be ${y-x-1}\choose{2}$ . Therefore the probability of this event will be ${y-x-1}\choose{2}$$(\frac{1}{6})^2 (\frac{5}{6})^{y-x-3}(\frac{1}{6})$
Therefore the joint PMF $p_{X,Y}(x,y)$ = $(\frac{5}{6})^{x-1}\frac{1}{6}$${y-x-1}\choose{2}$$(\frac{1}{6})^2 (\frac{5}{6})^{y-x-3}(\frac{1}{6}) $