It is safe to clear `~/.cache/`, new user accounts start with an empty directory anyway. You might want to log out after doing this though since programs might still use this directory. These programs can be found with this command:
find ~/.cache -print0 | xargs -0 lsof -n
In my case I would most likely be fine with just closing Firefox before removal.