grep searches file contents, not the files' names.
The shell is typically what you use to select the set of files to work on. To name all the files in the current directory with a single quote, you'd use the pattern `*\'*`. (A backslash is used to escape characters that otherwise would have meaning to the shell.)
To list them:
ls *\'*
To delete them:
rm *\'*