> I suspect the outermost pair of quotes is not part of the filename
I suspect the filename is just a single tab character. I think you gave us the output of `ls`. Compare _Why is`ls` suddenly wrapping items with spaces in single quotes?_ In your case `ls` goes even further and uses ANSI-C quoting (`$'…'`) to denote the tab character as `\t`.
If you type exactly `rm ''$'\t'` (in Bash or another shell that supports `$'…'`) and hit `Enter` then you will remove the file. The two leading single-quote characters enclose an empty string and they can be omitted. I used the literal string you got from `ls`, but it could as well be `$'\t'`.
Alternatively (if your shell does not support `$'…'`) `rm '``Ctrl`+`v`,`Tab``'` should allow you to put a tab character inside single-quotes. This command should also work.