Catalina has separate controls to limit access to Desktop, Documents and Downloads.
* go to System Preferences
* open "Security & Privacy"
* click on the "Privacy" tab
* in the list on the left select "Files and Folders"
* in list on the right look for "Terminal" (or iTerm if that is what you are using)
* enable the checkbox next to "Downloads Folder"