gnu parallel - speed up command agains remote servers that waits I'm using gnu parallel to generate backups for about ~1500 web sites on pantheon.io, using their terminus CLI. The `terminus backup:create` command does not finish until a response is received that it has completed on the remote end. I'm wondering if there is any way to better speed this up with parallel so that more sites can be backing up while waiting on previous ones to complete or just run more overall if not. If it makes any difference, this is being run from a Jenkins CI job. Thank you. #!/bin/bash +x backup_sites() { BACKUP=$(terminus backup:create "$*".live) echo "$*": "$BACKUP" } SITE_LIST=$(terminus site:list --field=name) export -f backup_sites echo "$SITE_LIST" | parallel backup_sites

Not specifying how many jobs to run in parallel will make it default to number of cpus.

From the manual:

> -j
>
> Number of jobslots on each machine. Run up to N jobs in parallel. 0 means as many as possible. Default is 100% which will run one job per CPU on each machine.

In general this is a safe bet, but you are waiting on network, not computation. So you can easily boost up the number. I would try -j 200. It should work quite well. You can tweak this parameter to get the speed you need.

So `echo "$SITE_LIST" | parallel -j 200 backup_sites` instead of `echo "$SITE_LIST" | parallel backup_sites`