Invoke a shell explicitly.
flock -x -w 5 ~/counter.txt sh -c 'COUNTER=$(cat counter.txt); echo $((COUNTER + 1)) > ~/counter.txt'
Note that any variable that you change is local to that shell instance. For example, the `COUNTER` variable will not be updated in the calling script: you'll have to read it back from the file (but it may have changed in the meantime), or as the output of the command:
new_counter=$(flock -x -w 5 ~/counter.txt sh -c 'COUNTER=$(cat counter.txt); echo $((COUNTER + 1)) | tee ~/counter.txt')