Tuesday, February 12, 2013

Multitask file downloader in Bash in 2 SECONDS

Thanks to +Ohad Lutzky, there is much better solution for this task in simple bash one-liner using xargs. Here is how:
MAXJOBS=50; cat $UrlsFile | xargs -I @ -P $MAXJOBS bash -c 'curl @ > ...'

Clean and simple. 100% Unix :)

P.S. I wrap curl with another bash command because I want separate per-URL output redirection.

No comments:

Post a Comment