An Introduction to cURL

curl is a cross-platform command line for getting and sending files using URL syntax. It is a recursive acronym that stands for Curl URL Request Library, and it is a very extensive program, supporting a wide range of internet protocols including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, LDAP, LDAPS, DICT, TELNET, FILE, IMAP, POP3, SMTP and RTSP.

curl has a ton of features and tricks, including user authentication, proxy support, FTP upload and download, HTTP post, cookies, file transfer resume, SSL connections and much more. This article will serve as an introduction to curl, for those unfamiliar with the program, while power users can also benefit.

To install curl on a Debian/Ubuntu system, run the following command:

sudo apt-get install curl

Install curl on a Debian/Ubuntu system.

At its most basic, curl expects a url argument, and it will attempt to fetch whatever file is available at the given url.

curl http://www.maketecheasier.com

By default, the fetched file contents are dumped to the command line. If an output file is specified, the program will show a progress meter which shows the amount of data transferred, transfer speeds, estimated time left, and time spent, among others. To save the fetched file to your system, with a given filename, use the -o option:

curl -o mte-index.html http://www.maketecheasier.com

To save the fetched file to your system, with a given filename, use the -o option.

To save a file with the same name as it has on the server, use the -O option:

curl -O ftp://ftp.kernel.org/pub/linux/kernel/v4.x/linux-4.0.tar.xz

To save a file with the same name as it has on the server, use the -O option.

To get a file from an ftp server that requires username and password:

curl -O ftp://username:password@ftp.kernel.org/pub/linux/kernel/v4.x/linux-4.0.tar.xz

To specify multiple URLs or parts of a URL, put the part sets in braces:

curl -O  http://www.maketecheasier.com/author/{obaro,ivana,vamsi}

You can also specify a range in an alphanumeric series by using square brackets:

curl -O ftp://ftp.numericals.com/file[1-100].txt
curl -O ftp://ftp.letters.com/file[a-z].txt
curl -O  http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html

You can also specify a range in an alphanumeric series by using square brackets.

To save the fetched files into multiple documents, use the ‘#’ symbol followed by a number in the filename specifier. This will then get replaced by the current string in the given URL

curl -o "file_#1.txt" http://{one,two}.site.com
curl -o "output_#1_#2" http://{site,host}.host[1-5].com

You can specify the User-Agent string to be used while identifying to http servers, for http connections use the -A flag:

curl -A "Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0"-o mte-index.html http://www.maketecheasier.com

To send data using HTTP POST, use the -d option:

curl -d "username=obaro" -d "password=password" http://maketecheasier.com

If you are, for example, logging in to a website, you most likely want to store the supplied cookies for later use. To store the received cookies, use the -c (or --cookie-jar) option:

curl -d "username=obaro" -d "password=password" -c mte-cookies http://maketecheasier.com

To use these stored cookies, use the -b option:

curl -b mte-cookies -d "hc_comment=This is a comment&submit=true" http://www.maketecheasier.com/monitor-hard-disk-health-linux/

To use this stored cookies, use the -b option.

To upload a file, use the -T option. For http(s) servers, the PUT command will be used instead:

curl -T "file1.jpg" http://www.uploadmania.com/upload

To force the use SSL/TLS for connections, use the --ssl-reqd flag. You can also specify multiple files in the same manner you specify multiple URLs:

curl --ssl-reqd -T "file[1-100].jpg" ftp://username:password@ftp.uploadmania.com/upload

In addition, you can specify the login credentials with the -u option:

curl -u username:password --ssl-reqd -T "file[1-100].jpg" ftp://ftp.uploadmania.com/upload

curl is a very simple, solid and yet feature-full program. There are so many different options available, and it handles a very large number of protocols, yet we have just barely scratched the surface of its abilities. This is an introduction to curl, for users who haven’t used it before, and a reminder for those who are familiar with it’s possibilities. Don’t forget to check out the manpages or type the following for more info:

man curl

If you have a particular use for this, please share with us in the comments.