How to Split and Download a Large File with cURL

curl-download-large-file-featured

With a fast Internet connection you will be able to download a large file without any issue. However, there are times when the network (or ISP) that you are connected to restricts your bandwidth or file download size. This is usually the case for educational institutions and places that offer free WiFi. So what can you do about it?

For such restrictive situations where you desperately need to download that large file to your computer, one of the solutions is to use cURL to split the file into smaller parts and combine them together again after all the parts are downloaded.

cURL is a cross-platform command line for getting and sending files using URL syntax. We have a detailed article on cURL usage, so I won’t go into detail on that.

Note: this tutorial is done on Ubuntu, though it will work on any other Linux distro as well as OS (including Windows and Mac OS X).

1. To get started, first make sure that cURL is installed in your system.

You can also download cURL packages and the installer here.

2. As an illustration, I will assume that my network has a 200MB file download limit, and I am going to download the Ubuntu 15.04 ISO file (Download link: http://mirror.pnl.gov/releases/15.04/ubuntu-15.04-desktop-amd64.iso) which is 1.1GB.

The plan is to split the ISO file into 6 parts, each of them 200MB. In the terminal the command is:

The --range flag tell cURL to download only the first 200MB of the file and save it as “ubuntu-iso.part1.”

We will do the same thing for the other parts.

You will notice that the last command doesn’t come with an end range. That means it will download from 1.0GB onward to the end of the file.

Once you are done downloading all the parts, you should now have 6 files in your computer.

curl-download-split-files

The last thing to do is to combine them back to a single file. This can be done with the cat command.

Note: for Windows users you can use the copy command instead of cat.

And a md5 checksum of the combined file shows that it is the same file as the one in the server.

curl-download-check-combine-file-checksum

There you have it: the large file that you wouldn’t be able to get using the usual download method.

6 comments

  1. 1. For those unaware, the “>” in the combining command should be a ‘>’, the greater-than symbol. Looks like Damien didn’t need to use the HTML entity in the code block :-)

    2. In another forum, someone was asking about recombining a file that was split into smaller pieces and since I’ve never done that under OS X/Linux, was wondering if ‘cat’ was the proper command to use. So, thanks for the confirmation, Damien! :-)

    3. An easier way, especially for Windows users, is to use a download manager. I preferred “DownThemAll” ( https://addons.mozilla.org/en-US/firefox/addon/downthemall/?src=ss ) for Firefox back in my Windows days, but there are others. Set the manager to split the download into as many sections (“segments”) as possible. It will then download and automatically recombine all the sections into the original file.

  2. is this command line entres or is it a windows app? having issues downloading win 10 . wondering if this might solve issues where it gets to say 79 % then quits.

Comments are closed.

Sponsored Stories