The Fastest Way to Load Any Dataset into Google Colab with CurlWget

Google Colaboratory, or "Colab" for short, has quickly become one of the most popular browser-based Jupyter notebook environments for data science and machine learning. Colab allows you to leverage the power of Google‘s cutting-edge CPUs, GPUs and TPUs entirely in the cloud, without having to set up anything on your local machine. It‘s an amazing free resource, especially if you‘re just getting started in data science or deep learning.

However, one significant drawback of Colab compared to running Jupyter locally is that uploading your datasets can be slow and cumbersome. When you‘re working with real-world datasets that can range from hundreds of megabytes to several gigabytes in size, waiting for them to upload to Colab‘s servers is a major productivity killer.

Luckily, there‘s a solution that allows you to load even multi-gigabyte datasets into Colab in just seconds. The key is a handy browser extension called CurlWget. In this guide, I‘ll walk you through exactly how to use CurlWget to transform your Colab workflow and spend less time waiting on uploads and more time extracting insights from your data.

What is CurlWget?

CurlWget is a lightweight Chrome extension that generates curl or wget command strings for downloading files and datasets. Since Google Colab runs on Linux-based servers under the hood, it can accept these commands to download data from around the web directly to its local storage.

Here‘s how to get started with CurlWget:

1. Install the CurlWget extension from the Chrome Web Store

2. Click the Extensions icon in the top right of your browser and pin CurlWget so it‘s easily accessible

3. That‘s it – you‘re ready to use CurlWget to load data into Colab!

Loading Datasets into Colab with CurlWget

Now let‘s walk through an example of using CurlWget to load a dataset into Colab. We‘ll use the popular Microsoft Malware Prediction dataset from Kaggle, which is about 1.5 GB in size.

Normally, you would have to download this dataset locally, then upload it to Colab. With CurlWget, here‘s what you do instead:

1. Go to the dataset‘s Kaggle page and click "Download" like you normally would. The download will start but don‘t wait for it to finish.

2. Cancel the download. Yes, you read that right – just cancel it!

3. Click the CurlWget extension in your browser. It will display the wget command for the file you were just downloading.

4. Click inside the CurlWget box to select the full wget command and copy it to your clipboard.

5. In Colab, create a new code cell and paste in the wget command. Be sure to put an exclamation point before it to run it as a shell command, like this:

!wget -O malware.zip https://kaggle.com/c/9863/download/malware.zip

6. Run the cell and watch in amazement as the entire 1.5 GB dataset downloads into Colab in a matter of seconds!

Using this technique, you can load almost any dataset of any size into Colab as long as it‘s accessible via a public URL. I‘ve personally used CurlWget to import datasets as large as 10 GB with no issues. The download speed will depend on Colab‘s servers but I typically see speeds of 50-100 MB/s. Compared to uploading from your local machine, it‘s night and day.

Handling Different File Formats

The Microsoft Malware dataset comes as a ZIP file, but CurlWget works just the same with any file format. Here‘s how to access your data in Colab once it‘s downloaded:

• For CSV files, you can load them into a Pandas DataFrame directly like this:

import pandas as pd
df = pd.read_csv(‘mydata.csv‘) 

• For ZIP files, unzip them first with this command:

!unzip mydata.zip

• For TAR files, extract them like this:

!tar -xvf mydata.tar

• For RAR files, first install the unrar utility and then extract:

!apt install unrar
!unrar x mydata.rar

Once you‘ve unpacked your data, you can access it from the file pane in the left sidebar of your Colab notebook. You can also get the full file path by clicking the three dot icon next to any file and selecting "Copy path". This is handy for loading data using libraries like PyTorch and TensorFlow.

Saving and Reusing Datasets

One downside of Colab is that its virtual machines and their local storage reset after you end your session. That means any data you download will be erased and you‘ll have to download it again next time.

To avoid this, I recommend saving your datasets (especially if you‘ve preprocessed them) to Google Drive so you can easily reuse them. Here‘s how:

1. In Colab, click the folder icon in the left sidebar and select "Mount Drive". Follow the authorization flow to connect your Google Drive.

2. Use shell commands to copy files from Colab‘s local storage to your Google Drive, like this:

!cp mydata.csv "drive/My Drive/project/mydata.csv"

3. Next time you need the data, mount your Google Drive and access it directly without needing to download it again:

from google.colab import drive
drive.mount(‘/content/drive‘)

df = pd.read_csv(‘drive/My Drive/project/mydata.csv‘)

Accessing Shared Datasets

If a colleague shares a dataset with you via Google Drive, you can access it in Colab as well. Just make sure they grant you access first.

By default, Drive files shared with you don‘t show up in your My Drive folder, which is what gets mounted in Colab. To access a shared file, navigate to it in Drive and select "Add shortcut to Drive" from the right click menu. Choose a location under My Drive and the dataset will become accessible from Colab.

Tips for Using CurlWget Effectively

To wrap things up, here are a few tips I‘ve learned to make the most of CurlWget for data science in Colab:

• If a download keeps timing out or failing, try adding the -c flag to the wget command to resume where it left off instead of starting over

• Use the -q flag to suppress wget‘s output and keep your Colab notebook tidy, like this:

!wget -q -O myfile.zip https://example.com/myfile.zip

• If a site is blocking wget, try changing your user agent by adding this flag:

!wget -U "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" https://example.com/myfile.zip

• Bookmark or save your wget/curl commands so you can easily rerun them without having to find the download URL again

I hope this guide has helped you see the power and convenience of using CurlWget to load datasets into Google Colab. It‘s truly a gamechanger compared to uploading data manually and has saved me countless hours. Give it a try yourself and see how much it improves your data science workflow in Colab.

Let me know in the comments if you have any other tips or feedback. Happy data wrangling!

Note: CurlWget commands shown in this article were current as of September 2022. While the core CurlWget functionality is unlikely to change, be sure to check the extension‘s documentation for the latest usage details.