Can We Upload Zip Files to Google Drive
How to successfully add together large information sets to Google Drive
And utilise them in Google Colab…
Introduction
In this post, I will explain how to add large data sets to Google Drive so they can be accessed from Google Colab for processing and modeling.
Whereas uploading a single file can exist done with the drag and drop interface of Google Bulldoze, it becomes more than difficult with a large number of files. Dragging the whole folder containing 1GB of files but fails and freezes Google Drive. The culling is to drag a zipped binder. This process is commonly successful and does not take even that long (a couple of minutes with 1GB file) simply the problem comes with unzipping the file in Google Drive itself which results in random files missing.
My trials for three days to successfully upload photo information for CNN training in lodge to utilise complimentary GPU led me to establish this culling. I am going to describe hither step by step how to upload successfully big data sets so they can be processed by Google Colab and take advantage of VMs provided by their services.
Steps
1. Zip the binder with the files. In my case, I had a folder called 'train' with 70257 .jpg files that were taking around 1GB.
ii. Upload the zipped file using Google Bulldoze Interface. I have uploaded mine to an empty directory called information.
3. Open up a new Google Colab file and mount it to Google Bulldoze to exist able to admission the zero file. If y'all practise non take Colab installed yous can follow this article that explains how to practise it.
The control below will start the mounting process.
from google.colab import bulldoze
drive.mount('/content/gdrive') You will exist asked to qualify access to Google Drive.
Follow instructions to give potency by copy-pasting the code and you should exist mounted.
4. Now excerpt files to the local environment with the following command.
!unzip gdrive/My\ Bulldoze/information/railroad train.zip Note that my train.zero file in 'data' folder is located in the Google Drive root directory. You volition need to modify the path accordingly to where your file is located.
Y'all should run into file unzipping.
It takes less than 1 minute to unzip 1GB and then you should non await also long. Once yous are comfy that this command is working you tin use the variation below that suppresses the output.
!unzip gdrive/My\ Drive/data/train.nothing > /dev/null One time the cell has executed y'all tin encounter the files take appeared in the local train folder. You tin can find information technology on the left-hand side of the colab interface.
five. You can use the files for annihilation correct now. Juts admission them from the new 'train' folder. In my case, I tin can display the first image using the following lawmaking.
import tensorflow every bit tf
img = tf.keras.preprocessing.image.load_img('train/abs_000.jpg')
img 
You can now use your data for anything yous wish. In my case, it was training a CNN using free GPU.
Summary
The process that I described above worked best for me! So please free to re-create it.
On the other hand, information technology would be much ameliorate to unzip the files in Google Drive, so they just stay there. This yet was 'mission impossible' for me for thee days in a row. Every time I have unzipped the files and saved them in Google Drive there were missing photos in the end.
I accept tried to practise it programmatically with Colab too as using Zip Extractor continued to Google Bulldoze. Both methods resulted in missing files with no alarm about information technology. I have likewise tried two different Google accounts and the problem persisted.
Afterward losing 3 days of trial and error. I came with the procedure described above and information technology works smoothly for me every fourth dimension I rerun the lawmaking.
I hope it helps others that endeavor to load large information sets to Google Drive in order to process them in Colab. If y'all are i that has constitute a ameliorate method or was able to unzip large files in Google Drive, I would exist curious how this has been done.
Happy coding!
PS: I am writing articles that explicate basic Data Science concepts in a simple and comprehensible manner on Medium and aboutdatablog.com . You can subscribe to my email listing to get notified every time I write a new article. And if you are non a Medium member still you lot tin can join hither .
Below there are some other posts y'all may savour:
Source: https://towardsdatascience.com/how-to-successfully-add-large-data-sets-to-google-drive-130beb320f1a
0 Response to "Can We Upload Zip Files to Google Drive"
Post a Comment