Python built-in zip library is commonly used to create archive. However, there is a concern when creating zip files using built-in library. Consider the case which we are zipping files larger than our available memory, we would easily run out of memory.
I was building a feature that requires zipping of files and upload to our Django backend storage. After digging around the internet, I summarize the logic I used to support this feature.
Use NamedTemporaryFile Instead Of Memory
A NamedTemporaryFile resides in secondary memory instead of main memory, thus using it does not consume extra memory.
1. Create a NamedTempFile
2. Create a ZipFile with NamedTempFile as file output
3. Write files into ZipFile
4. Move the cursor of the NamedTempFile back to the beginning
5. Wrap it with Django File
6. Inject a file name
7. Upload it to storage
Assume we have a model as in models.py, the logic of using secondary memory to create zip files lies in zipping.py.
I hope this helps. Please leave me a comment if you have better idea on reducing memory load when zipping files.
Your clap will definitely drive me further. Give me a clap if you like this post.
Reducing Memory Footprint While Creating Archive in Django was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.