When writing Python programs, there are times when you need to temporarily store data without consuming much memory, or when you need to write files to the filesystem for subsequent programs to read. In such cases, creating temporary files is a safe and convenient approach that doesn't interfere with the local filesystem.

By using the standard library tempfile in Python, you can easily perform operations related to temporary files. There are two main use cases:

Creating Temporary Files

One of the classic uses of tempfile is to use TemporaryFile() to gether with the with context manager. This creates a file with a random name in the temporary file directory of the local filesystem. Once the code inside the with block finsihes executing, the temporary file is automatically deleted:

import os
import tempfile

with tempfile.TemporaryFile() as f:
    print(f.name)
    print(os.path.exists(f.name))

print(os.path.exists(f.name))

Based on this feature, you can apply it to many scenarios. For example, when you want to convert a table-formatted string into a pandas DataFrame, you can do it like this:

import pandas as pd
import tempfile

# Suppose we have a CSV-formatted string
data = "a,b,c\n1,2,3\n4,5,6"

# Write the string to a temporary file and read it back as a DataFrame
with tempfile.TemporaryFile(mode='w+t') as f:
    f.write(data)
    f.seek(0)
    df = pd.read_csv(f)

print(df)

Creating Temporary Directories

The previous example demonstrated creating temproary files. In some scenarios, you need to create temporary directories. This can be achieved using TemporaryDirectory(), which behaves similarly to TemporaryFile():

import tempfile
import os

with tempfile.TemporaryDirectory() as tmpdir:
    print(tmpdir)
    print(os.path.exists(tmpdir))

print(os.path.exists(tmpdir))

A typical application is to use TemporaryDirectory() together with shutil to generate compressed archive files:

import shutil
import zipfile
import numpy as np
import pandas as pd
import tempfile
import os

with tempfile.TemporaryDirectory() as tmpdir:
    # Simulate writing multiple files to the temporary directory
    for i in range(100):
        (
            pd.DataFrame(np.random.rand(10, 3), columns=list('ABC'))
            .to_csv(os.path.join(tmpdir, f'demo{i+1}.csv'), index=False)
        )
    
    # Convert the temporary directory into a zip archive
    shutil.make_archive('./demo', 'zip', tmpdir)

# List files inside the created archive
with zipfile.ZipFile('demo.zip', 'r') as zf:
    print([file.filename for file in zf.filelist])