The universal_newlines parameter in Pyhton's subprocess module enables text mode processing for standard input, output, and error streams. When set to True, Python handles these streams as text rather than raw bytes, simplifying text data manipulation.
In text mode:
- Standard input (stdin): Text data sent to the subprocess gets encoded to bytes before being passed to the child process
- Standard output (stdout) and standard error (stderr): Output from the subprocess gets decoded from bytes to text, making it easier to process in Python
This parameter essentially converts the communication channels between parent and child processes from binary to text mode, which is particularly useful when working with textual data.
Here's an example demonstrating text mode processing:
import subprocess
# Execute command and redirect output to file
with open("output.txt", "w") as output_file:
process = subprocess.Popen(
["ls", "-l"],
stdout=output_file,
stderr=subprocess.PIPE,
bufsize=1,
universal_newlines=True # Enable text mode
)
# Wait for process completion
stdout, stderr = process.communicate()
In this example, the output from the ls -l command is written to output.txt as text rather than bytes.
Python's subprocess module offers different buffering options through the bufsize parameter:
- Unbuffered I/O (
bufsize=0): Data is processed immediately without any buffering - Line-buffered I/O (
bufsize=1): Output is written whenever a newline character is encountered - Fully buffered I/O (default,
bufsize>1): Data is buffered until the buffer reaches a certain size
Here's an example of unbuffered I/O:
import subprocess
# Execute command with unbuffered output
with open("unbuffered.log", "w") as log_file:
process = subprocess.Popen(
["ping", "-c", "3", "example.com"],
stdout=log_file,
stderr=subprocess.PIPE,
bufsize=0 # Unbuffered mode
)
# Wait for process to complete
stdout, stderr = process.communicate()
And here's an example of line-buffered I/O:
import subprocess
# Execute command with line-buffered output
with open("linebuffered.log", "w") as log_file:
process = subprocess.Popen(
["tail", "-f", "/var/log/syslog"],
stdout=log_file,
stderr=subprocess.PIPE,
bufsize=1 # Line-buffered mode
)
# Wait for process to complete
stdout, stderr = process.communicate()
When working with subprocesses, sending signals to terminate them might not always work as expected. The send_signal() method sends a signal to the subprocess, but the process might not terminate if:
- The subprocess doesn't have a proper signal handler for the sent signal
- The subprocess is blocked in a system call and doesn't check for signals
- The subprocess is running in a shell and the shell doesn't forward the signal
- There are permission issues preventing signal delivery
- The subprocess has already terminated but the Popen object hasn't updated its status
If a subprocess doesn't respond to SIGINT (Ctrl+C), you can try alternative approaches:
- Use
SIGTERMinstead ofSIGINTwithprocess.send_signal(signal.SIGTERM) - Force termination with
process.terminate()orprocess.kill() - Ensure you're not using
shell=Truewhich can interfere with signal handling - Verify the subprocess has proper signal handling implemented
Here's an example of signal handling:
import subprocess
import signal
import time
# Start a long-running process
process = subprocess.Popen(
["sleep", "60"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
# Send interrupt signal after 2 seconds
time.sleep(2)
process.send_signal(signal.SIGINT)
# Check if process terminated
if process.poll() is None:
print("Process didn't respond to SIGINT, terminating...")
process.terminate()