Python Subprocess Tutorial: Master run() and Popen() Commands (with Examples)

What is the Python subprocess module?

Did you ever want to run a shell command from within a Python script? Whether it’s automating file backups, deploying code, or monitoring system performance, Python’s subprocess module is the tool for the job.

But hold on, what is a subprocess? A subprocess is essentially a secondary process launched by a main application, here, your Python script. Think of it as your Python program asking your operating system to run another program on its behalf.

The Python’s subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes from within your Python scripts. It provides two primary interfaces: subprocess.run() for basic command execution and subprocess.Popen() for advanced process management with real-time interaction capabilities.

Where is the subprocess module useful?

• Automate system tasks: Automate backups, start or stop services, or schedule recurring jobs using shell commands.

• Run shell utilities: Execute tools like grep, sed, or awk and process their output directly in Python.

• Launch external programs: Run executables like .exe files on Windows or shell scripts on Unix systems.

• Start background scripts: Trigger scripts to run in the background without blocking your main Python program.

• Use non-Python scripts: Run Bash, Perl, or Ruby scripts as part of a Python workflow and capture their output.

• Enable parallel execution: Launch multiple subprocesses for tasks like scraping or processing large datasets.

• Integrate into deployment flows: Automate test, build, and deployment steps from Python-based CI/CD pipelines.

• Monitor system resources: To track system health, use commands like df, top, or custom monitoring scripts.

Integrating shell-level power with Python’s logic makes the subprocess module essential to any automation, scripting, or systems programming toolkit.

Now that we know why the subprocess module is so valuable, let’s explore subprocess.run(), the most beginner-friendly interface.

Using subprocess.run() for basic commands

The subprocess.run() function is the basic way to run external commands from within a Python script. Introduced in Python 3.5, it provides a powerful yet straightforward interface to execute a command, wait for it to finish, and then return a CompletedProcess object. This object contains essential details such as the command’s return code (returncode), standard output (stdout), and standard error (stderr). It can also raise an exception if the command fails, capture output to variables, or run commands via the system shell.

This function is ideal for short, one-off shell tasks without advanced interaction, such as installing dependencies, listing files, or automating routine jobs.

Syntax of subprocess.run():

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None, **other_popen_kwargs) 

Parameters:

• args: A list or string of the command and its arguments to run.

• stdin: Specifies the standard input to the process (e.g., subprocess.PIPE).

• input: Data to be sent to the process’s standard input.

• stdout: Specifies where the standard output goes. Use subprocess.PIPE to capture it.

• stderr: Specifies where the standard error goes. Use subprocess.PIPE to capture it.

• capture_output: A shortcut to set both stdout and stderr to subprocess.PIPE.

• shell: If True, runs the command through the shell (like Bash or CMD).

• cwd: Set the current working directory before executing the command.

• timeout: Maximum time in seconds to wait before killing the process.

• check: If True, raises a CalledProcessError for non-zero exit codes.

• encoding / errors: Manage how output bytes are converted to strings and how errors are handled.

• text or universal_newlines: If True, treats output and input as strings (instead of bytes).

• env: A dictionary defining environment variables for the new process.

Example 1: Running shell commands with output

In the following example, we run a Windows command (dir) and capture the output as follows:

import subprocess 

result = subprocess.run(["dir"], shell=True, capture_output=True, text=True) 
print(result.stdout) 

Copy to clipboard
Copy to clipboard

A sample output produced by this code is:

Volume in drive C has no label. 
Volume Serial Number is F48A-4644 
  
Directory of C:\Users\LENOVO\OneDrive\Desktop\Codecademy - Python 

14-05-2025  19:49    <DIR>          . 
14-05-2025  13:26    <DIR>          .. 
21-03-2025  15:06    <DIR>          data 
19-12-2024 15:17              1,087 pet_adoption_center.py 
14-05-2025  19:49    <DIR>          python-modules 
14-05-2025  19:49    <DIR>          pytorch - tensors 
14-05-2025  19:48             123   test.py 
12-12-2024  16:36    <DIR>          __pycache__ 
                                    2 File(s)          1,210 bytes 
                                    6 Dir(s)  110,465,581,056 bytes free 

Copy to clipboard
Copy to clipboard

Note: This code’s output will vary depending on your operating system, current working directory, and the files and folders present at runtime.

In this code:

• ["dir"] runs the directory listing command.
• shell=True allows running shell built-in commands like dir.
• capture_output=True captures both stdout and stderr.
• text=True converts the output to a string.

Example 2: Running a Python script

We execute an external Python script named my_python_file.py in this example.

The content inside the my_python_file.py will be:

print("This is the output from subprocess module") 

Copy to clipboard
Copy to clipboard

Content inside the main script file:

import subprocess 

result = subprocess.run(["python", "my_python_file.py"], capture_output=True, text=True) 
print(result.stdout) 

Copy to clipboard
Copy to clipboard

The output of this code will be:

This is the output from subprocess module 

Copy to clipboard
Copy to clipboard

The subprocess.run() function is used here to execute another Python file. capture_output=True stores the output so it can be printed or processed later.

Example 3: Capturing output and errors separately

Run a command and separately capture both its output and errors:

import subprocess 

result = subprocess.run( 
    ["python", "--version"], 
    stdout=subprocess.PIPE, 
    stderr=subprocess.PIPE, 
    text=True 
) 

print("STDOUT:", result.stdout) 
print("STDERR:", result.stderr) 

Copy to clipboard
Copy to clipboard

Possible output of this code will be:

STDOUT: Python 3.11.2 
STDERR: 

Copy to clipboard
Copy to clipboard

Here:

• stdout=subprocess.PIPE and stderr=subprocess.PIPE capture the respective outputs.

• text=True returns the output as readable strings.

When to use shell=True (and its risks)

Setting shell=True allows running shell commands as raw strings and enables shell-specific features like variable expansion and redirection:

subprocess.run("echo %USERNAME%", shell=True) 

Copy to clipboard
Copy to clipboard

Note: shell=True can be dangerous if you pass user input directly, risking shell injection attacks.

An unsafe example:

# Do NOT do this with user input  
user_input = "some_command; rm -rf /"  
subprocess.run(user_input, shell=True) 

Copy to clipboard
Copy to clipboard

Avoid shell=True unless necessary. Prefer list-style arguments like ['echo', 'hello'], and sanitize all input if shell usage is unavoidable.

subprocess.run() is excellent for basic command execution and output capturing. But when you need real-time interaction, parallel processing, or advanced piping, it’s time to level up with subprocess.Popen().

Using subprocess.Popen() for advanced control

While subprocess.run() is ideal for simple, one-off commands, Python’s subprocess.Popen() class offers more flexibility. It is better suited for advanced use cases requiring real-time interaction, streaming data, or handling multiple processes.

The Popen() class launches a new process and gives you direct control over its input/output/error streams. Unlike run(), it doesn’t wait for the command to finish immediately, you can interact with it as it runs.

Syntax of Popen():

subprocess.Popen(args, bufsize=-1, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=True, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0, text=None) 

Parameters:

• args: A list or string of the command to execute.

• stdin / stdout / stderr: Specifies how to handle input/output/error streams. Use subprocess.PIPE to access them from Python.

• shell: If True, the command will be run through the shell.

• cwd: Set the working directory for the new process.

• env: A dictionary of environment variables for the new process.

• text / universal_newlines: If True, treats input/output as text strings instead of bytes.

• bufsize: Buffering policy. 0 for unbuffered, 1 for line-buffered, or any integer for block buffering.

Example 1: Reading output and sending input

import subprocess  

process = subprocess.Popen('more', stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True, shell=True)  
output, _ = process.communicate('Hello from Python!\n')  
print(output) 

Copy to clipboard
Copy to clipboard

The output of this code is:

Hello from Python! 

Copy to clipboard
Copy to clipboard

Here:

• 'more' is a shell command that paginates input. Here, it is run with shell=True so Windows recognizes it.

• stdin=subprocess.PIPE lets us send input data.

• stdout=subprocess.PIPE captures the output.

• text=True means inputs and outputs are treated as strings, not bytes.

• process.communicate() sends the input string and waits for process completion, returning the output.

• The print(output) displays what the subprocess returned.

Example 2: Managing multiple processes

Popen() also enables chaining commands by connecting their inputs and outputs using subprocess.PIPE. In the example below, we are piping findstr and sort on Windows:

import subprocess 

p1 = subprocess.Popen('findstr Python', stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True, shell=True) 
p2 = subprocess.Popen('sort', stdin=p1.stdout, stdout=subprocess.PIPE, text=True, shell=True) 

p1.stdin.write("Hello\nPython\nWorld\nPythonista\n") 

p1.stdin.close()   

output, _ = p2.communicate() 
print(output) 

Copy to clipboard
Copy to clipboard

Here, the output of findstr (searches for lines containing “Python”) is piped to sort to order the results. The output of this code is:

Python 
Pythonista 

Copy to clipboard
Copy to clipboard

Popen() provides granular control over process execution, input/output streaming, and chaining multiple processes, making it ideal for interactive or complex subprocess workflows.

Now, you must be wondering which one to choose. Let’s compare run() and Popen() to help you decide which better fits your use case.

Difference between subprocess.run() and subprocess.Popen()

When working with subprocesses in Python, choosing between subprocess.run() and subprocess.Popen() depends mainly on your task complexity and interaction needs. Here is a table highlighting their differences:

In short:

• Use run() for simple commands where you just want to execute a process and wait for it to complete. It’s easy to use and returns results directly.

• Use Popen() when you need advanced control: continuous input/output interaction, streaming data, managing multiple processes at once, or running long-lived processes asynchronously.

Common pitfalls like deadlocks, unresponsive scripts, or lost output can be avoided by choosing the right tool.

Next, let’s look at some common mistakes to avoid and best practices when using the subprocess module.

Python subprocess pitfalls and best practices

So far, we’ve covered how to use subprocess.run() for straightforward commands and subprocess.Popen() for advanced process control. Both are powerful tools for integrating system commands within Python, but they have some important caveats to remember. Here are some best practices to keep your code safe:

• Avoid shell=True with untrusted input: Using shell=True can expose your program to shell injection vulnerabilities. Always sanitize inputs or avoid shell=True when running external commands.

• Always handle errors and exit codes: Check process return codes (returncode) to detect failures. Use check=True with run() to raise exceptions on errors and inspect Popen() return codes to ensure your subprocesses succeed.

• Use shlex.split() to safely parse command strings: When you have a command as a single string (not a list), use shlex.split() to safely split it into a list of arguments, preventing injection issues and parsing errors.

• Manage cross-platform compatibility (Windows vs. Unix): Commands like ls, cat, or grep differ between operating systems. Write OS-aware code or use Python-native libraries, when possible, for portability.

Understanding and applying these best practices ensures your subprocess usage is secure and robust.

Conclusion

In this article, we explored how to use Python’s subprocess module effectively. We started with the basic yet powerful subprocess.run() function for running commands and capturing output. Then, we looked at the more advanced subprocess.Popen() for greater control over processes, including interacting with input/output streams and managing multiple processes. We also discussed when to use each method, common pitfalls to avoid, and best practices to ensure your code is secure and reliable.

If you’re interested in further exploring Python scripting and automation, check out Codecademy’s Learn Python 3 course.

Frequently asked questions

1. How do I pipe the output from one subprocess to another?

You can use subprocess.Popen() with stdout=subprocess.PIPE in the first process and pass that pipe as stdin to the second process. This allows chaining commands similar to shell pipelines.

2. What are alternatives to subprocess in Python?

Alternatives include:

• os.system() for simple commands (but less secure and flexible)
• The sh or plumbum third-party libraries for more Pythonic interfaces
• asyncio.create_subprocess_exec() for asynchronous subprocess management
• pexpect for interactive subprocess control, especially for commands requiring user input

3. What is the difference between call() and run() in subprocess?

subprocess.call() runs a command and returns its return code. subprocess.run() (introduced in Python 3.5) is more versatile, returning a CompletedProcess object containing return code, stdout, and stderr, and supports more parameters.

4. What is the difference between the main process and a subprocess?

The main process is the Python program you start, while a subprocess is a separate process spawned by your program to execute external commands or scripts. Subprocesses run concurrently and can communicate with the main process via input/output streams.