Python is an interpreted language, which means it typically runs slower than compiled languages like C or C++. However, Python's performance is often not as poor as many assume. This article presents several techniques to accelerate Python code execution.
Code Optimization Principles
Before diving into specific optimization techniques, it's important to understand some fundamental principles.
Principle 1: Avoid Premature Optimization
Many developers start optimizing from the beginning, but "making a correct program faster is easier than making a fast program correct." Optimization should only begin after the code works correctly. Premature optimization can lead to missing the big picture, and optimizing the wrong sections wastes effort.
Principle 2: Consider the Cost of Optimization
Optimization has costs. Solving all performance issues is usually impossible. You often face tradeoffs between time and space, and development time must also be considered.
Principle 3: Don't Optimize Unimportant Sections
Optimizing every single part of your code makes it harder to read and maintain. If your code is slow, first identify the bottlenecks—typically inner loops—and focus optimization efforts there. Minor inefficiencies elsewhere usually don't matter.
1. Avoid Global Variables
# Not recommended. Execution time: 26.8 seconds
import math
size = 10000
for x in range(size):
for y in range(size):
z = math.sqrt(x) + math.sqrt(y)
Developers new to Python often write scripts using global variables by default. However, code defined at module level executes slower than code inside functions due to differences in how Python handles global versus local variable lookups. Moving script logic into functions typically yields a 15-30% speed improvement.
# Recommended. Execution time: 20.6 seconds
import math
def main():
size = 10000
for x in range(size):
for y in range(size):
z = math.sqrt(x) + math.sqrt(y)
main()
2. Minimize Attribute Access
2.1 Avoid Module and Function Attribute Lookups
# Not recommended. Execution time: 14.5 seconds
import math
def calculateSquareRoots(n: int):
results = []
for i in range(n):
results.append(math.sqrt(i))
return results
def main():
n = 10000
for _ in range(n):
results = calculateSquareRoots(n)
main()
Every attribute access using the dot operator triggers special methods like __getattribute__() and __getattr__(), which involve dictionary lookups and add overhead. Using from import eliminates these attribute lookups.
# First optimization. Execution time: 10.9 seconds
from math import sqrt
def calculateSquareRoots(n: int):
results = []
for i in range(n):
results.append(sqrt(i))
return results
def main():
n = 10000
for _ in range(n):
results = calculateSquareRoots(n)
main()
Since local variable lookups are faster than global variable lookups, assigning frequently accessed functions to local variables speeds up execution.
# Second optimization. Execution time: 9.9 seconds
import math
def calculateSquareRoots(n: int):
results = []
root = math.sqrt
for i in range(n):
results.append(root(i))
return results
def main():
n = 10000
for _ in range(n):
results = calculateSquareRoots(n)
main()
Beyond math.sqrt, the computeSqrt function also contains dot operators when calling list.append. Assigning this method to a local variable eliminates all dot operations inside the for loop.
# Recommended. Execution time: 7.9 seconds
import math
def calculateSquareRoots(n: int):
results = []
push = results.append
root = math.sqrt
for i in range(n):
push(root(i))
return results
def main():
n = 10000
for _ in range(n):
results = calculateSquareRoots(n)
main()
2.2 Avoid Class Property Access
# Not recommended. Execution time: 10.4 seconds
import math
from typing import List
class DataProcessor:
def __init__(self, val: int):
self._val = val
def computeRoots(self, n: int) -> List[float]:
results = []
push = results.append
root = math.sqrt
for _ in range(n):
push(root(self._val))
return results
def main():
n = 10000
for _ in range(n):
processor = DataProcessor(n)
results = processor.computeRoots(n)
main()
The principle of minimizing dot operations also applies to class properties. Accessing self._val is slower than accessing a local variable. Assigning frequently accessed properties to local variables improves performance.
# Recommended. Execution time: 8.0 seconds
import math
from typing import List
class DataProcessor:
def __init__(self, val: int):
self._val = val
def computeRoots(self, n: int) -> List[float]:
results = []
push = results.append
root = math.sqrt
current_val = self._val
for _ in range(n):
push(root(current_val))
return results
def main():
n = 10000
for _ in range(n):
processor = DataProcessor(n)
processor.computeRoots(n)
main()
3. Avoid Unnecessary Abstraction
# Not recommended. Execution time: 0.55 seconds
class ConfigManager:
def __init__(self, val: int):
self.val = val
@property
def val(self) -> int:
return self._val
@val.setter
def val(self, x: int):
self._val = x
def main():
n = 1000000
for i in range(n):
config = ConfigManager(n)
value = config.val
config.val = i
main()
Using additional layers like decorators, properties, or descriptors adds overhead. Often, property accessors are unnecessary—using getter/setter functions is a habit carried over from C/C++ programming. When not strictly needed, use simple public attributes.
# Recommended. Execution time: 0.33 seconds
class ConfigManager:
def __init__(self, val: int):
self.val = val
def main():
n = 1000000
for i in range(n):
config = ConfigManager(n)
value = config.val
config.val = i
main()
4. Avoid Unnecessary Data Copying
4.1 Remove Meaningless Copies
# Not recommended. Execution time: 6.5 seconds
def main():
n = 10000
for _ in range(n):
numbers = range(n)
num_list = [x for x in numbers]
squared = [x * x for x in num_list]
main()
The num_list variable in the code above is completely unnecessary, creating an extra data structure and copying values unnecessarily.
# Recommended. Execution time: 4.8 seconds
def main():
n = 10000
for _ in range(n):
numbers = range(n)
squared = [x * x for x in numbers]
main()
Another issue stems from overconfidence in Python's memory model. Some developers overuse functions like copy.deepcopy() with out understanding or trusting Python's reference semantics. Usually, these copies can be eliminated.
4.2 Swap Values Without Temporary Variables
# Not recommended. Execution time: 0.07 seconds
def main():
n = 1000000
for _ in range(n):
a = 3
b = 5
temp = a
a = b
b = temp
main()
The code above creates a temporary variable for swapping values. Python's tuple unpacking makes this more concise and faster.
# Recommended. Execution time: 0.06 seconds
def main():
n = 1000000
for _ in range(n):
a = 3
b = 5
a, b = b, a
main()
4.3 Use join() Instead of + for String Concatenation
# Not recommended. Execution time: 2.6 seconds
import string
from typing import List
def joinStrings(texts: List[str]) -> str:
output = ''
for txt in texts:
output += txt
return output
def main():
texts = list(string.ascii_letters * 100)
for _ in range(10000):
result = joinStrings(texts)
main()
When concatenating strings with +, Python creates a new immutable string object, allocates memory, and copies both strings into the new space. For n concatenations, this creates n-1 intermediate strings, each requiring memory allocation and copying. Using join() calculates the total required memory upfront and allocates it once, then copies all strings into the buffer.
# Recommended. Execution time: 0.3 seconds
import string
from typing import List
def joinStrings(texts: List[str]) -> str:
return ''.join(texts)
def main():
texts = list(string.ascii_letters * 100)
for _ in range(10000):
result = joinStrings(texts)
main()
5. Leverage Short-Circuit Evaluation
# Not recommended. Execution time: 0.05 seconds
from typing import List
def filterAbbreviations(texts: List[str]) -> str:
abbrevs = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
output = ''
for txt in texts:
if txt in abbrevs:
output += txt
return output
def main():
for _ in range(10000):
texts = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
result = filterAbbreviations(texts)
main()
Short-circuit evaluation means for if a and b, when a is False, Python returns immediately without evaluating b. For if a or b, when a is True, it returns without evaluating b. To save execution time, for or conditions, place the more likely-to-be-True expression first. For and conditions, place the more likely-to-be-False expression first.
# Recommended. Execution time: 0.03 seconds
from typing import List
def filterAbbreviations(texts: List[str]) -> str:
abbrevs = {'cf.', 'e.g.', 'ex.', 'etc.', 'flg.', 'i.e.', 'Mr.', 'vs.'}
output = ''
for txt in texts:
if txt[-1] == '.' and txt in abbrevs:
output += txt
return output
def main():
for _ in range(10000):
texts = ['Mr.', 'Hat', 'is', 'Chasing', 'the', 'black', 'cat', '.']
result = filterAbbreviations(texts)
main()
6. Loop Optimization
6.1 Use for Loops Instead of while Loops
# Not recommended. Execution time: 6.7 seconds
def sumUp(n: int) -> int:
total = 0
i = 0
while i < n:
total += i
i += 1
return total
def main():
n = 10000
for _ in range(n):
total = sumUp(n)
main()
Python's for loops are significantly faster than while loops.
# Recommended. Execution time: 4.3 seconds
def sumUp(n: int) -> int:
total = 0
for i in range(n):
total += i
return total
def main():
n = 10000
for _ in range(n):
total = sumUp(n)
main()
6.2 Replace Explicit for Loops with Implicit Alternatives
For the above example, using built-in functions with implicit loops improves performance further.
# Recommended. Execution time: 1.7 seconds
def sumUp(n: int) -> int:
return sum(range(n))
def main():
n = 10000
for _ in range(n):
total = sumUp(n)
main()
6.3 Hoist Invariant Computations Out of Inner Loops
# Not recommended. Execution time: 12.8 seconds
import math
def main():
n = 10000
root = math.sqrt
for x in range(n):
for y in range(n):
z = root(x) + root(y)
main()
In the code above, sqrt(x) appears in the inner loop and gets recalculated on every iteration, causing unnecessary overhead.
# Recommended. Execution time: 7.0 seconds
import math
def main():
n = 10000
root = math.sqrt
for x in range(n):
root_x = root(x)
for y in range(n):
z = root_x + root(y)
main()
7. Use numba.jit
Using the previous example, we can apply numba.jit to JIT-compile the Python function to native machine code, dramatically improving execution speed.
# Recommended. Execution time: 0.62 seconds
import numba
@numba.jit
def sumUp(n: float) -> int:
total = 0
for i in range(n):
total += i
return total
def main():
n = 10000
for _ in range(n):
total = sumUp(n)
main()
8. Choose Appropriate Data Structures
Python's built-in data structures—str, tuple, list, set, and dict—are all implemented in C and are extremely fast. Creating custom data structures to match this performance is nearly impossible.
list is similar to C++'s std::vector, a dynamic array that pre-allocates memory. When the allocated space fills up, it allocates a larger block, copies all existing elements, and frees the old memory. When the used space drops below half the allocated space, it reallocates to a smaller block. Therefore, frequent insertions and deletions with large batch sizes make lists inefficient. In such cases, consider collections.deque, a double-ended queue that supports O(1) insertions and deletions at both ends.
List lookups are also expensive. When you need to frequently search for elements or access them in order, use the bisect module to maintain a sorted list with binary search capabilities.
For finding minimum or maximum values repeatedly, the heapq module can transform a list into a heap, enabling O(1) retrieval of the smallest or largest element.
The Python Wiki provides time complexity information for common built-in data structures: