Functions are defined as blocks of code that accept parameters as inputs, perform processing involving these inputs, and return a value as output. When a function takes another function as input or returns another function as output, these are called higher-order functions.

Functions like map(), reduce(), and filter() are all higher-order functions. Functional programming emphasizes treating functions as first-class objects. Today, we'll explore several higher-order functions in the functools library that are used to create and modify functions.

Introduction to the functools Module

The functools module is part of Python's standard library, implemented for higher-order functions to enhance function capabilities.

import functools
import logging

logging.basicConfig(level=logging.INFO)
logging.info(functools)
logging.info(functools.__doc__)
logging.info(dir(functools))

This information shows that the functools module provides a series of tools for handling functions and callable objects.

Here's a detailed explanation of the main methods included in the functools module:

1. cached_property: A decorator that converts a method into a read-only property, computing and caching the value on first access.
2. cmp_to_key: A utility for converting old-style comparison functions to key functions.
3. cache: A decorator that provides a caching mechanism for function results to improve performance.
4. lru_cache: A decorator that provides a function decorator with a least recently used (LRU) cache to improve performance.
5. partial: A function for partially applying a function's arguments and returning a new function.
6. partialmethod: Similar to partial, but specifically for partially applying class method arguments.
7. reduce: A function that performs cumulative operations on elements in a sequence.
8. singledispatch: A decorator for creating generic functions that dispatch based on the type of the first argument.
9. singledispatchmethod: Similar to singledispatch, but specifically for class methods.
10. total_ordering: A class decorator that automatically generates all comparison methods from a subset of methods.
11. update_wrapper: A function that updates a function object's attributes like __doc__, __name__, and __module__.
12. wraps: A decorator that preserves a function's metadata when applying another decorator.

These tools can help Python developers improve efficiency and flexibility when working with functions.

functools.cached_property

This function converts a class method into a property that is cached after the first computation and used as a regular attribute throughout the instance's lifecycle.

It's similar to property() but adds caching functionality, making it very useful for instance attributes that require significant computational resources but are otherwise effectively immutable.

from functools import cached_property
import logging

logging.basicConfig(level=logging.INFO)

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n - 1) + fibonacci(n - 2)

class DataProcessor:
    @cached_property
    def expensive_calculation(self):
        # This is an expensive calculation we want to execute only once and cache
        logging.info('Computing expensive_calculation')
        return fibonacci(20)

# Usage:
processor = DataProcessor()
logging.info(processor.expensive_calculation)  # Result will be cached
logging.info(processor.expensive_calculation)  # Result will be retrieved from cache

functools.cached_property is available in Python 3.8 and later, allowing you to cache class properties. After evaluation, the property won't be evaluated again.

functools.cache

functools.cache is used as a decorator that caches a function's return values based on its inputs. It's available in Python 3.9 and later.

The cache size is unlimited.

from functools import cache

@cache
def fibonacci(n):
    if n < 2:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

print(fibonacci(3))
print(fibonacci(10))

functools.lru_cache

functools.lru_cache allows you to cache recursive function calls in a least recently used cache. This can optimize functions with multiple recursive calls, like the Fibonacci sequence.

@lru_cache(maxsize=10) means only the 10 most recently used entries will be kept in the cache. When new entries arrive, the oldest cached entries will be discarded.

import logging
from functools import lru_cache

logging.basicConfig(level=logging.INFO)

# Define a global variable to track function call count
call_count = 0

@lru_cache(maxsize=10)
def fibonacci(n):
    global call_count
    call_count += 1

    if n in [0, 1]:
        return n
    else:
        return fibonacci(n - 1) + fibonacci(n - 2)

logging.info(fibonacci(4))
logging.info(f"Total function called {call_count} times.")

logging.info("Second call to fibonacci(1), should hit cache")
logging.info(fibonacci(1))
logging.info(f"Total function called {call_count} times.")

This decorator caches different call results in memory, so attention should be paid to memory usage.

functools.cmp_to_key()

functools.cmp_to_key is a function that converts a comparison function (cmp function) into a key function. In Python 2, comparison functions were widely used for sorting and related operations, but in Python 3, the use of comparison functions was limited due to the removal of the cmp paramter. To continue using comparison functions for sorting in Python 3, you can use functools.cmp_to_key to convert the comparison function into a key function and then pass it to sorting functions.

A comparison function is any callable that accepts two arguments, compares them, and returns a negative number for less than, zero for equal, or a positive number for greater than.

Here's an example demonstrating how to use functools.cmp_to_key:

import functools

# Define a comparison function (directly usable as a sorting function in Python 2)
def custom_compare(x, y):
    return (x > y) - (x < y)

# Convert the comparison function to a key function
key_function = functools.cmp_to_key(custom_compare)

# Use the converted key function for sorting
sorted_list = sorted([3, 1, 4, 1, 5, 9], key=key_function)

print(sorted_list)  # Output: [1, 1, 3, 4, 5, 9]

In this example, custom_compare is a comparison function that takes two parameters and returns -1, 0, or 1 to indicate their relationship. Then, functools.cmp_to_key is used to convert this comparison function into a key function key_function. Finally, by passing key_function to the sorted function, the list can be sorted using this key function.

That is, when sorting, the function specified by key is called for each element first, and then sorting is performed. The cmp_to_key function is used to convert old-style comparison functions into key functions. Functions that use the key parameter also include sorted(), min(), max(), heapq.nlargest(), itertools.groupby(), etc.

functools.total_ordering

The total_ordering decorator is used to define operator classes that can implement various comparison operations, suitable for subclasses of numbers.Number and semi-numeric classes.

functools.total_ordering is a decorator that allows you to define only a small subset of comparison methods when defining a class, and it will automatically complete the remaining comparison methods for you. This way, you can easily define a complete ordered class without manually implementing all comparison methods.

This decorator requires that the class defines at least one of __lt__, __le__, __gt__, or __ge__ methods, and must also define the __eq__ method.

Here's an example demonstrating how to use the functools.total_ordering decorator:

import logging
from functools import total_ordering

logging.basicConfig(level=logging.INFO)

@total_ordering
class Student:
    def __init__(self, name, score):
        self.name = name
        self.score = score

    def __eq__(self, other):
        return self.score == other.score

    def __lt__(self, other):
        return self.score < other.score

# Usage example
alice = Student("Alice", 85)
bob = Student("Bob", 75)

logging.info(alice > bob)  # True, because Alice's score is higher
logging.info(alice == bob)  # False

In this example, we define a Student class and decorate it with the total_ordering decorator. We only define the __eq__ and __lt__ methods without defining other comparison methods. Then, the total_ordering decorator automatically completes the __le__, __gt__, and __ge__ methods for us. This way, we can use all comparison operators to compare Student objects by their scores.

If the __eq__ method is not defined, it's impossible to determine if two objects are equal, which can lead to unexpected behavior when using the total_ordering decorator. Therefore, to ensure the class behaves as expected, you must provide an __eq__ method. The inequality comparison method __ne__() is generated based on __eq__() by default.

functools.partial

Its purpose is to fix some parameters of a function.

That is, it's generally used for: generating a new function based on an old function and some of its parameters.

Here's a simple example:

import logging
from functools import partial

logging.basicConfig(level=logging.INFO)

def power(base, exponent):
    return base ** exponent

# Create a new function with the exponent parameter preset to 2
square = partial(power, exponent=2)

# Use the new function
result = square(base=3)  # Equivalent to power(3, 2), returns 9
logging.info(result)

# Create another new function with the base parameter preset to 2
square_two = partial(power, base=2)

# Use the new function
result_two = square_two(exponent=3)  # Equivalent to power(2, 3), returns 8
logging.info(result_two)

The partial() function is mainly used to "freeze" some parameters of a function, returning a function object with fewer parameters that is simpler to use.

Application scenario: When a function is executed, it must be callled with all necessary parameters, but some parameters can be known before the function is called. In this case, these known function parameters can be used in advance so that the function can be called with fewer parameters.

Example:

urlunquote = functools.partial(urlunquote, encoding='latin1')

When calling urlunquote(args, *kargs), it's equivalent to urlunquote(args, *kargs, encoding='latin1')

A practical example:

import json
import datetime
from functools import partial
import logging

logging.basicConfig(level=logging.INFO)

def json_serial_fallback(obj):
    """JSON serializer for objects not serializable by default json code"""
    if isinstance(obj, (datetime.datetime, datetime.date)):
        return str(obj)
    if isinstance(obj, bytes):
        return obj.decode("utf-8")
    raise TypeError(f"{obj} is not JSON serializable")

# Define json_dumps function
json_dumps = partial(json.dumps, default=json_serial_fallback)

# Create a dictionary containing datetime and bytes
data = {
    "timestamp": datetime.datetime.now(),
    "binary_data": b"example binary data"
}

# Convert the dictionary to a JSON format string
json_string = json_dumps(data)
logging.info(json_string)  # {"timestamp": "2024-02-24 11:16:25.016089", "binary_data": "example binary data"}

functools.partialmethod

functools.partialmethod is a function in Python's standard library used to create partially applicable methods. It's similar to functools.partial, but differs in that it's used for partially applying methods rather than functions.

Partial application is a concept from functional programming that allows you to fix some parameters when calling a function, thereby creating a new function that uses these fixed parameters in subsequent calls. functools.partialmethod plays the same role in object-oriented programming but is used for partially applying methods.

Here's a simple example of functools.partialmethod:

import functools

class Greeter:
    def __init__(self, greeting):
        self.greeting = greeting

    def greet(self, name, *args):
        return f"{self.greeting}, {name} {''.join(args)}"

    # Create a partially applied method
    greet_hello = functools.partialmethod(greet, 'Hello')

# Create a Greeter instance
greeter = Greeter('Bonjour')

# Correctly call the partially applied method
print(greeter.greet_hello("Alice!"))  # Output: Bonjour, Hello Alice!

In this example, the Greeter class defines a greet method for greeting a given name. Then, using functools.partialmethod, a partially applied method greet_hello is created, passing 'Hello' as a fixed parameter to the greet method, thereby creating a new method. When calling greeter.greet_hello('Alice'), it's actually calling greeter.greet('Hello', "Alice!"), so it outputs Bonjour, Hello Alice!.

functools.reduce

The function's purpose is to reduce a sequence to a single output reduce(function, sequence, startValue)

import logging
from functools import reduce

logging.basicConfig(level=logging.INFO)

data_list = range(1, 50)
logging.info(reduce(lambda x, y: x + y, data_list))  # 1225

Importance of Initial Values in functools.reduce

Setting initial values is important for both map() and reduce() functions.

Initial values are important when using the functools.reduce function, especially when dealing with empty sequences. Let's illustrate this with an example:

Suppose we have a list, and we want to calculate the cumulative product of all elements in the list.

from functools import reduce

numbers = [1, 2, 3, 4, 5]

# Calculate cumulative product
result = reduce(lambda x, y: x * y, numbers)
print(result)

In this example, we didn't provide an initial value, so the reduce function uses the first element of the list as the initial cumulative value. Therefore, the calculation process is as follows:

1. Initialize the cumulative value with the first element of the list: accumulator = 1
2. For each element in the list, multiply it by the cumulative value: accumulator = 1 * 2 = 2
3. Continue with the cumulative product for the remaining elements: accumulator = 2 * 3 = 6, accumulator = 6 * 4 = 24, accumulator = 24 * 5 = 120

Therefore, the final result is 120.

Now, let's consider what happens when we have an empty list:

from functools import reduce

numbers = []

# Calculate cumulative product
result = reduce(lambda x, y: x * y, numbers)
print(result)

In this example, because the list is empty, the reduce function cannot determine the initial cumulative value. If no initial value is provided, it will result in a TypeError: reduce() of empty sequence with no initial value error.

To avoid this error, we can provide an initial value, such as 1, to ensure it works correctly even when dealing with empty lists:

from functools import reduce

numbers = []

# Calculate cumulative product with initial value of 1
result = reduce(lambda x, y: x * y, numbers, 1)
print(result)  # Output: 1

In this example, we provided an initial value of 1, so even if the list is empty, the reduce function can correctly return the initial value as the result. This illustrates the importance of providing an initial value when using the reduce function, especially when dealing with empty sequences.

If you don't set an initial value, the reduce() function uses the first value of the sequence as the initial value, and this value won't be passed to the convolution function, leading to calculation errors.

Now, let's define some built-in reduction functions using the reduce() higher-order function:

import logging
from functools import reduce
from typing import Callable, Any, Union

logging.basicConfig(level=logging.INFO)

# Example data list
data = [1, 2, 3, 4, 5]
data2 = []

# Calculate sum of squares
sum_of_squares: Callable[[Any], int] = lambda iterable: reduce(lambda x, y: x + y ** 2, iterable, 0)
logging.info(f"Sum of squares: {sum_of_squares(data)}")  # Output: Sum of squares: 55
logging.info(f"Sum of squares: {sum_of_squares(data2)}")  # Output: Sum of squares: 0

# Calculate total sum
total_sum: Callable[[Any], int] = lambda iterable: reduce(lambda x, y: x + y, iterable, 0)
logging.info(f"Total sum: {total_sum(data)}")  # Output: Total sum: 15
logging.info(f"Total sum: {total_sum(data2)}")  # Output: Total sum: 0

# Count elements
count_elements: Callable[[Any], Union[int, Any]] = lambda iterable: reduce(lambda x, y: x + 1, iterable, 0)
logging.info(f"Count of elements: {count_elements(data)}")  # Output: Count of elements: 5
logging.info(f"Count of elements: {count_elements(data2)}")  # Output: Count of elements: 0

# Find minimum value
minimum_value: Callable[[Any], Any] = lambda iterable: reduce(lambda x, y: x if x < y else y,
                                                             iterable) if iterable else None

logging.info(f"Minimum value: {minimum_value(data)}")  # Output: Minimum value: 1
logging.info(f"Minimum value: {minimum_value(data2)}")  # Output: Minimum value: None

# Find maximum value
maximum_value: Callable[[Any], Any] = lambda iterable: reduce(lambda x, y: x if x > y else y,
                                                             iterable) if iterable else None
logging.info(f"Maximum value: {maximum_value(data)}")  # Output: Maximum value: 5
logging.info(f"Maximum value: {maximum_value(data2)}")  # Output: Maximum value: None

functools.update_wrapper

functools.update_wrapper is a function used to manually update a wrapper function's attributes to match those of the wrapped function. It's typically used with custom decorators to ensure the wrapper function behaves consistently with the original function and preserves its metadata.

Here's an example demonstrating the usage of functools.update_wrapper:

import functools

def another_decorator(func):
    def wrapper(*args, **kwargs):
        print("Another thing is happening before the function is called.")
        result = func(*args, **kwargs)
        print("Another thing is happening after the function is called.")
        return result

    # Use update_wrapper to update wrapper function's attributes to match func's attributes
    functools.update_wrapper(wrapper, func)
    return wrapper

@another_decorator
def say_hello():
    """A simple greeting function."""
    print("Hello!")

say_hello()
print(say_hello.__name__)  # Output: say_hello
print(say_hello.__doc__)  # Output: A simple greeting function.

In this example, another_decorator is a decorator that doesn't use the functools.wraps decorator to preserve the original function's metadata. Instead, it uses the functools.update_wrapper function to manually update the wrapper function's attributes to match the func function's attributes, thereby preserving the original function's metadata.

The wraps function is a wrapper around update_wrapper that makes it easier to copy the signature of the decorated function in decorators.

functools.wraps

functools.wraps is a decorator in Python's functools module used to create well-behaved decorators. Decorators are functions that modify the behavior of other functions or methods. When you apply a decorator to a function or method, the original function's metadata, such as its name, docstring, and parameter list, may be lost or modified. functools.wraps helps preserve this metadata.

Here's an example of using functools.wraps:

import functools

def my_decorator(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print("Something is happening before the function is called.")
        result = func(*args, **kwargs)
        print("Something is happening after the function is called.")
        return result
    return wrapper

@my_decorator
def say_hello():
    """A simple greeting function."""
    print("Hello!")

say_hello()
print(say_hello.__name__)  # Output: say_hello
print(say_hello.__doc__)   # Output: A simple greeting function.

In this example, my_decorator is a decorator that decorates the function say_hello. Without using functools.wraps, accessing say_hello.__name__ or say_hello.__doc__ would not yield the expected results because they would reflect the metadata of the wrapper function rather than say_hello. However, by using @functools.wraps(func), the original function say_hello's metadata is preserved, making the decorator behave as expected.

functools.singledispatch

The singledispatch decorator is used for function overloading, converting a function into a single-dispatch generic function.

Dispatch occurs based on the type of the first argument.

from functools import singledispatch
import logging

logging.basicConfig(level=logging.INFO)

@singledispatch
def process_data(arg1, arg2):
    logging.info(f"default implementation of process_data - {arg1, arg2}")

@process_data.register
def process_string(arg1: str, arg2):
    logging.info(f"【process_string】with first argument as string - {arg1, arg2}")

@process_data.register
def process_int(arg1: int, arg2):
    logging.info(f"【process_int】with first argument as int - {arg1, arg2}")

process_data(1.34, "hi")
process_data("test", "hello")
process_data(1, "hello")

logging.info(process_data.registry)
logging.info(process_data.registry.keys())

functools.singledispatchmethod

Converts a method into a single-dispatch generic function.

When defining functions with @singledispatchmethod, note that dispatch occurs based on the type of the first non-self or non-cls parameter.

from functools import singledispatchmethod

class Calculator:
    @singledispatchmethod
    def compute(self, arg1, arg2):
        print(f"Default implementation with arg1 = {arg1} and arg2 = {arg2}")

    @compute.register
    def compute_int(self, arg1: int, arg2: int):
        print(f"Sum with integers. {arg1} + {arg2} = {arg1 + arg2}")

    @compute.register
    def compute_float(self, arg1: float, arg2: float):
        print(f"Sum with floats. {arg1} + {arg2} = {arg1 + arg2}")

calc = Calculator()
calc.compute(2, 3)
calc.compute(2.1, 3.4)
calc.compute("hi", 3.4)

"""
Output:
Sum with integers. 2 + 3 = 5
Sum with floats. 2.1 + 3.4 = 5.5
Default implementation with arg1 = hi and arg2 = 3.4
"""

Overloading class methods:

from functools import singledispatchmethod

class DataProcessor:
    @singledispatchmethod
    @classmethod
    def process(cls, arg):
        print(f"Default implementation. arg - {arg}")

    @process.register(int)
    @classmethod
    def process_int(cls, arg: int):
        print(f"Integer implementation. arg - {arg}")

    @process.register(bool)
    @classmethod
    def process_bool(cls, arg):
        print(f"Boolean implementation. arg - {arg}")

DataProcessor.process(4)
DataProcessor.process(True)
DataProcessor.process("hi")

"""
Integer implementation. arg - 4
Boolean implementation. arg - True
Default implementation. arg - hi
"""