Table of Contents

• The Calibration Problem
• Understanding the Sampling Distortion
• Mathematical Foundation via Bayes' Theorem
• Establishing the Connection Between Distributions
• Deriving the Calibration Equation
• Evidence-Based Interpretation
• Integration with Model Bias
• Implementation Example
  • (1) Setup
  • (2) Creating Synthetic Data
  • (3) Performing Negative Downsampling
  • (4) Model Architecture and Training
  • (5) Pre-Calibration Assessment
  • (6) Applying Calibration Correction
  • (7) Post-Calibration Assessment
  • (8) Visual Comparison

The Calibration Problem

In numerous applications, we require not just class predictions but reliable probability estimates. However, raw model outputs often fail to represent true likelihoods. Probability calibration addresses this gap by recalibrating predictions to align with actual event frequencies.

Misaligned probabilities frequently emerge in click-through rate prediction, fraud detection, and medical screening—domains characterized by extreme class imbalance. Consider advertising systems where click rates hover around 1%, leaving 99% as non-clicks. Such severe imbalance creates two critical challenges:

1. Majority class bias: Models discover a "shortcut" by predominantly predicting the majority class, achieving high accuracy while poorly identifying rare events.
2. Computational inefficiency: Redundant negative examples inflate training time and resource consumption.

The standard remedy involves negative downsampling: retain all positive instances while randomly selecting a fraction \(r = 10\%\) of negatives for training.

This sampling strategy fundamentally alters the class distribution. The model learns from a "distorted" dataset where positive instances appear disproportionate frequent. Consequently, the model's predicted probability \(p' = P'(y=1|x)\) reflects this artificial distribution rather than reality. Without correction, \(p'\) becomes systematically inflated and unsuitable for business decisions.

We denote \(p\) as the true-world probability and \(p'\) as the sampled-world probability, with \(r\) representing the negative sampling ratio (downsampled negatives divided by original negatives).

Calibration objective: Transform the probability \(p'\) learned from sampled data back to its true-world equivalent \(p\).

Understanding the Sampling Distortion

Negative downsampling creates a mismatch between training and deployment environments. During training, the model observes an artificially inflated positive rate, causing it to output probabilities calibrated to this distorted distribution rather than the actual population.

Consider a scenario with 1% true positive rate. After 10% negative downsampling, the training set contains approximately 9.1% positives (assuming all positives retained). The model learns that a 9.1% positive rate is "normal," so its predicted probabilities become biased upward. When deployed on the true population with only 1% positives, these inflated predictions become unreliable.

Mathematical Foundation via Bayes' Theorem

Bayes' theorem expresses posterior probability as:

[P(A|B) = \frac{P(A)P(B|A)}{P(B)}. ]

Applying this to both our true and sampled distributions:

In the true distribution:

[P(y=1|x) = \frac{P(y=1)P(x|y=1)}{P(x)}, ] [P(y=0|x) = \frac{P(y=0)P(x|y=0)}{P(x)}, ]

Taking the ratio yields the posterior odds in the true world, denoted \(o\):

[o = \frac{P(y=1|x)}{P(y=0|x)} = \frac{P(x|y=1)P(y=1)}{P(x|y=0)P(y=0)}. ]

In the sampled distribution:

[P'(y=1|x) = \frac{P'(y=1)P'(x|y=1)}{P'(x)}, ] [P'(y=0|x) = \frac{P'(y=0)P'(x|y=0)}{P'(x)}, ]

The ratio gives sampled posterior odds, denoted \(o'\):

[o' = \frac{P'(y=1|x)}{P'(y=0|x)} = \frac{P'(x|y=1)P'(y=1)}{P'(x|y=0)P'(y=0)}. ]

Odds represents the ratio of probability of occurrence to non-occurrence for an event.

Establishing the Connection Between Distributions

Examining the relationship between true-world and sampled-world odds components:

(1) Likelihood \(P(x|y)\):

\(P'(x|y=1) = P(x|y=1)\): Since all positives are retained, their intra-class distribution remains unchanged.

\(P'(x|y=0) = P(x|y=0)\): Random downsampling preserves the intra-class distribution of negatives.

(2) Prior \(P(y)\):

The prior probability is precisely what sampling modifies.

Expressing the prior odds relationship between distributions:

[\frac{\frac{P'(y=1)}{P'(y=0)}}{\frac{P(y=1)}{P(y=0)}} = \frac{1}{r}. ]

This relationship is intuitive: prior odds represent the positive-to-negative count ratio. Downsampling multiplies negative counts by \(r\), making the sampled prior odds \(\frac{1}{r}\) times the true prior odds.

Deriving the Calibration Equation

Using the prior odds ratio, we derive the calibration formula via posterior odds:

[\begin{aligned} \frac{o'}{o} & = \frac{\frac{P'(y=1|x)}{P'(y=0|x)}}{\frac{P(y=1|x)}{P(y=0|x)}} = \frac{\frac{P'(x|y=1)P'(y=1)}{P'(x|y=0)P'(y=0)}}{\frac{P(x|y=1)P(y=1)}{P(x|y=0)P(y=0)}} \ & = \frac{\frac{P'(y=1)}{P'(y=0)}}{\frac{P(y=1)}{P(y=0)}} = \frac{1}{r}. \end{aligned} ]

This yields a remarkably elegant relationship:

[o = o' \cdot r. ]

Converting odds back to probabilities, where \(o' = \frac{p'}{1-p'}\) and \(o = \frac{p}{1-p}\):

[\frac{p}{1 - p} = \frac{p'}{1 - p'} \cdot r, ]

Solving for \(p\):

[p = \frac{p'}{p' + \frac{1 - p'}{r}}. ]

This constitutes the prior calibration formula.

Evidence-Based Interpretation

The calibration formula admits an intuitive "evidence" interpretation:

[p = \frac{p'}{p' + \frac{1 - p'}{r}}, ]

In the sampled world, \(p'\) represents positive "evidence" while \(1-p'\) represents negative "evidence." The posterior probability equals the ratio of positive evidence to total evidence:

[\text{Posterior Probability} = \frac{\text{Positive Evidence}}{\text{Positive Evidence} + \text{Negative Evidence}}, ] [p' = \frac{p'}{p' + (1 - p')}. ]

Since negatives were downsampled to fraction \(r\), we must amplify their evidence back to true scale by dividing by \(r\). The calibrated negative evidence becomes \(\frac{1-p'}{r}\) while positive evidence stays at \(p'\).

Recalculating the posterior with scaled evidence produces the calibrated probability \(p\):

[p = \frac{p'}{p' + \frac{1 - p'}{r}}. ]

Integration with Model Bias

For logistic regression, prior calibration can be absorbed into the bias term.

The logistic function takes the form:

[p' = \frac{1}{1 + e^{-z}} = \frac{e^z}{1 + e^z}, ]

where \(z\) is the logit. Logit equals log of odds:

[z = \log\frac{p'}{1 - p'}, ]

and \(e^z\) represents the odds.

Rearranging the calibration formula to expose the odds term:

[\begin{aligned} p & = \frac{p'}{p' + \frac{1 - p'}{r}} \ & = \frac{\frac{p'}{1 - p'}}{\frac{p'}{1 - p'} + \frac{1}{r}} = \frac{e^z}{e^z + \frac{1}{r}} \ & = \frac{1}{1 + \frac{1}{e^z \cdot r}} = \frac{1}{1 + (e^{z - \log r})^{-1}} \ & = \frac{1}{1 + e^{-(z + \log r)}}. \end{aligned} ]

Thus, prior calibration requires only adding \(\log r\) to the model bias—no additional computation or post-processing needed.

This equivalence reveals that prior calibration preserves ranking order, meaning it does not affect sample ordering and therefore does not change AUC.

Implementation Example

The following demonstration illustrates prior calibration in action. Run this in a Jupyter environment.

(1) Setup

import copy
import torch
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score, roc_auc_score
import matplotlib.pyplot as plt

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using device: {device}')

(2) Creating Synthetic Data

Generate a highly imbalanced dataset simulating real-world conditions where positive are rare.

def generate_synthetic_data(
    n_samples: int,
    class_weights: list[float],
    seed: int = 42
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Binary classification dataset generator using sklearn
    
    :param n_samples: Total sample count
    :param class_weights: Class proportions, e.g., [0.99, 0.01]
    :param seed: Random seed for reproducibility
    """
    X, y = make_classification(
        n_samples=n_samples,
        n_features=20,
        n_informative=10,
        n_redundant=5,
        n_classes=2,
        weights=class_weights,
        flip_y=0.05,
        random_state=seed
    )
    X = torch.tensor(X, dtype=torch.float32)
    y = torch.tensor(y, dtype=torch.int64)
    return X, y


# Generate complete population dataset
X_all, y_all = generate_synthetic_data(n_samples=100000, class_weights=[0.99, 0.01])

# Analyze dataset characteristics
actual_positive_rate = y_all.float().mean().item()
print(f'--- Population Dataset Summary ---')
print(f'Total samples: {len(y_all)}')
print(f'Positive instances: {(y_all == 1).sum().item()}')
print(f'Negative instances: {len(y_all) - y_all.sum().item()}')
print(f'Actual positive rate (true p): {actual_positive_rate:.4f}')

Output:

--- Population Dataset Summary ---
Total samples: 100000
Positive instances: 3445
Negative instances: 96555
Actual positive rate (true p): 0.0344

(3) Performing Negative Downsampling

Apply downsampling to create a training subset with manageable class ratios.

def downsample_negatives(
    features: torch.Tensor,
    labels: torch.Tensor,
    sample_ratio: float
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Downsample negative class (y=0)
    
    :param sample_ratio: Fraction of negatives to retain (r)
    """
    pos_mask = (labels == 1).nonzero(as_tuple=True)[0]
    neg_mask = (labels == 0).nonzero(as_tuple=True)[0]
    n_neg_retained = int(len(neg_mask) * sample_ratio)
    sampled_neg = neg_mask[torch.randperm(len(neg_mask))[:n_neg_retained]]
    combined_indices = torch.cat([pos_mask, sampled_neg])
    shuffled_indices = combined_indices[torch.randperm(len(combined_indices))]
    return features[shuffled_indices], labels[shuffled_indices]


# Downsampling ratio
NEG_DOWNSAMPLE_RATIO = 0.1
# Construct training subset
X_train, y_train = downsample_negatives(X_all, y_all, NEG_DOWNSAMPLE_RATIO)

# Analyze training subset
training_positive_rate = y_train.float().mean().item()
print(f'--- Downsampled Training Set Summary ---')
print(f'Negative sampling ratio (r): {NEG_DOWNSAMPLE_RATIO}')
print(f'Training set size: {len(y_train)}')
print(f'Training positives: {y_train.sum().item():.0f}')
print(f'Training negatives: {len(y_train) - y_train.sum().item()}')
print(f"Training positive rate (distorted p'): {training_positive_rate:.4f}")

Output:

--- Downsampled Training Set Summary ---
Negative sampling ratio (r): 0.1
Training set size: 13100
Training positives: 3445
Training negatives: 9655
Training positive rate (distorted p'): 0.2630

(4) Model Architecture and Training

Train a logistic regression model on the downsampled data.

class BinaryLogisticRegressor(torch.nn.Module):
    def __init__(self, input_dim: int):
        super().__init__()
        self.linear = torch.nn.Linear(input_dim, 1)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.linear(x).squeeze(dim=-1)


def train_classifier(
    model: torch.nn.Module,
    X: torch.Tensor,
    y: torch.Tensor,
    device: torch.device,
    epochs: int,
    learning_rate: float,
):
    """Training loop for binary classification"""
    X, y = X.to(device), y.to(device)
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=1e-3)
    model.train()
    for epoch in range(epochs):
        optimizer.zero_grad()
        logits = model(X)
        loss = torch.nn.functional.binary_cross_entropy_with_logits(logits, y.float())
        loss.backward()
        optimizer.step()
        if (epoch + 1) % 100 == 0:
            print(f'Epoch [{epoch + 1}/{epochs}], Loss: {loss.item():.4f}')


# --- Training Phase ---
model = BinaryLogisticRegressor(input_dim=X_train.size(1))
print('Training on downsampled dataset...')
train_classifier(model, X_train, y_train, epochs=500, learning_rate=0.01, device=device)
print('Training complete!')

Output:

Training on downsampled dataset...
Epoch [100/500], Loss: 0.5467
Epoch [200/500], Loss: 0.5459
Epoch [300/500], Loss: 0.5458
Epoch [400/500], Loss: 0.5457
Epoch [500/500], Loss: 0.5457
Training complete!

(5) Pre-Calibration Assessment

Evaluate the model on full population data. Predicted probabilities will substantially exceed the true positive rate.

def compute_probabilities(model: torch.nn.Module, X: torch.Tensor) -> torch.Tensor:
    """Extract predicted probabilities from model"""
    model.eval()
    with torch.no_grad():
        X = X.to(device)
        logits = model(X)
        probs = torch.sigmoid(logits)
    return probs.cpu()


# Obtain pre-calibration predictions
predictions_before = compute_probabilities(model, X_all)
mean_pred_before = predictions_before.mean()
print('--- Pre-Calibration Results (Full Population) ---')
print(f'Mean predicted probability: {mean_pred_before:.4f}')
print(f'Actual positive rate: {actual_positive_rate:.4f}')
print()
print(f'Accuracy: {accuracy_score(y_all, (predictions_before > 0.5).int()):.4f}')
print(f'AUC: {roc_auc_score(y_all, predictions_before):.4f}')

Output:

--- Pre-Calibration Results (Full Population) ---
Mean predicted probability: 0.2484
Actual positive rate: 0.0344

Accuracy: 0.9655
AUC: 0.6150

The model exhibits severely inflated probabilities due to training on the artificially enriched dataset.

(6) Applying Calibration Correction

# Extract current bias value
original_bias = model.linear.bias.item()
print(f'Original Bias: {original_bias:.4f}')

# Compute calibration adjustment log(r)
calibration_adjustment = torch.log(torch.tensor(NEG_DOWNSAMPLE_RATIO, dtype=torch.float32))
print(f'Calibration term log(r): {calibration_adjustment.item():.4f}')

# Apply calibration to model
calibrated_model = copy.deepcopy(model)
calibrated_model.linear.bias.data += calibration_adjustment.to(device)
print(f'Calibrated Bias: {calibrated_model.linear.bias.item():.4f}')

Output:

Original Bias: -0.5210
Calibration term log(r): -2.3026
Calibrated Bias: -2.8236

(7) Post-Calibration Assessment

Re-evaluate on full population data. The mean prediction should now closely match the true positive rate.

# Obtain post-calibration predictions
predictions_after = compute_probabilities(calibrated_model, X_all)
mean_pred_after = predictions_after.mean()

print('--- Post-Calibration Results (Full Population) ---')
print(f'Mean predicted probability: {mean_pred_after:.4f}')
print(f'Actual positive rate: {actual_positive_rate:.4f}')
print()
print(f'Accuracy: {accuracy_score(y_all, (predictions_after > 0.5).int()):.4f}')
print(f'AUC: {roc_auc_score(y_all, predictions_after):.4f}')

Output:

--- Post-Calibration Results (Full Population) ---
Mean predicted probability: 0.0337
Actual positive rate: 0.0344

Accuracy: 0.9656
AUC: 0.6150

Calibration successfully aligned predictions with reality. Notably, AUC remains unchanged since calibration preserves ranking order.

(8) Visual Comparison

import matplotlib
matplotlib.rcParams['font.sans-serif'] = ['DejaVu Sans']
matplotlib.rcParams['axes.unicode_minus'] = False

plt.figure(figsize=(8, 5), dpi=300)
# Pre-calibration distribution
plt.hist(
    predictions_before,
    color='skyblue',
    label=f'Before (mean={mean_pred_before:.4f})',
    density=True,
    bins=100
)
# Post-calibration distribution
plt.hist(
    predictions_after,
    color='lightgreen',
    label=f'After (mean={mean_pred_after:.4f})',
    density=True,
    bins=100
)
# Ground truth reference line
plt.axvline(
    actual_positive_rate,
    color='red',
    linestyle='--',
    linewidth=2,
    label=f'True Rate ({actual_positive_rate:.4f})'
)

plt.title('Prior Calibration Effect Comparison', fontsize=16)
plt.xlabel('Predicted Probability', fontsize=12)
plt.ylabel('Density', fontsize=12)
plt.legend()
plt.show()

Visual analysis reveals:

• Blue histogram (pre-calibration): Severely right-skewed distribution with mean far exceeding the true rate.
• Green histogram (post-calibration): Distribution shifts left, with mean aligned to the true rate.
• Red dashed line: Represents the target mean probability in the full population.

Prior calibration effectively corrects prediction bias introduced by sample imbalance, producing statistically meaningful probability estimates suitible for decision-making.