Index & Column Management

Direct attribute access via .index and .columns yields Index objects that behave like mutable lists. Labels can be reassigned directly, for example: df.index = [f'entry_{i}' for i in range(len(df))]. Dynamic column filtering avoids manual enumeration:

filtered_df = df[[col for col in df.columns if col not in exclude_list]]

Note that .values returns a NumPy view; attempting direct assignment triggers an error. Use .shape, .size, and .dtypes to inspect structure and memory layout. The .empty attribute safely validates whether the frame contains any records.

Display Configuration

Control output truncation during exploratory phases:

pd.options.display.max_columns = 50
pd.options.display.width = 120
pd.options.display.max_colwidth = 40
pd.options.display.unicode.east_asian_width = True

Data Inspection Methods

.info() provides a concise memory and schema overview. .describe() computes statistical summaries for numeric types. Specify include='object' for categorical strings or include='all' to cover mixed dtypes. Output includes percentiles, central tendency, and dispersion metrics.

Type Casting & Memory Optimizaton

String dtype aliases map directly to NumPy equivalents ('i1' → np.int8, 'i8' → np.int64). Iterate over columns rather than hardcoding conversions:

type_map = {'numeric': float, 'text': str, 'default': int}
for col in target_df.columns:
    if col in numeric_fields:
        target_df[col] = target_df[col].astype(type_map['numeric'])
    elif col in string_fields:
        target_df[col] = target_df[col].astype(type_map['text'])
    else:
        target_df[col] = target_df[col].astype(type_map['default'])

Finding Min/Max Indices

.idxmin() and .idxmax() locate row labels corresponding to extreme values:

lowest_point = metrics_df['revenue'].idxmin()
highest_values = metrics_df.idxmax() 

Sorting Strategies

Order data by index labels or column values. Handle mutability carefully to avoid chained assignment warnings:

sorted_asc = raw_data.sort_index(ascending=True)
sorted_desc = raw_data.sort_values(by=['score', 'id'], ascending=[False, True])
# Preferred pattern: reassign after sorting to bypass SettingWithCopyWarning
processed = raw_data.sort_values(...).reset_index(drop=True)

Frequency Analysis & Unique Values

.value_counts() operates exclusively on Series. Combine with groupby for ratio calculations:

freq_series = dataset['category'].value_counts()
group_sums = dataset.groupby('category')['target_val'].sum()
proportions = round(group_sums / freq_series, 2)

For element counts, .nunique(dropna=True) ignores nulls by default. Set dropna=False to include them. Remove constant columns efficiently:

dataset.loc[:, dataset.nunique() > 1]

Reindexing & Renaming Operations

.reindex() aligns data to a new axis specification, inserting NaNs for missing keys. .rename() applies dictionary mappings. These methods return new objects unless inplace=True is specified:

aligned_df = base_df.reindex(index=[0, 5, 12], columns=['X', 'Y'])
renamed_df = base_df.rename(columns={'old_x': 'X_axis'}, index={0: 'start'})
clean_order = sorted_df.reset_index(drop=True)

Eliminating Rows and Columns

Drop columns using .drop(labels, axis=1), del df['label'], or .pop('label'). Conditionally remove rows:

cleaned = original.drop(original[original['flag'] == 0].index)
unique_records = master_df.drop_duplicates(subset=['user_id'], keep='first')
dup_stats = master_df['session_id'].duplicated().value_counts()

Boolean Masking & Filtering

Boolean indexing requires aligned indices between the mask and the target frame. Wrap multiple conditions in parentheses and use bitwise operators &, |, ~:

filtered = data[(data['age'] >= 18) & (~data['status'].isin(['inactive']))]
valid_range = data[data['timestamp'].between(start_dt, end_dt)]

Applying boolean logic directly to the entire DataFrame replaces non-matching cells with NaN rather than filtering rows.

Pattern Extraction from Strings

.str.extract() captures regex groups into separate columns. Prefix patterns with r'' to prevent escape character interpretation:

extracted = records['description'].str.extract(r'(\d{4})\s([A-Z]+)', expand=True)

Missing Value Resolution

Identify gaps via .isnull().sum(). Imputation depends on distribution characteristics:

• Forward/backward fill: .fillna(method='ffill')
• Statistical fill: .fillna(dataset['col'].median())
• KNN imputation requires external modules:

from sklearn.impute import KNNImputer
imputer = KNNImputer(n_neighbors=5)
filled_array = imputer.fit_transform(dataset.select_dtypes('number'))
dataset_numeric = pd.DataFrame(filled_array, columns=dataset.select_dtypes('number').columns)

Sequential metrics may use Lagrange interpolation, though sparse series require careful validation.

Outlier Detection

Interquartile Range (IQR) clipping isolates extremes:

Q1, Q3 = col.quantile(0.25), col.quantile(0.75)
IQR = Q3 - Q1
bounds = (Q1 - 1.5*IQR, Q3 + 1.5*IQR)
clipped = col.clip(lower=bounds[0], upper=bounds[1])

Correlation & Deep Copying

Pearson correlation uses .corr(). Interpretation thresholds: |r|<0.3 (weak), 0.3–0.5 (moderate), 0.5–0.8 (strong), >0.8 (very strong). Prevent shallow reference bugs by cloning frames: df.copy(deep=True).

Functional Mapping & Transformation

.pipe() chains DataFrame operations. .apply() targets rows (axis=1) or columns. .map() processes individual Series elements:

def scale_by_base(val, multiplier):
    return val * multiplier
scaled = base_df.pipe(scale_by_base, multiplier=100)
category_mapped = base_df['region'].map(lambda x: 'Zone_' + str(x))

GroupBy Aggregations

Combine .groupby() with .agg() for multi-step summaries. Dictionary syntax assigns distinct functions per column:

summary = transactions.groupby('customer_id').agg({
    'amount': np.sum,
    'count': 'size',
    'rating': lambda x: x.mean()
})
grouped_flat = transactions.groupby('id', as_index=False)['val'].mean()

Pivot Tables & Cross Tabulations

.pivot_table() reshapes hierarchically. Use fill_value=0 to handle empty combinations. pd.crosstab() generates frequency matrices for categorical pairs:

pivot = pd.pivot_table(records, index='quarter', columns='segment', values='revenue', aggfunc=np.mean, fill_value=0)
freq_matrix = pd.crosstab(records['source'], records['channel'])

Encoding & Binning

Binary mapping: df['label'] = df['raw'].map({'A': 0, 'B': 1}). Multi-class encoding: pd.get_dummies(df['category'], prefix='cat'). Interval discretization relies on pd.cut() (fixed ranges) or pd.qcut() (equal-frequency quantile bins).

Merging & Concatenation

pd.merge() joins on shared keys with explicit join strategies (inner, left, right, outer). Apply suffixes to resolve overlapping names. Stack frames vertically/horizontally via pd.concat():

combined = pd.concat([frame_a, frame_b], ignore_index=True, sort=False)
joined = pd.merge(master, reference, on='ref_key', how='left')

I/O Parsing Nuances

read_csv() accepts explicit parameters for robust loading:

loaded = pd.read_csv('source.csv', index_col='id', dtype={'price': np.float64}, parse_dates=['log_date'])
raw_data = pd.read_csv('data.txt', engine='python', header=None, names=['col1', 'col2'])

Temporal Data Handling

Generate date sequences with pd.date_range(). Extract components via the .dt accessor: series.dt.year, series.dt.month. Calculate differences using .dt.total_seconds() or division by pd.Timedelta:

deltas_days = (end_times - start_times).dt.days
duration_hours = (end_times - start_times) / pd.Timedelta(hours=1)
parsed_ts = pd.to_datetime('2023-Q1', format='%Y-%Q')

DataFrame Constructor Syntax

Positional arguments follow (data, index, columns) order. Explicit keyword naming improves readability and prevents parameter shifting errors during refactoring.