Environment Configuration & Dependencies

Successful deployment requires a specific stack of machine learning, graph processing, and database libraries. The following Python packages and versions were validated for compatibility:

psycopg2-binary==2.9.9
tqdm==4.66.4
scikit-learn==1.5.0
networkx==3.3
xxhash==3.4.1
graphviz==0.20.3
pytz==2024.1
torch==2.3.0+cu121
torchvision==0.18.0+cu121
torch_geometric==2.5.3
pyg_lib==0.4.0+pt23cu121
torch_scatter==2.1.2+pt23cu121
torch_sparse==0.6.18+pt23cu121
torch_cluster==1.6.3+pt23cu121
torch_spline_conv==1.2.2+pt23cu121

Supported evaluation datasets include CADETS (E3/E5), THEIA (E3/E5), ClearScope (E3/E5), and OpTC.

Relational Database Schema Initialization

PostgreSQL serves as the backend for storing provenance events and entity mappings. Initialize the service and configure the schema as follows:

sudo systemctl status postgresql
sudo systemctl start postgresql
sudo -i -u postgres
psql

Execute the following SQL commands to provision the database and define the relational structure:

-- Provision database
CREATE DATABASE kairos_provenance_db;
\c kairos_provenance_db;

-- Event logging table
CREATE TABLE system_events (
    source_entity   VARCHAR,
    src_ref_id      VARCHAR,
    action_type     VARCHAR,
    target_entity   VARCHAR,
    dst_ref_id      VARCHAR,
    event_epoch     BIGINT,
    record_id       SERIAL PRIMARY KEY
);
ALTER TABLE system_events OWNER TO postgres;
CREATE UNIQUE INDEX idx_event_record ON system_events(record_id);
GRANT ALL PRIVILEGES ON system_events TO postgres;

-- File entity registry
CREATE TABLE file_assets (
    entity_uuid VARCHAR NOT NULL,
    digest_id   VARCHAR NOT NULL,
    file_path   VARCHAR,
    CONSTRAINT pk_file_assets PRIMARY KEY (entity_uuid, digest_id)
);
ALTER TABLE file_assets OWNER TO postgres;

-- Network connection tracking
CREATE TABLE network_connections (
    entity_uuid VARCHAR NOT NULL,
    digest_id   VARCHAR NOT NULL,
    origin_ip   VARCHAR,
    origin_port VARCHAR,
    dest_ip     VARCHAR,
    dest_port   VARCHAR,
    CONSTRAINT pk_net_connections PRIMARY KEY (entity_uuid, digest_id)
);
ALTER TABLE network_connections OWNER TO postgres;

-- Process/Subject tracking
CREATE TABLE process_subjects (
    entity_uuid VARCHAR,
    digest_id   VARCHAR,
    exec_path   VARCHAR
);
ALTER TABLE process_subjects OWNER TO postgres;

-- Global entity mapping
CREATE TABLE entity_registry (
    digest_id   VARCHAR PRIMARY KEY,
    class_type  VARCHAR,
    raw_payload VARCHAR,
    seq_index   BIGINT
);
ALTER TABLE entity_registry OWNER TO postgres;
CREATE UNIQUE INDEX idx_registry_digest ON entity_registry(digest_id);

Temporal Graph Learning Pipeline

Directory & Configuration Setup

Prepare the workspace and adjust connection parameters in the configuration module:

cd DARPA/CADETS_E3/
mkdir -p ./artifacts/
sudo systemctl start postgresql

Update config.py to point to the raw dataset directory and database credentials:

RAW_DATA_DIR = '/path/to/cadets_e3_raw'
DB_HOST = '127.0.0.1'
DB_CREDENTIALS = 'your_secure_password'

Feature Hashing Adjustment

The default feature generation routine may raise a type mismatch when FeatureHasher receives a flat string instead of an iterbale sequence. Modify the vectorization step in gen_feature:

# Original: vec = FH_string.transform([i]).toarray()
# Corrected:
vec = FH_string.transform([[i]]).toarray()

Training Loop Adaptation

PyTorch Geometric 2.5.3 deprecated the native seq_batches iterator. Implement a custom chunking generator to feed temporal slices into the model:

class TemporalSlice:
    def __init__(self, src, dst, ts, feats):
        self.src = src
        self.dst = dst
        self.t = ts
        self.msg = feats
        self.num_events = len(src)

def yield_temporal_chunks(graph_obj, chunk_sz):
    total = graph_obj.num_events
    for start in range(0, total, chunk_sz):
        end = min(start + chunk_sz, total)
        yield TemporalSlice(
            src=graph_obj.src[start:end],
            dst=graph_obj.dst[start:end],
            t=graph_obj.t[start:end],
            feats=graph_obj.msg[start:end]
        )

# Usage in training loop:
for mini_batch in yield_temporal_chunks(training_graph, chunk_sz=BATCH_SIZE):
    # forward pass & optimization
    pass

Inference Stream Handling

Direct execution of the evaluation script triggers an AttributeError due to missing sequence batching methods. Replace the deprecated call with TemporalDataLoader:

# Replace: for batch in eval_data.seq_batches(batch_size=BATCH):
stream_loader = TemporalDataLoader(dataset=eval_data, batch_size=BATCH_SIZE)
for eval_batch in stream_loader:
    # reconstruction & scoring
    pass

Anomaly Detection & Evaluation Workflow

Generate the anomaly queue and compute detection metrics:

python build_anomaly_queue.py
python compute_metrics.py

The evaluation script processes reconstruction errors across temporal windows, outputting precision, recalll, and threshold calibration logs.

Attack Investigation & Graph Visualization

Dependency resolution is required before running the investigation module:

pip install python-louvain
sudo apt-get install graphviz
dot -V
python run_investigation.py

The script applies Louvain community detection to partition the provenance graph, filters edges based on dynamic reconstruction thresholds, and exports a condensed attack summary graph for manual inspection.

Experimental Benchmarking & Analysis

The framework leverages scikit-learn for hierarchical feature hashing, PyG for temporal graph representation learning, and NetworkX for community detection. GraphViz handles final visualization. All benchmarks were executed on a CentOS 7.9 server equipped with a 20-core Intel Xeon Silver 4210 (2.20 GHz) and 64 GB RAM.

Dataset Configuration

Experiments utilize DARPA TC (E3/E5), OpTC, and the Manzoor et al. provenance datasets.

• Manzoor et al.: Contains SystemTap-captured provenance graphs across six operational scenarios. One benign graph per scenario trains the model, 24 validate thresholds, and the remainder (75 benign + 100 attack) form the test set. Enables direct comparison with Unicorn and ThreaTrace, though fine-grained attack ground truth is unavailable for reconstruction validation.
• DARPA TC & OpTC: E3/E5 provide public provenance traces. OpTC spans 500 Windows hosts over 7 days of benign activity plus 3 days of mixed APT operations. Six hosts are randomly sampled: one day trains, one validates, and all three attack days test. DARPA provides node/edge-level ground truth, enabling precise reconstruction verification and demonstrating scalability in enterprise networks.

Detection Methodology & Performance

Models train exclusively on benign traces. During testing, data streams are replayed to simulate runtime monitoring. Metrics are computed per temporal window. Windows containing attack activity correctly flagged count as True Positives (TP); benign windows incorrectly flagged count as False Positives (FP).

The system achieves 100% recall across all attack scenarios. False positives primarily stem from two sources: lingering high reconstruction errors on edges influenced by prior attacker activity, and concept drift caused by newly introduced legitimate applications whose behavioral patterns deviate from the training distribution. To mitigate drift, the pipeline supports incremental retraining using only FP windows, allowing the model to adapt to evolving benign baselines without full retraining.

Baseline Comparisons

• Unicorn: Models system behavior via incrementally updated graph sketches. Classifies entire graphs rather than temporal windows. When adapted to window-level evaluation, Unicorn generates higher FP rates and lacks localized attack footprint extraction, forcing analysts to manually backtrack through massive graphs.
• ThreaTrace: Builds per-node-type models to flag anomalous entities. Evaluation converts node-level alerts to graph-level flags using thresholding. While node-level detection offers some insight, it fails to reconstruct cohesive attack narratives and becomes impractical on large graphs with high FP node counts.
• StreamSpot & Frappuccino: Prior studies and internal TC dataset evaluations confirm StreamSpot fails to trigger alerts on modern TC traces. Consequently, these are excluded from detailed comparison.

Hyperparameter Sensitivity

• Node Embedding Dimension (|Φ|): Set to 16. Larger dimensions induce sparsity and memory bloat without improving detection.
• Node State Dimension (|s(v)|): Set to 100. Balances historical context retention against outdated noise and memory consumption.
• Neighborhood Sample Size (|N|): Set to 20. Optimally captures structural roles across diverse datasets.
• Edge Embedding Dimension (|z|): Set to 200. Preserves sufficient temporal-structural information for the decoder while maintaining generalization.
• Temporal Window Length (|tw|): Set to 15 minutes. Longer windows (60m) dilute attack signals with benign noise, increasing false negatives. Shorter windows (5m) restrict contextual accuracy. Streaming architecture ensures memory overhead remains stable regardless of window size.

Attack Graph Reduction & Throughput

A core objective is condensing massive provenance traces into actionable summaries. The pipeline achieves reduction factors up to five orders of magnitude. For example, on E3-Theia, edge counts drop from 3.4 million in flagged windows to just 31 in the final summary graph.

Processing latency remains well within real-time constraints. On E3-Theia, a 15-minute window containing ~2.5 million edges processes in 228.8 seconds (25.4% of the window duration). While raw edge throughput (~11K edges/sec) slightly trails StreamSpot (~14K edges/sec), the detection accuracy and reconstruction capabilities are substantially superior. Given that production provenance collectors typically generate <10K edges/sec under heavy load, the system operates comfortably within streaming boundaries without falling behind.