Overview

The TNO Multiband Image Collection serves as a foundational resource for developing and evaluating image fusion algorithms in surveillance contexts. Despite sustained research interest in multi-spectral image fusion, few publicly accessible datasets provide registered multi-band imagery, particular dynamic sequences. This collection addresses that gap by offering synchronized visual (390–700 nm), near-infrared (700–1000 nm), and longwave infrared (8–12 μm) nighttime imagery across varied military and surveillance scenarios.

Dataset Composition

The collection comprises three primary subsets:

1. TNO Image Fusion Dataset: Contains static registered pairs of intensified visual, NIR, and LWIR images from diverse outdoor environments including urban and rural settings.
2. Kayak Image Fusion Sequence: Provides time-series data with registered visual, NIR, and LWIR frames capturing approaching vessels in maritime environments, enabling assessment of temopral fusion techniques.
3. TRICLOBS Dynamic Multiband Image Dataset: Features motion sequences combining visual (400–700 nm), NIR (700–1000 nm), and LWIR (8–14 μm) imagery from dynamic urban surveillance situations, supplemented with reference color photographs.

Technical Specifications

Applications

This dataset supports development in:

• Static and dynamic image fusion methodologies
• Color-remapped fusion approaches
• Multispectral object detection and classification systems
• Low-contrast target identification algorithms

Directory Structure Analysis

TRICLOBS Subset

Located within Triclobs_images, this subset organizes scenes into dedicated folders containing wavelength-specific captures:

Triclobs_images/
├── Balls          # Outdoor metallic sphere
├── barbed_wire_1   # Person crossing barrier
├── barbed_wire_2   # Running figure at fence
├── Bosnia         # Rural building scene
├── Farm           # Agricultural structures
├── House          # Thermal signature near lighting
├── houses_with_3_men
├── jeep_in_smoke
├── Kaptein_*      # Nighttime tactical scenarios
├── Marne_*        # Daylight residential captures
├── Movie_*        # Urban street activities
├── pancake_house
├── Reek           # Building facade
├── soldier_behind_smoke
├── soldiers_with_jeep
├── square_with_houses
├── Veluwe         # Telescopic observation
└── Vlasakkers     # Residential architecture

Each folder typically includes:

• VIS.* : Visible spectrum capture
• NIR.* : Near-infrared band
• LWIR.*: Longwave infrared data
• photo.*: Reference photograph where applicable

TNO Primary Subset

Contained in Athena_images, featuring tactical surveillance imagery:

Athena_images/
├── 2_men_in_front_of_house
├── airplane_in_trees
├── APC_*/           # Armored vehicle perspectives
├── bunker
├── heather
├── helicopter
├── lake
├── man_in_doorway
└── soldier_*        # Sniper and trench positions

Files follow naming conventions indicating wavelength and acquisition context.

Kayak Sequence Subset

Organized into continuous frame sequences:

DHV_images/
├── Fire_sequence/   # Pyrotechnic events
│   ├── part_1/      # RAD & DHV frame sets
│   ├── part_2/
│   └── part_3/
├── bench
├── sandpath
└── wall

FEL_images/
├── Duine_sequence   # Road-based activity
├── Nato_camp_sequence
└── Tree_sequence    # Forest environment

Custom Dataset Implementation

from torch.utils.data import Dataset
from torchvision import transforms
from pathlib import Path
from PIL import Image
import zipfile
import hashlib
import requests

class TNOMultibandDataset(Dataset):
    """
    PyTorch-compatible dataset loader for TNO Multiband Image Collection.
    Supports selective loading based on modality and data type.
    """
    
    DOWNLOAD_URL = "https://figshare.com/ndownloader/files/1475454"
    CHECKSUM = "1ab9ce3b84dd3e5894630c13abd20953"
    ARCHIVE_NAME = "tno_dataset.zip"
    
    MODALITY_MAP = {
        'visible': ['VIS', 'vis', 'R'],
        'near_ir': ['NIR', 'nir', 'G'],
        'longwave_ir': ['LWIR', 'IR', 'B']
    }
    
    def __init__(self, 
                 root_path, 
                 modalities=['visible', 'longwave_ir'],
                 split_type='static',
                 transform=None,
                 auto_download=True):
        """
        Initialize dataset handler.
        
        Args:
            root_path (str): Base storage directory
            modalities (list): Required imaging bands
            split_type (str): 'static', 'sequence', or 'all'
            transform (callable): Preprocessing pipeline
            auto_download (bool): Enable automatic fetching
        """
        self.root = Path(root_path)
        self.modalities = modalities
        self.split = split_type
        self.transform = transform
        self.data_index = []
        
        if auto_download and not self._verify_local():
            self._fetch_dataset()
        
        self._build_index()
    
    def _verify_local(self):
        """Check local archive integrity"""
        archive_path = self.root / self.ARCHIVE_NAME
        if not archive_path.exists():
            return False
        
        hash_md5 = hashlib.md5()
        with open(archive_path, "rb") as f:
            for chunk in iter(lambda: f.read(4096), b""):
                hash_md5.update(chunk)
        return hash_md5.hexdigest() == self.CHECKSUM
    
    def _fetch_dataset(self):
        """Download and extract dataset archive"""
        self.root.mkdir(parents=True, exist_ok=True)
        archive_path = self.root / self.ARCHIVE_NAME
        
        response = requests.get(self.DOWNLOAD_URL)
        with open(archive_path, 'wb') as f:
            f.write(response.content)
        
        with zipfile.ZipFile(archive_path, 'r') as zip_ref:
            zip_ref.extractall(self.root)
    
    def _build_index(self):
        """Construct internal sample mapping"""
        # Simplified indexing logic - production version would parse actual directory structure
        base_dirs = [
            self.root / "Triclobs_images",
            self.root / "Athena_images",
            self.root / "DHV_images"
        ]
        
        for base in base_dirs:
            if base.exists():
                for item in base.rglob("*.bmp"):  # Extend to other formats
                    entry = {'path': item}
                    # Parse modality from filename
                    for mod, keys in self.MODALITY_MAP.items():
                        if any(key.lower() in item.stem.lower() for key in keys):
                            entry['modality'] = mod
                            break
                    if 'modality' in entry and entry['modality'] in self.modalities:
                        self.data_index.append(entry)
    
    def __len__(self):
        return len(self.data_index)
    
    def __getitem__(self, idx):
        """Retrieve processed sample tensor"""
        entry = self.data_index[idx]
        image = Image.open(entry['path']).convert('L')
        
        if self.transform:
            image = self.transform(image)
            
        return {
            'data': image,
            'modality': entry.get('modality'),
            'source_file': str(entry['path'])
        }

# Usage example
if __name__ == "__main__":
    # Configure preprocessing
    preprocess = transforms.Compose([
        transforms.Resize((256, 256)),
        transforms.ToTensor()
    ])
    
    # Instantiate dataset
    dataset = TNOMultibandDataset(
        root_path="./datasets/tno",
        modalities=['visible', 'longwave_ir'],
        transform=preprocess
    )
    
    print(f"Available samples: {len(dataset)}")
    
    # Access first record
    sample = dataset[0]
    print(f"Sample modality: {sample['modality']}")
    print(f"Tensor shape: {sample['data'].shape}")

Practical Considerations

When working with this dataset:

1. Naming inconsistencies: File naming schemes vary significantly between subcollections requiring careful parsing logic
2. Quality variations: Some infrared images are stored in incorrect color spaces necessitating post-processing corrections
3. Resolution mismatches: Frame dimensions differ across capture devices requiring standardization
4. Temporal alignment: Sequential data requires synchronization verification before use in time-sensitive applications

These characteristics make the TNO collection valuable for benchmarking robust preprocessing pipelines alongside fusion algorithm development.