Environment Setup

Begin by cloning the repository from GitHub - zhengxuJosh/DATR. The experimental environment mirrors that of Trans4PASS, with configuration details available in the previous article. Note that this project requires Segformer's codebase for auxiliary experiments. The Segformer implementation can be found at GitHub - NVlabs/SegFormer: Official PyTorch implementation of SegFormer.

Dataset Preparation

Cityscapes Dataset

Cityscapes provides high-quality dense pixel annotations where overlayed colors represent semantic classes. Individual instances of traffic participants are annotated separately.

SynPASS Dataset

SynPASS contains 9080 panoramic images (1024x2048) across 22 categories. Scenes encompass cloudy, foggy, rainy, sunny, and day/night conditions.

DensePASS Dataset

Organized as follows:

datasets/
|--- cityscapes
|   |___ gtfine
|   |___ leftImg8bit
|--- SynPASS
|   |--- img
|   |   |___ cloud
|   |   |___ fog
|   |   |___ rain
|   |   |___ sun
|--- semantic
|   |___ cloud
|   |___ fog
|   |___ rain
|   |___ sun
|--- DensePASS
|   |___ gtfine
|   |___ leftImg8bit

Paper Analysis

Title and Approach

The paper titled "Look at the Neighbor: Distortion-aware Unsupervised Domain Adaptation for Panoramic Semantic Segmentation" presents a method addressing limitations in existing approaches for panoramic semantic segmentation under domain adaptation. It focuses on the distortion introduced by Equirectangular Projection (ERP), which maps spherical images onto flat surfaces, causing geometric inconsistencies.

Abstract

Prior works suffer from poor generalization due to ERP projection artifacts. This study proposes a distortion-aware attention mechanism (DA) that captures neighboring pixel distributions without geometric constraints, alongside a class feature aggregation (CFA) module that iteratively updates feature representations through memory banks. These improvements achieve state-of-the-art performance with 80% fewer parameters compared to previous methods.

Introduction

ERP introduces distortions because spherical distances differ from planar projections. By reducing receptive fields, the approach efficiently addresses these distortions. Controlling neighborhood size becomes crucial for balancing receptive field and distortion trade-offs.

Related Work

UDA methods fall into three categories: adversarial learning, pseudo-label generation, and feature prototype adaptation. Unlike multi-stage strategies, this work introduces CFA to aggregate and iteratively update class prototypes, enhancing domain-specific representation.

ERP distortion mitigation has been addressed through deformable kernels and adaptive CNNs. However, these often require significant computational resources. In contrast, this approach leverages ERP neighbor relationships to reduce distortion with minimal parameters.

Self-attention mechanisms have been used for long-range dependencies in 360° imagery, but they struggle with ERP-induced distortions. This work instead focuses on local neighborhoods using DA modules to capture inter-domain pixel distribution differences.

Methodology

ERP Distortion Analysis

ERP mapping causes uneven pixel spacing, leading to feature extraction inaccuracies. The distortion coefficient measures deviation from uniform distribution, suggesting that horizontal disparities are more significant than vertical ones—potential for further refinement.

Distortion-Aware UDA Framework

Integrates positional encoding within attention mechanisms to constrain errors. The core idea involves minimizing segmentation loss while optimizing target domain features through iterative CFA processes.

Code Structure

Dataset Handling

The dataset folder contains preprocessing utilities for handling panoramic images. The eq2tangent function transforms 360-degree images into equi-angular patches for consistent spatial processing.

Metrics Module

Computes evaluation metrics like mIoU. No special modifications required beyond standard implementations.

Model Architecture

DATR Network

Definnes a segmentation network combining backbone encoders (DATRM, DATRT, DATRS) with a SegFormer decoder. Pre-trained weights from SegFormer are loaded to initialize the encoder components.

import torch
import torch.nn as nn
import torch.nn.functional as F 
from .decoder import SegFormerHead
from .encoder import DATRM, DATRT, DATRS

class DATR(nn.Module):
    def __init__(self, backbone, num_classes=20, embedding_dim=256, pretrained=None):
        super().__init__()
        self.num_classes = num_classes
        self.embedding_dim = embedding_dim
        self.feature_strides = [4, 8, 16, 32]

        if backbone == 'DATRM':
            self.encoder = DATRM()
            if pretrained:
                state_dict = torch.load('/hpc/users/CONNECT/tpan695/DATR/models/ptmodel/mit_b0.pth')
                state_dict.pop('head.weight')
                state_dict.pop('head.bias')
                self.encoder.load_state_dict(state_dict,strict=False)
        elif backbone == 'DATRT':
            self.encoder = DATRT()
            if pretrained:
                state_dict = torch.load('/hpc/users/CONNECT/tpan695/DATR/models/ptmodel/mit_b1.pth')
                state_dict.pop('head.weight')
                state_dict.pop('head.bias')
                self.encoder.load_state_dict(state_dict,strict=False)
        elif backbone == 'DATRS':
            self.encoder = DATRS()
            if pretrained:
                state_dict = torch.load('/hpc/users/CONNECT/tpan695/DATR/models/ptmodel/mit_b2.pth')
                state_dict.pop('head.weight')
                state_dict.pop('head.bias')
                self.encoder.load_state_dict(state_dict,strict=False)
        self.in_channels = self.encoder.embed_dims

        self.backbone = backbone
        self.decoder = SegFormerHead(feature_strides=self.feature_strides, in_channels=self.in_channels, embedding_dim=self.embedding_dim, num_classes=self.num_classes)
        
        self.classifier = nn.Conv2d(in_channels=self.in_channels[-1], out_channels=self.num_classes, kernel_size=1, bias=False)

    def _forward_cam(self, x):
        cam = F.conv2d(x, self.classifier.weight)
        cam = F.relu(cam)
        return cam

    def get_param_groups(self):
        param_groups = [[], [], []]
        for name, param in list(self.encoder.named_parameters()):
            if "norm" in name:
                param_groups[1].append(param)
            else:
                param_groups[0].append(param)
        for param in list(self.decoder.parameters()):
            param_groups[2].append(param)
        param_groups[2].append(self.classifier.weight)
        return param_groups

    def forward(self, x):
        _, _, height, width = x.shape
        _x = self.encoder(x)
        feature =  self.decoder(_x)
        pred = F.interpolate(feature, size=(height,width), mode='bilinear', align_corners=False)
        return pred, _x

Encoder Implementation

Uses Neighborhood Attention (NATTEN) exclusively at the highest embedding dimension. Other components follow standard conventions similar to Trans4PASS.

Feature Aggregation Function

import torch.nn as nn
import torch.nn.functional as F

def IFV(feat_1, feat_2, target0, target1, cen_bank_1, cen_bank_2, epoch):
    size_f = (feat_1.shape[2], feat_1.shape[3])
    tar_feat_0 = nn.Upsample(size_f, mode='nearest')(target1.unsqueeze(1).float()).expand(feat_1.size())
    tar_feat_1 = nn.Upsample(size_f, mode='nearest')(target0.unsqueeze(1).float()).expand(feat_2.size())
    center_feat_S = feat_1.clone()
    center_feat_T = feat_2.clone()
    for i in range(19):
        mask_feat_0 = (tar_feat_0 == i).float()
        mask_feat_1 = (tar_feat_1 == i).float()
        center_feat_S = (1 - mask_feat_0) * center_feat_S + mask_feat_0 * ((mask_feat_0 * feat_1).sum(-1).sum(-1) / (mask_feat_0.sum(-1).sum(-1) + 1e-6)).unsqueeze(-1).unsqueeze(-1)
        center_feat_T = (1 - mask_feat_1) * center_feat_T + mask_feat_1 * ((mask_feat_1 * feat_2).sum(-1).sum(-1) / (mask_feat_1.sum(-1).sum(-1) + 1e-6)).unsqueeze(-1).unsqueeze(-1)

    center_feat_S = ((1 - 1 / (epoch + 1)) * cen_bank_1 + center_feat_S * 1 / (epoch + 1)) * 0.5
    center_feat_T = ((1 - 1 / (epoch + 1)) * cen_bank_2 + center_feat_T * 1 / (epoch + 1)) * 0.5
    
    cos = nn.CosineSimilarity(dim=1)
    pcsim_feat_S = cos(feat_1, center_feat_S)
    pcsim_feat_T = cos(feat_2, center_feat_T)

    mse = nn.MSELoss()
    loss = mse(pcsim_feat_S, pcsim_feat_T)
    return loss, center_feat_S, center_feat_T

Learning Rate Adjustment

import torch.optim as optim

def adjust_learning_rate_poly(optimizer, epoch, num_epochs, base_lr, power):
    lr = base_lr * (1-epoch/num_epochs)**power
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
    return lr

Training Script

The training script implements domain adaptation via pseudo-labeling and feature alignment. It initializes models with SegFormer pre-trained weights and performs iterative optimization using both supervised and unsupervised losses.

import torch
import torch.nn as nn
import torch.optim as optim
from models.segformer.segformer import Seg

# Initialize model
args.backbone = 'b1'
model1 = Seg(num_classes=13, phi=args.backbone, pretrained=True)
model_path = "/home/yaowen-chang/DATR/DATR-main/models/segformer/backbone/mit_b1.pth"
model1.load_state_dict(torch.load(model_path, map_location=torch.device("cpu")), strict=False)
model1 = model1.to(device)

# Define loss and optimizer
optimizer1 = optim.AdamW(model1.parameters(), lr=args.lr, weight_decay=0.0001)
criterion_sup = nn.CrossEntropyLoss(reduction='mean', ignore_index=255)

# Training loop
for it in range(1, args.iterations + 1):
    # Load batches from both datasets
    s_img, s_gt, _, _ = next(syn_sup_loader)
    p_img, p_gt, _, _ = next(pass_img_loader)
    
    s_img, s_gt = s_img.to(device), s_gt.to(device)
    p_img, p_gt = p_img.to(device), p_gt.to(device)
    
    # Forward pass
    syn_pred, syn_feat = model1(s_img)
    pass_pred, pass_feat = model1(p_img)
    
    # Compute losses
    loss_sup_1 = criterion_sup(syn_pred, s_gt)
    loss_sup_2 = criterion_sup(pass_pred, p_gt)
    loss_fa_3, cen_1, cen_2 = IFV(pass_feat[3], syn_feat[3], p_gt, s_gt, cen_bank_1, cen_bank_2, epoch)
    
    loss_1 = loss_sup_1 + loss_sup_2 + loss_fa_3
    
    # Update parameters
    optimizer1.zero_grad()
    loss_1.backward()
    optimizer1.step()
    
    # Adjust learning rate
    if it <= 1500:
        lr_ = base_lr * (it / 1500)
        for param_group in optimizer1.param_groups:
            param_group['lr'] = lr_
    else:
        lr_ = adjust_learning_rate_poly(optimizer1, it - 1500, it, args.lr, 1)
    
    # Validation
    if it % pass_length == 0 or it == 1:
        print(f'iter:{it};Model1 Total loss: {loss_1:.4f}')
        model1.eval()
        validation(num_classes, NUM_CLASSES, NAME_CLASSES, device, testloader, model1,
                  best_performance_dp, save_path, epoch, 'densepass')
        epoch += 1
        model1.train()

The experiment requires additional setup steps including proper installation of NATTEN library and preparation of pseudo-labels for target domain training.