Introduction

Bird's Eye View (BEV) representation is critical for autonomous driving perception tasks. Two primary approaches exist: explicit depth estimation methods (top-down) and transformer-based BEV query techniques (bottom-up). The Lift-Splat-Shoot (LSS) framework provides an end-to-end solutoin for multi-sensor fusion by eliminating post-processing steps and enabling direct feature transformation.

Lift Operation

Key perception parameters define the operational space:

img_height = 128
img_width = 352
x_range = [-50.0, 50.0, 0.5]
y_range = [-50.0, 50.0, 0.5]
z_range = [-10.0, 10.0, 20.0]
depth_bins = [4.0, 45.0, 1.0]
feat_height = img_height // 16
feat_width = img_width // 16

Frustum Construction

A 3D viewing volume is created using discrete depth values:

def generate_view_volume(self):
    depth_values = torch.arange(*self.config['depth_bins'], dtype=torch.float).view(-1, 1, 1)
    depth_values = depth_values.expand(-1, self.feat_h, self.feat_w)
    
    x_coords = torch.linspace(0, self.orig_width-1, self.feat_w, dtype=torch.float)
    x_coords = x_coords.view(1, 1, -1).expand(depth_values.shape[0], self.feat_h, self.feat_w)
    
    y_coords = torch.linspace(0, self.orig_height-1, self.feat_h, dtype=torch.float)
    y_coords = y_coords.view(1, -1, 1).expand(depth_values.shape[0], self.feat_h, self.feat_w)
    
    volume = torch.stack((x_coords, y_coords, depth_values), -1)
    return volume

Camera Feature Encoding

Image features are extracted and combined with depth distributions:

class CameraEncoder(nn.Module):
    def __init__(self, depth_bins, feat_channels, reduction_factor):
        super().__init__()
        self.depth_bins = depth_bins
        self.feat_channels = feat_channels
        
        self.backbone = EfficientNet.from_pretrained("efficientnet-b0")
        self.upsample = Up(320+112, 512)
        self.feature_net = nn.Conv2d(512, depth_bins + feat_channels, kernel_size=1)
    
    def get_depth_weights(self, x):
        return x.softmax(dim=1)
    
    def forward(self, x):
        base_features = self.backbone(x)
        features = self.upsample(base_features)
        output = self.feature_net(features)
        
        depth_weights = self.get_depth_weights(output[:, :self.depth_bins])
        spatial_features = depth_weights.unsqueeze(1) * output[:, self.depth_bins:].unsqueeze(2)
        return spatial_features

Splat Operation

Coordinate Transformation

Image coordinates are projected into 3D ego space:

def project_to_ego(self, rotations, translations, intrinsics, post_rots, post_trans):
    batch_size, num_cams, _ = translations.shape
    adjusted_points = self.view_volume - post_trans.view(batch_size, num_cams, 1, 1, 1, 3)
    
    inverse_rots = torch.inverse(post_rots).view(batch_size, num_cams, 1, 1, 1, 3, 3)
    adjusted_points = inverse_rots.matmul(adjusted_points.unsqueeze(-1)).squeeze(-1)
    
    combined_transform = rotations.matmul(torch.inverse(intrinsics))
    ego_points = combined_transform.view(batch_size, num_cams, 1, 1, 1, 3, 3).matmul(adjusted_points.unsqueeze(-1))
    ego_points = ego_points.squeeze(-1) + translations.view(batch_size, num_cams, 1, 1, 1, 3)
    
    return ego_points

Feature Pooling

Features are aggregated into voxel grids:

def pool_features(self, ego_coords, features):
    batch_size, num_cams, depth_bins, feat_h, feat_w, channels = features.shape
    total_points = batch_size * num_cams * depth_bins * feat_h * feat_w
    
    flat_features = features.view(total_points, channels)
    voxel_coords = ((ego_coords - (self.voxel_origin - self.voxel_size/2)) / self.voxel_size).long()
    flat_coords = voxel_coords.view(total_points, 3)
    
    batch_indices = torch.cat([torch.full([total_points//batch_size, 1], i) for i in range(batch_size)])
    combined_coords = torch.cat((flat_coords, batch_indices), 1)
    
    valid_mask = (combined_coords[:, 0] >= 0) & (combined_coords[:, 0] < self.grid_size[0]) & \
                 (combined_coords[:, 1] >= 0) & (combined_coords[:, 1] < self.grid_size[1]) & \
                 (combined_coords[:, 2] >= 0) & (combined_coords[:, 2] < self.grid_size[2])
    
    filtered_features = flat_features[valid_mask]
    filtered_coords = combined_coords[valid_mask]
    
    sorting_key = filtered_coords[:, 0] * (self.grid_size[1] * self.grid_size[2] * batch_size) + \
                  filtered_coords[:, 1] * (self.grid_size[2] * batch_size) + \
                  filtered_coords[:, 2] * batch_size + \
                  filtered_coords[:, 3]
    
    sorted_indices = sorting_key.argsort()
    sorted_features = filtered_features[sorted_indices]
    sorted_coords = filtered_coords[sorted_indices]
    
    pooled_voxels = torch.zeros((batch_size, channels, self.grid_size[2], self.grid_size[0], self.grid_size[1]))
    pooled_voxels[sorted_coords[:, 3], :, sorted_coords[:, 2], sorted_coords[:, 0], sorted_coords[:, 1]] = sorted_features
    
    return pooled_voxels

BEV Feature Processing

The final BEV features are processed through an encoder network:

class BEVEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(64, 64, kernel_size=7, stride=2, padding=3)
        self.norm1 = nn.BatchNorm2d(64)
        self.activation = nn.ReLU()
        
        self.block1 = ResidualBlock(64, 64)
        self.block2 = ResidualBlock(64, 128, stride=2)
        self.block3 = ResidualBlock(128, 256, stride=2)
        
        self.upsample1 = UpsampleBlock(256, 64)
        self.upsample2 = UpsampleBlock(64, 1)
    
    def forward(self, x):
        x = self.activation(self.norm1(self.conv1(x)))
        x1 = self.block1(x)
        x = self.block2(x1)
        x = self.block3(x)
        x = self.upsample1(x, x1)
        return self.upsample2(x)

Advantages and Limitations

LSS provides a unified approach for multi-sensor BEV feature extraction but has notable dependencies:

• Advantages: Enables integrated perception tasks in BEV space and supports multi-modal sensor fusion
• Limitations: Requires accurate depth estimation and has significant computational requirements