Automated Framework Design for Lightweight Visual Models The transition from manually engineered convolutional architectures to algorithmically discovered topologies has accelerated through Neural Architecture Search (NAS) methodologies. The FBNet lineage demonstrates a progressive refinement of this paradigm, shifting from pure macro-structure discovery to multi-dimensional constraint optimization and ultimately to joint topology-training pipeline co-design. Each iteration addresses specific bottlenecks in computational budget, memory footprint, and hardware deployment efficiency.

V1: Layer-Wise Differentiable Macro Search Unlike cell-centric approaches that treat architecture design as a graph of reusable modules, the initial iteration focuses on macro-architectural configuration. The search space remains fixed across most layers, with targeted variability introduced at specific depths. This layer-wise selection mechanism allows distinct operational blocks to be assigned per stage while preserving overall connectivity patterns. The core challenge lies in optimizing discrete structural choices without resorting to exhaustive enumeration. To enable gradient-based optimization, the candidate block distribution is modeled as a stochastic mixture. Sampling probabilities are governed by learnable routing parameters $\theta_l$. Rather than applying hard selections that break differentiability, a continuous relaxation is achieved through Gumbel-Softmax tempering: $m_{l,i} = \frac{\exp((\theta_{l,i} + g_{l,i})/\tau)}{\sum_j \exp((\theta_{l,j} + g_{l,j})/\tau)}$ where $g_{l,i}$ represents independent noise drawn from a Gumbel$(0,1)$ distribution and $\tau$ controls the sharpness of the approximation. During training, $\tau$ decays toward zero, gradually enforcing sparse, one-hot-like selections. Hardware efficiency is explicitly incorporated into the objective function. Instead of relying solely on accuracy metrics, the optimization penalizes deviation from target hardware latency using a lookup table that aggregates pre-measured block execution times: $L(a, w_a) = CE(a, w_a) \cdot \alpha \log(\text{LAT}(a)^\beta)$ This formulation transforms architecture selection into a differentiable problem, allowing simultaneous backpropagation through both model weights and structural routing coefficients.

class HybridSupernetCriterion(nn.Module):
    def __init__(self, alpha_factor=1.0, beta_lat=0.5):
        super().__init__()
        self.alpha = alpha_factor
        self.beta = beta_lat
        self.ce_criterion = nn.CrossEntropyLoss(reduction='mean')
        
    def forward(self, logits, ground_truth, estimated_latency, buffer_dict, batch_size):
        accuracy_loss = self.ce_criterion(logits, ground_truth)
        latency_term = torch.log(torch.pow(estimated_latency, self.beta))
        
        buffer_dict['ce'].update(accuracy_loss.item(), batch_size)
        buffer_dict['lat'].update(latency_term.item(), batch_size)
        
        combined_cost = self.alpha * accuracy_loss * latency_term
        return combined_cost

class AdaptiveRoutingLayer(nn.Module):
    def __init__(self, available_ops, op_costs, temperature_init=1.0):
        super().__init__()
        self.operations = nn.ModuleList([op() for _ in range(len(available_ops))])
        self.cost_lookup = torch.tensor(op_costs, dtype=torch.float32)
        # Initialize routing weights uniformly
        self.routing_params = nn.Parameter(torch.ones(len(available_ops)))
        self.temperature = temperature_init
        
    def forward(self, input_tensor, current_temp, accumulated_delay_buffer):
        soft_selection_weights = nn.functional.gumbel_softmax(self.routing_params, temp=current_temp, hard=False)
        
        # Weighted combination of parallel branches
        branch_outputs = [op(input_tensor) for op in self.operations]
        fused_feature = sum(w * out for w, out in zip(soft_selection_weights, branch_outputs))
        
        # Accumulate latency proportionally to selection probability
        step_delay = torch.dot(soft_selection_weights, self.cost_lookup)
        accumulated_delay_buffer += step_delay.item()
        
        return fused_feature, accumulated_delay_buffer

V2: Constraint-Aware Width and Spatial Resolution Scaling Expanding the search space to include channel dimensions and input resolutions introduces combinatorial complexity and memory overhead. Directly instentiating all combination would require $O(N^2)$ forward passes. To mitigate this, a masking strategy is introduced where alternative channel configurations share identical convolution kernels. The output feature map is derived by multiplying the base computation result with a learned binary mask vector $M$, effectively selecting desired widths without redundant weight copies. Spatial dimension variations present unique alignment challenges. Standard zero-padding along tensor boundaries causes pixel misalignment and distorts receptive fields. The solution involves strategic dispersion of padding elements followed by secondary downsampling before convolution. This preserves the intended geometric coverage while maintaining compatibility across resolutions. During the training phase, fractional channel counts arise due to weighted mixing. To accurately estimate computational cost without discretization bias, an "effective shape" metric is tracked continuously: $\overline{C}{out}^l = \sum_i g^l_i \cdot C^l{i,out}$ This non-integer representation feeds directly into FLOP and parameter accounting routines, enabling precise budget control even when the architecture explores intermediate states. Only post-convergence does the final integer configuration extract itself from the optimized routing probabilities.

def normalize_dimension(value, divisor=8, minimum=None):
    rounded = int(value)
    min_bound = minimum if minimum else divisor
    if value % divisor != 0:
        rounded = int(round(value / divisor)) * divisor
        if rounded < 0.95 * value:
            rounded += divisor
    return max(rounded, min_bound)


class BottleneckInvertedResidual(nn.Module):
    def __init__(self, in_ch, mid_ch, out_ch, kernel_dim, stride, activation_fn, se_enabled=True, prob_drop=0.0):
        super().__init__()
        self.skip_connection = (stride == 1) and (in_ch == out_ch)
        
        self.expansion = nn.Identity() if in_ch == mid_ch else nn.Conv2d(in_ch, mid_ch, 1, bias=False)
        self.depthwise = nn.Conv2d(mid_ch, mid_ch, kernel_dim, stride, groups=mid_ch, padding=kernel_dim//2, bias=False)
        self.activation = activation_fn
        self.se_module = SELayer(mid_ch, reduction=4) if se_enabled else nn.Identity()
        self.projection = nn.Conv2d(mid_ch, out_ch, 1, bias=False)
        self.dropout_path = DropPath(prob_drop) if prob_drop and self.skip_connection else nn.Identity()

    def forward(self, x):
        residual = x
        y = self.expansion(x)
        y = self.depthwise(y)
        y = self.activation(y)
        y = self.se_module(y)
        y = self.projection(y)
        
        if self.skip_connection:
            y = self.dropout_path(y) + residual
        return y


class ScalableMobileNet(nn.Module):
    def __init__(self, stage_configs, num_classes, drop_rate=0.0, drop_prob=0.0):
        super().__init__()
        curr_channels = stage_configs['start_ch']
        head_channels = stage_configs['head_ch']
        
        self.initial_block = nn.Conv2d(3, curr_channels, 3, stride=2, padding=1, bias=False)
        
        for idx, cfg in enumerate(stage_configs['blocks']):
            block_layers = []
            for k, c, s, depth, expansion_ratio, use_se, act_name in cfg:
                expanded = normalize_dimension(curr_channels * expansion_ratio)
                
                if k == 1:
                    block_layers.append(nn.Conv2d(curr_channels, c, 1, s, bias=False))
                else:
                    block_layers.append(BottleneckInvertedResidual(curr_channels, expanded, c, k, s, act_name, use_se, drop_prob))
                curr_channels = c
            setattr(self, f'level_{idx+1}', nn.Sequential(*block_layers))
            
        if drop_prob > 0:
            self._apply_ascending_drop(drop_prob)
            
        self.polarization = nn.Conv2d(curr_channels, curr_channels * 6, 1, bias=False)
        self.reduction = nn.AdaptiveAvgPool2d(1)
        self.classifier_head = nn.Conv2d(curr_channels * 6, head_channels, 1, bias=False)
        self.drop = nn.Dropout(drop_rate)
        self.out_proj = nn.Conv2d(head_channels, num_classes, 1, bias=True)

    def forward(self, x):
        x = self.initial_block(x)
        x = self.level_1(x)
        x = self.level_2(x)
        x = self.level_3(x)
        x = self.level_4(x)
        x = self.level_5(x)
        
        x = self.polarization(x)
        x = self.reduction(x)
        x = self.classifier_head(x)
        if self.drop.training:
            x = self.drop(x)
        return self.out_proj(x).squeeze(-1).squeeze(-1)

V3: Joint Topology and Training Hyperparameter Co-Search Architectural elegance alone rarely guarantees peak performance; mismatched training schedules frequently degrade generalization. Recognizing this, the latest iteration formalizes a dual-domain optimization framework. Instead of fixing training recipes arbitrarily, the system simultaneously navigates structural configurations alongside hyperparameter settings including optimizer types, learning rate decay profiles, regularization strengths, mixup interpolation rates, dropout frequencies, exponential moving average toggles, and input resolutions. The workflow splits into two complementary phases. The first constructs a surrogate regression model capable of forecasting validation accuracy given arbitrary architecture-hyperparameter tuples. An MLP encoder ingests one-hot encoded structural descriptors alongside numeric training flags, projecting them into a compact embedding space. Dual prediction heads operate in tandem: an auxiliary head estimates computational proxies (FLOPs, parameters), while the primary head outputs expected top-1 scores. Early-stopping thresholds are dynamically calibrated by measuring rank correlation between truncated and full training runs against a reference sample set. Once the predictor achieves sufficient fidelity, the second phase deploys a constrained evolutionary optimizer. Candidate solutions are mutated and recombined within strict resource boundaries. The surrogate model rapidly scores each generation, terminating once convergence stagnation occurs. This hybrid methodology drastically reduces wall-clock requirements while yielding robust deployments adaptable to varying compute budgets without retraining the predictor.

def align_channel_count(base_value, mod=8, floor_limit=None):
    target = int(base_value)
    lower_bound = floor_limit if floor_limit else mod
    remainder = base_value % mod
    if remainder != 0:
        target = int(round(base_value / mod)) * mod
        if target < 0.95 * base_value:
            target += mod
    return max(target, lower_bound)


class OptimizedBottleneckBlock(nn.Module):
    def __init__(self, ch_in, ch_mid, ch_out, kern_sz, stride, activation_func, se_cfg, drop_p=0.0):
        super().__init__()
        self.has_skip = (stride == 1) and (ch_in == ch_out)
        
        self.bump = nn.Identity() if ch_in == ch_mid else nn.Conv2d(ch_in, ch_mid, 1, bias=False)
        self.spacial_filter = nn.Conv2d(ch_mid, ch_mid, kern_sz, stride, groups=ch_mid, padding=kern_sz//2, bias=False)
        self.nonlin = activation_func
        self.attention_unit = SELayer(ch_mid, **se_cfg) if se_cfg else nn.Identity()
        self.final_conv = nn.Conv2d(ch_mid, ch_out, 1, bias=False)
        self.stochastic_path = DropPath(drop_p) if drop_p and self.has_skip else nn.Identity()

    def forward(self, inp):
        res = inp
        feat = self.bump(inp)
        feat = self.spacial_filter(feat)
        feat = self.nonlin(feat)
        feat = self.attention_unit(feat)
        feat = self.final_conv(feat)
        
        if self.has_skip:
            feat = self.stochastic_path(feat) + res
        return feat


class CoSearchMobileNet(nn.Module):
    def __init__(self, blueprint, classes, p_drop=0.0, p_path=0.0):
        super().__init__()
        active_ch = blueprint['init_width']
        endpoint_ch = blueprint['end_width']
        
        self.frontend = nn.Conv2d(3, active_ch, 3, 2, 1, bias=False)
        
        for lvl_idx, layer_spec in enumerate(blueprint['phases']):
            sublayers = []
            for params in layer_spec:
                k, c, s, rep, exp, se_opt, act_fn = params
                inner_ch = align_channel_count(active_ch * exp)
                
                if k == 1:
                    sublayers.append(nn.Conv2d(active_ch, c, 1, s, bias=False))
                else:
                    sublayers.append(OptimizedBottleneckBlock(active_ch, inner_ch, c, k, s, act_fn, se_opt, p_path))
                    active_ch = c
                    
                    for _ in range(rep - 1):
                        inner_ch = align_channel_count(active_ch * exp)
                        sublayers.append(OptimizedBottleneckBlock(active_ch, inner_ch, c, k, 1, act_fn, se_opt, 0.0))
                        
            setattr(self, f'phase_{lvl_idx+1}', nn.Sequential(*sublayers))
            
        if p_path > 0:
            self._init_depth_decay(p_path)
            
        self.broadcaster = nn.Conv2d(active_ch, active_ch * 6, 1, bias=False)
        self.pooler = nn.AdaptiveAvgPool2d(1)
        self.head_mapper = nn.Conv2d(active_ch * 6, endpoint_ch, 1, bias=False)
        self.regulator = nn.Dropout(p_drop)
        self.pred_layer = nn.Conv2d(endpoint_ch, classes, 1, bias=True)

    def forward(self, x):
        x = self.frontend(x)
        x = self.phase_1(x)
        x = self.phase_2(x)
        x = self.phase_3(x)
        x = self.phase_4(x)
        x = self.phase_5(x)
        
        x = self.broadcaster(x)
        x = self.pooler(x)
        x = self.head_mapper(x)
        if self.regulator.training:
            x = self.regulator(x)
        return self.pred_layer(x).flatten(1)