A Cluster-Level Admission Control Policy for Multi-Tenant ML Training with Predictable Tail Latency in E-Commerce Marketplace ML Platforms

Abstract

Modern e-commerce marketplace platforms rely on frequent model refresh cycles for ranking, retrieval, fraud detection, and ads, which in turn depend on shared multi-tenant GPU clusters that run heterogeneous training jobs. As clusters become more consolidated, the dominant source of unpredictability shifts from raw training throughput to tail latency, where a small fraction of jobs experience large queueing and interference-induced slowdowns that violate operational deadlines. This paper studies cluster-level admission control for multi-tenant ML training with an explicit objective of predictable tail latency under capacity fragmentation, bursty arrivals, and non-stationary job size distributions. We formalize end-to-end training latency as the sum of admission delay, placement-induced startup delay, and runtime dilation caused by resource contention and straggler amplification. Building on this decomposition, we introduce a risk-budgeted admission control policy that uses quantile-aware load estimation and virtual-queue enforcement to bound tail-SLO violation probability while preserving utilization. The policy combines a conservative upper confidence bound on effective resource demand with a per-tenant tail-risk regulator that converts percentile constraints into stabilizing feedback. We provide analytical conditions under which the resulting closed-loop system maintains bounded tail-risk backlogs and yields predictable high-percentile completion times for admitted jobs. A prototype integration with a generic cluster scheduler is described, along with trace-driven experiments that demonstrate reductions in p99 queueing delay and fewer deadline misses at comparable utilization relative to throughput-centric baselines. The results suggest that cluster-level admission control can serve as a practical lever for tail predictability in marketplace ML platforms when coupled with quantile-sensitive estimation and enforceable risk budgets.