Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes

In offline reinforcement learning (RL), the absence of active explorationcalls for attention on the model robustness to tackle the sim-to-real gap,where the discrepancy between the simulated and deployed environments cansignificantly undermine the performance of the learned policy. To endow thelearned policy with robustness in a sample-efficient manner in the presence ofhigh-dimensional state-action space, this paper considers the sample complexityof distributionally robust linear Markov decision processes (MDPs) with anuncertainty set characterized by the total variation distance using offlinedata. We develop a pessimistic model-based algorithm and establish its samplecomplexity bound under minimal data coverage assumptions, which outperformsprior art by at least Õ(d), where d is the feature dimension. Wefurther improve the performance guarantee of the proposed algorithm byincorporating a carefully-designed variance estimator.