Author: Hongseok Namkoong

Publisher:

Published: 2019

Total Pages:

ISBN-13:

As machine learning systems increasingly get applied in high-stake domains such as autonomous vehicles and medical diagnosis, it is imperative that they maintain good performance when deployed. Modeling assumptions rarely hold due to noisy inputs, shifts in environment, unmeasured confounders, and even adversarial attacks to the system. The standard machine learning paradigm that optimize average performance is brittle to even small amounts of noise, and exhibit poor performance on underrepresented minority groups. We study \emph{distributionally robust} learning procedures that explicitly protect against potential shifts in the data-generating distribution. Instead of doing well just on average, distributionally robust methods learn models that can do well on a range of scenarios that are different to the training distribution. In the first part of thesis, we show that robustness to small perturbations in the data allows better generalization by optimally trading between approximation and estimation error. We show that robust solutions provide asymptotically exact confidence intervals and finite-sample guarantees for stochastic optimization problems. In the second part of the thesis, we focus on notions of distributional robustness that correspond to uniform performance across different subpopulations. We build procedures that balance tail-performance alongside classical notions of average performance. To trade these multiple goals \emph{optimally}, we show fundamental trade-offs (lower bounds), and develop efficient procedures that achieve these limits (upper bounds). Then, we extend our formulation to study partial covariate shifts, where we are interested in marginal distributional shifts on a subset of the feature vector. We provide convex procedures for these robust formulations, and characterize their non-asymptotic convergence properties. In the final part of the thesis, we develop and analyze distributionally robust approaches using Wasserstein distances, which allows models to generalize to distributions that have different support than the training distribution. We show that for smooth neural networks, our robust procedure guarantees performance under imperceptible adversarial perturbations. Extending such notions to protect against distributions defined on learned feature spaces, we show these models can also improve performance across unseen domains.