Conformal and Probabilistic Prediction

Methods for conformal/probabilistic prediction (Conformal Predictors, Venn-Predictors) are wrappers around Machine Learning algorithms, that provide guarantees about their predictions; generally, their sole assumption is exchangeability on data (weaker than standard i.i.d.). They were first proposed in the book “Algorithmic Learning in a Random World” (Vovk, Gammerman, Shafer; 2005).

Conformal Predictors (CP) allow limiting the errors committed by a learning algorithm (“underlying algorithm”), in a multi-label classification setting, to a desired significance level $\varepsilon$: their accuracy is guaranteed to be at least $1-\varepsilon$.

Venn-Predictors (VP) output a set of probability distributions on the labels, as a prediction for a new object $x$; one of these distributions is guaranteed to be perfectly calibrated.

Code