Confident Prediction
A collection of works on Conformal Prediction
Methods for conformal/probabilistic prediction (Conformal Predictors, VennPredictors) are wrappers around Machine Learning algorithms, that provide guarantees about their predictions; generally, their sole assumption is exchangeability on data (weaker than standard i.i.d.). They were first proposed in the book “Algorithmic Learning in a Random World” (Vovk, Gammerman, Shafer; 2005).
Conformal Predictors (CP) allow limiting the errors committed by a learning algorithm (“underlying algorithm”), in a multilabel classification setting, to a desired significance level \(\varepsilon\): their accuracy is guaranteed to be at least \(1\varepsilon\).
VennPredictors (VP) output a set of probability distributions on the labels, as a prediction for a new object \(x\); one of these distributions is guaranteed to be perfectly calibrated.
Conformal Prediction for Hidden Markov Models (CPHMM) are an alternative to HMM and the Viterbi algorithm for supervised sequence learning problems (Cherubin, Nouretdinov, 2016), giving error guarantees (validity) under the sole exangeability assumption. They are more robust than the standard maximum likelihood approach when facing arbitrary distributions.
Code
 Python implementation of Conformal Prediction with scikitlearn integration: https://github.com/donlnz/nonconformist
 My Rust implementation of Conformal Predictionrelated methods, aiming for correctness and performance: https://github.com/gchers/randomworld
 My implementation of CPHMM https://github.com/gchers/cphmm
 (Unmaintained) My old Python implementation of Conformal Prediction: https://github.com/gchers/cpy
Projects
A list of works on CP.

Approximating full conformal prediction at scale via influence functions In Proceedings of the AAAI Conference on Artificial Intelligence 2023 [Paper]

How do the performance of a Conformal Predictor and its underlying algorithm relate? In Conformal and Probabilistic Prediction with Applications 2023 [Paper]

Conformal Clustering and Its Application to Botnet Traffic In Statistical Learning and Data Sciences  Third International Symposium, SLDS 2015, Egham, UK, April 2023, 2015, Proceedings 2015 [Slides]