Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability $\epsilon$, together with a method (e.g. a Neural Net) that makes a prediction $\hat{y}$ of a label $y$, it produces a set of labels, typically containing $\hat{y}$, that also contains $y$ with probability $1 - \epsilon$. ### From a High Level How good is your prediction $\hat{y}$? If you are predicting the label $y$ of a new object, how confident are you that $y = \hat{y}$? Conformal prediction uses *past experience* to determine precise levels of confidence in predictions. Given a method (e.g. a Neural Net) for making a prediction $\hat{y}$, conformal prediction produces a 95% *prediction region* - a set $\Gamma^{0.05}$ that contains $y$ with probability at least 95%. Typically $\Gamma^{0.05}$ also contains the prediction $\hat{y}$. We call $\hat{y}$ the **point prediction** and $\Gamma^{0.05}$ the **region prediction**. There are two main components to the conformal prediction process: 1. Starting from the method for point prediction, we construct a **nonconformity measure**, which measures how *unusual* an example looks relative to previous examples 2. The **conformal algorithm** turns this nonconformity measure into prediction regions. ### The Conformal Algorithm, V1 --- Date: 20220522 Links to: Tags: References: * [GitHub - valeman/awesome-conformal-prediction: A professionally curated list of awesome Conformal Prediction videos, tutorials, books, papers, PhD theses, articles and open-source libraries.](https://github.com/valeman/awesome-conformal-prediction) * [Conformal Prediction Tutorial - YouTube](https://www.youtube.com/watch?v=0MsGri8nmJQ)