# ML Confidence Scores The problem is straightforward: given a prediction of some output value, we wish to know how *confident* our model is about that prediction. Generally, in the case of a neural net or logistic regression, a simple confident score would be to take the maximum value of the output layer and use that as your confidence. For instance, if we are trying to predict what animal is in an image, and our classifier returns a vector containing a vector of probabilities corresponding to each animals likelihood of being in the image, then we can take the max of that vector and use that as our confidence. For example: ``` y_hat = [0.1, 0.05, 0.7, 0.05, 0.1] ``` Here we would take our confidence score to be 70% that the image contained the animal associated with our third index. There are other notes related to this in [Bayesian Analysis](Bayesian%20Analysis.md) and [Bayesian Inference](Bayesian%20Inference.md). ### Distance Based Confidence Score (via embeddings) See more [here](https://www.youtube.com/watch?v=BCtkMHiwINs). The general idea is that $xs from the same class should reside close together in the embedding space (embedding layer of a neural network for instance). Visually this looks like: ![500](Screen%20Shot%202022-08-15%20at%207.29.21%20AM.png) ### My Personal Thoughts on Confidence scores I often want to know *how many times we have seen similar $x$‘s in the past*? For instance, if we have seen 100 times an $x$ similar to that which we are observing now, and every one of those 100 times the corresponding $y$ was $5$, then I have more confidence that $y$ will be close to $5$ now. --- Date: 20220815 Links to: [003-Data-Science-MOC](003-Data-Science-MOC) Tags: #review References: * [Distance-based Confidence Score for Neural Network Classifiers - Amit Mandelbaum - YouTube](https://www.youtube.com/watch?v=BCtkMHiwINs) * [TensorFlow Probability: Learning with confidence (TF Dev Summit '19) - YouTube](https://www.youtube.com/watch?v=BrwKURU-wpk&t=205s) * [Maria Navarro: Quantifying uncertainty in Machine Learning predictions | PyData London 2019 - YouTube](https://www.youtube.com/watch?v=r6bhm_A-YcQ&t=1136s) * Bayesian Methods for hackers * [A Tutorial on Conformal Prediction - YouTube](https://www.youtube.com/watch?v=nql000Lu_iE) * [Jupyter Notebook Viewer](https://nbviewer.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter5_LossFunctions/Ch5_LossFunctions_PyMC3.ipynb#Machine-Learning-via-Bayesian-Methods)