Bayesian Inference and Decision Theory

Bayesian inference and decision theory is the engine behind how AI systems handle uncertainty and make choices. When you understand this, you can grasp how a spam filter decides an email is spam, how a self-driving car decides to brake, how a medical AI recommends a diagnosis, and how a language model like ChatGPT chooses the most likely next word. This unlocks the entire Bayesian methods family — including Gaussian processes, variational inference, and generative models — which power modern recommendation systems, robotics, and predictive analytics. It also connects to information geometry and Fisher information, which you will meet later as you sharpen your statistical toolkit.

Imagine you have a belief about something, like the chance it will rain tomorrow. You start with a guess — maybe 30% based on the season. But then you look out the window and see dark clouds. That is new evidence. In Bayesian inference, you update your belief systematically: the clouds make rain more likely, so you raise your estimate to 60%. If you then hear the weather forecast says sunny, you might lower it a bit. Each piece of evidence adjusts your belief. Bayes' rule is the precise arithmetic of this update. It tells you how to combine what you believed before (the prior) with what the evidence says (the likelihood) to get a new, better belief (the posterior).\n\nBut having a belief is not the end. You also have to act — carry an umbrella or not, approve a loan or not. Decision theory is the part that says: given what you believe, what action is best? You weigh the costs and benefits of being wrong. If it might rain, carrying an umbrella costs a little discomfort but avoids getting soaked. If it likely won't rain, leaving it at home is fine. Decision theory formalises this trade-off. You choose the action that minimises your expected loss — the average cost of being wrong, weighted by your updated beliefs.

Think of a doctor diagnosing a patient. Initially, the doctor has a prior belief about how common a disease is in the general population — say 1 in 100 (1%). Then the patient reports a symptom, say a persistent cough. The doctor knows from medical studies that this symptom appears in 80% of people who have the disease, and in 10% of people who don't. Using Bayes' rule, the doctor can combine these to get a posterior probability: given the cough, the chance the patient has the disease. This is the updated belief.\n\nBut the doctor's job is not just to believe — it's to act. The doctor must decide whether to prescribe a treatment or run more tests. This is where decision theory comes in. The doctor considers the loss of a false negative (missing the disease, which could be serious) versus a false positive (unnecessary treatment, with side effects). If the disease is dangerous and the treatment is mild, the doctor might prescribe even if the posterior probability is only 20%. If the treatment has severe side effects, the doctor might wait for a confirmatory test. The doctor chooses the action with the lowest expected loss, balancing the probability of each outcome with its cost.\n\nThis analogy works well for Bayesian inference and decision theory, but it breaks down in one way: in real medicine, doctors don't always have precise numbers for the likelihoods, and patients are unique. In AI, these numbers are often estimated from large datasets, and the decision rules are applied consistently, which can be a strength (consistency) and a weakness (lack of human judgment).

Bayesian inference is the process of updating a prior belief (a probability distribution) about an unknown quantity into a posterior belief by incorporating new evidence via Bayes' rule; decision theory is the subsequent step of choosing the action that minimises the expected loss computed with respect to that posterior belief.

You have not studied any other topics yet, so this is your first formal idea. But this concept will later connect to Fisher Information (how sharply the data identifies a parameter), Information Geometry (the geometry of probability distributions), and other Bayesian methods like Gaussian processes. Everything you learn here — priors, posteriors, expected loss — will reappear throughout your AI journey, especially in probabilistic machine learning.