A First Course in Probability

📚 Content Summary

An elementary introduction to the theory of probability for students in mathematics, statistics, engineering, and the sciences. It covers the basic principles of combinatorial analysis, probability axioms, conditional probability, random variables, and limit theorems.

A classic, comprehensive foundation to the mathematical theory and applications of probability.

Author: Sheldon Ross

Acknowledgments: Hossein Hamedani, Joe Blitzstein, Peter Nuesch, Ivan Ardestani, and several university reviewers/contributors are credited for accuracy and feedback.

🎯 Learning Objectives

1. Apply the Basic and Generalized Principles of Counting to multi-stage experiments.
2. Differentiate between and calculate permutations and combinations for both distinct and indistinguishable objects.
3. Prove combinatorial identities using algebraic induction and logical combinatorial arguments.
4. Define sample spaces and events for diverse experiments and apply DeMorgan's laws to set operations.
5. Calculate probabilities using the three fundamental Axioms of Probability and simple propositions (complements, unions, and subsets).
6. Solve complex combinatorial problems involving equally likely outcomes, such as poker hands, the Matching problem, and the Birthday problem.
7. Define and calculate conditional probabilities using the formula P(E|F) = \frac{P(EF)}{P(F)}.
8. Apply Bayes's Formula to solve complex problems involving multiple hypotheses and diagnostic testing.
9. Distinguish between independent and conditionally independent events in practical scenarios like genetics and engineering.
10. Define discrete random variables and compute their PMFs and CDFs.

🔹 Lesson 1: Combinatorial Analysis

Overview: This lesson covers the fundamental mathematical theory of counting, known as combinatorial analysis. It progresses from the basic principles of multiplication for independent experiments to the formal study of permutations and combinations. Students will master the binomial and multinomial theorems, explore various proof techniques, and solve complex distribution problems using integer-valued equations.

Learning Outcomes:

• Apply the Basic and Generalized Principles of Counting to multi-stage experiments.
• Differentiate between and calculate permutations and combinations for both distinct and indistinguishable objects.
• Prove combinatorial identities using algebraic induction and logical combinatorial arguments.

🔹 Lesson 2: Axioms of Probability

Overview: This lesson establishes the formal mathematical foundation of probability theory, beginning with the definition of sample spaces and events. It introduces Kolmogorov’s three Axioms of Probability and derived propositions, such as the Inclusion-Exclusion Principle. The content extends into combinatorial applications including the Birthday Problem and Matching Problem.

Learning Outcomes:

• Define sample spaces and events for diverse experiments and apply DeMorgan's laws to set operations.
• Calculate probabilities using the three fundamental Axioms of Probability and simple propositions.
• Solve complex combinatorial problems involving equally likely outcomes, such as poker hands, the Matching problem, and the Birthday problem.

🔹 Lesson 3: Conditional Probability and Independence

Overview: This lesson explores how the probability of an event is revised in light of new information. It transitions from unconditional probabilities to conditional frameworks, formalizing the relationship between dependent and independent events through Bayes's Formula and Laplace's Rule of Succession. Students will learn to update prior probabilities with empirical evidence to reach posterior conclusions.

Learning Outcomes:

• Define and calculate conditional probabilities using the formula P(E|F) = \frac{P(EF)}{P(F)}.
• Apply Bayes's Formula to solve complex problems involving multiple hypotheses and diagnostic testing.
• Distinguish between independent and conditionally independent events in practical scenarios like genetics and engineering.

🔹 Lesson 4: Discrete Random Variables

Overview: This lesson explores the fundamental theory and application of discrete random variables, which are variables whose set of possible values is finite or countably infinite. We define their probability mass functions (PMF) and cumulative distribution functions (CDF), while establishing core measures of central tendency and dispersion. Finally, the lesson examines specific distribution families used to model real-world phenomena.

Learning Outcomes:

• Define discrete random variables and compute their PMFs and CDFs.
• Calculate the Expected Value and Variance of a random variable and its functions.
• Identify and apply the appropriate discrete probability distribution to solve complex word problems.

🔹 Lesson 5: Continuous Random Variables

Overview: This lesson explores the properties and applications of continuous random variables, focusing on their expectations, variances, and specific probability distributions. Students will learn to model real-world phenomena using Uniform, Normal, Exponential, Gamma, Weibull, Cauchy, and Beta distributions. The lesson also covers techniques for approximating discrete distributions and transforming random variables.

Learning Outcomes:

• Calculate the expectation and variance for continuous random variables and functions of those variables.
• Apply the Normal distribution and its approximation to the Binomial distribution using the continuity correction.
• Analyze reliability and lifetimes using Exponential distributions, the memoryless property, and hazard rate functions.

🔹 Lesson 6: Jointly Distributed Random Variables

Overview: This lesson explores the mathematical framework for handling multiple random variables simultaneously. It covers the transition from individual distributions to joint probability density/mass functions, the rigorous definition of independence, and the behavior of sums of independent variables. Furthermore, the curriculum extends into advanced topics including order statistics and transformations of random vectors using Jacobians.

Learning Outcomes:

• Compute marginal distributions and conditional densities for continuous and discrete joint random variables.
• Apply factorization criteria to determine if random variables are independent and model complex processes.
• Use the Jacobian determinant method to find the joint distribution of functions of random variables and calculate the distributions of order statistics.

🔹 Lesson 7: Properties of Expectation

Overview: This lesson explores the advanced properties of mathematical expectation, moving beyond simple averages to the linearity of sums, covariance, and the power of conditioning. Students will learn to apply these tools to algorithmic analysis, probabilistic bounds, and predictive modeling.

Learning Outcomes:

• Apply the linearity of expectation to complex sums, including indicator variables and infinite series.
• Calculate and interpret covariance, correlation, and the variance of sums for dependent and independent variables.
• Utilize conditional expectation and variance to simplify the analysis of compound random variables and solve optimization problems.

🔹 Lesson 8: Limit Theorems

Overview: This lesson covers the fundamental asymptotic results of probability theory, specifically how the sum and average of random variables behave as the number of observations grows to infinity. We explore the Weak and Strong Laws of Large Numbers and the Central Limit Theorem. Specific probability bounds like the one-sided Chebyshev inequality are also addressed.

Learning Outcomes:

• Distinguish between the Weak and Strong Laws of Large Numbers in terms of convergence criteria.
• Apply the Central Limit Theorem (CLT) to approximate probabilities for sums of random variables using the normal distribution.
• Utilize the one-sided Chebyshev inequality to provide upper bounds for tail probabilities.

🔹 Lesson 9: Stochastic Processes and Entropy

Overview: This lesson explores the mathematical frameworks for modeling random events over time and quantifying information. It covers the Poisson Process and its interarrival times, the structure and long-term behavior of Markov Chains, and the fundamental principles of Information Theory. Specifically, it addresses entropy and its application to optimal coding.

Learning Outcomes:

• Define and calculate probabilities for the Poisson Process and determine the distribution of interarrival times.
• Construct transition matrices for Markov Chains and utilize the Chapman-Kolmogorov equations to find n-step probabilities.
• Calculate limiting probabilities for Ergodic Markov Chains and solve random walk problems.

🔹 Lesson 10: Simulation Techniques

Overview: This lesson explores the principles and applications of simulation for empirically determining probabilities and expected values. It covers the generation of random permutations, techniques for simulating continuous and discrete random variables, and advanced methods for variance reduction. These techniques improve the efficiency and accuracy of simulation estimates.

Learning Outcomes:

• Understand the role of pseudorandom number generators and seeds in simulation.
• Implement algorithms to generate random permutations and simulate variables from both discrete and continuous distributions.
• Apply the Polar Method for generating unit normals and simulate chi-squared variables.