Lecture 3 - Part 2

Contents

• Well-Known Distributions
  • Bernoulli Distribution
  • Binomial Distribution
  • Geometric Distribution
  • Poisson Distribution
  • Uniform Distribution
  • Normal (Gaussian) Distribution
  • Exponential Distribution
  • Rayleigh Distribution
  • Gamma Distribution
  • Chi-Square (χ²) Distribution
  • Generating Random Variables in Python
    • Method 1: Using Inverse Transform approach
    • Method 2: Using Library Functions
• Revisiting Gaussian Distribution
  • Multivariate Gaussian Distribution
  • Covariance Matrix
• Understanding Covariance through Contour Plots
  • Positive Covariance
  • Negative Covariance
  • Zero Covariance
• Reference

Well-Known Distributions

In the last part, we covered most of the fundamental concepts which are required to understand the detection theory. The only remaining bit is introduction to some commonly encountered probability distribution and methods for simulating the random variables. We will start with these two concepts and then do a quick tour of detection theory in this part.

Bernoulli Distribution

• Mean: $p$
• Variance: $p(1-p)$

Binomial Distribution

Number of successes in $n$ independent Bernoulli trials:

• Mean: $np$
• Variance: $np(1-p)$

Geometric Distribution

Number of trials until first success:

• Mean: $\tfrac{1}{p}$
• Variance: $\tfrac{1-p}{p^2}$

Poisson Distribution

Models rare events:

• Mean: $\lambda$
• Variance: $\lambda$

Uniform Distribution

• Mean: $\tfrac{a+b}{2}$
• Variance: $\tfrac{(b-a)^2}{12}$

Normal (Gaussian) Distribution

• Mean: $\mu$
• Variance: $\sigma^2$

Exponential Distribution

• Mean: $\tfrac{1}{\lambda}$
• Variance: $\tfrac{1}{\lambda^2}$

Rayleigh Distribution

A continuous probability distribution for non-negative random variables, often used to model the magnitude of a vector with two independent and identically distributed Gaussian components (e.g., fading in wireless channels).
PDF:

Gamma Distribution

A two-parameter family of continuous probability distributions that generalizes the exponential distribution and is often used to model waiting times.
PDF:

where $\Gamma(k)$ is the Gamma function.

Chi-Square (χ²) Distribution

A special case of the Gamma distribution with shape parameter $k/2$ and scale parameter $2$. It arises as the distribution of a sum of squares of $k$ independent standard normal random variables.
PDF:

Generating Random Variables in Python

Method 1: Using Inverse Transform approach

Here’s how we do it:

1. Step 1: Generate $𝑈∼\text{Uniform}(0,1)$
2. Step 2: Transform using the inverse CDF (quantile function) of the distribution.

Below is Python code for all 10 distributions using inverse transform sampling:

No output yet. Click "Run Python" to execute.

Method 2: Using Library Functions

Not all RVs have closed-form PDF and CDFs, sometimes its better to perform Rejection Sampling to generate the variable. We do not dwell into the details in these notes. For the sake of this book, you can use builtin python libraries to generate the RVs as follows:

No output yet. Click "Run Python" to execute.

Revisiting Gaussian Distribution

The Gaussian probability density function (PDF) (also referred to as Normal PDF) is defined as

where $\mu$ is the mean and $\sigma^2$ is the variance. It is denoted by $\mathcal{N}(\mu,\sigma^2)$ and we indicate $X \sim \mathcal{N}(\mu,\sigma^2)$ to say that $X$ follows Normal distribution. If $\mu=0$ then the moments are given by:

If $\mu>0$ then, we can define the $n^{\text{th}}$ moment as:

with $\mathbb{E}(X^k)$ can be computed from the previous equation (Eq. (12)).

As can be seen from this interactive graph, changing the mean changes the location of the distribution, while changing the variance controls the spread. The cumulative distribution of standard Normal distribution ($\mu=0$ and $\sigma^2=1$) is given by:

Often it is more convenient to use complementary CDF, i.e., $1-F_X(z)= 1-\Phi(z)$ the probability that the $X$ exceeds a certain value $z$. This is also known as right tail probability. Then we can express,

Multivariate Gaussian Distribution

In previous section, we explored univariate Gaussian distribution. This can be generalised easily to multivariate distribution. Consider, $\boldsymbol{x}=[x_1,x_2,...,x_n]$ where $x_i \sim \mathcal{N}(\mu_i,\sigma_i^2)$ then vector $\boldsymbol{x}$ has multivariate Gaussian distribution. Mathematically,

where $\boldsymbol{X}=[x_1,x_2,..., x_n]$ is the vector collection of RVs, $\boldsymbol{\mu}=[\mu_1,\mu_2,...,\mu_N]$ is a mean vector, and $\boldsymbol{\Sigma}$ is a $N\times N$ covariance matrix. We will come back to the covariance matrix in few moments. For now consider it to be matrix containing relationship between $x_i$'s. When $N=2$, we have special case of Bi-variate Gaussian distribution. A Bi-variate Gaussian distribution can be visualised in 3D as follows:

No output yet. Click "Run Python" to execute.

In the plot above, try changing the 0.5 values to 0, and observe what happens to the shape of the distribution. To understand what is happening here, let us dive into concept of covariance.

Covariance Matrix

The covariance matrix's entries can be written as follows:

So essentially, the co-variance matrix contains how two random variable say $x_i$ and $x_j$ change together. Covariance quantifies how much two variables are linearly related. The covariance only captures the linear relationship and not any other types of relationship between these random variables. Consider the term $(x_i-\mathbb{E}(x_i))(x_j-\mathbb{E}(x_j))$:

1. If both $x_i$ and $x_j$ are above their respective mean values this term is positive. Showing that both of these variables move together, in a straight line sense. Essentially, $\text{Cov}(x_i,x_j)>0$.
2. If one of these goes down while another one up, then they are negatively correlated. Essentially, $\text{Cov}(x_i,x_j)<0$.
3. If both $x_i$ and $x_j$ are uncorrelated then $\text{Cov}(x_i,x_j)=0$. Notice that if two RVs are independent then $\text{Cov}(x_i,x_j)=0$ but other way around is not true.

Understanding Covariance through Contour Plots

Explore the figure below showing a contour plot. You can observe how the sign of covariance affects the shape and orientation of the contours.

Positive Covariance

• The contours are elliptical and tilted upward.
• This indicates that as X increases, Y tends to increase too.
• In other words, the two variables move together.

Negative Covariance

• The contours are elliptical but tilted downward.
• This indicates that as X increases, Y tends to decrease.
• The variables move in opposite directions.

Zero Covariance

• The contours are circular (or axis-aligned ellipses).
• This means there is no linear relationship between X and Y.
• They may still have non-linear dependence, but linearly they are independent.

You can also visualize the same covariance in a 3D surface plot, which makes the tilt of the density even more visually obvious.

When $\boldsymbol{\Sigma}=\sigma^2 \boldsymbol{I}_N$ with

we often say that the $\boldsymbol{X}$ has isotropic Gaussian distribution. We will come back to these concepts again when we look at Time-Series Analysis in future discussion. For now, we have explored the fundamental concepts to build our background and are all set to explore basic detection problems.

Reference

[1] Kay, Steven M. Fundamentals of statistical signal processing: Detection theory. Prentice-Hall, Inc., 1993. Read Chapter 2 to familiarise yourself with some of the concepts discussed here.