Note for Gamma Distribution

Jun 30, 2022 3 min read Probability, tutorial

Motivation for Gamma Function
Definition of Gamma Function
Gamma Distribution
- How to remember the Gamma pdf?
- Gamma & Exponential Connection

Motivation for Gamma Function

We all know how to compute the factorial of integer. BUT what is the factorial of 1/2?

In other words, how to interpolate the factorial function?

The gamma function can be seen as a solution to the following interpolation problem:

“Find a smooth curve that connects the points (x, y) given by y = (x − 1)! at the positive integer values for x.”

More details, check the wiki page.

Definition of Gamma Function

Definition 1 (Gamma Function) $\begin{array}{r} Γ (z) = \int_{0}^{\infty} x^{z - 1} e^{- x} d x, z \in R^{+} \end{array}$

For the the Gamma function, it is enough to know the following properties for now.

Lemma 1 $\begin{aligned} Γ (z + 1) = z Γ (z), z \in R^{+} \\ Γ (n) = (n - 1)!, n = 1, 2, 3, . . . \end{aligned}$

Easy to prove using integration by parts.

Now, what is $Γ (\frac{1}{2})$ ?

$Γ (\frac{1}{2}) = \int_{0}^{\infty} x^{- 1 / 2} e^{- x} d x = ?$

Recall that, $\int_{0}^{\infty} e^{- x^{2}} = \frac{1}{2} \sqrt{π}$ , let $u = x^{2}$ and will get the result $Γ (\frac{1}{2}) = \sqrt{π}$ .

Gamma Distribution

From the Gamma function, it is pretty natural to get Gamma pdf. JUST normalizing!

Clearly, $1 = \int_{0}^{\infty} \frac{x^{r - 1} e^{- x}}{Γ (r)} d x = \int_{0}^{\infty} f_{X} (x) d x, X := G a m m a (r, 1)$

What is the pdf for the general $G a m m a (r, λ)$ ? Let

$Y = \frac{X}{λ}, Y \sim G a m m a (r, λ)$ $f_{Y} (y) = f_{X} (x) \frac{d x}{d y}$

We’ll get

$f (y; r, λ) = \frac{λ^{r} y^{r - 1} e^{- λ y}}{Γ (r)}$

Here, $r$ is called the shape parameter and $λ$ is called the rate parameter.

How to remember the Gamma pdf?

That’s my trick: exponential density times the power rise to (shape-1), then divided by normalizer.

Exponential density (very familiar): $λ e^{- λ x}$
power rise to (shape-1): $(λ x)^{r - 1}$
normalizing constant (using shape): $Γ (r)$

$\begin{aligned} f (x; r, λ) & = \frac{exp density \cdot {power}^{shape-1}}{n o r m a l i z e r} \\ = \frac{λ e^{- λ x} (λ x)^{r - 1}}{Γ (r)} \\ = \frac{λ^{r} x^{r - 1} e^{- λ x}}{Γ (r)} \end{aligned}$

Another way to remember is this:

Exponential key part: $e^{- λ x}$
Add Power part: $λ^{◻} x^{◻} e^{- λ x}$
- multiply the power part in exponential
- rate rises to shape
- variable rises to shape-1
$λ^{r} x^{r - 1} e^{- λ x}$
Add Normalizing part: $\frac{λ^{r} x^{r - 1} e^{- λ x}}{Γ (r)}$

Gamma & Exponential Connection

Let’s recall the Poisson Process, $N_{t} = number of arrials up to time t \sim P o i s (λ t)$ The number of arrivals in the disjoint intervals are independent.

Let $T_{1}$ be the time of 1st arrival, $P (T_{1} > t) = P (N_{t} = 0) = e^{- λ t} ⟹ T_{1} \sim E x p (λ)$ Now, let $T_{n}$ be the time of nth arrival, that is, $T_{n} = \sum_{i = 1}^{n} X_{i}$ , where $X_{i} \overset{iid}{\sim} E x p (λ)$

What is the pdf of $T_{n}$ ? Answer is Gamma!

Proposition 1 Gamma is the sum of iid Exponentials.

Proof:

Since $M_{X} (t) = \frac{λ}{λ - t}$ , where $t < λ$ ,

$M_{\sum_{i = 1}^{n} X_{i}} (t) = (M_{X} (t))^{n} = (\frac{λ}{λ - t})^{n}$

It is enough to show the MGF of Gamma equals the above value.

Let $Y \sim G a m m a (n, λ)$ ,

$\begin{aligned} M_{Y} (t) = E (e^{t y}) & = \int_{0}^{\infty} e^{t y} \frac{1}{Γ (n)} λ^{n} e^{- λ y} y^{n - 1} d y \\ = \frac{λ^{n}}{Γ (n)} \int_{0}^{\infty} e^{- (λ - t) y} y^{n - 1} d y, let u = (λ - t) y \\ = \frac{λ^{n}}{Γ (n)} (\frac{1}{λ - t})^{n} \int_{0}^{\infty} e^{- u} u^{n - 1} d u \\ = \frac{λ^{n}}{Γ (n)} (\frac{1}{λ - t})^{n} Γ (n) \\ = (\frac{λ}{λ - t})^{n} \end{aligned}$

Proved!

Remark. The Exponential is the continous analog of the Geometric. Similarly, the Gamma is the continous analog of the Negative Binomial.

Probability prediction CLV latex gamma function gamma distribution

Chen Xing

Founder & Data Scientist

Enjoy Life & Enjoy Work!