Probability & Bayesian Statistics | Chen Xing

Notes for Variational Inference

Sun, 27 Apr 2025 00:00:00 +0000

Introduction

In modern Bayesian statistics, we often face posterior distributions that are difficult to compute. Let $p(z)$ be prior density and $p(x \mid z)$ be likelihood. The standard approach to compute posterior $p(z \mid x)$ is to use MCMC (like Metropolis-Hastings, Gibbs sampling and HMC). But MCMC has downsides:

Slow for big datasets or complex models.
Difficult to scale in the era of massive data.

Variational inference (VI) offers a faster alternative. The key difference between MCMC and VI is:

MCMC sample a Markov chain
VI solve an optimization problem

The main idea behind VI is to use optimization. Specifically,

step 1: we posit a family of densities $Q$
step 2: find a member in $Q$ to minimize the KL divergence $$ q^*(\mathbf{z})=\underset{q(\mathbf{z}) \in \mathcal{Q}}{\arg \min } \mathrm{KL}(q(\mathbf{z}) | p(\mathbf{z} \mid \mathbf{x})) . $$

Key Idea: Rather than sampling, VI optimizes — it finds a best guess distribution by minimizing a divergence.

VI vs MCMC: When to Use Which?

Pros and Cons of VI

Pros:
- Much faster than MCMC.
- Easy to scale with stochastic optimization and distributed computation.
Cons:
- VI underestimates posterior variance (it tends to be “overconfident”).
- It does not guarantee exact samples from the true posterior.

Variational Inference

Recall that in the Bayesian framework, $$p(z \mid x) = \frac{p(z,x)}{p(x)} \propto p(x \mid z) p(z)$$ We try to avoid calculating the denominator (marginal likelihood), $p(x)$, also called evidence, as it requires us to calculate high dimensional integrals.

Variational inference turns Bayesian inference into an optimization problem by minimizing KL divergence within a simpler family of distributions, typically using coordinate ascent to maximize the evidence lower bound (ELBO).

First, the optimization goal is

$$ q^*(\mathbf{z})=\underset{q(\mathbf{z}) \in \mathcal{Q}}{\arg \min } KL(q(\mathbf{z}) | p(\mathbf{z} \mid \mathbf{x})), $$

where $q^*$ is the best approximation.

$$ \begin{aligned} KL(q(\boldsymbol{z}) \| p(\boldsymbol{z} \mid \boldsymbol{x})) & =\int_z q(\boldsymbol{z}) \log \left[\frac{q(\boldsymbol{z})}{p(\boldsymbol{z} \mid \boldsymbol{x})}\right] d \boldsymbol{z} \\ & =\int_{\boldsymbol{z}}[q(\boldsymbol{z}) \log q(\boldsymbol{z})] d \boldsymbol{z}-\int_{\boldsymbol{z}}[q(\boldsymbol{z}) \log p(\boldsymbol{z} \mid \boldsymbol{x})] d \boldsymbol{z} \\ & =\mathbb{E}_q[\log q(\boldsymbol{z})]-\mathbb{E}_q[\log p(\boldsymbol{z} \mid \boldsymbol{x})] \\ & =\mathbb{E}_q[\log q(\boldsymbol{z})]-\mathbb{E}_q\left[\log \left[\frac{p(\boldsymbol{x}, \boldsymbol{z})}{p(\boldsymbol{x})}\right]\right] \\ & =\mathbb{E}_q[\log q(\boldsymbol{z})]-\mathbb{E}_q[\log p(\boldsymbol{x}, \boldsymbol{z})]+\mathbb{E}_q[\log p(\boldsymbol{x})] \\ & =\mathbb{E}_q[\log q(\boldsymbol{z})]-\mathbb{E}_q[\log p(\boldsymbol{x}, \boldsymbol{z})]+\log p(\boldsymbol{x}) \end{aligned} $$

Note that, $\log p(\boldsymbol{x})$ does not contain $q(\cdot)$, so we can ignore it in the optimization. We define evidence lower bound (ELBO) as

$$ \operatorname{ELBO}(q)=\mathbb{E}_q[\log p(\boldsymbol{x}, \boldsymbol{z})]-\mathbb{E}_q[\log q(\boldsymbol{z})] $$

This value is called evidence lower bound because it is the lower bound of “log evidence”.

$$ \begin{aligned} \log p(\boldsymbol{x}) & =\operatorname{ELBO}(q)+KL(q(\boldsymbol{z}) \| p(\boldsymbol{z} \mid \boldsymbol{x})) \\ & \geq \operatorname{ELBO}(q) \end{aligned} $$

The second line holds because $KL(\cdot | \cdot) \ge 0$ by Jensen’s inequality.

Therefore, maximizing ELBO is equivalent to minimizing KL divergence.

$$ \begin{aligned} q^*(\boldsymbol{z}) & =\underset{q(\boldsymbol{z}) \in \mathcal{Q}}{\operatorname{argmin}} KL(q(\boldsymbol{z}) \| p(\boldsymbol{z} \mid \boldsymbol{x})) \\ & =\underset{q(\boldsymbol{z}) \in \mathcal{Q}}{\operatorname{argmax}} \operatorname{ELBO}(\mathrm{q}) \\ & =\underset{q(\boldsymbol{z}) \in \mathcal{Q}}{\operatorname{argmax}}\left\{\mathbb{E}_q[\log p(\boldsymbol{x}, \boldsymbol{z})]-\mathbb{E}_q[\log q(\boldsymbol{z})]\right\} \end{aligned} $$

What is the intuition for $\operatorname{ELBO}(q)$?

$$ \begin{aligned} \operatorname{ELBO}(q) & \triangleq \mathbb{E}_q[\log p(\boldsymbol{x}, \boldsymbol{z})]-\mathbb{E}_q[\log q(\boldsymbol{z})] \\ & =\mathbb{E}[\log p(\mathbf{z})]+\mathbb{E}[\log p(\mathbf{x} \mid \mathbf{z})]-\mathbb{E}[\log q(\mathbf{z})] \\ & =\mathbb{E}[\log p(\mathbf{x} \mid \mathbf{z})]-\mathrm{KL}(q(\mathbf{z}) \| p(\mathbf{z})) . \end{aligned} $$

first term is try to “maximize the likelihood”
second term is try to encourage density $q(\cdot)$ close to prior
balance between likelihood and prior

Mean-Field Variational Family

In mean field variational inference, we assume that the variational family factorizes,

$$ q(z_1, \cdots, z_m) = \prod_{j=1}^m q_j(z_j), $$ Each variable is independent.

Coordinate ascent algorithm

We will use coordinate ascent inference, iteratively optimizing each variational distribution holding the others fixed.

The ELBO converges to a local minimum. Use the resulting $q$ is as a proxy for the true posterior.

There is a strong relationship between this algorithm and Gibbs sampling.

In Gibbs sampling we sample from the conditional
In coordinate ascent variational inference, we iteratively set each factor to

$$ \text{distribution of } z_k \propto \exp {\mathbb{E}[\log(\text{conditional})]} $$

Reference

Blei, D. M., Kucukelbir ,Alp, & and McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518), 859–877. https://doi.org/10.1080/01621459.2017.1285773
Looking for a nice summary? Check this first: Variational Inference - Princeton CS tutorial
For derivation details, check this: Introduction to Variational Inference

Using Stan to do MCMC

Thu, 20 Apr 2023 00:00:00 +0000

What is Stan?

Stan is an intuitive, yet sophisticated, probabilistic programming language that provides an interface to a recently proposed extension to Hamiltonian Monte Carlo (HMC), known as No U-Turn Sampler (NUTS).

Stan is what is known as an imperative programming language. This is not true for BUGS and JAGS, which are known as declarative languages.

Why choose Stan?

Stan is popular and has huge range of resources generated by users and developers, so when you get stuck, you can find someone to get help.

Bayesian Statistics Self-Study Resources

Sat, 11 Mar 2023 00:00:00 +0000

Here are some good resources for Bayesian Statistics self-learning.

Textbooks

“A Student’s Guide to Bayesian Statistics”: Easy to understand, also with problem and solution available on the author’s personal blog. There are also lots of other useful Bayesian resources on his blog, for example, the Bayesian lecture slides, definitely deserve to check!
“Bayesian Data Analysis”: Sort of “Bayesian Bible”. Comprehensive coverage of Bayesian methods.
“Statistical Rethinking A Bayesian Course with Examples in R and STAN”: Very Special flavor … includes a strong focus on using the statistical software R and the modeling language Stan.
Bayes Rules! An Introduction to Applied Bayesian Modeling

Video Lectures

A Student’s Guide to Bayesian Statistics textbook also provides a series of video lectures on YouTube.
Statistical Rethinking with video lectures. “Statistical Rethinking: A Bayesian Course with Examples in R and Stan” by Richard McElreath. This book is very practical and focuses on applied Bayesian statistics, and would be a good choice if you want to get your hands dirty with real-world problems and data. It uses the R and Stan software tools, which are popular among statisticians and data scientists.
Bayesian statistics: a comprehensive course: This playlist provides a complete introduction to the field of Bayesian statistics and covers a wide range of topics, including Bayesian inference, prior and posterior distributions, conjugate priors, hierarchical models, and more.
MIT Open Course Ware: “Statistics for Applications” Lecture 17 and lecture 18 give a brief introduction to Bayesian statistics.
Bayesian Statistics YouTube playlist provides us some short videos, around 10 mins per video, to illustrate some key concepts in Bayesian stats.
S22 MATH 347 Bayesian Statistics at Vassar College video lectures with GitHub repo containing the course material.

Miscellaneous

A Story Behind Maximum Likelihood

Wed, 31 Aug 2022 00:00:00 +0000

TL;DR

What are we actually doing on MLE? What is the motivation for the Maximum Likelihood Estimation? In this post, we will go over the story behind the maximum likelihood.

We’ll first talk about the total variation distance, which is a very intuitive measure to tell you how “close” our estimator is from the true parameter.

Next, let’s move to KL-divergence used to replace the total variation distance.

In the end, we’ll minimize the KL-divergence to get the “good” estimator. In this process, the maximum likelihood principle will come in.

Total Variation Distance

How to define a “good” estimator?

A “good” estimator should be very “close” to the true parameter isn’t it?

Motivation

Click to view the formal setting

Definition of Total Variation Distance

Def (TV distance)

This definition is very intuitive but pretty strong and hard to calculate right? Because you have to find the maximum over all possible sets. The good news is, we also have this formulation:

Click to view the formal statement.

Before the proof, let’s use the graph to get some feeling for the equation. Where does the 1/2 come from?

Area between the curves

Proof:

Click to view the proof

Proof of TV equation

Note that, the key part in the proof is the observation: $$\int f = \int g = 1 \implies \int f - g = 0,$$ Then we have $$\int_{\{x: f-g > 0\}} f - g = \int_{\{x: f-g < 0\}} g - f.$$

Properties of TV

Total variation is symmetric, non-negative, definite and satisfies the triangle inequality.

Properties of Total Variation

Unclear how to estimate TV!

Our goal is to find the “good” estimator.

If using Total Variation distance to describe the “close”, we need to :

Build an estimator $\widehat{TV}(\mathbb{P}_{\theta}, \mathbb{P}_{\theta^*})$.
Find $\hat{\theta}$ that minimize the function.

However, it is unclear how to build $\widehat{TV}(\mathbb{P}_{\theta}, \mathbb{P}_{\theta^*})$!

We don’t know the $f_{\theta^*}$ (the true parameter $\theta^*$ is unknown), and it is very hard to manipulate the integral of density difference.
The common strategy is to replace the expectation ($E(\cdot)$ ) with the average ($\frac{1}{n}\sum_n(\cdot)$ ), but there is no clear expectation in $TV(\cdot)$.

Due to above difficulties, we need a more convenient distance between probability measure to replace total variation. This is the motivation for DL divergence.

REMARK:

The total variation distance, describing the “worst” scenario, is very intuitive and has a clear interpretation. However, it is hard to build an estimator. That’s why we move to KL divergence.

KL divergence

Let’s check the definition first,

KL divergence

Properties of KL-divergence

For the second property (non-negative), we can use Jensen’s inequality to show it.

KL 🤝 MLE

Now we are ready to introduce maximum likelihood principle using the KL-divergence.

Maximum Likelihood Connection with KL-divergence

Summary

What are we actually doing on MLE?

The short answer is we are minimizing the KL-divergence.

Remember: Maximum Likelihood Estimation is just the empirical version of trying to minimize the KL-divergence!

Reference

YouTube video: MIT Parametric Inference (cont.) and Maximum Likelihood Estimation.

Beta Distribution — Intuition, Derivation, and Examples

Wed, 03 Aug 2022 00:00:00 +0000

Motivation

Model probabilities

The Beta distribution is a probability distribution on probabilities.

The Beta distribution can be understood as representing a distribution of probabilities, that is, it represents all the possible values of a probability when we don’t know what that probability is. For example,

the Click-Through Rate of your advertisement
the conversion rate of customers actually purchasing in your store
how likely the customer will become “inactive”

Because the Beta distribution models a probability, its domain is bounded between 0 and 1.

Generalization of Uniform Distribution

Give me a continuous and bounded random variable except the Uniform Distribution. This is another way to look at beta distribution, continuous and bounded between 0 and 1; also the density is not flat.

$$ X \sim Beta(a, b), \text{ where } a>0, \ b>0. $$ $$ f_X(x) = c \cdot x ^{a-1}(1-x)^{b-1}, \text{ where } x>0. $$

What is $c$ ? Just a normalization constant! We’ll find the value of $c$ later.

Conjugate Prior

The Beta distribution is the conjugate prior for the Bernoulli, binomial, negative binomial and geometric distributions (seems like those are the distributions that involve success & failure) in Bayesian inference.

Computing a posterior using a conjugate prior is very convenient, because you can avoid expensive numerical computation involved in Bayesian Inference.

Conjugate prior = Convenient prior

For example, the beta distribution is a conjugate prior to the binomial. If we choose to use the beta distribution Beta(α, β) as a prior, during the modeling phase, we already know the posterior will also be a beta distribution. Therefore, after carrying out more experiments, you can compute the posterior simply by adding the number of successes (x), and failures (n-x) to the existing parameters α, β respectively, instead of multiplying the likelihood with the prior distribution. The posterior also becomes a Beta distribution with parameters (x+α, n-x+β).

What is the Intuition?

The intuition for the beta distribution comes into play when we look at it from the lens of the binomial distribution.

The difference between the binomial and the beta is that the former models the number of successes (x), while the latter models the probability (p) of success.

In other words, the probability is a parameter in binomial; In the Beta, the probability is a random variable.

Interpretation of α, β

You can think of α-1 as the number of successes and β-1 as the number of failures, just like n & n-x terms in binomial.

You can choose the α and β parameters however you think they are supposed to be.

If you think the probability of success is very high, let’s say 90%, set 90 for α and 10 for β.
If you think otherwise, 90 for β and 10 for α.

As α becomes larger (more successful events), the bulk of the probability distribution will shift towards the right, whereas an increase in β moves the distribution towards the left (more failures).

Also, the distribution will narrow if both α and β increase, for we are more certain.

Dr. Bognar at the University of Iowa built the calculator for Beta distribution, which I found useful and beautiful. You can experiment with different values of α and β and visualize how the shape changes.

Derivation

In this section, we’ll derive Beta distribution using the Beta-Gamma Connections.

We can regard Beta distribution as the fraction of waiting time.

Fraction of Waiting Time

Let $X$ be the waiting time at Bank,

$$ X \sim Gamma(n_1, \lambda) $$

Let $Y$ be the waiting time at Post Office,

$$ Y \sim Gamma(n_2, \lambda) $$

Assume $X$ and $Y$ are independent. What is the distribution of the proportion $\frac{X}{X+Y}$ ?

Solution:

Let $T := X+Y$ be the total waiting time. Clearly, $T \sim Gamma(n_1+n_2, \lambda)$, you can prove it by MGF.

Let $W =: \frac{X}{X+Y}$ be the proportion of waiting time at Bank to the total waiting time. We need to find the PDF of $W$.

The idea is to find the joint PDF $f_{T,W}(t,w)$ at first, and then get the marginal distribution.

$$ \begin{aligned}f_{T,W}(t,w) &= f_{X,Y}(x,y) \left | \frac{\partial(x,y)}{\partial(t,w)} \right|\\ &= \frac{1}{\Gamma(n_1)}\lambda^{n_1}x^{n_1 - 1}e^{-\lambda x} \frac{1}{\Gamma(n_2)}\lambda^{n_2}x^{n_2 - 1}e^{-\lambda y} \left|-t\right|\\ &= \lambda^{n_1+n_2}t^{n_1+n_2-1}e^{-\lambda t} \frac{1}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1}\\ &= \frac{\lambda^{n_1+n_2}t^{n_1+n_2-1}e^{-\lambda t}}{\Gamma(n_1+n_2)} \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1}\\ &= f_T(t) \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1}\end{aligned} $$

Integrating $t$ out to get the marginal:

$$ \begin{aligned}f_W(w) &= \int_0^\infty f_{T,W}(t,w) dt \\&= \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} \cdot\int_0^\infty f_T(t)dt \\&= \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} \end{aligned} $$

Here we have Beta,

$$W \sim Beta(n_1, n_2)$$ REMARK: the above result also proves W and T and independent!

Beta Function as a normalizing constant

Note that, $f_W(w)$ is a PDF needed to be integrated to 1,

$$ \int_0^1\frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} dw \equiv 1 $$

So the normalization constant should be,

$$ c = \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)} := \frac{1}{B(n_1, n_2)} $$

Mean of Beta distribution

As a byproduct in the above derivation, we get that fact that W and T are independent. Then, we can use this to derive the mean of Beta distribution,

$$E(WT) = E(W)E(T) \implies E(X) =E\left(\frac{X}{X+Y}\right)E(X+Y),$$

rearrange to get,

$$E\left(\frac{X}{X+Y}\right) = \frac{E(X)}{E(X+Y)}$$

This result is clear Not True in general, but under our setting, we have this interesting result.

We can use this result to find the mean of $W \sim Beta(a, b)$ without the slightest trace of calculus.

$$E(W)=E\left(\frac{X}{X+Y}\right)=\frac{E(X)}{E(X+Y)}=\frac{a / \lambda}{a / \lambda+b / \lambda}=\frac{a}{a+b}$$

Getting Beta parameters in practice

The Beta distribution is the conjugate prior for many common distributions. We use it a lot. But in practice, how to figure out its parameters? It is sometimes useful to estimate quickly the parameters of the Beta distribution using the method of moments:

$$X \sim Beta(\alpha, \beta),$$ $$\alpha + \beta = \frac{E(X)(1-E(X))}{Var(X)} - 1,$$ $$\alpha = (\alpha + \beta)E(X),$$ $$\beta = (\alpha + \beta)(1 - E(X))$$

Here is the R code:

# calculate beta params using method of moments
cal_beta_params <- function(meanX, sdX) {
  varX <- sdX^2
  sum_ab <- meanX * (1 - meanX) / varX - 1
  a <- sum_ab * meanX
  b <- sum_ab * (1 - meanX)
  # return
  c("shape" = a, "scale" = b)
}

# example
cal_beta_params(meanX = 0.136, sdX = 0.103)

##    shape    scale 
## 1.370320 8.705559

Summary

To summarize, the bank–post office story tells us that: when we add independent Gamma r.v.s $X$ and $Y$ with the same rate $\lambda$ ,

the total $X+Y$ has a Gamma distribution;
the fraction $\frac{X}{X+Y}$ has a Beta distribution;
the total is independent of the fraction.

Examples

The PDF of Beta distribution can be U-shaped with asymptotic ends, bell-shaped, strictly increasing/decreasing or even straight lines. As you change α or β, the shape of the distribution changes.

I. Bell-Shape

The PDF of a beta distribution is approximately normal if α + β is large enough and α & β are approximately equal.

Intuition behind Bell-Shape

Why would Beta(2,2) be bell-shaped?

If you think:

α-1 as the number of successes
β-1 as the number of failures
Beta(2,2) means you got 1 success and 1 failure

So it makes sense that the probability of the success is highest at 0.5.

Also, Beta(1,1) would mean you got zero for the head and zero for the tail. Then, your guess about the probability of success should be the same throughout [0,1]. The horizontal straight line confirms it.

II. Straight Lines

α = 1 or β = 1, the beta PDF can be a straight line.

III. U-Shape

When $\alpha < 1, \beta<1$ the PDF of the Beta is U-shaped.

Reference

Tutorial Notes for Bayesian Statistics

Tue, 26 Jul 2022 00:00:00 +0000

Bayesian Inference

Posterior Distribution

The intuition behind the posterior distribution:

“It is a weighted version of likelihood! … just weighting the likelihood using my prior belief on theta …”

Useful Resource

17. Bayesian Statistics MIT 18.650 Statistics for Applications, Fall 2016
Statistical Rethinking (2022 Edition) github, this is a recommended bayesian textbook with video lectures.

Beta distribution as a prior

Mon, 25 Jul 2022 00:00:00 +0000

Introduction

The Beta distribution is a useful probability distribution when you want model uncertainty over a parameter bounded between 0 and 1.

In this post, we’ll explore how the two parameters of the Beta distribution determine its shape.

One way to see how the shape parameters of the Beta distribution affect its shape is to generate a large number of random draws using the rbeta(n, shape1, shape2) function and visualize these as a histogram.

Beta(1,1)

# Explore using the rbeta function
beta_sample <- rbeta(n = 1000000, shape1 = 1, shape2 = 1)

# Visualize the results
hist(beta_sample)

A Beta(1,1) distribution is the same as a uniform distribution between 0 and 1. It is useful as a so-called non-informative prior as it expresses than any value from 0 to 1 is equally likely.

Discover shape parameters

What happened if you set shape parameters to negative numbers?

# Modify the parameters
beta_sample <- rbeta(n = 1000000, shape1 = -1, shape2 = 1)

# Explore the results
head(beta_sample)

## [1] NaN NaN NaN NaN NaN NaN

Yes, NaN stands for not a number and the reason you got a lot of NaNs is that the Beta distribution is only defined when its shape parameters are positive.

# Modify the parameters
beta_sample <- rbeta(n = 1000000, shape1 = 100, shape2 = 100)

# Visualize the results
hist(beta_sample)

So the larger the shape parameters are, the more concentrated the beta distribution becomes. When used as a prior, this Beta distribution encodes the information that the parameter is most likely close to 0.5 .

# Modify the parameters
beta_sample <- rbeta(n = 1000000, shape1 = 100, shape2 = 20)

# Visualize the results
hist(beta_sample)

So the larger the shape1 parameter is the closer the resulting distribution is to 1.0 and the larger the shape2 the closer it is to 0.

Some Note for Pareto Distribution

Sat, 16 Jul 2022 00:00:00 +0000

Power Law Distribution

log-log-scale of cCDF showing you a straight line ?

This is the signature of the Power Law distribution.

R Code

library(zetaEDA)
library(zetaclv)
enable_zeta_ggplot_theme()

# transactional data for cohort 2019
cohort19 <- eg_trans_data %>%
  with_groups(
    cust,
    mutate,
    fp_yr = min(lubridate::year(date))
  ) %>%
  filter(fp_yr == 2019) %>%
  select(-fp_yr)

# build cbs data
dcbs <- generate_cbs(cohort19, timeUnit = "weeks")

## Note that: time unit is in < weeks >

head(dcbs)

##      cust x      t.x     litt sales sales.x      first     T.cal
## 1 uid0001 1 20.00000 2.995732  4644    1174 2019-12-02  79.00000
## 2 uid0005 0  0.00000 0.000000  1169       0 2019-08-08  95.57143
## 3 uid0006 1 50.71429 3.926208  1430     922 2019-04-20 111.28571
## 4 uid0010 0  0.00000 0.000000  2820       0 2019-02-15 120.42857
## 5 uid0011 0  0.00000 0.000000  6460       0 2019-01-15 124.85714
## 6 uid0012 0  0.00000 0.000000   473       0 2019-10-07  87.00000

Note that t.x is the Time between first and last transactions. This is the “observed” part of lifetime. Let’s look at the distribution of t.x.

dtmp <- dcbs %>%
  # remove single purchase customers
  filter(t.x > 0) %>%
  # get value of cdf, P(X <= x)
  mutate(cdf = ecdf(t.x)(t.x)) %>%
  # get ccef, P(X > x)
  mutate(ccdf = 1 - cdf)


dtmp %>%
  ggplot(aes(x = t.x, y = ccdf)) +
  geom_point(color = "red") +
  geom_line()

References

Youtube video: Network Analysis. Lecture 2. Power laws.

Note for Beta Distribution

Sat, 30 Jan 2021 00:00:00 +0000

This post is out of date, please check the new post named “Beta Distribution — Intuition, Derivation, and Examples”.

1 Why Beta Distribution?

1.1 Model probabilities

The short story is that the Beta distribution can be understood as representing a distribution of probabilities, that is, it represents all the possible values of a probability when we don’t know what that probability is.

1.2 Generalization of uniform

Give me a continuous and bounded random variable, em, except the Uniform distribution. That is another way to look at beta distribution, continuous and bounded between 0, 1; also the density is not flat. \[ X \sim Beta(a, b), \text{ where } a>0, \ b>0. \]

\[ f_X(x) = c \cdot x ^{a-1}(1-x)^{b-1}, \text{ where } x>0. \]

What is $c$? Just a normalization constant. We’ll get the value of $c$ later.

2 Construction

2.1 Bank and Post Office Story

Let $X$ be the waiting time at Bank,

\[ X \sim Gamma(n_1, \lambda) \]

Let $Y$ be the waiting time at Post Office,

\[ Y \sim Gamma(n_2, \lambda) \]

Assume $X$ and $Y$ are independent.

Then, what is the distribution of the proportion $\frac{X}{X+Y}$?

Define $T = X+Y$ be the total waiting time.

Clearly, $T \sim Gamma(n_1+n_2, \lambda)$, proved by MGF.

Define $W = \frac{X}{X+Y}$ , the proportion of waiting time at Bank to the total waiting time.

What is the distribution of $W$?

We need to derive $f_W(w)$, but first let’s find the joint pdf $f_{T,W}(t,w)$ \[ \begin{align} f_{T,W}(t,w) &= f_{X,Y}(x,y) \left | \frac{\partial(x,y)}{\partial(t,w)} \right|\\ &= \frac{1}{\Gamma(n_1)}\lambda^{n_1}x^{n_1 - 1}e^{-\lambda x} \frac{1}{\Gamma(n_2)}\lambda^{n_2}x^{n_2 - 1}e^{-\lambda y} \left|-t\right| \\ &= \lambda^{n_1+n_2}t^{n_1+n_2-1}e^{\lambda t} \frac{1}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} \\ &= \frac{\lambda^{n_1+n_2}t^{n_1+n_2-1}e^{\lambda t}}{\Gamma(n_1+n_2)} \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1}\\ &= f_T(t) \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} \end{align} \]

Then we find the marginal,

\[\begin{aligned} f_W(w) &= \int_0^\infty f_{T,W}(t,w) dt \\ &= \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} \cdot\int_0^\infty f_T(t)dt \\ &= \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} \end{aligned}\]

Since $f_W(w)$ is the pdf needed to be integrated to 1,

\[ \int_0^1\frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)}w^{n_1 - 1}(1-w)^{n_2-1} dw \equiv 1 \]

so the normalization constant should be

\[ c = \frac{\Gamma(n_1+n_2)}{\Gamma(n_1)\Gamma(n_2)} := \frac{1}{B(n_1, n_2)} \]

2.1.1 Summary

The connection between Gamma and Beta distribution helps us to find the normalization constant in Beta. In summary,

If $X \sim Gamma(\alpha, \lambda)$ and $Y \sim Gamma(\beta, \lambda)$ are independent, then $\frac{X}{X+Y} \sim Beta(\alpha, \beta)$.

2.2 plots

library(zetaEDA)
library(ggfortify)
enable_zeta_ggplot_theme()

Let’s check Beta density for some different parameters value.

How about $a = b = 1$? The Beta(1,1) is just the Unif(0,1).

ggdistribution(func = dbeta, x = seq(0, 1, .01), shape1 = 1, shape2 = 1) +
  labs(title = "Beta Density with a = 1, b = 1")

How about $a = b = \frac{1}{2}$?

How about $a = b = 2$?

How about $a= 2, \ b = 1$?

One more,

p <- ggdistribution(func = dbeta, x = seq(0, 1, .01), shape1 = 1.5, shape2 = 5, colour = "tomato", linetype = "dashed")

ggdistribution(func = dbeta, x = seq(0, 1, .01), shape1 = 5, shape2 = 1.5, colour = "blue", p = p) +
  labs(title = "Red: a = 1.5, b = 5\n Blue: a = 5, b = 1.5")

For more checking, click this link and try some parameters to check the density curve.

Have fun!

3 Reference

Note for Gamma Distribution

Sat, 30 Jan 2021 00:00:00 +0000

Motivation for Gamma Function

We all know how to compute the factorial of integer. BUT what is the factorial of 1/2?

In other words, how to interpolate the factorial function?

The gamma function can be seen as a solution to the following interpolation problem:

“Find a smooth curve that connects the points (x, y) given by y = (x − 1)! at the positive integer values for x.”

More details, check the wiki page.

Definition of Gamma Function

Definition 1 (Gamma Function) \[\begin{align} \Gamma(z) = \int_{0}^{\infty}x^{z-1}e^{-x}dx, \ \ z \in \mathbb{R}^+ \end{align}\]

For the the Gamma function, it is enough to know the following properties for now.

Lemma 1 \[\begin{align} & \Gamma(z+1) = z\Gamma(z), \ \ z \in \mathbb{R}^+ \\ & \Gamma(n) = (n-1)!, \ \ n = 1,2,3,... \end{align}\]

Easy to prove using integration by parts.

Now, what is $\Gamma(\frac{1}{2})$?

\[ \Gamma(\frac{1}{2}) = \int_{0}^{\infty}x^{-1/2}e^{-x}dx = ? \]

Recall that, $\int_{0}^{\infty} e^{-x^2} = \frac{1}{2}\sqrt{\pi}$, let $u = x^2$ and will get the result $\Gamma(\frac{1}{2} )= \sqrt{\pi}$.

Gamma Distribution

From the Gamma function, it is pretty natural to get Gamma pdf. JUST normalizing!

Clearly, \[ 1 = \int_0^\infty \frac{x^{r-1}e^{-x}}{\Gamma(r)}dx = \int_0^\infty f_X(x)dx, \ \ \ X := Gamma(r, 1) \]

What is the pdf for the general $Gamma(r, \lambda)$ ? Let

\[ Y = \frac{X}{\lambda}, \ Y \sim Gamma(r, \lambda) \] \[ f_Y(y) = f_X(x)\frac{dx}{dy} \]

We’ll get

\[ f(y; r, \lambda ) = \frac{\lambda ^{r}y^{r-1}e^{-\lambda y}}{\Gamma(r)} \]

Here, $r$ is called the shape parameter and $\lambda$ is called the rate parameter.

How to remember the Gamma pdf?

That’s my trick: exponential density times the power rise to (shape-1), then divided by normalizer.

Exponential density (very familiar): $\lambda e^{-\lambda x}$
power rise to (shape-1): $(\lambda x)^{r-1}$
normalizing constant (using shape): $\Gamma(r)$

\[ \begin{align} f(x; r, \lambda ) &= \frac{\text{exp density} \cdot \text{power}^\text{shape-1} }{normalizer} \\ & = \frac{\lambda e^{-\lambda x}(\lambda x)^{r-1}}{\Gamma(r)} \\ & = \frac{\lambda ^{r}x^{r-1}e^{-\lambda x}}{\Gamma(r)} \end{align} \]

Another way to remember is this:

Exponential key part: \[ e^{-\lambda x} \]
Add Power part: \[ \lambda^{\square} x^{\square} e^{- \lambda x} \]
- multiply the power part in exponential
- rate rises to shape
- variable rises to shape-1
\[ \lambda^{r} x^{r-1} e^{- \lambda x} \]
Add Normalizing part: \[ \frac{\lambda^{r} x^{r-1} e^{- \lambda x}}{\Gamma(r)} \]

Gamma & Exponential Connection

Let’s recall the Poisson Process, \[ N_t = \text{number of arrials up to time t} \sim Pois(\lambda t) \] The number of arrivals in the disjoint intervals are independent.

Let $T_1$ be the time of 1st arrival, \[ P(T_1 > t) = P(N_t = 0) = e^{-\lambda t} \ \implies T_1 \sim Exp(\lambda) \] Now, let $T_n$ be the time of nth arrival, that is, $T_n = \sum_{i=1}^n X_i$ , where $X_i \overset{\text{iid}}{\sim}Exp(\lambda)$

What is the pdf of $T_n$? Answer is Gamma!

Proposition 1 Gamma is the sum of iid Exponentials.

Proof:

Since $M_X(t) = \frac{\lambda}{\lambda-t}$, where $t < \lambda$,

\[ M_{\sum_{i=1}^n X_i}(t) = (M_X(t))^n = (\frac{\lambda}{\lambda-t})^n \]

It is enough to show the MGF of Gamma equals the above value.

Let $Y \sim Gamma(n, \lambda)$,

\[ \begin{align} M_Y(t) = E(e^{ty}) &= \int_0^\infty e^{ty} \frac{1}{\Gamma(n)} \lambda^{n}e^{-\lambda y}y^{n-1} dy \\ &= \frac{\lambda^n}{\Gamma(n)} \int_0^\infty e^{-(\lambda - t)y}y^{n-1}dy, \text{ let } u = (\lambda - t)y \\ &= \frac{\lambda^n}{\Gamma(n)} (\frac{1}{\lambda-t})^n \int_0^\infty e^{-u}u^{n-1}du \\ &= \frac{\lambda^n}{\Gamma(n)} (\frac{1}{\lambda-t})^n\Gamma(n) \\ &= (\frac{\lambda}{\lambda-t})^n \end{align} \]

Proved!

Remark. The Exponential is the continuous analog of the Geometric. Similarly, the Gamma is the continuous analog of the Negative Binomial.

Probability Theory Study Notes

Thu, 17 Dec 2020 00:00:00 +0000

Online Courses

Probability Theory and Stochastic Processes (video lecture).
- Description: These lectures encompass a full-year course in probability theory and stochastic processes, as taught at the University of California, San Diego (as Math 280). These lectures were produced in the 2020/2021 school year, in the midst of the Covid-19 pandemic.
- YouTube Video List here.
Graduate-level probability course by Bilkent University (in Turkey) (video lecture)
- Description: As explained by the professor in the 1st lecture, it uses the measure-theoretic approach to introduce probability concepts.
- YouTube Video List here.

Notes for LP Inequality

Wed, 24 May 2017 00:00:00 +0000

Here, I summarize some useful LP Inequalities with proofs.

Download PDF file here.

Deriving the conditional distributions of a multivariate normal distribution

Fri, 31 Mar 2017 00:00:00 +0000

Overall, the intuition behind the conditional distribution of a bivariate normal is that even though the two variables are correlated in the joint distribution, they can still be treated as independent when you’re looking at the distribution of one variable, given a fixed value of the other variable. This allows you to use the normal distribution, which is a well-understood and widely-used probability distribution, to model the conditional distribution of a bivariate normal.

More generally,

Assume we know there’s a theorem that says all conditional distributions of a multivariate normal distribution are normal.