In our previous blog, we have discussed about Variational Bayesian Inference model, its importance and application areas. In this part, we are going explain the concept of this model with mathematical foundation.
In practice, evaluation of the posterior is usually difficult because we can’t easily evaluate, especially when analytical solutions are not available and numerical integration is too expensive. The goal of the Variational Bayesian approximation is to approximate posterior (θ) by a simpler distribution q(z) for which marginalization is tractable. q(z) is a probability density function (pdf) of hidden variables that is maximally similar to the true posterior.
Given observed (incomplete) data x, latent or hidden variables z and parameters θ, maximize the log-likelihood with respect to θ:
where we have written the integral over the joint distribution for x and z.
Using Jensen’s inequality (Jensen, 1906), any distribution over hidden variables q(z) gives :
F(q, Θ) is a lower bound on the log-likelihood.
The difference between the log-likelihood and the bound
As we have said earlier, q(z) is any probabilistic density function over the hidden variables gives rise to a lower bound on . KL(q||p) is the Kullback-leibler divergence between p(z|x;θ) and q(z). Since KL(q||p)>=0, we can say ≥ F(q, θ). Hence F(q, θ) is a lower bound on Θ) i.e., the log-likelihood. The idea in Variational Bayes (VB) approximation is to select a simple approximation q(z) to the complicated conditional density p(x| Θ). The Kullback-Leibler divergence gives us a measure of the discrepancy between q(z) and true posterior, p(z|x;Θ); so the goal is to find an approximation q(z) which minimizes KL(q||p), hence maximizes F(q, Θ).
From the above discussion, we can conclude that Variational Bayesian inference based model is better than other conventional techniques.