Muon spin spectroscopy

2014-01-07T09:06:00Z

129.129.157.152:

'''Variational message passing''' (VMP) is an [[approximate inference]] technique for continuous- or discrete-valued [[Bayesian networks]], with [[conjugate exponents|conjugate-exponential]] parents, developed by John Winn. VMP was developed as a means of generalizing the approximate [[Variational Bayesian methods|variational methods]] used by such techniques as [[Latent Dirichlet allocation]] and works by updating an approximate distribution at each node through messages in the node's [[Markov blanket]].

==Likelihood Lower Bound==

Given some set of hidden variables <math>H</math> and observed variables <math>V</math>, the goal of approximate inference is to lower-bound the probability that a graphical model is in the configuration <math>V</math>. Over some probability distribution <math>Q</math> (to be defined later),

:<math> \ln P(V) = \sum_H Q(H) \ln \frac{P(H,V)}{P(H|V)} = \sum_{H} Q(H) \Bigg[ \ln \frac{P(H,V)}{Q(H)} - \ln \frac{P(H|V)}{Q(H)} \Bigg] </math>.

So, if we define our lower bound to be

:<math> L(Q) = \sum_{H} Q(H) \ln \frac{P(H,V)}{Q(H)} </math>,

then the likelihood is simply this bound plus the [[relative entropy]] between <math>P</math> and <math>Q</math>. Because the relative entropy is non-negative, the function <math>L</math> defined above is indeed a lower bound of the log likelihood of our observation <math>V</math>. The distribution <math>Q</math> will have a simpler character than that of <math>P</math> because marginalizing over <math>P</math> is intractable for all but the simplest of [[graphical models]]. In particular, VMP uses a factorized distribution <math>Q</math>:

:<math> Q(H) = \prod_i Q_i(H_i), </math>

where <math>H_i</math> is a disjoint part of the graphical model.

==Determining the Update Rule==

The likelihood estimate needs to be as large as possible; because it's a lower bound, getting closer <math>\log P</math> improves the approximation of the log likelihood. By substituting in the factorized version of <math>Q</math>, <math>L(Q)</math>, parameterized over the hidden nodes <math>H_i</math> as above, is simply the negative [[relative entropy]] between <math>Q_j</math> and <math>Q_j^*</math> plus other terms independent of <math>Q_j</math> if <math>Q_j^*</math> is defined as

:<math>Q_j^*(H_j) = \frac{1}{Z} e^{\mathbb{E}_{-j}\{\ln P(H,V)\}} </math>,

where <math>\mathbb{E}_{-j}\{\ln P(H,V)\}</math> is the expectation over all distributions <math>Q_i</math> except <math>Q_j</math>. Thus, if we set <math>Q_j</math> to be <math>Q_j^*</math>, the bound <math>L</math> is maximized.

==Messages in Variational Message Passing==

Parents send their children the expectation of their [[sufficient statistic]] while children send their parents their [[natural parameter]], which also requires messages to be sent from the co-parents of the node.

==Relationship to Exponential Families==

Because all nodes in VMP come from [[Exponential family|exponential families]] and all parents of nodes are [[Conjugate prior|conjugate]] to their children nodes, the expectation of the [[sufficient statistic]] can be computed from the [[normalization factor]].

==VMP Algorithm==

The algorithm begins by computing the expected value of the sufficient statistics for that vector. Then, until the likelihood converges to a stable value (this is usually accomplished by setting a small threshold value and running the algorithm until it increases by less than that threshold value), do the following at each node:
# Get all messages from parents
# Get all messages from children (this might require the children to get messages from the co-parents)
# Compute the expected value of the nodes sufficient statistics

==Constraints==

Because every child must be conjugate to its parent, this limits the types of distributions that can be used in the model. For example, the parents of a [[Gaussian distribution]] must be a [[Gaussian distribution]] (corresponding to the [[Mean]]) and a [[gamma distribution]] (corresponding to the precision, or one over <math>\sigma</math> in more common parameterizations). Discrete variables can have [[Dirichlet distribution|Dirichlet]] parents, and [[Poisson distribution|Poisson]] and [[Exponential distribution|exponential]] nodes must have [[gamma distribution|gamma]] parents. However, if the data can be modeled in this manner, VMP offers a generalized framework for providing inference.

==External links==
* [http://research.microsoft.com/infernet Infer.NET]: an inference framework which includes an implementation of VMP with examples.
* [http://dimple.probprog.org/ dimple]: an open-source inference system supporting VMP.
* An [http://vibes.sourceforge.net/ older implementation] of VMP with usage examples.

==References==
*{{cite journal |first1=J.M. |last1=Winn |first2=C. |last2=Bishop |title=Variational Message Passing |journal=Journal of Machine Learning Research |volume=6 |pages=661–694 |year=2005 |url=http://johnwinn.org/Publications/papers/VMP2005.pdf |format=PDF}}
*{{Cite thesis |type=PhD |title=Variational Algorithms for Approximate Bayesian Inference |url=http://www.cs.toronto.edu/~beal/thesis/beal03.pdf |last=Beal |first=M.J. |year=2003 |publisher=[http://www.gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit], University College London }}

[[Category:Bayesian networks]]

formulasearchengine - User contributions [en]

Muon spin spectroscopy