Graphical Models, Exponential Families, and Variational Inference

by Martin J. Wainwright and Michael I. Jordan

Abstract

The formalism of probabilistic graphical models provides a unifying frameworkforcapturingcomplexdependenciesamongrandom variables,andbuildinglarge-scalemultivariate statisticalmodels. Graphical models have become a focus of research in many statisti-cal, computational and mathematical fields, including bioinformatics, communicationtheory, statisticalphysics,combinatorialoptimiza-tion, signal and image processing, information retrieval and statistical machine learning. Many problems that arise in specific instances — including the key problems of computing marginals and modes of probability distributions — are best studied in the general setting. Working with exponential family representations, and exploiting the conjugate duality between the cumulant function and the entropy for exponential families, we develop general variational representa-tions of the problems of computing likelihoods, marginal probabili-ties and most probable configurations. We describe how a wide variety ofalgorithms—amongthemsum-pro duct,clusterv ariationalmeth-o ds,exp ectation-propagation,meanfieldmetho ds,max-pro ductand linearprogrammingrelaxation,asw ellasconicprogrammingrelax-ations—canallb eundersto o dintermsofexactorappro ximateforms ofthesev ariationalrepresen tations.Thev ariationalapproac hpro vides acomplemen taryalternativ etoMark o vc hainMon teCarloasageneral sourceofappro ximationmetho dsforinferenceinlarge-scalestatistical models