Undirected graphical models are a popular class of statistical models, used in a wide variety of applications. In many settings, however, it might not be clear which subclass of graphical models to use, particularly for non-Gaussian and non-categorical data. In this paper, we consider a general sub-class of graphical models where the node-wise conditional distributions arise from exponential families. This allows us to derive multivariate graphical model distributions from univariate exponential family distributions, such as the Poisson, negative binomial, and exponential distributions. Our key contributions include a class of $M$-estimators to fit these graphical model distributions; and rigorous sparsistency analysis for these $M$-estimators.