We propose a Bayesian approach to the compressive sensing problem. In our formulation, the measurements are modeled as a known linear combination of the sparse coefficients plus additive white Gaussian noise of known variance, and the sparse coefficients are modeled as independent Gaussian random variables whose unknown means and variances are drawn independently from a finite set according to known priors. So that the coefficient vector is sparse, the most a priori probable {mean,variance} combination can be set to {0,0}. Leveraging the discrete nature of the Gaussian mixture parameters, we propose an efficient tree-search which finds the mixture vectors that account for the dominant posterior probabilities. The MMSE estimate of the sparse-coefficient vector then follows directly from an average over these dominant posteriors. A key feature of our tree-search is a fast metric update which operates on the "matched filter outputs" (i.e., the inner products between the measurement vector and the columns of the combining matrix), hence the name ``Fast Bayesian Matching Pursuit.''