Measuring similarity is a crucial task that is richer and broader than learning metric functions. For example, similarity can arise from the process of aggregating the decisions of multiple latent components, where each compares data in its own way by focusing on a different set of features. In this paper, we propose Similarity Component Analysis (SCA), a probabilistic graphical model that discovers those latent components from data.  The final similarity measure is then obtained by combining the local similarity values with a (noisy-)OR gate. We derive an EM-based algorithm for fitting the model parameters with similarity-annotated data from pairwise comparisons. We validate the SCA model on both synthetic and real datasets for classification and exploratory data analysis.