We propose an algorithm, called orthogonalizing EM algorithm, for estimating penalized regression models. This algorithm is based on two simple ideas: (1) active expansion of an arbitrary regression matrix to an orthogonal matrix and (2) iterative imputation of missing data. Despite its simple form, theory and numerical examples will demonstrate that the proposed algorithm performs well for massive data.