In statistical inference from a given data set, if the inference task is known (e.g., inferring the value of unknown parameter), one can perform data reduction by constructing sufficient statistics. In this work, we consider the problem of data reduction for the scenario where the inference tasks are not pre-specified. This scenario arises, for example, when different persons want to infer different quantities from the compressed dataset. We propose to perform data reduction by carefully selecting a subset of data points. We show that the KL distance between the pdf estimated from the original dataset with Kernal density estimation and the pdf estimated from the selected subset is a submodular function. Using this property, we construct efficient algorithm for data reduction.