This work investigates the phase retrieval problem, which aims to recover an unknown (possibly high-dimensional) signal from its magnitude measurements. Such a problem has wide applications in imaging science such as X-ray crystallography and coherent diffraction imaging. This research is along the line of the nonconvex method recently proposed to solve phase retrieval problems, but employs a lower-order nonconvex nonsmooth loss function for designing fast algorithms. We showed that the new algorithm still enjoys the best possible statistical efficiency and linear convergence rate, and is more robust to random measurements, easy to implement, and most importantly, runs much faster than all existing algorithms of the same kind. We then developed accelerated algorithm and stochastic algorithm, which have substantially reduced runtime in practice, and further designed the median-based algorithms, which are robust to outliers in the measurements. We show the convergence guarantee for all these algorithms.