Although widely used in practice, the practical behavior and accuracy of the popular module identification technique called "modularity maximization" is not well understood. Here, we show that, when applied to modular networks, the modularity function Q exhibits extreme degeneracies in which there are an exponential number of structurally dissimilar high-scoring solutions and that there is typically no clear global maximum. Using three real-world metabolic networks as examples, we further show that the degenerate solutions can fundamentally disagree on many, but not all, partition properties such as the composition of the largest modules and the distribution of module sizes. These results significantly clarify the behavior of this method in scientific contexts, as well as explains why so many heuristics perform well in practice at finding high-scoring partitions and why different heuristics often disagree on the modular composition the same network. Time allowing, we'll conclude with some forward-looking thoughts about the general problem of identifying network modules from connectivity data alone.