How about a description in words instead? I think the mathematics is not the point so much. Other answers lay out the mechanics in detail. The short answer is that the theory has infinitely many divergences which cannot be absorbed by fixing finitely many couplings to their observed values. This means it is not predictive, or in more fancy words does not provide algorithmic compression: the amount of input it requires equals the amount of output it produces.
But the talk about divergences masks the physics of the situation, which I'll try to describe:
Any theory has couplings which describe the possible interactions of the system. When you probe the system at different length scales, or energy scales, those couplings change in a calculable way. This is the process of renormalization, which fundamentally has nothing to do with infinities.
Renormalizable theory is one that can be continued to high energies, or short distances, without encountering any difficulties. This means all couplings remain small, and all calculations are reliable. In non-renormalizable theories, the higher energies you probe the system, the stronger the couplings become, at some stage they become infinite. This means that you cannot do reliable calculations, which is normally an indication that there are some physical effects that you are missing. There are many examples where such physical effects are well-understood, for example new degrees of freedom (which are not visible at low energies) become important at those energy scales.
In gravity, when you calculate in perturbation theory, for small perturbations around flat space, it looks like the coupling becomes strong and something is missing. The mechanics of seeing this is straightforward and not that illuminating. Perhaps one intuitive way to understand this is that in gravity the interaction coupling is governed by the energy, so almost by definition it grows with energy...Most likely scenario is that we need new degrees of freedom to complete the theory at short distances. One suggestion for "UV completion" is string theory.
There is another scenario which may apply to gravity. Once the couplings become strong, you cannot any more reliably calculate anything, and that includes their scale dependence. So it is a logical possibility that they will stabilize at some finite value, and don't go all the way to infinity. This scenario is called asymptotic safety. In my mind this is very unlikely, and there is no indication* this scenario applies to gravity, but it remains a logical possibility.
(*I know about the various claims, this is a statement of my personal opinion.)
This post imported from StackExchange Physics at 2014-04-01 16:30 (UCT), posted by SE-user user566