As is known, the presence of gauge anomalies leads to breakdown of the unitarity of the gauge theory.
One way to understand this is to involve the BRST quantization of the gauge field theory. It reads as follows. The gauge invariance describes in fact the redundancy of the Hilbert space of gauge variant rays to the space of gauge-invariant rays. In the result, the correct state in gauge theory is defined to be invariant under the gauge transformation. In a path integral formulation of the gauge theory, this redundancy is nothing but reducing the integration over all gauge fields configuration to the integration over ones satisfying the gauge fixing condition; the latter defines a surface (gauge orbit) in the space of gauge fields configurations.
For particular choices of the gauge fixing condition this redundancy leads to generating the ghosts action. The ghoats are unphysical states with indefinite norm in the Hilbert space. Although they mediate the physical processes, they can't be in- or out- states, so their indefinite norm doesn't break unitarity. They cannot contribute to the physical state because of the Slavnov-Taylor identities; the latter are a direct consequence of underlying gauge invariance.
If, however, the gauge anomaly is present, then the Slavnov-Taylor identities are broken. Therefore the ghosts contribute in the Hilbert space of physical states, and unitarity is broken.
It is always possible to choose the gauge fixing in a way that ghosts are not present? In abelian gauge theories an example is the Lorentz gauge. In non-abelian gauge theories, an example is the so-called auxiliary gauge. With this choice of gauge fixing conditions, there are no intermediate states with indefinite norm whose presence leads to the violation of unitarity in case of a gauge anomaly. So where exactly is the unitarity breakdown hidden in the case of fixing the gauge condition in a way such that the ghosts are absent?
In fact, although gauge invariance says us that all gauge fixing conditions are equivalent, and one might say that the unitarity has to be preserved for all possible choices. However, I may say that the gauge anomaly requires the quantization by using the ghost-free choices of fixing condition, so that unitarity is preserved (as long as I don't see where the unitarity breakdown is hidden).