Actually even the quantum fields support ("are") representations of Poincaré group acting on a suitable vector bundle whose sections are, in fact, the considered fields:

$$\phi^A(x) \to S^A_B(\Lambda) \phi^B (\Lambda x + t)$$

where $(\Lambda, t)$ is the generic element of the Poincaré group. Notice that $S$ sees only the Lorentz part of the Poincaré group and defines a vector representation on its own right.

When you fix an event in spacetime, i.e., you deal with a fiber of the vector bundle only, the Lorentz group part of the semidirect product you wrote acts on that fiber, by means of the representation

$$\phi^A(x) \to S^A_B(\Lambda) \phi^B (x)$$

leaving the fiber fixed. $S$ is a finite dimensional representation of the Lorentz group, since the fiber has finite dimension as a vector space (the range of the index $A$ if finite). However this fiber does not admits a Hilbert space structure invariant under that representation, for this reason the representation is not unitary and it can be finite dimensional.

It is fundamental to notice that, in QFT, $\phi^A$ is **also** an operator in the Hilbert space of the theory, and Poincaré group is a continuous symmetry of the physical system: It leaves the transition probabilities invariant. Essentially due to Wigner theorem, one has that this symmetry can be implemented **unitarily** in the Hilbert space of the theory by means of a (strongly) continuous unitary representation $U_{(\Lambda, t)}$.

It is natural tu assume that, under this unitary representation of Poincaré group *acting in the Hilbert space*, the quantum fields viewed as operators, transform covariantly with respect the other representation *acting in the spacetime*:

$$U_{(\Lambda,t)} \phi^A(x) U_{(\Lambda,t)}^\dagger = S^A_B(\Lambda) \phi^B (\Lambda x + t)\:.$$