A unitary representation of a group $G$ is a mapping $U$ from $G$ to the algebra of unitary linear operators on a Hilbert space $V$. This space is called the representation space, and one says that $G$ acts unitarily (or is unitarily represented) on $V$. Thus for every $g\in G$, the representation defines a mapping $\psi\to U(g)\psi$ with the natural compatibility properties. In particular, unit vectors are mapped to unit vectors. In quantum mechanics, the elements of $V$ (or only the unit vectors, depending on the author) are referred to as the states, and the thing that acts on the states is the group element.

Saying that states of a certain form form a representation is loose talk for saying that all states of this certain form form a Hilbert space (with inner product taken from the context) on which $G$ acts unitarily (in the obvious way, or in the way defined by the context).

Finding a representation means finding a Hilbert space $V$ and the action of $G$ on it. Typically, one pieces $V$ together from constituents already known. This defines the states of interest. When the construction is elegant, the notation used for the states is such that the group action is obvious; otherwise the group action has to be defined explicitly and one must prove that products behave correctly.

Matrices are just the special case when $V$ is the Hilbert space of complex column vectors. The representation of states as column vectors is appropriate for an $N$-level system and the unitary group $U(n)$, but for other groups it is usually preferable to use a different notation for the states that is adapted to the group.