Lets begin by defining more precisely the objects which we are dealing with there. Let \(\phi=(X^\mu):\Sigma\longrightarrow \mathcal M\) an embedding of our world-sheet \(\Sigma\) in spacetime \(\mathcal M\) with metric \(\eta\), \(h = \phi^*g\in \mathrm{sec}T^{(0,2)}(\Sigma )\) the metric induced on the world-sheet \(\Sigma\) by the background metric \(\eta\), and let \(\gamma = \mathrm{sec}T^{(0,2)}(\Sigma )\) be a metric on the world-sheet. Observe that \(h\) and \(\gamma\) are unrelated: the former is induced by the extrinsic geometry, the latter defines the intrinsic world-sheet geometry. The world-sheet is oriented by the 2-form \(\omega \in \bigwedge^2 (\Sigma)\) defining our element of volume. The Hodge dual \(\star_\gamma \) is then the duality operator define on the oriented pseudo-Riemannian surface \((\Sigma,\gamma,\omega)\) (and not the Hodge dual induced on the bundle of differential forms of the background metric!). We may then define the action functional

\(S[X^\mu,\gamma]=\frac{1}{2}\int_\Sigma g_{\mu \nu} [X] dX^\mu \wedge \star_\gamma dX^\nu\)

where \((g_{\mu \nu})\) is family of functions living on \(\Sigma\) depending on the non-linear sigma model you are dealing with. (For a bosonic string moving in a curved background \(\mathcal M\) with curved metric \(G_{\mu\nu}\), we let \(g_{\mu\nu}=G_{\mu \nu}\). To study the interaction with gravitons, let \(G_{\mu\nu}=\eta_{\mu\nu}+\xi_{\mu\nu}\) for a small metric pertubation \(\xi\) . See Polchinski 3.7 for details).

Since you only want to know how to study the calculus of variations with this formalism, lets choose the linear sigma model for which \(g_{\mu\nu}=\eta_{\mu\nu}\). Observe that from the calculus of exterior differential forms, the component version of this action is actually

\[S[X^\mu,\gamma]=\frac{1}{2}\int_\Sigma \eta_{\mu \nu} \gamma(dX^\mu, dX^\nu)\omega =\frac{1}{2}\int_\Sigma \eta_{\mu \nu} \gamma^{ab} \partial _aX^\mu \partial_bX^\nu|\gamma|^{1/2}d\sigma \wedge d\tau\]

This is just the Polyakov action. The fact is that \(\alpha \wedge \star_\gamma \beta = \gamma(\alpha,\beta)\omega\) for all \(\alpha,\beta\in \bigwedge\Sigma\) of the same degree. So, in the enunciation of this question, the author forgot to include the volume element, and the corresponding metric determinant, in the second line of the second paragraph).

To do the variation using geometric formalism, consider the variation \(X^\mu \mapsto X^\mu+\bar{\delta}X^\mu\). (The bar on the delta will distinguish the variation symbol to the coderivative, to be introduced below). The action becomes, after using the Leibnitz rule for the exterior derivative, Stokes theorem and ignoring boundary (\(\in \partial \Sigma\)) terms,

\[\bar{\delta}_XS[X,\gamma]=-\int_\Sigma \eta_{\mu \nu} \bar{\delta}X^{\mu}\wedge d \star_\gamma d X^\nu=0\]

Recalling that, for surfaces of Lorentz signatures the coderivative is \(\delta=-\star_\gamma d \star_\gamma\) we get the equation of motion \(\delta d X^\mu=0\). If \(\square = d\delta + \delta d\) denotes the Laplace-Beltrami operator on the world-sheet, our equations of motion are just the wave-equations

\[\square X^\mu=0\]

since \(\delta X=0\) for all 0-forms \(X\in \bigwedge^0\Sigma\).

To carry the world-sheet metric variation now, I will introduce the notation \(\langle T|V \rangle\) for the contraction of the tensors \(T,V\) of the same degree, and \(\alpha \lrcorner \beta \) for the contraction of the forms \(\alpha,\beta\) of the same degree. Observe that by varying the world-sheet metric \(\gamma\), one is changing both the volume element \(\omega\) and the Hodge star \(\star_\gamma\). The variation of terms involving the Hodge dual is not so trivial, and in fact, can be proven to be just

\[\bar{\delta}(\alpha \wedge \star_\gamma \beta)=\big\langle \frac{1}{2}(\alpha\lrcorner\beta)\gamma^{-1}-\alpha\otimes\beta\big|\bar{\delta}\gamma\big\rangle\omega\]

where by \(\gamma^{-1}\) I actually denote the inverse metric of \(\gamma\) (namely, whose elements are \((\gamma^{ab})\) in world-sheet coordinates). This formula can be used to perform the variation of the Einstein-Hilbert action in differential forms which I commented in another question here and is based on Thirring's course on mathematical physics. I have also seen it been derived in Göckeler & Schücker and in this paper. If someone is really interested, I can provide a more easier derivation of this result upon request. Anyway, applying this to our equation gives:

\[\bar{\delta}_\gamma S[X,\gamma]=-\int_\Sigma \big\langle \frac{1}{2}(dx^\mu\lrcorner dX_\mu)\gamma^{-1}-dX^\mu\otimes dX_\mu\big|\bar{\delta}\gamma\big\rangle\omega\]

and the equation of motion completely fix our (auxiliary) world-sheet metric as

\[\frac{1}{2} (dX^\mu\lrcorner dX_\mu) \gamma^{-1}=dX^\mu\otimes dX_\mu\]

By recalling that \(h_{ab}=\partial_aX^\mu \partial_bX_\mu\) is our induced metric on the world-sheet, the above equation just becomes the well-known formula

\[\frac{1}{2}\gamma^{cd}\gamma^{ab}h_{ab}=h^{cd}\]

that you may grab your copy of Polchinski and see Eq.(1.2.16) (with slight changes in index positioning). So, to summarize, in differential forms, the equations of motion for the bosonic Polyakov action are just

\[\square X^\mu=0 \\ \frac{1}{2} (dX^\mu\lrcorner dX_\mu) \gamma^{-1}=h\]