Simple physics comments are welcome, for example, if you are confused on a point you can say "what's going on in Section 3?" as a comment. You can say "The results are compatible with the work I know from ref A and ref B, but the methods are different, so it is not clear that the thing is accurate" and so on. The idea is to keep focused on the technical content, so that comments about bad style or confusing presentation are better dealt with by adding a rephrasing of the text in the question/summary of the paper, or commenting to get a clarification.
But at this early stage, I personally was hoping that we could set the tone for the site on a high level by making the reviews as complete as you would for a journal. That's not a requirement, it's just a request for the first dozen or so reviews. This hopefully doesn't require duplicative work--- if you have old journal reviews you can import the papers and post them here with only minor edits, even anonymously. But then as the review section grows, and people expect full reviews with content as a matter of routine, other reviews which are more probing or incomplete will be just as welcome, but less upvoted.
On a voting site, the standards for voting up/down are dynamically established at the early stages, and then frozen in. The idea is to set a high tone so that when the site grows, the voting is professional, and so that people will automatically feel disinclined to post political statements with no physics content.