The questions within the category "Reviews" and its subcategories will be a special type of questions in any case (two related questions or only one). As dimension10 already mentioned, we should not start his category before this new type is defined and available. This to avoid later ugly handmade synch corrections. This type should fit somehow in the Q2A framework to avoid a completely new development. In order to approach a more detailed specification of your requirements, I try to formulate some more detailed explanations and questions.
1) To ease the implementation I would prefer to have only one question per paper. I already made a proposition on how such a question could look like. To keep the information concentrated, I copy my proposition here again:
The text includes a link to the paper on ArXiV and the papers title. Two voting inputs are possible, one for originality and one for accuracy/quality. Authorized users may give one vote on each of them. A score value is computed from the total of each vote criteria and is displayed within the question.
2) The asking user (in the Q2A sense) in the bottom line will be the bot for papers imported automatically or a real user, if imported manually. I have grayed the usual votes on the top left in my proposition, because it is unclear to me, who should earn the points from such a question. The bot? The editing user? Nobody?
3) Should there be a mechanism to change this asking user, if a question imported by a bot is edited by a real user? What if multiple users edit such questions?
4) Who exactly earns the votes for originality and quality and/or the score value and how? The authors of the papers? How would you "link them individually to authors by hand when the authors join the site"? By direct access to the database? Q2A does not provide any means to distribute votes to multiple users.
5) What shall the semantics of answers and comments to review questions be? At least earning votes is easy there, we can use the normal Q2A behavior.
6) I assume that in any case, we need a plugin to import papers into review questions manually. OK?
You help best for a fast implementation of your demands, if you provide precise requests to me.
Changing the voting system or the tagging system in Q2A is not only a core-hack, it's a hardcore hack! The related code is located deep in the framework and percolated over many files. If somebody is interested in details, you may download the code here, it is open source. 5 minutes installation time mentioned there are true, but not for a reviewing system. Propositions are welcome.