# A suggestion on how to review and evaluate ongoing research on PhysicsOverflow

+ 3 like - 0 dislike
366 views

As Ron Maimon said here, it would be nice to have the possibility to obtain honest feedback on and discuss specific ongoing research (papers) unhindered by any "political" obstructions on PhysicsOverflow.

My idea about how to implement this on PhysicsOverflow is to dedicate a special category (I called it Refereeing) which can be choosen if needed when asking a question, to the process of peer reviewing.

In this category, a user can for example post in a question his newest work (or a paper of other authors he likes to be discussed and evaluated) and summarize / explain it. Other members of the PhysicsOverflow community can then vote on this paper (the question) and put their reviews and evaulations down as answers (which can be voted on too).  Both, the paper in the question and the reviews in the answers can be further discussed in the comments.

retagged Jul 11, 2014

+ 2 like - 0 dislike

I think the proper model is to automatically create a link to every paper on arxiv, either at the beginning when the site goes live, updated daily with new submissions, or on a case by case basis, as they are referenced by others, whose "question" is simply the arxiv paper title, the question body is simply a link to the arxiv page or to the pdf directly, and whose "answers" are simply refereeing comments regarding the paper's accuracy. This is extremely useful for two reasons:

1. It is value added--- it immediately distinguishes this site from stackexchange, in that refereeing is not Q/A. This is important, as a clone of existing sites is not original enough to get widespread attention and acceptance, you need something which adds new value.

2. It explains the "free speech" aspects of political moderation--- you need to be free to criticize when you are refereeing, and to retract wrong statements when you make a mistake. Scientific refereeing is extremely hostile (when work is wrong, or duplicative of earlier literature).

3. It will draw in EVERYBODY. I mean EVERYBODY. The moment you referee a Maldacena paper, Maldacena will be here within a matter of days. If you make a comment on an 't Hooft paper, 't Hooft will be here in days. You saw this on stackexchange. It is coercive, it forces the whole physics community to pay attention immediately.

Because of this coercive aspect, there is zero chance that this site will be ignored, or wither or die. It will be important, simply because if too few physicists are contributing, all you have to do is go to the refereeing section and criticize some of their mistakes in the published literature, or point out insufficient attribution to earlier work, and boy, will then come. If they have referee reports lying around, or if they have some things that they remember wanting to say about this or that paper, then they will do so at the appropriate location.

The paper-questions themselves can spin off questions related to the paper, by adding related questions, so as to clarify the content. When the paper is old, not freely available, then the question body can and should include a complete summary of the contents of the paper, edited wiki-style, by various people, but mostly belonging to one person. So, for example, if you want to cite a result of Newton in the "Principia", or talk about this work, you can add it to the refereeing section, with a summary of the relevant result, working out all the calculations. The same for an old Polyakov paper, or something from the 1970s by Wilson, or Jackiw or someone like this.

In this case, I think it is important to be able to criticize summaries for their accuracy somewhere, so that a good summary emerges. If the questions are edited community style, with honest comments, this shouldn't be hard, as the work is not original, and all you need to do is stay faithful to the original. But there are always simplifications possible, it is usually possible to summarize old literature very well with newer notations and methods that were developed later. This adds important forward-references to future work to old literature, something which you don't get from following the references in the literature. This is also value added, and it is better than a reverse-citation index, because it is curated by humans, and points to the relevant clarifications immediately.

In doing this, it is perhaps useful to add a bot to the site which makes a question automatically for every arxiv paper every day, and adds them to the "referee" section.

For the referee section, I think a good guideline is that the upvoting/downvoting should be in two parts:

1. Originality: this is the originality of the paper, how original is it?

2. Significance: this is the importance of the paper--- how important is it?

Then the answers can point out various mistakes in the paper, and these can be upvoted/downvoted according to accuracy, as usual. This allows the site to substitute for a journal with very little modification to the software--- simply automatic addition of papers.

But I am going to say up front that the site will get VERY heavy traffic the moment you do this, because every single academic will worry about comments about their paper. You should try to avoid people writing nice things on these pages, as these will be the authors, or their colleagues patting each other on the back. The comments should be for explanations, reworking of results, pointing out errors, pointing out missing citations, the usual negative refereeing.

This is an atomic bomb onthe academic community, because they can't stop you from doing it, and they can't help participating when it happens.

answered Feb 21, 2014 by (7,720 points)
edited Feb 21, 2014

Having a ton of arxiv links everywhere is a lot of noise. It's a lot easier to answer questions than read papers. Do you really think people will referee papers? I wouldn't.

It would be fine to have this as a separate section of the site, but let's focus on a Q&A dealy first.

Refereeing is only "noise" for those that think that stackexchange "question/answer" model is ideal. I think it sucks, for one, but it's the best available thing right now. Since there is a new site being made, which needs to differentiate itself by stackexchange in as many ways as possible, this is the ideal opportunity to make the "science 2.0" site that people have been trying so hard to do for the past 5 years, and failing to do. It only requires minor modifications to the stackexchange format.

If you don't want to participate, stay on the Q/A section, it won't be much different than stackexchange, except for higher quality Q/A (if you use refereeing to suck in the professionals). I promise to cogently sincerely negatively referee ~1 paper a day, I'll look for mistakes, people make them once in a while, and maybe Lubos's can do the same (he does it on his blog anyway, it won't be any different). Negative refereeing means the author will come (and they will come) to answer your comments, and they can make their own reports on others' papers, it will be easier if we allow them to comment anonymously. In this way, there can easily be a positive chain reaction coefficient, and suddenly, all the physicists are here, commenting on each others' papers.

This is so much of an advantage, that it would be foolish not to take advantage of the fact that nobody has managed to do it yet. The reason others have failed (they tried!) is that they made a closed commenting model, where you had to be invited (!) to comment on a paper, and if you weren't classified as an "expert" you couldn't comment on a paper. It's unbelievable stupidity on the part of the other science 2.0 sites, but it means there is a great opportunity for this site. Don't squander it.

Remember, if you don't want to referee, don't go to the referee section. But I predict, after about a dozen papers are refereed by folks here, there will be a dozen people commenting on other people's papers (if the commenting mechanism is transparent), and then it's a chain reaction, those people will comment, and so on. The key is to give positive and negative feedback, with technical meat. I can do this on a handful of papers per day, but it's enough to start the process off.

This happened on stackexchange, regarding 'tHooft's papers on deterministic QM (which I criticized somewhat, although I supported the general program. Lubos also criticized, but less constructively), and then, whaddayaknow, there was 't Hooft, answering the criticisms! It was useful. Also, Lubos criticized the loop guys, and then one of the loop guys came over to rebut the criticism. This is a professor trap, criticizing their papers is coercive, it WILL bring them here, 100% guaranteed, but you have to KEEP them here, with a good site organization.

Once people see it works (it does work), about 6-12 months later, all the other sites are going to try to do the same thing themselves. The first site to do it has an advantage, and I would like this site to be the one, because I know the moderation here will not exclude eccentric people, and probably will not censor views that are contrarian, it will just give them less prominence when they are wrong. I don't think power can go to Dilaton's head, or to Dimension10's, they are pretty much aware of the issue of moderation abuse, and will probably not succumb to it.

Thanks for these nice additional explanations Ron, it makes me personally very excited about the refereeing section as a professor trap too! I can almost see the exchanges between Lumo and LQG guys in advance ... :-) To be honest, I was temporarily a bit worried because I thought your intention is to completely replace doing higher-level daily Q&A by refereeing... But I think we should keep the higher-level Q&A section for people to clarify questions on shorter than refereeing time scales too. The environment on Physics SE for advanced students who want to learn higher-level topics at a technical level is going from hostil to really toxic now, so they need a place to go ... But if I understand you right now, you agree that we can do both in different sections? There can even be different moderators, experts, etc assigned to different sections (categories) in Q2A if needed... Cheers :-)

Hi Ron. Thanks for the thoughtful response. If it is a separate section of the site and does not take development attention too far from the Q&A site then I am all for it. Revolutionizing how we do science is an exciting prospect. Let's remember however that a Q&A site is also a way of changing the game, and no mathoverflow currently exists for physics.

You can use an arxiv mirror, but that warning has been there since 1995 at least, and it doesn't prevent crawlers from indexing arxiv, so it is probably just a "beware of dog" sign with no dog.

 Please use answers only to (at least partly) answer questions. To comment, discuss, or ask for clarification, leave a comment instead. To mask links under text, please type your text, highlight it, and click the "link" button. You can then enter your link URL. Please consult the FAQ for as to how to format your post. This is the answer box; if you want to write a comment instead, please use the 'add comment' button. Live preview (may slow down editor)   Preview Your name to display (optional): Email me at this address if my answer is selected or commented on: Privacy: Your email address will only be used for sending these notifications. Anti-spam verification: If you are a human please identify the position of the character covered by the symbol $\varnothing$ in the following word:p$\hbar\varnothing$sicsOverflowThen drag the red bullet below over the corresponding character of our banner. When you drop it there, the bullet changes to green (on slow internet connections after a few seconds). To avoid this verification in future, please log in or register.