• Register
PhysicsOverflow is a next-generation academic platform for physicists and astronomers, including a community peer review system and a postgraduate-level discussion forum analogous to MathOverflow.

Welcome to PhysicsOverflow! PhysicsOverflow is an open platform for community peer review and graduate-level Physics discussion.

Please help promote PhysicsOverflow ads elsewhere if you like it.


PO is now at the Physics Department of Bielefeld University!

New printer friendly PO pages!

Migration to Bielefeld University was successful!

Please vote for this year's PhysicsOverflow ads!

Please do help out in categorising submissions. Submit a paper to PhysicsOverflow!

... see more

Tools for paper authors

Submit paper
Claim Paper Authorship

Tools for SE users

Search User
Reclaim SE Account
Request Account Merger
Nativise imported posts
Claim post (deleted users)
Import SE post

Users whose questions have been imported from Physics Stack Exchange, Theoretical Physics Stack Exchange, or any other Stack Exchange site are kindly requested to reclaim their account and not to register as a new user.

Public \(\beta\) tools

Report a bug with a feature
Request a new functionality
404 page design
Send feedback


(propose a free ad)

Site Statistics

205 submissions , 163 unreviewed
5,054 questions , 2,207 unanswered
5,345 answers , 22,719 comments
1,470 users with positive rep
818 active unimported users
More ...

  A suggestion on how to review and evaluate ongoing research on PhysicsOverflow

+ 3 like - 0 dislike

As Ron Maimon said here, it would be nice to have the possibility to obtain honest feedback on and discuss specific ongoing research (papers) unhindered by any "political" obstructions on PhysicsOverflow.

My idea about how to implement this on PhysicsOverflow is to dedicate a special category (I called it Refereeing) which can be choosen if needed when asking a question, to the process of peer reviewing.

In this category, a user can for example post in a question his newest work (or a paper of other authors he likes to be discussed and evaluated) and summarize / explain it. Other members of the PhysicsOverflow community can then vote on this paper (the question) and put their reviews and evaulations down as answers (which can be voted on too).  Both, the paper in the question and the reviews in the answers can be further discussed in the comments.

What do people think about this?

asked Feb 21, 2014 in Discussion by Dilaton (6,240 points) [ revision history ]
retagged Jul 11, 2014 by dimension10

1 Answer

+ 2 like - 0 dislike

I think the proper model is to automatically create a link to every paper on arxiv, either at the beginning when the site goes live, updated daily with new submissions, or on a case by case basis, as they are referenced by others, whose "question" is simply the arxiv paper title, the question body is simply a link to the arxiv page or to the pdf directly, and whose "answers" are simply refereeing comments regarding the paper's accuracy. This is extremely useful for two reasons:

1. It is value added--- it immediately distinguishes this site from stackexchange, in that refereeing is not Q/A. This is important, as a clone of existing sites is not original enough to get widespread attention and acceptance, you need something which adds new value.

2. It explains the "free speech" aspects of political moderation--- you need to be free to criticize when you are refereeing, and to retract wrong statements when you make a mistake. Scientific refereeing is extremely hostile (when work is wrong, or duplicative of earlier literature).

3. It will draw in EVERYBODY. I mean EVERYBODY. The moment you referee a Maldacena paper, Maldacena will be here within a matter of days. If you make a comment on an 't Hooft paper, 't Hooft will be here in days. You saw this on stackexchange. It is coercive, it forces the whole physics community to pay attention immediately.

Because of this coercive aspect, there is zero chance that this site will be ignored, or wither or die. It will be important, simply because if too few physicists are contributing, all you have to do is go to the refereeing section and criticize some of their mistakes in the published literature, or point out insufficient attribution to earlier work, and boy, will then come. If they have referee reports lying around, or if they have some things that they remember wanting to say about this or that paper, then they will do so at the appropriate location.

The paper-questions themselves can spin off questions related to the paper, by adding related questions, so as to clarify the content. When the paper is old, not freely available, then the question body can and should include a complete summary of the contents of the paper, edited wiki-style, by various people, but mostly belonging to one person. So, for example, if you want to cite a result of Newton in the "Principia", or talk about this work, you can add it to the refereeing section, with a summary of the relevant result, working out all the calculations. The same for an old Polyakov paper, or something from the 1970s by Wilson, or Jackiw or someone like this.

In this case, I think it is important to be able to criticize summaries for their accuracy somewhere, so that a good summary emerges. If the questions are edited community style, with honest comments, this shouldn't be hard, as the work is not original, and all you need to do is stay faithful to the original. But there are always simplifications possible, it is usually possible to summarize old literature very well with newer notations and methods that were developed later. This adds important forward-references to future work to old literature, something which you don't get from following the references in the literature. This is also value added, and it is better than a reverse-citation index, because it is curated by humans, and points to the relevant clarifications immediately.

In doing this, it is perhaps useful to add a bot to the site which makes a question automatically for every arxiv paper every day, and adds them to the "referee" section.

For the referee section, I think a good guideline is that the upvoting/downvoting should be in two parts:

1. Originality: this is the originality of the paper, how original is it?

2. Significance: this is the importance of the paper--- how important is it?

Then the answers can point out various mistakes in the paper, and these can be upvoted/downvoted according to accuracy, as usual. This allows the site to substitute for a journal with very little modification to the software--- simply automatic addition of papers.

But I am going to say up front that the site will get VERY heavy traffic the moment you do this, because every single academic will worry about comments about their paper. You should try to avoid people writing nice things on these pages, as these will be the authors, or their colleagues patting each other on the back. The comments should be for explanations, reworking of results, pointing out errors, pointing out missing citations, the usual negative refereeing.

This is an atomic bomb onthe academic community, because they can't stop you from doing it, and they can't help participating when it happens.

answered Feb 21, 2014 by Ron Maimon (7,720 points) [ no revision ]
edited Feb 21, 2014 by Ron Maimon
Most voted comments show all comments

Refereeing is only "noise" for those that think that stackexchange "question/answer" model is ideal. I think it sucks, for one, but it's the best available thing right now. Since there is a new site being made, which needs to differentiate itself by stackexchange in as many ways as possible, this is the ideal opportunity to make the "science 2.0" site that people have been trying so hard to do for the past 5 years, and failing to do. It only requires minor modifications to the stackexchange format.

If you don't want to participate, stay on the Q/A section, it won't be much different than stackexchange, except for higher quality Q/A (if you use refereeing to suck in the professionals). I promise to cogently sincerely negatively referee ~1 paper a day, I'll look for mistakes, people make them once in a while, and maybe Lubos's can do the same (he does it on his blog anyway, it won't be any different). Negative refereeing means the author will come (and they will come) to answer your comments, and they can make their own reports on others' papers, it will be easier if we allow them to comment anonymously. In this way, there can easily be a positive chain reaction coefficient, and suddenly, all the physicists are here, commenting on each others' papers.

This is so much of an advantage, that it would be foolish not to take advantage of the fact that nobody has managed to do it yet. The reason others have failed (they tried!) is that they made a closed commenting model, where you had to be invited (!) to comment on a paper, and if you weren't classified as an "expert" you couldn't comment on a paper. It's unbelievable stupidity on the part of the other science 2.0 sites, but it means there is a great opportunity for this site. Don't squander it.

Remember, if you don't want to referee, don't go to the referee section. But I predict, after about a dozen papers are refereed by folks here, there will be a dozen people commenting on other people's papers (if the commenting mechanism is transparent), and then it's a chain reaction, those people will comment, and so on. The key is to give positive and negative feedback, with technical meat. I can do this on a handful of papers per day, but it's enough to start the process off.

This happened on stackexchange, regarding 'tHooft's papers on deterministic QM (which I criticized somewhat, although I supported the general program. Lubos also criticized, but less constructively), and then, whaddayaknow, there was 't Hooft, answering the criticisms! It was useful. Also, Lubos criticized the loop guys, and then one of the loop guys came over to rebut the criticism. This is a professor trap, criticizing their papers is coercive, it WILL bring them here, 100% guaranteed, but you have to KEEP them here, with a good site organization.

Once people see it works (it does work), about 6-12 months later, all the other sites are going to try to do the same thing themselves. The first site to do it has an advantage, and I would like this site to be the one, because I know the moderation here will not exclude eccentric people, and probably will not censor views that are contrarian, it will just give them less prominence when they are wrong. I don't think power can go to Dilaton's head, or to Dimension10's, they are pretty much aware of the issue of moderation abuse, and will probably not succumb to it.

Thanks for these nice additional explanations Ron, it makes me personally very excited about the refereeing section as a professor trap too! I can almost see the exchanges between Lumo and LQG guys in advance ... :-) To be honest, I was temporarily a bit worried because I thought your intention is to completely replace doing higher-level daily Q&A by refereeing... But I think we should keep the higher-level Q&A section for people to clarify questions on shorter than refereeing time scales too. The environment on Physics SE for advanced students who want to learn higher-level topics at a technical level is going from hostil to really toxic now, so they need a place to go ... But if I understand you right now, you agree that we can do both in different sections? There can even be different moderators, experts, etc assigned to different sections (categories) in Q2A if needed... Cheers :-)

Hi Ron. Thanks for the thoughtful response. If it is a separate section of the site and does not take development attention too far from the Q&A site then I am all for it. Revolutionizing how we do science is an exciting prospect. Let's remember however that a Q&A site is also a way of changing the game, and no mathoverflow currently exists for physics.

I agree with both goals, I was not meaning that Q/A should be excluded, it is also necessary, and it is the primary goal of the site organizers. But I just think it is important to take advantage of the refereeing potential of this format to do something totally new that can really make a big splash, it's something other people are trying to do, but haven't figured out yet, and the folks here already know exactly what to do.

If you succeed in this, then you can get a cash-flow from various non-intrusive mechanisms, remember how much cash Elsevier STILL suckers people for! Then you can pay polarkernel a proper wage for his or her labor, hire a programmer or two full time, become a nice non-profit, with good moderators, so long as someone with more resources doesn't take the idea and run first, because any other person, other than the people here, who are nice, is definitely going to be a shithead. Guaranteed.

You can use an arxiv mirror, but that warning has been there since 1995 at least, and it doesn't prevent crawlers from indexing arxiv, so it is probably just a "beware of dog" sign with no dog.

Most recent comments show all comments

Hi Ron,

maybe you are interested in describing the refereeing section in our FAQ thread too? I unfortunately had to split Dimension10's original post into the different chapters because of the limited number of characters allowed in in a post (answer) ... :-/

I definitely hope this is possible, but I'm not sure if this prevents this: http://arxiv.org/help/robots Of course using an RSS feed may help this.  

Your answer

Please use answers only to (at least partly) answer questions. To comment, discuss, or ask for clarification, leave a comment instead.
To mask links under text, please type your text, highlight it, and click the "link" button. You can then enter your link URL.
Please consult the FAQ for as to how to format your post.
This is the answer box; if you want to write a comment instead, please use the 'add comment' button.
Live preview (may slow down editor)   Preview
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
Anti-spam verification:
If you are a human please identify the position of the character covered by the symbol $\varnothing$ in the following word:
Then drag the red bullet below over the corresponding character of our banner. When you drop it there, the bullet changes to green (on slow internet connections after a few seconds).
Please complete the anti-spam verification

user contributions licensed under cc by-sa 3.0 with attribution required

Your rights