# Is the current scoring scheme for reviews appropriate?

+ 1 like - 0 dislike
822 views

It seems that if the accuracy is negative, then the more positive the originality, the lower total score there will be. I'm not sure how "originality" is interpreted by most, but to me a positive originality has a positive connotation. For example, if I give a paper one upvote on originality and one downvote on accuracy, I would be thinking: "interesting idea, too bad it's wrong." It deserves a better score than the case of "bad/cliche/crazy idea, and totally wrong/not even wrong", in which case I would give one downvote on both originality and accuracy, corresponding to a less negative score however.

What do you guys think?

For reference, here is our current formula of calculating the total score: http://www.physicsoverflow.org/faq#a4807

edited Jul 18, 2014

It's supposed to work that way, original and inaccurate means wild speculation without mathematical basis/with faulty mathematical basis, commonly associated with the term "crackpot" : )

+ 2 like - 0 dislike

Hm as I underststand it, far reaching new claims (high originality score), are no good if they can not be backed up by a corresponding accurate (at best mathematical) justification and reasoning.

For example the ideas in the first ER/EPR paper are very original and they are soundly backed up by already accepted mathematical/theoretical knowledge. So I voted (+1,+1) on that one based on Lumo's review.

But far reaching claims, for example to have found a new ToE or being able to generally dispense with the concept of renormaliztion, may be original, but if the argument or calculations the author applies are weak and out of thin air or even worse, contradict established theoretical / experimental knowledge, the final score should IMHO be negative for such  (+1,-1) papers

So in my opinion, the formula behaves as it should concerning papers with positive originality score, as far reaching claims or new ideas (originality +1) are only rated positive if they are soundly backed up and do not contradict established knowledge. To me this seems a good way to punish crackpots, wannabe new Einsteins, crazy surfer dudes, etc who want to overthrow establised knowledge for no good reason for example.

But I noted a strange (possibly wrong behavior) of the formula in another situation. I will further investigate this and then mention it in this thread too....

answered Jul 18, 2014 by (6,040 points)

" To me this seems a good way to punish crackpots, wannabe new Einsteins, crazy surfer dudes, etc who want to overthrow establised knowledge for no good reason for example." I doubt if such people can receive any upvote on "originality" here.

Yes, they should not. And in such cases we should in my personal opinion not too much worry about censorship, as PhysicsOverflow is meant to be a serious physics site.

Maybe the best way to answer this question would be to systematically investigate the behavior of the formula and find out if there are places in the 2d parameter space (originality and accuracy) where it does not behave is ist should. Not sure, if Dimension10 and Ron have already done this and I am the only one who can not do it in my head without using Matlab for example ... ;-P

In my case "the accuracy factor "made my score negative, not because I made a mistake, but because my proofs and conclusions support the Dirac's opinion on the reason of renormalization success (a fluke). People do not tolerate such things anymore. In my case it turned out to be a "factor of tolerance".

@JiaYiyang They should! I upvoted originality on Vladimir's paper but downvoted the accuracy. If we assume that people vote sensibly, and not downvote everything because they hate the paper, or such, then crackpot works will get punished as they should.

@Dilaton I notice a strange problem for unoriginal, accurate papers... Is this the same strange behaviour you encounter?

@dimension10, I think we really interpret "originality" differently, it could be just semantics but the word "original" has a strongly positive connotation to me, such feeling makes it very hard for me to vote up something like, proposing a theory based on, say Santa Clause, with almost zero mathematics and claim it's physics. For the record I did not vote up or down Vladimir's paper.

The formula behaved rather reasonalble where I thought it would break down...

What is the issue with unoriginal ut accurate papers? They should be rather positively received, as not every paper has to introduce a completely new point of view or framework to positively contribute to research. In fact, most good papers do rather incremental progress, as Lumo often says too.

@Dilaton To me it seems that downvoting for originality indicates plagiarism or something similar.

I don't understand why someone would downvote a paper with accurate calculations. The question of "accuracy" was, I thought, meant to indicate "accuracy within the assumptions of the paper", not "accuracy in describing nature" necessarily, or "concordance with other literature", or anything political. It is to assess whether the calculations or results are soundly obtained. The question of agreement with experiment is separate.

I thought for a paper to be "accurate", the calculations have to be in accordance with the assumptions and with the conclusions/claims issued too? I mean presenting a good calculation that is valid in certain situations and under specific assumptions and then issue far reaching claims based on this (that are not justified by the derivation in the paper) is not good (or not "accurate")?

If you have a beef with the conclusions, then comment on the paper. Getting hung up on the wording of introductions and conclusions is a total waste of time, it's usually arguing over philosophy. If the deductions are accurate, upvote accuracy, and originality, but if there are overreaching conclusions from these accurate deductions, just say so in your review, and your review will get upvoted. The voting and review system is designed to complement the literature, not enforce political voting.

I don't think it is a good idea to restrict the conclusions people are allowed to come to by politics. If Schwinger calculates the magnetic moment of the electron, and then says "this calculations shows that fundamentally, electrons really love the number pi, and they made friends with Archimedes", who cares? Just upvote the calculation, upvote the originality, and say "Schwinger is totally off base about the conclusions at the end there" in the content of the review, problem solved. A positive upvote can come with a review critical of the conclusions, the two activities, voting and commenting, are separate acts unlike in a journal refereeing process.

This has nothing to do with policy, censorship, or whatelse ...

Throwing out too far overreaching claims (not to confuse with hypotheses, or interesting further issues to study) that are not backed up by the beaf of the paper (or other acceptable sources) seems unscientific to me.

Did I get this wrong when thinking that the votes should be accompanied by a (partial) review or at least a comment, which can then be upvoted by other people who agree with the point stated, etc ...?

If you guys are talking about my Toy model, I want to add two obvious things. First, my model is easily promoted to QM by inserting its Lagrangians into the path integral. Next, my finite corrections correspond (to a great extent) to finite corrections in QFT with cut-off. So my conclusions, although obtained (on purpose) with a classical mechanics problem, have their akin counterparts in QFT.

The problem they have is not with the model itself, but with the conclusions that you draw from it regarding other people's calculation methods. The quesiton is whether these conclusions, which are in this case inflammatory and really unjustified, mean that the paper deserves a downvote on accuracy. I don't think so, I think that people are allowed to express their opions, the work's technical accuracy in the model is what should matter for a review. I would upvote your paper on accuracy, except I haven't gone through it in detail yet.

Thanks, Ron. I would love to listen what is inflammatory and unjustified too. Will be waiting for your review. Take your time, though.

+ 2 like - 0 dislike

Based on the answers by Dilaton and Ron, I think we do need a instruction message in every review post on what originality and accuracy mean. In fact, my definition of "original" disagrees with Dilaton and dimension10's, and my definition of "accurate" disagrees with Ron's. Once the instruction is ready, I will follow what the instruction says.

answered Jul 21, 2014 by (2,640 points)
Maybe we should also say there that people are free to decide about their votes as they see fit from their best knowledge and judgement, as any votes can be explained in reviews or comments as needed. We should avoid the impression that we want to patronize the votes of our members by bureaucracy and policies, as it can be observed on other physics Q&A sites. So I think you do not necessarily have to wait for official definitions before voting in the reviews section, if you feel familiar enough with the topic and paper at hand ... ;-)

Well, I think it's important to have a common definition of what the votes mean, because they will be aggregated. People can happily disagree on whether a paper is "original" or not, but the meaning of "original" should be the same for everyone.

I think the point is that it might be best though to leave it nebulous until the community congeals on a common definition. For me, the definition of "original" is the degree to which a paper is unrelated to other work, with completely new papers with no relation to previous work scoring very high, papers extending previous work scoring medium, and plagiarized or duplicative papers scoring negatively. The "correctness" to me means that the calculations, experiments, deductions, are accurate within their context. A false deduction leading to an overbroad conclusion is also a failure of accuracy, but not as bad as totally made-up equations with totally false calculations.

+ 0 like - 0 dislike

I don't think that a good idea that's wrong deserves a downvote on accuracy, so long as the predictions of the idea are accurately worked out, and honestly compared with experiment. If a theoretical paper presents an original idea, with accurate calculations, it should be upvoted on both counts, even if it disagrees with experiment.

The point of these categories is to make sure that people vote on accuracy, not on politics. If a paper is original, accurate, and disagrees with experiment (for example, steady state model, or Brans-Dicke theory), this paper should be upvoted. The accuracy votes are designed to reflect accurate calculations and deductions, not agreement with experiment.

For an experimental paper, on the other hand, accuracy means that the results are measured well, accurately, and the originality means that the experiment is of a novel kind, and reveals something interesting about nature.

answered Jul 21, 2014 by (7,720 points)
I almost agree with you, but what in my opinion does not deserve any upvotes, are papers of the kind "I have found a better method to do/calculate X under the assumtion A in Situatatuon B AND other people who use the established method Y generally are wrong/deluded/etc ...". Papers that do the first part before the AND are perfectly good and may deserve many upvotes on accuracy if done correctly, while the unjustified overreach, often done by crackpots paired with attacks against established physics/physicists, I would rather downvote on accuracy.

@dilaton: The point of the accuracy vote is to remove any considerations of  the paper's tone. When someone does something new, they are often motivated by wrongheaded considerations, for example, Dirac insisted that the field equation for the electron must be linear because the wavefunction can't obey a second order equation, and dismissed the work of Klein and Gordon for this reason. Does this make the Dirac equation wrong? No. It just means that he had a wrong motivation. The motivation is largely irrelevant, you can examine the calculations, and see whether the deductions are accurate, and ultimately if there is a wrong conclusion, it will be unsupported by the calculations.

Saying "all the physicists using method Y are wrong/deluded" is often true. In the case of the renormalization papers you are thinking about, by V. Kalitviansky, his grip is that using traditional renormalization you aren't producing a way to deal with the classical self-field of the electron at large distances, rather you are dealing with the log-divergences at short distances without considering the classical self-field problem at all. This is true, it is one of the miracles of QED that the short-distance divergences are not like in classical electrodynamics, and so the perturbative renormalization does not need to consider the self-field problem in the classical setting, of long-distance soft modes excited by electron motion. It also means that physicists are often deluded about how well they understand the self-field, and think that the problems of QED are completely solved with renormalization.

But they are not deluded that the short-distance problems of QED are solved. These problems really are solved by renormalization (up to Landau issues at ultra-tiny distances). But the motivation to consider this old problem leads Vladimir to consider interesting models for the long-distance self-field, which is important for understanding better the infraparticle properties of QED, and self-field issues, which appear as infrared divergences, not ultraviolet divergences. Because these problems are involved, it is natural to assume that those who don't consider these problems important are deluded, etc, even if it is not important for them, and ultraviolet renormalization is completely fine.

A downvote on accuracy should not be a downvote on politics. If you don't like the political tone, say something, but an accurate paper can come with hostility and annoyance, because someone with a new point of view, or even just a different point of view, always sees things differently, and is therefore automatically hostile to everyone else who doesn't.

Hi Ron, are you sure that Wilson's more general view of systematically derive appropriate EFTs does not allow to correctly deal with the classical self field either? This is what I am not sure about and Vladimir always talks about the rather limited and dated point of view while ignoring Wilson' s insights completely...

He doesn't care about Wilsonian field theory. The Kadanovian Fisherite Wilsonian theory is just a more careful, nonperturbative, version of what Feynman, Dyson and Schwinger and all the others did in the 1950s. It just gives you a way to understand what's going on at short distances. At long distances, the problems of definition remain. I'll explain the issue--- what is an electron state? The bare electron is trivial, it's a free particle. But a dressed electron comes with a 1/r field which extends out to infinity, and if you just tap it slightly, it creates a shower of infinitely many soft photons at enormous wavelengths to fix-up the long-distance field.

A proper treatment of a dressed electron seems like it needs to consider the field as part of the particle. Wilsonian approaches are local, and only fold in the self-mass and so on in a local approximation, and ignore the field at long distances, where "long" mean "longer than the compton wavelength of the electron". This is not a solution to the long-distance issue that you don't have a good definition of a single electron state, and no definition for the S-matrix. These problems are relatively involved compared to renormalizing QED.

The models VK advocates are all about using new variables to treat the electron+self-field as a unit somehow. This is interesting for this problem. Feynman et al had a different approach, which was to just give the photon an infinitesimal mass, thereby cutting off the infrared divergences. But you have to be careful here, and the story with the infrared divergences is generally forgotten, precisely because it is not important for the more fundamental Wilsonian issues.

VK doesn't agree with Wilsonian renormalization, and he's totally off base here, but his point of view makes it that he is trying all sorts of different approaches to the long-range problem of infrared divergences.

Also, to respond to a hidden comment, I agree with not telling anyone how to vote. My opinions  on how to vote are solely mine, and I can't impose them on anyone else in any way.

I personally solve this problem by considering the single electron state in the theory of massive QED with an infinitesimal photon mass. A single particle exact eigenstate obeys a free Dirac equation with no assumptions, just by Poicare symmetry and consideration of spin. So this eigenstate's motion in response to an external slow electromagnetic field can be described by a free field theory to an approximation, and in the massive photon case, the approximation is that the frequency of external perturbations is not sufficient to create even one photon. The "phenomenological equations", meaning the free equations, then describe the single electron state, and the mass in these equations is the free electron mass. Making the electron interact with the quantum field just involves choosing the field perturbation to come with local counterterms that deal with the short distance renormalization of the mass in response to the self-field. This short distance thing is separate from the problem of the long-range field, the long-range field of the electron only makes a negligible (about 1/alpha) contribution to the electron mass. The only problem of principle that remains is how to describe the electron states in the limit of zero photon mass, when every change in momentum is accompanied by infinitely many soft photons. For this problem, it is useful to consider variables which include the self-field around the electron in some way, but I don't know how exactly, because it is the difficult problem.

Your complaints are not with the methods that people use, but with the philosophical words that they use to justify these methods. Ignore the words, and look at the calculations themselves, they can be justified properly, without the words that tick you off.

 Please use answers only to (at least partly) answer questions. To comment, discuss, or ask for clarification, leave a comment instead. To mask links under text, please type your text, highlight it, and click the "link" button. You can then enter your link URL. Please consult the FAQ for as to how to format your post. This is the answer box; if you want to write a comment instead, please use the 'add comment' button. Live preview (may slow down editor)   Preview Your name to display (optional): Email me at this address if my answer is selected or commented on: Privacy: Your email address will only be used for sending these notifications. Anti-spam verification: If you are a human please identify the position of the character covered by the symbol $\varnothing$ in the following word:p$\hbar$ysicsOverfl$\varnothing$wThen drag the red bullet below over the corresponding character of our banner. When you drop it there, the bullet changes to green (on slow internet connections after a few seconds). To avoid this verification in future, please log in or register.