I understand you concerns VK, however a system when in every upvote or downvote NEEDS to be explained is not a viable one. Not every one had patience or often even in depth understanding to write a review.
I tend to up vote Papers if I think the general direction is a good one, for instance I up voted both the Amplituhedron papers on this site. I am far from understanding it deeply enough to review it.
I think the current system is fine. However, I would like it if the voting was not so easily swung. It is due to the strange multiplicative formula that is being used I think. Though, that is not something I would worry about.