I understand that by publishing this post I may be giving ammunition to the anti-science crowd. But we can't just hide in the ivory tower and tell people that science is perfect as it is. We need deep reforms in the way science is done.
What is truth? These famous words come not from a scientist but from a politician, Pontius Pilate, governor of Palestine in Roman times. As a politician, Pilate knew very well how truth could be twisted, stretched, sliced, cooked, flavored, and rearranged in many ways in order to be sold to people. Things are not different, today. In politics, truth is what you perceive to be true. After all, isn't it true that we can create our own reality? (a US government official is reported to have said that at the time of the invasion of Iraq, in 2003).
Eventually, the Roman Empire drowned in its own lies, it was an epistemological collapse. Something similar may happen to us: we cannot continue for long to ignore reality, believing that we can manufacture our own, and deceive everyone in the process.
But how about science? Isn't science able to tell us what reality is according to its much-praised "scientific method?" In principle, yes, but science is far from being a perfect truth-seeking machine. The attacks that science is receiving from all quarters have some justification: as scientists, we cannot claim to be able to save the world if we don't clean up our act first.
A critical element of the soft belly of modern science is the process called "Peer Review." If you are not familiar with this procedure, let me explain it to you. The idea is that when scientists want to diffuse the results of their studies in the form of a "paper", they will submit it to a "peer-reviewed" journal. Their manuscript will be sent to a number (typically 2 or 3) anonymous reviewers - scientists working in the same field - who will recommend rejection or publication and, in the latter case, changes to improve the paper. (for details, see this excellent paper by Jon Tennant)
So far, so good: if everyone does their best to do the reviewing job, the process might provide good results. And, indeed, peer review is supposed to be the "golden standard" of science. The typical accusation that climate scientists direct to their critics is that their papers are not peer-reviewed: they are often published in politically motivated blogs, and they lack the rigor that real scientific papers have. This is often a correct viewpoint, climate science is one of the most advanced and vital fields of science, today, and the criticism made at it is normally of poor quality and politically biased.
But there is a problem: the gold standard called Peer Review is neither gold nor a standard. First of all, it doesn't prevent bad science from filtering through. It is always possible with some effort and some patience to find a favorable combination of reviewers and editors and manage to publish on a serious journal a paper that's wrong from top to bottom. It has happened and some cases are truly a scandal. You can see this one, for instance, where the authors invented out of whole cloth a completely new nuclear physics based on shaky (to say the least) experimental evidence, and all in order to explain phenomena that had other, perfectly good, explanations.
At this point, get ready for a surprise: Scientific Journals have NO WAY to remedy a mistake. Once a paper is published in a peer-reviewed journal, it is enshrined in the corpus of "officially approved science." Unless there are evident scams involved, such as plagiarism or fake data, the fact that most scientists in the field think that the paper is wrong is not sufficient to have it removed. The best that other scientists can do is to submit a comment to the editors - who will be usually as happy to publish it as they are happy to see their dentist. Then, the authors of the commented paper will be able to submit their counter-comments and the whole process will only serve to give them more visibility - and that's exactly what happened with the paper I was telling you about.
On the other side of the peer review process, the filter makes it nearly impossible to publish innovative ideas, especially for young researchers. Reviewers are a self-selected group, often formed of elderly gentlemen, whose purpose seems to be to make sure that nothing that contrasts with their views goes through the filter. To say nothing about the incredibly slow, time-consuming, and frustrating process of dealing with reviewers who understood nothing of one's work but nevertheless think that they can demolish it, even poking fun at it. Reviewers can always manage to transform a perfectly good paper into an ugly mongrel just because they want to throw their weight around. The worst is when they won't be happy until they have imposed their views on the authors, forcing them to write the paper the way they (the reviewers) want.
How about standards in reviewing? Again, brace for a surprise: there are none, zero, zilch, null. The whole process is carried out in secrecy, the authors don't know who is the person who has somehow gained the right to abuse them from a comfortable anonymity position, there is no standard for what kind of criticism is supposed to be acceptable or not, nor for what kind of rebuttal is supposed to be acceptable or not. The editors can do whatever they want with a submission and normally there is no procedure that an author can follow to protest against what they see as an unfair treatment of their paper.
Now, would you assign to someone the job of - say - designing a plane on the basis of this review method? Would you fly in it once it is built? So, if you are a scientist, do you think that you can save the world in this way? And peer review is not the only problem plaguing modern science!
You may wonder how come that scientists - who are supposed to be so smart - behave in such a sloppy manner when it is question of publishing their results. I am baffled by myself on this point; the only thing I can propose is that they are good at whatever they are specialized in, but not necessarily in all fields. In other words, many scientists can be defined as "Idiot Savants," interested only in their narrow specialized field. The final result is something like the Brancaleone Army I was mentioning at the beginning of this post.
Fortunately, science is not yet the scam it is accused to be, for instance, by those engaged in rejecting climate science. But, if we don't do something fast to improve, we risk seeing science perceived by everyone to be a scam. And didn't we say that in politics truth is what you perceive to be true?
In Italy, we have a term for those who engage in a task much too big and too difficult for them. We call them a "Brancaleone Army" (Armata Brancaleone), a term coming from the title of a wonderful 1966 Italian movie where an Italian self-styled knight tries to lead a ragtag army of incompetent fighters. The sad conditions of science nowadays sometimes look to me like the story of the Brancaleone army.
What is truth? These famous words come not from a scientist but from a politician, Pontius Pilate, governor of Palestine in Roman times. As a politician, Pilate knew very well how truth could be twisted, stretched, sliced, cooked, flavored, and rearranged in many ways in order to be sold to people. Things are not different, today. In politics, truth is what you perceive to be true. After all, isn't it true that we can create our own reality? (a US government official is reported to have said that at the time of the invasion of Iraq, in 2003).
Eventually, the Roman Empire drowned in its own lies, it was an epistemological collapse. Something similar may happen to us: we cannot continue for long to ignore reality, believing that we can manufacture our own, and deceive everyone in the process.
But how about science? Isn't science able to tell us what reality is according to its much-praised "scientific method?" In principle, yes, but science is far from being a perfect truth-seeking machine. The attacks that science is receiving from all quarters have some justification: as scientists, we cannot claim to be able to save the world if we don't clean up our act first.
A critical element of the soft belly of modern science is the process called "Peer Review." If you are not familiar with this procedure, let me explain it to you. The idea is that when scientists want to diffuse the results of their studies in the form of a "paper", they will submit it to a "peer-reviewed" journal. Their manuscript will be sent to a number (typically 2 or 3) anonymous reviewers - scientists working in the same field - who will recommend rejection or publication and, in the latter case, changes to improve the paper. (for details, see this excellent paper by Jon Tennant)
So far, so good: if everyone does their best to do the reviewing job, the process might provide good results. And, indeed, peer review is supposed to be the "golden standard" of science. The typical accusation that climate scientists direct to their critics is that their papers are not peer-reviewed: they are often published in politically motivated blogs, and they lack the rigor that real scientific papers have. This is often a correct viewpoint, climate science is one of the most advanced and vital fields of science, today, and the criticism made at it is normally of poor quality and politically biased.
But there is a problem: the gold standard called Peer Review is neither gold nor a standard. First of all, it doesn't prevent bad science from filtering through. It is always possible with some effort and some patience to find a favorable combination of reviewers and editors and manage to publish on a serious journal a paper that's wrong from top to bottom. It has happened and some cases are truly a scandal. You can see this one, for instance, where the authors invented out of whole cloth a completely new nuclear physics based on shaky (to say the least) experimental evidence, and all in order to explain phenomena that had other, perfectly good, explanations.
At this point, get ready for a surprise: Scientific Journals have NO WAY to remedy a mistake. Once a paper is published in a peer-reviewed journal, it is enshrined in the corpus of "officially approved science." Unless there are evident scams involved, such as plagiarism or fake data, the fact that most scientists in the field think that the paper is wrong is not sufficient to have it removed. The best that other scientists can do is to submit a comment to the editors - who will be usually as happy to publish it as they are happy to see their dentist. Then, the authors of the commented paper will be able to submit their counter-comments and the whole process will only serve to give them more visibility - and that's exactly what happened with the paper I was telling you about.
On the other side of the peer review process, the filter makes it nearly impossible to publish innovative ideas, especially for young researchers. Reviewers are a self-selected group, often formed of elderly gentlemen, whose purpose seems to be to make sure that nothing that contrasts with their views goes through the filter. To say nothing about the incredibly slow, time-consuming, and frustrating process of dealing with reviewers who understood nothing of one's work but nevertheless think that they can demolish it, even poking fun at it. Reviewers can always manage to transform a perfectly good paper into an ugly mongrel just because they want to throw their weight around. The worst is when they won't be happy until they have imposed their views on the authors, forcing them to write the paper the way they (the reviewers) want.
How about standards in reviewing? Again, brace for a surprise: there are none, zero, zilch, null. The whole process is carried out in secrecy, the authors don't know who is the person who has somehow gained the right to abuse them from a comfortable anonymity position, there is no standard for what kind of criticism is supposed to be acceptable or not, nor for what kind of rebuttal is supposed to be acceptable or not. The editors can do whatever they want with a submission and normally there is no procedure that an author can follow to protest against what they see as an unfair treatment of their paper.
Now, would you assign to someone the job of - say - designing a plane on the basis of this review method? Would you fly in it once it is built? So, if you are a scientist, do you think that you can save the world in this way? And peer review is not the only problem plaguing modern science!
You may wonder how come that scientists - who are supposed to be so smart - behave in such a sloppy manner when it is question of publishing their results. I am baffled by myself on this point; the only thing I can propose is that they are good at whatever they are specialized in, but not necessarily in all fields. In other words, many scientists can be defined as "Idiot Savants," interested only in their narrow specialized field. The final result is something like the Brancaleone Army I was mentioning at the beginning of this post.
Fortunately, science is not yet the scam it is accused to be, for instance, by those engaged in rejecting climate science. But, if we don't do something fast to improve, we risk seeing science perceived by everyone to be a scam. And didn't we say that in politics truth is what you perceive to be true?
This post was inspired by a paper by Jon Tennant. http://fossilsandshit.com/the-state-of-the-art-in-peer-review/ and by a post by Jem Bendell
And here is the original Brancaleone in action!
