Monday, August 27, 2018

So, You Think Science Will Save the World? Are You Sure?

I understand that by publishing this post I may be giving ammunition to the anti-science crowd. But we can't just hide in the ivory tower and tell people that science is perfect as it is. We need deep reforms in the way science is done.



In Italy, we have a term for those who engage in a task much too big and too difficult for them. We call them a "Brancaleone Army" (Armata Brancaleone), a term coming from the title of a wonderful 1966 Italian movie where an Italian self-styled knight tries to lead a ragtag army of incompetent fighters. The sad conditions of science nowadays sometimes look to me like the story of the Brancaleone army.


What is truth? These famous words come not from a scientist but from a politician, Pontius Pilate, governor of Palestine in Roman times. As a politician, Pilate knew very well how truth could be twisted, stretched, sliced, cooked, flavored, and rearranged in many ways in order to be sold to people. Things are not different, today. In politics, truth is what you perceive to be true. After all, isn't it true that we can create our own reality? (a US government official is reported to have said that at the time of the invasion of Iraq, in 2003).

Eventually, the Roman Empire drowned in its own lies, it was an epistemological collapse. Something similar may happen to us: we cannot continue for long to ignore reality, believing that we can manufacture our own, and deceive everyone in the process.

But how about science? Isn't science able to tell us what reality is according to its much-praised "scientific method?" In principle, yes, but science is far from being a perfect truth-seeking machine. The attacks that science is receiving from all quarters have some justification: as scientists, we cannot claim to be able to save the world if we don't clean up our act first.

A critical element of the soft belly of modern science is the process called "Peer Review." If you are not familiar with this procedure, let me explain it to you. The idea is that when scientists want to diffuse the results of their studies in the form of a "paper", they will submit it to a "peer-reviewed" journal. Their manuscript will be sent to a number (typically 2 or 3) anonymous reviewers - scientists working in the same field - who will recommend rejection or publication and, in the latter case, changes to improve the paper. (for details, see this excellent paper by Jon Tennant)

So far, so good: if everyone does their best to do the reviewing job, the process might provide good results. And, indeed, peer review is supposed to be the "golden standard" of science. The typical accusation that climate scientists direct to their critics is that their papers are not peer-reviewed: they are often published in politically motivated blogs, and they lack the rigor that real scientific papers have. This is often a correct viewpoint, climate science is one of the most advanced and vital fields of science, today, and the criticism made at it is normally of poor quality and politically biased.

But there is a problem: the gold standard called Peer Review is neither gold nor a standard. First of all, it doesn't prevent bad science from filtering through. It is always possible with some effort and some patience to find a favorable combination of reviewers and editors and manage to publish on a serious journal a paper that's wrong from top to bottom. It has happened and some cases are truly a scandal. You can see this one, for instance, where the authors invented out of whole cloth a completely new nuclear physics based on shaky (to say the least) experimental evidence, and all in order to explain phenomena that had other, perfectly good, explanations.

At this point, get ready for a surprise: Scientific Journals have NO WAY to remedy a mistake. Once a paper is published in a peer-reviewed journal, it is enshrined in the corpus of "officially approved science." Unless there are evident scams involved, such as plagiarism or fake data, the fact that most scientists in the field think that the paper is wrong is not sufficient to have it removed. The best that other scientists can do is to submit a comment to the editors - who will be usually as happy to publish it as they are happy to see their dentist. Then, the authors of the commented paper will be able to submit their counter-comments and the whole process will only serve to give them more visibility - and that's exactly what happened with the paper I was telling you about.

On the other side of the peer review process, the filter makes it nearly impossible to publish innovative ideas, especially for young researchers. Reviewers are a self-selected group, often formed of elderly gentlemen, whose purpose seems to be to make sure that nothing that contrasts with their views goes through the filter. To say nothing about the incredibly slow, time-consuming, and frustrating process of dealing with reviewers who understood nothing of one's work but nevertheless think that they can demolish it, even poking fun at it. Reviewers can always manage to transform a perfectly good paper into an ugly mongrel just because they want to throw their weight around. The worst is when they won't be happy until they have imposed their views on the authors, forcing them to write the paper the way they (the reviewers) want.

How about standards in reviewing? Again, brace for a surprise: there are none, zero, zilch, null. The whole process is carried out in secrecy, the authors don't know who is the person who has somehow gained the right to abuse them from a comfortable anonymity position, there is no standard for what kind of criticism is supposed to be acceptable or not, nor for what kind of rebuttal is supposed to be acceptable or not. The editors can do whatever they want with a submission and normally there is no procedure that an author can follow to protest against what they see as an unfair treatment of their paper.

Now, would you assign to someone the job of - say - designing a plane on the basis of this review method? Would you fly in it once it is built? So, if you are a scientist, do you think that you can save the world in this way? And peer review is not the only problem plaguing modern science!

You may wonder how come that scientists - who are supposed to be so smart - behave in such a sloppy manner when it is question of publishing their results. I am baffled by myself on this point; the only thing I can propose is that they are good at whatever they are specialized in, but not necessarily in all fields. In other words, many scientists can be defined as "Idiot Savants," interested only in their narrow specialized field. The final result is something like the Brancaleone Army I was mentioning at the beginning of this post.

Fortunately, science is not yet the scam it is accused to be, for instance, by those engaged in rejecting climate science. But, if we don't do something fast to improve, we risk seeing science perceived by everyone to be a scam. And  didn't we say that in politics truth is what you perceive to be true?




This post was inspired by a paper by Jon Tennant. http://fossilsandshit.com/the-state-of-the-art-in-peer-review/  and by a post by Jem Bendell


And here is the original Brancaleone in action!

20 comments:

  1. You've got the right target, in my view: the peer review process. It is flawed. However, like democracy being the worst form of constitution apart from all the others, we need to offer a viable alternative. That's hard -- but I'll join with you or others to work out some good enough ways to do peer review properly. I honestly believe it is very hard but possible, so an extremely worthy project.

    ReplyDelete
  2. try to write a critical thesis or work in research and being critical of the holy Grail of energy production in Austria: aka hydro power or biomass like wood-pellets or biogas-electrification of maize silage. Or talking about the energy taboos like the Jevons effect or the rebound of efficiency.

    ReplyDelete
  3. My thoughts here: https://blog.wozukunft.de/2018/08/28/a-brief-defense-of-the-haphazard-scientific-article-review-method/

    ReplyDelete
    Replies
    1. Gregor, your thoughts echo with mine above, and I see only the orientation as different. Yes, we are right to appreciate some aspects of peer review. But to me, the question is, how can we do better, using our information and communications tools to better effect? After all, the peer review system could (and did) work by snail mail.

      Delete
  4. A very related work:

    Imre Lakatos & Paul Feyerabend:
    For and Against Method
    University of Chicago Press, 1999

    I strongly recommend reading and discussing

    ReplyDelete
  5. Thanks for providing this great insight for the paper! Hope it is useful in inspiring some discussion :)

    ReplyDelete
  6. Back in the early 1990's I wrote a paper on an esoteric matter in stress analysis. It went thru peer review. I presented it at a conference; the audience stood up to applaud. Best of all, I was the proud recipient of a few beers that evening!

    A few months later I got my comeuppance. A man called for advice on applying the results of my published paper, particularly a graphic he didn't quite understand. The reason was that it was wrong! It was rotated 90° from its correct orientation! With that matter cleared up, he went ahead and modified the design of… a gigantic machine for strip-mining coal! (The ASME kindly published a brief note on the error and its correction. Still, horribly embarrassing. And yeah, peer review is a leaky defense.)

    ReplyDelete
  7. When and where does anonymous ever work? A person, especially a scientist, should be held to account for their work, even or especially if it is a critique, good or bad, of science.

    It is high time that people be held accountable for their expressed thoughts, actions, and inaction with regard to the wholesale destruction of the biosphere that supports life on this planet. I can't believe I need to say that.

    ReplyDelete
  8. The limit of human brain to process information pose a hard limit to the capacity to compreend complex system, opening the review to everyone is a strange proposal:some can't understand but are free to review based on bias, some can review posing observation based on different approach. The process is theorically sound but have a low signal noise ratio even in absence of specific interest, more open is the review more noise you get, editors too have to get hi rating so add up personal agenda.
    Solution, back to the basic, scientific method is based on the repetition of the effect, time consuming but necessary, the crowning is the engeneering applications.
    Problems arise to predictive science, in this field pubblic predictions can be ferified ex post on multiple predictions, again time consuming.....
    The timeframe can be similar to the adoption of new complex military systems (they need to be really thougth tested),few years for incremental and few decades for radical or new.

    ReplyDelete
    Replies

    1. The Limit is to Understand! What we can not or do not want to understand - we try to approximate. We then can not identify - it at best looks like - something we can imagine. As Pythagoras already made clear: Akkusmatoi: to believe what we say and hear...

      Delete
  9. As it is me, if you follwed my comments, you can allready guess what I have to say about this topic. Science has a problem because capitalism creeped into the scientific method and undermined it.

    There are many examples where big money and corporate funding successfully undermined science, I want to put the spotlight on one.

    Ignacio Chapela is (was?) a very distinguished professor in his field (microbiology / UC-Berkely) and his story is quite astonishing.

    Chapela had advised mexican scientists about examining transgene intrusions into mexican corn by US gene-modified industrial seeds. When trying to educate students how to find traces of modified genes in mexico they took an ostensibly clean sample to compare it with monsanto seeds. To their surprise that clean sample already had been contaminated.

    After examining mexican corn for a while he published his findings in Nature magazine, the flagship of all scientific publications. The paper was peer reviewed and published, but as it turned out, Nature later considered this publication to have been "an error". In the next issue, Nature wrote an editorial about how they would have preferred to not have published Chapela's paper.

    So what happened? Was the paper scientifically flawed? Not at all, the science had received much praise from many sources, but after publishing the paper, monsanto had put in its considerable influence and power into pressuring Nature magazine, forcing them to write that editorial, a process that realy hurt the magazines standing, but which they still decided to go for.

    Usually the power of money seems to be enough to stop publications like Chapelas bevor they appear int journals, but this time, something has gone wrong and the Chapela's paper creeped through the filter process that companies like monsanto had in place in universities and in journals.

    Today there is not a single institute in microbiology where companies like bayer or monsanto do not spend millions. They do finance science, but they also control it, and there is ample proof for that.

    When this is not enough, scientists are directly attacked and ruined, which is exactly what happened to Prof Arpad Pusztai, who had given a 150 sec interview with british television critical of monsanto.

    To save science, science needs to reclaim its independence from large corporations.

    Source "Scientists under Attack - genetic engineering in the magnetic field of money" https://www.denkmal.film/index.php?page=gekaufte-wahrheit-4&l=en

    Quote:
    Árpád Pusztai and Ignacio Chapela have two things in common. They are distinguished scientists, and their careers are in ruins. Both chose to investigate the phenomenon of genetic engineering. Both made important discoveries. Both are now suffering the fate of those who criticise the powerful vested interests that dominate big business and scientific research.

    Statements made by scientists themselves suggest that 95% of those researching in the area of genetic engineering are funded to do so by industry. Only 5% are independent. This situation is a major threat to the freedom of science – and to our democracy. Can we, the public, still trust our scientists?

    ReplyDelete
    Replies
    1. Yes, so we need to set up a peer review and publication system that is immune from financial influence. How? Would you like to contribute to imagining it in detail?

      Delete
    2. The answer is simple enough. Its digital online self publishing by scientists and the "open science peer review".

      Nobody really needs journals anymore.

      Delete
    3. @alien_observer: what do you mean by "open science peer review"? In my view, the sum of people's reactions on the searchable web is way, way short of being useful, because given any controversial subject, the communication space will be swamped by those with an ideological, not scientific, view. If you know of some "open science peer review" process that is more reliable, please let us know!

      Delete
    4. For those who have not followed that movement here are some links:

      In Nature, in January, 2017, a group of scholars known for advocating open science published a "manifesto" for open science:
      https://www.nature.com/articles/s41562-016-0021

      There is a wikipedia entry about it:
      https://en.wikipedia.org/wiki/Open_science

      I think journals act as a wall between the public, that has funded most of the published science, and the science community. This leeds to a growing disconnect between public and scientists.

      More transparency in the scientific publishing process could also result in more control of science by the public, adding new layers to peer review processes. In fields where corporate interest has grown to dominate science, this control might help mitigate the effects.

      Delete
    5. Obviously, what the perfect method for soemthing like an "open peer review" process should look like is an ongoing debate:
      https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5437951/

      Delete
  10. As a Biologist my experience is, already in the 1970s it emerged, Science research became so vastly complex, even the Specialists have trouble to follow and to understand. We may have passed “Peak Mind” - and what we do, may become impossible to understand anymore. This creates the Führer-Syndrome, when 1 claims, to know it all - and Science is Religion.. and the Heretics will be burned...

    ReplyDelete
    Replies
    1. Maybe one can offer its own piece of approximation to the other people "sine ira nec studio" (without hidden motivations) and accept criticisms and make criticisms in the same spirit. If the issue is in "the hidden motivations" then the problem is not the scientific method (or any other method ) but a problem of trust - as Prof. Bardi pointed out. You can be persuaded by an argument or not and you shouldn't easily abandon your convinctions. But you should be open to listen and ready to change your opinion if it is the case. Its a problem of attitude toward the truth rather than a problem of method.

      Delete
  11. A long time ago, in a galaxy far, far away (late 1980s) when I was a graduate student pursuing my Master’s in archaeology I became interested in the scientific process and epistemology. It was prompted by reading such authors as Stephen Jay Gould, Thomas Kuhn, and Clifford Geertz (and a lot of others). I began exploring the topics of textual criticism, philology, and hermeneutics, particularly as they pertained to trying to understand the ‘other’. The post-modernist era was in full swing and was impacting my outlook greatly.
    The issue of academic freedom and its influence on the practise of science rose its head at my university (McMaster University, Hamilton, Ontario, Canada) when our university paper published an article outlining an argument by one of the professors that academics needed to be granted far more money and time away from teaching in order to lessen the external political and social pressures present.
    Being who I was (am?) and what I was learning through my readings, I had to challenge the implicit argument that more money and less teaching time would ensure scientific ‘objectivity’. I held and still believe humans are incapable of such objectivity.
    As I argued in my letter published by the paper, “...science is a socially-embedded activity in which research is pervasively influenced by the sociocultural milieu within which it operates. Various psychological and sociocultural factors serve to guide scientific research in predetermined directions. This belief debases the stereotypical view of science as a purely objective enterprise. In fact, some researchers have taken a more radical stand and argued that truth itself is just what a particular scientific community passes at a particular time; that facts are created and we make them fit into our predetermined categories; and that truth is merely the truth of those in power. Alternatively, it can be argued that the facts are real enough but in the interpretation that necessarily follows empirical observations, hard ‘facts’ are tainted by external pressures. These pressures are believed to influence research and its conclusions, even to the point of ‘cheating’. Because of such pervasive pressures within the scientific community, self-policing by academics is not an adequate solution.”
    And, “...Since research is a subjective and interpretive enterprise, interpretations will inevitably be pluralistic in nature and there is no monopoly on truth. A diversity of interpretations is, therefore, both inevitable and necessary. However, this should not be construed as academic anarchy. Scholars should attempt to understand their own subjective biases and how their sociocultural milieu influences their work. It is only by doing this that they may become more sensitive to the restrictions that are imposed upon their interpretations. Perhaps this endeavour would result in a useful balance between the outdated view of science as objective and the radical notion of a total lack of truth.”
    To steal a line from the X-Files, the ‘truth’ is out there but I wonder whether humans can ever fully comprehend it given our biases, prejudices, and predilections. We are social animals that are greatly influenced by our surroundings, regardless of attempts to isolate ourselves and our thinking.
    My full letter can be seen here at my website: http://olduvai.ca/?p=29956

    ReplyDelete
  12. Ugo: Thanks or this post. I have spent quite a bit of time thinking about this and, while not rejecting science, I have come to the conclusion that it's effects in the current system are not in the best interests of humanity as a whole.

    There you go, awkward silence all around.

    What we are talking about when we talk about "Science" is a social construct more resembling a church than most people are willing to admit. The Bishops and Archbishops usually don't think about much except for their own perquisites and honors.

    Science, and it's handmaiden technology brought us to this strait. I live a comfortable life and will probably manage to die being overweight. My son states that he stands a pretty good chance of dying from an effect of climate change. He is probably right

    The trouble is, science, while useful for searching out the mysteries, once it figures out the mysteries turns the tools developed in the process over to the greedy and rapacious.

    If that isn't enabling the destruction, I don't know what is.

    As always, your work allows me to think. I wrote more about this subject over at my place should anyone wish to visit.

    John
    https://mightaswellliebackandenjoyit.blogspot.com/2018/09/lineage.html

    ReplyDelete

Who

Ugo Bardi is a member of the Club of Rome and the author of "Extracted: how the quest for mineral resources is plundering the Planet" (Chelsea Green 2014). His most recent book is "The Seneca Effect" (Springer 2017)