Happy Talk About Published Scientific Error and Fraud

Posted: January 4, 2013 | Author: | Filed under: ethics, research | 2 Comments

Over at the American Scientist (in an overall interesting Jan-Feb. 2013 issue) we have a column arguing that there’s no need to worry about a contagion of fraud and error in scientific publication, even though the number of publications has exploded and the number of retractions has exploded along with them. The basic pitch: the scientific literature is wonderfully self-correcting. The evidence given: the ratio of voluntary corrections to retractions for fraud looks kind of high, and journals with more aggressive and welcoming policies toward corrections have more of them. I kid you not.

But wait, you say. How is that evidence at all probative? Good question, as one says when the student goes right where we want to take the discussion. At the very least, we’d want to see if the rate of retractions is going up over time, but somehow those figures and graphs don’t appear in the article. But what we’d really like to know is how many non-retracted, non-corrected, and non-commented articles are in fact erroneous or misleading despite peer review, and here the article is silent. It’s evidence is almost completely non-responsive to the question it purports to address. But the problem goes deeper.

Recent public concerns, including on this blog, have noted pressures for sensationalism, publication bias, data snooping and experimental tuning bias,  and many similar causally based arguments. John Ionnadis has made a pretty good career pounding on these issues and trying to place upper and lower bounds on the problem. The devastating Begley and Ellis study of “landmark” papers in preclinical cancer research found that only 6 of 53 had reproducible results, even after going back to the original investigators and sometimes even after the original investigators themselves tried to reproduce their published results. Here is what the latter authors think about the health of the peer-reviewed publishing system in pre-clinical cancer research:

The academic system and peer-review process tolerates and perhaps even inadvertently encourages such conduct5. To obtain funding, a job, promotion or tenure, researchers need a strong publication record, often including a first-authored high-impact publication. Journal editors, reviewers and grant-review committees often look for a scientific finding that is simple, clear and complete — a ‘perfect’ story. It is therefore tempting for investigators to submit selected data sets for publication, or even to massage data to fit the underlying hypothesis.

Of this substantial and growing literature on the prevalence of error and publication of invalid results, the American Scientist article is entirely innocent. Instead, it uses a single Wall Street Journal article as its target for attack, and even there ignores the non-anecdotal parts of the story–evidence that retractions have been growing faster than publications since 2001 (up 1500% vs. a 44% increase in papers), that the time lag between publication and retraction is growing, and that retractions in biomedicine related to fraud have been growing faster than those due to error and constitute about 75% of the total retractions.

Perhaps a corrigendum is in order over at the Am Sci.

UPDARE:

A September 2012 article in PNAS found that most retractions are caused by misconduct rather than error:

A detailed review of all 2,047 biomedical and life-science research articles indexed by PubMed as retracted on May 3, 2012 revealed that only 21.3% of retractions were attributable to error. In contrast, 67.4% of retractions were attributable to misconduct, including fraud or suspected fraud (43.4%), duplicate publication (14.2%), and plagiarism (9.8%). Incomplete, uninformative or misleading retraction announcements have led to a previous underestimation of the role of fraud in the ongoing retraction epidemic. The percentage of scientific articles retracted because of fraud has increased ∼10-fold since 1975. Retractions exhibit distinctive temporal and geographic patterns that may reveal underlying causes.

The Fallacy of Equivocation at the Economist

Posted: January 3, 2013 | Author: | Filed under: are you kidding me?, business school, education, ethics, rants | 6 Comments

In a remarkably shoddy example of anti-market propaganda emanating from the Nottingham Business School, the Economist runs a screed that starts out with the debatable but reasonable premise that business leaders exaggerate their omniscience. It somehow ends up with the unsupported conclusion that business schools should abandon economics, finance, and the pursuit of profit for the cant trio of “sustainability,” “social responsibility,” and “leadership for all not for the few.”

The crude equivocating shifts from intellectual humility to moral humility to altruism would qualify for an F in any class on composition, much less philosophy. The vague assertions about “business excess” (entirely unsupported or even defined), the implicit attribution of these excesses to the teachings of business schools (ditto), and the wild leap at the end (replacing business school education with an agora-like setting in which sophists mingle with scientists and philosophers with philistines to figure out what are “social needs”), all conduce to a massive loss of reader brain cells per sentence. This article might be useful as a sort of mine detector–anyone who finds it congenial is best separated from responsibility for educating or commenting on business or economic issues.

 

Genetically Engineered Human Capital?

Posted: November 10, 2012 | Author: | Filed under: ethics, Healthcare, human capital | Leave a comment

NPR reports that geneticists have crossed a line that has been considered taboo: They changed human DNA in a way that can be passed down to future generations. The researchers at Oregon Health & Science University say they took the step to try to prevent women from giving birth to babies with genetic diseases.

Applied to such health issues, over a long haul, it could make richer nations genetically predisposed to better health. Stronger health, in turn, may create economic opportunities that might not otherwise exist. One can imagine that this could widen existing gaps between emerging economies where such technologies are less likely to be applied. Of course, it may also exacerbate such gaps within wealthy nations where income inequality is already a hot-button issue.

This assumes all that the technology is not applied to more controversial traits like enhancing intelligence (which we can’t even measure very well much less identify a gene that would have such an effect).

You can listen to the story by clicking here.

The History Tax

Posted: May 10, 2012 | Author: | Filed under: current events, economics, ethics, law and society, political science, rants, Uncategorized | 2 Comments

Try to guess the context for this piece of writing. Is it part of a scholarly study on the history of convention centers? A tourist guidebook? Is it the catalogue to a museum display on convention-center architecture?

In order to attract growing numbers of conventions in the
second half of the twentieth century, cities incorporated
convention center construction within urban renewal and
redevelopment schemes, usually at the edge of core urban
areas where space would be available for construction of
large buildings with contiguous, flat-floor space.

Read the rest of this entry »

Manipulating journal impact factors

Posted: February 28, 2012 | Author: | Filed under: ethics, incentives, research | 3 Comments

Via several folks on Facebook (e.g., Marcel Bogers, Der Chao Chen) – here’s a short blog post on how journals are manipulating their impact factors: coerced citations and manipulated impact factors – dirty tricks of academic journals.

Here’s the Science piece that the above blog post refers to: Coercive citation in academic publishing.  Here’s the data (pdf).

Fraud in the ivory tower (and a big one too)

Posted: December 5, 2011 | Author: | Filed under: business school, ethics, incentives | 14 Comments

The fraud of Diederik Stapel – professor of social psychology at Tilburg University in the Netherlands – was enormous. His list of publications was truly impressive, both in terms of the content of the articles as well as its sheer number and the prestige of the journals in which it was published: dozens of articles in all the top psychology journals in academia with a number of them in famous general science outlets such as Science. His seemingly careful research was very thorough in terms of its research design, and was thought to reveal many intriguing insights about fundamental human nature. The problem was, he had made it all up…

For years – so we know now – Diederik Stapel made up all his data. He would carefully reiterature, design all the studies (with his various co-authors), set up the experiments, print out all the questionnaires, and then, instead of actually doing the experiments and distributing the questionnaires, made it all up. Just like that.

He finally got caught because, eventually, he did not even bother anymore to really make up newly faked data. He used the same (fake) numbers for different experiments, gave those to his various PhD students to analyze, who then in disbelief slaving away in their adjacent cubicles discovered that their very different experiments led to exactly the same statistical values (a near impossibility). When they compared their databases, there was substantial overlap. There was no denying it any longer; Diederik Stapel, was making it up; he was immediately fired by the university, admitted to his lengthy fraud, and handed back his PhD degree.

In an open letter, sent to Dutch newspapers to try to explain his actions, he cited the huge pressures to come up with interesting findings that he had been under, in the publish or perish culture that exist in the academic world, which he had been unable to resist, and which led him to his extreme actions.

There are various things I find truly remarkable and puzzling about the case of Diederik Stapel.

  • The first one is the sheer scale and (eventually) outright clumsiness of his fraud. It also makes me realize that there must be dozens, maybe hundreds of others just like him. They just do it a little bit less, less extreme, and are probably a bit more sophisticated about it, but they’re subject to the exact same pressures and temptations as Diederik Stapel. Surely others give in to them as well. He got caught because he was flying so high, he did it so much, and so clumsily. But I am guessing that for every fraud that gets caught, due to hubris, there are at least ten other ones that don’t.
  • The second one is that he did it at all. Of course because it is fraud, unethical, and unacceptable, but also because it sort of seems he did not really need it. You have to realize that “getting the data” is just a very small proportion of all the skills and capabilities one needs to get published. You have to really know and understand the literature; you have to be able to carefully design an experiment, ruling out any potential statistical biases, alternative explanations, and other pitfalls; you have to be able to write it up so that it catches people’s interest and imagination; and you have to be able to see the article through the various reviewers and steps in the publication process that every prestigious academic journal operates. Those are substantial and difficult skills; all of which Diederik Stapel possessed. All he did is make up the data; something which is just a small proportion of the total set of skills required, and something that he could have easily outsourced to one of his many PhD students. Sure, you then would not have had the guarantee that the experiment would come out the way you wanted them, but who knows, they could.
  • That’s what I find puzzling as well; that at no point he seems to have become curious whether his experiments might actually work without him making it all up. They were interesting experiments; wouldn’t you at some point be tempted to see whether they might work…?
  • Truly amazing I also find the fact that he never stopped. It seems he has much in common with Bernard Madoff and his Ponzi Scheme, or the notorious traders in investments banks such as 827 million Nick Leeson, who brought down Barings Bank with his massive fraudulent trades, Societe Generale’s 4.9 billion Jerome Kerviel, and UBS’s 2.3 billion Kweku Adoboli. The difference: Stapel could have stopped. For people like Madoff or the rogue traders, there was no way back; once they had started the fraud there was no stopping it. But Stapel could have stopped at any point. Surely at some point he must have at least considered this? I guess he was addicted; addicted to the status and aura of continued success.
  • Finally, what I find truly amazing is that he was teaching the Ethics course at Tilburg University. You just don’t make that one up; that’s Dutch irony at its best.

Strategic analysis of strategist certification

Posted: November 15, 2011 | Author: | Filed under: collective behavior, conferences, ethics, law and society, rants, Uncategorized | 9 Comments

At the recent Strategic Management Society meetings in Miami, I attended a session devoted to creating an SMS strategy certificate. (Apparently this is an ongoing initiative that started a year ago or so, although I hadn’t been paying attention.) The idea is to offer a written exam that consultants can take (for a fee) in order to become SMS-certified strategists. (I would put in links to the SMS website for all this–they even have a forum where members can view the tentative list of exam topics and leave comments–but the hamsters that power the site appear either to be on strike or allergic to Chrome.)

My first reaction to this proposed exam was to be reminded of the old story about the grocer who observes a shopper sniffing the meat for freshness and responds, “Lady, could you pass that test?” They had a laundry list of topics forming a kind of core and then planned “electives” in different specialized areas of strategy. Many of the topics are things I’ve heard of but don’t know much about. Others are things that I know about but believe to be vacuous or fatally flawed. It looked like a flat-file version of one of those giant multicolored management textbooks used by undergraduate business majors, which have always depressed me with their pretension and lack of coherence. I’m not sure if I espied Miles and Snow’s categories among the topics flashing by on the Powerpoint, but they did have SWOT analysis, generic strategies, the BCG matrix, vision/mission statements, and a variety of other forms of management Laetrile. Can you imagine being certified in SWOT? In vision statements? It’s almost as embarrassing as Louisiana’s tests for licensing flower arrangers that were mostly repealed under pressure from the Institute for Justice.

Perhaps to maintain buy-in from the heavily academic constituency of SMS, the program is being sold as having no effect on academic curricula or research. The influence is supposed to go entirely from academia to consulting and practice, with no one’s course being pressured to meet the certification content.

It was a peculiar meeting in the Neptune room of the Loews. A working group had been beavering away on a proposed curriculum for a year and was ostensibly soliciting our feedback, but 1) didn’t want to engage in the specifics of what they had come up with and 2) didn’t really want to address the basic question of whether the whole enterprise makes sense. Those in charge took notes on what the people in the room said but it felt like one of those government “request for public comment” setups, where the fix is in and no meaningful reconsideration of the project is possible. One person told me afterward that he had never been in a meeting with such an undercurrent of fear and suppressed tension. There was indeed a whiff of preference falsification in the air.

I was as diplomatic as possible, but expressed some of my concerns. Afterwards, a few people commented to me that they thought that this was a terrible idea but had expected/hoped that its intrinsic hideousness would have killed it off by now. I see no signs of such a spontaneous abortion. Rather, the meetings keep going on and the “process” keeps rolling forward, despite the instantaneously queasy feeling it causes in everyone with whom I discuss it.

Why would the SMS want to do this?

Read the rest of this entry »

MBA Oath

Posted: October 31, 2011 | Author: | Filed under: education, ethics | 2 Comments

There’s lots of talk about the MBA Oath these days. A few links: