Bad Research

[xoxoxoBruce]

This is depressing. Do they get their grant money for the next fraud anyway?

US scientists are significantly more likely to publish fake research than scientists from elsewhere, finds a trawl of officially withdrawn (retracted) studies, published online in the Journal of Medical Ethics.
Fraudsters are also more likely to be "repeat offenders," the study shows.

The study author searched the PubMed database for every scientific research paper that had been withdrawn -- and therefore officially expunged from the public record -- between 2000 and 2010.
A total of 788 papers had been retracted during this period. Around three quarters of these papers had been withdrawn because of a serious error (545); the rest of the retractions were attributed to fraud (data fabrication or falsification).

The highest number of retracted papers were written by US first authors (260), accounting for a third of the total. One in three of these was attributed to fraud.

link

[glatt]

The article is only talking about total numbers, and doesn't put them in context. The US had the most papers withdrawn, but the article doesn't say if it also published the most. I suspect it did. A better number would be the percentage of papers withdrawn. This article is of very little value without the rest of the numbers to put this in context.

I've struck out fewer times than Babe Ruth. Maybe two dozen times total in my life. Therefore, I'm a better baseball player. Absolute BS.

[Spexxvet]

It makes you wonder if this study is fraudulent.

[Lamplighter]

Since my career did involve scientific publications, I am not a neutral observer here.

But a headline of "Bad Research" with no critical examination is bothersome for me.
What is especially so in this case is that the links provided do not allow direct access
to the original publication, so evaluation of source material is not possible.

I agree with Glatt. it is inappropriate to imply that US researchers
are more (intentionally?) fallible from these meager data,
particularly since the authors say the number of "repeat offenders" is signficant.

Here is an earlier report with a few raw numbers:

As the article below says, unintentional mistakes and errors do occur.
But in science the widely accepted ethical thing to do is publicly acknowledge such errors,
and do whatever is reasonable to correct the information.
Even financial reimbursement of the $-funding agencies is expected.
Such ethics are not always the case in other areas of public endeavor.

How many scientific papers should be retracted?
Murat Cokol, Ivan Iossifov, Raul Rodriguez-Esteban & Andrey Rzhetsky

Published scholarly articles commonly contain imperfections: punctuation errors, imprecise wording
and occasionally more substantial flaws in scientific methodology, such as mistakes in experimental design,
execution errors and even misconduct (Martinson et al, 2005).

These imperfections are similar to manufacturing defects in man-made machines:
most are not dangerous but a small minority have the potential to cause a disaster
(Wohn & Normile, 2006; Stewart & Feder, 1987).

Retracting a published scientific article is the academic counterpart of recalling a flawed industrial product
(Budd et al, 1998).

PubMed provides us with a ‘paleontological’ record of articles published in 4,348 journals with a known impact factor (IF).
Of the 9,398,715 articles published between 1950 and 2004, 596 were retracted.
This wave of retraction hits high-impact journals significantly harder than lower-impact journals (Fig 1A),
suggesting that high-impact journals are either more prone to publishing flawed manuscripts or
scrutinized much more rigorously than low-impact journals.

Our analysis indicates that although high-impact journals tend to have fewer undetected flawed articles
than their lower-impact peers, even the most vigilant journals potentially host papers that should be retracted.
However, the positive relationship between visibility of research and post-publication scrutiny suggests that the technical
and sociological progress in information dissemination—
the internet, omnipresent electronic publishing and the open access initiative—
inadvertently improves the self-correction of science by making scientific publications more visible and accessible.

[ZenGum]

Direct from New Scientist:

Which nation's scientists are most prone to falsifying their results? A statistical debate has broken out this week, following the publication of a paper claiming that US researchers "are significantly more prone to engage in data fabrication or falsification than scientists from other countries".

To reach this conclusion, published in the Journal of Medical Ethics, Grant Steen of Medical Communication Consultants in Chapel Hill, North Carolina, searched the PubMed database of the biomedical literature for papers retracted between 2000 and 2010, recorded whether the retractions were due to error or fraud, and calculated a fraud-to-error ratio.

But as Richard Van Noorden pointed out on Nature's Great Beyond blog, this doesn't necessarily mean that US-based scientists engage in more fraud - they might simply publish fewer sloppy errors. By looking at total papers published, he calculated US researchers have to retract a smaller proportion because of fraud than those from China, India and South Korea.

Bob O'Hara, a statistician and ecologist at the University of Helsinki in Finland, then crunched the numbers on his Nature Network blog. The retraction rate due to fraud for US-based researchers was slightly above the norm, but nowhere near as high as in China (about three times above average) and India (five times). The differences between the US and its Asian competitors were highly statistically significant.

French researchers emerged as the least prone to fraud in this analysis. But as O'Hara explains, we shouldn't read too much into any of the figures, as there's a "missing data" problem: retractions only measure the rate at which people get caught out for fraud, not the rate at which it is committed.

Indeed, as New Scientist's long-running investigation into stem cell research at the University of Minnesota has shown, papers containing manipulated data may remain unchallenged for many years, until someone takes the time to pick them apart.

There are other biases to consider. Steen's paper confirms earlier findings that rates of retraction are higher in more prominent journals. That makes sense, as these papers will be subject to greater scrutiny.

In June, New Scientist found that US-based researchers were more successful than those elsewhere at getting their papers into high-impact journals in the hot field of cellular reprogramming. If that's also true in other areas of biomedical research, it may make US-based researchers seem more fraudulent than they actually are.

As I read this, the original study compared fraud-retractions to error-retractions.
A much better measure is retractions of either type compared to total publications. On that, the US comes out about normal.

So is the original study just badly done? or downright fraudulent?

[xoxoxoBruce]

It says nothing about the original study. We haven't read the original study so we don't know how it was presented, or what conclusions were purported. All we know is what the Science Daily reporter plucked from the study and the conclusions he made. That's why it's important to look at what people who have seen it have to say.

I find it depressing that there are cases of fraud. Mistakes I can understand, and hope they would be discovered and corrected. But the cases of fraud make all research papers suspect, to me.

[DanaC]

Originally Posted by xoxoxoBruce

I find it depressing that there are cases of fraud. Mistakes I can understand, and hope they would be discovered and corrected. But the cases of fraud make all research papers suspect, to me.

That seems completely unreasonable to me. Every industry and every profession has its share of shady operators. Every product and every line has its quality control issues. No quality control is 100% effective.

What is heartening is how much of that fraud actually gets picked up and the work retracted. The more important and relevant a paper, the more scrutiny it will inevitably attract; the more likely a fraudulent paper in that area will be discovered.

[classicman]

Originally Posted by xoxoxoBruce

I find it depressing that there are cases of fraud. Mistakes I can understand, and hope they would be discovered and corrected. But the cases of fraud make all research papers suspect, to me.

That seems completely reasonable to me since every industry and every profession has its share of shady operators. Every product and every line has its quality control issues. No quality control is 100% effective. Therefore they all MUST BE suspect.

[ZenGum]

Bruce, Classic, good news.

Firstly, the number of fraudulent articles per decade is in the hundreds. There are over 100,000 academic journals, each publishing either yearly, quarterly, monthly, or even weekly. Each issue will have a dozen or more articles. We're talking something between one and five million articles per year. Maybe a few dozen are fraudulent. Compare that to other industries - how many badly made cars or building are done each year? How many unsafely cooked meals are served in restaurants? etc etc. as a percentage, it is minute. What percentage of news media reports are bull#$%#?

And for Classic, every article is suspect. That's the point of publication. Anyone else working in that field will read the article and try to pick it apart. If it seems interesting, they will replicate the experiments. If no-one can reproduce the results, people start scrutinising the original article very closely.

You're right, though, to not get carried away when we read in the paper that "scientists have discovered that ...". Always wait for checking and so on.

I notice that this is the second thread we've had in a few months about criticisms of academic and scientific research. It didn't start in the cellar, but I'm wondering if some vested interests are deliberately trying to undermine confidence in research.

[DanaC]

I think we need a better word than 'suspect'. Perhaps one less loaded with negative connotations. All research papers should be viewed critically. But the idea that because some small minority of research scientists have engaged in dishonest practice all research should be considered 'morally' suspect, which is what we are in fact talking about, is unreasonable.

All research is 'suspect' only in terms of it having to be proved and scrutinised and not taken at face value without some kind of critical appraisal of methodology and results. To approach all scientific research papers with one eye open to fraud is an over reaction to a very small problem.

[eta]

Originally Posted by ZenGum

I notice that this is the second thread we've had in a few months about criticisms of academic and scientific research. It didn't start in the cellar, but I'm wondering if some vested interests are deliberately trying to undermine confidence in research.

Not just 'research'. Academia in general. There have been a lot of negative academia stories in the news recently; the kind that basically suggest that academia is a big-business con trick and academicians somehow less worthy and more distant than other professionals.

Which is probably why my tone may have been unnecessarily abrasive. There is a little cultural war being played out, over here and in the US; don't know about other places. It's no coincidence that respect ratings for teachers (along with wages in the public/non-fee sector) are lower here than in most other countries.

You're right that the scientific world seems to come in for special approbation at times. They're clearly perceived as dangerous; whilst other academics are simply not valued.

[Sundae]

Dana! You retracted!

[Aliantha]

It's very difficult to find a scientist who isn't biased one way or another. There are very few altruistic scientists out there. They all have their own beliefs and requirements for their studies, and in the end, most of them have to pay the bills one way or another anyway.

Science is no more black and white than any other field. There are many shades of grey.

My suggestion is that the reason science is coming under scrutiny is that every tom, dick or harry can get their hands on scientific research these days thanks to the internet, so everyone thinks they're smarter and more informed than ever before. While this may be true in some cases, in others, people simply do not have the basic skills to disseminate the information in front of them properly and come to wildly unfounded conclusions based on, yep, you guessed it. Their very own personal biases.

[monster]

The thing about science is it's almost impossible to develop a good experiment without a pre-conceived idea about the result. Hence the need fo a null hypothesis. But you still know what you want/expect to happen, and scientists are human. And money can make them more so.....

[piercehawkeye45]

Originally Posted by classicman

That seems completely reasonable to me since every industry and every profession has its share of shady operators. Every product and every line has its quality control issues. No quality control is 100% effective. Therefore they all MUST BE suspect.

Right now there is a very large amount of pressure on professors and researchers to get grants and put out studies and mistakes and fraud will be an unfortunate consequence of it.

Another large problem is that the reasons for "mistakes" are so varied. It is really tough to tell the difference between outright fraud for profit, mistakes, and people in the middle who feel they have to publish something or risk not getting tenure.

[classicman]

I agree Pierce. You'll note that the post you quoted was basically Dana's response to xoB, with my interpretation.