Wikipedia talk:Identifying reliable sources (medicine)

This is the talk page for discussing improvements to the Identifying reliable sources (medicine) page.

Put new text under old text. Click here to start a new topic.
New to Wikipedia? Welcome! Learn to edit; get help.

WT:MEDRS

Archives: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31: 60 days

Ideal sources for Wikipedia's health content are defined in the guideline Wikipedia:Identifying reliable sources (medicine) and are typically review articles. Here are links to possibly useful sources of information about Identifying reliable sources (medicine).

PubMed provides review articles from the past five years (limit to free review articles)
The TRIP database provides clinical publications about evidence-based medicine.
Other potential sources include: Centre for Reviews and Dissemination and CDC

view · edit

Frequently asked questions

Shortcut

WP:MEDFAQ

This is an explanatory essay about the Wikipedia:Identifying reliable sources (medicine) policy.

This page provides additional information about concepts in the page(s) it supplements. This page is not one of Wikipedia's policies or guidelines as it has not been thoroughly vetted by the community.

These are some Frequently Asked Questions about Wikipedia's guidelines on sourcing for medical content, manual of style for medicine-related articles, and how the guidelines and policies apply to biomedical content.

General

Does Wikipedia have special rules for medical information?

Yes, but the guidelines for medical information follow the same broad principles as the rest of Wikipedia. Examples of this include the requirement for reliable sources and the preference for secondary sources over primary sources. These apply to both medical and non-medical information. However, there are differences in the details of the guidelines, such as which sources are considered reliable.

Why do you have special rules for medical information?

Different types of sources have different strengths and weaknesses. A type of source that is good for scientific information is not usually as reliable for political information, and vice versa. Since Wikipedia's readers may make medical decisions based on information found in our articles,^[1] we want to use high-quality sources when writing about biomedical information. Many sources that are acceptable for other types of information under Wikipedia's general sourcing guideline, such as the popular press, are not suitable sources for reliable medical information.^[2]^[3]^[4] (See also: WP:MEDPOP and WP:WHYMEDRS)

When do I need to follow MEDRS?

MEDRS-compliant sources are required for all biomedical information. Like the policy on the biographies of living people ("BLP"), MEDRS applies to statements and not to articles: biomedical statements in non-medical articles need to comply with MEDRS, while non-medical statements in medical articles do not need to follow MEDRS. Also like BLP, the spirit of MEDRS is to err on the side of caution when making biomedical statements. Content about human biochemistry or about medical research in animals is also subject to MEDRS if it is relevant to human health.

Sourcing

I used a peer-reviewed source, but it was reverted, and the editor said I needed to use a review. I did, didn't I?

Probably not. Most peer-reviewed articles are not review articles. The very similar names are easily confused. For most (not all) purposes, the ideal source is a peer-reviewed review article.

Why can't I use primary sources?

Primary sources aren't completely banned, but they should only be used in rare situations. An individual primary source may be flawed, such as being a clinical trial that uses too few volunteers. There have been cases where primary sources have been outright fraudulent. Furthermore, a single primary source may produce a different result to what multiple other primary sources suggest, even if it is a high-quality clinical trial. Secondary sources serve two purposes: they combine the results of all relevant primary sources and they filter out primary sources that are unreliable. Secondary sources are not infallible, but they have less room for error than a primary source.

This follows a principle that guides the whole of Wikipedia. If a company announces a notable new product, Wikipedia would not cite a press release on the company's website (a primary source) but instead would cite a newspaper article that covers it (a secondary source). The difference with medical information is that the popular press are not suitable sources.

Whenever possible, you should cite a secondary source such as:

a review article
a meta-analysis
a high-quality textbook

So if primary sources can be used in rare cases, what are those rare cases?

Primary sources might be useful in these common situations:

when writing about a rare disease, uncommon procedure, etc., for which no high-quality secondary literature is available, or for which the available secondary sources do not cover all of the information normally included in an encyclopedia article.
when mentioning a famous paper or clinical trial that made a recognized substantial impact, as part of a purely historical treatment of a topic.
when describing major research that has made a significant impact (i.e., continued and substantial coverage). While recent research results are normally omitted, it is sometimes necessary to include them for WP:DUE weight. In this case, it is usually preferable to read and cite the primary scientific literature in preference to WP:PRIMARYNEWS sources. Later, these primary sources can be replaced or supplemented with citations to high-quality secondary sources.

Why can't I use articles from the popular press?

The popular press includes many media outlets which are acceptable sources for factual information about current events, sometimes with significant caveats. It also includes media outlets which are discouraged in all cases because the quality of their journalism is inadequate. However, even high-quality media outlets have disadvantages in the context of medicine.

Firstly, news articles on medicine will frequently be reporting a new medical primary source, such as the results of a new study. This means that they are effectively acting as a primary source, which as explained above makes those articles generally unsuitable for medical information. These articles also tend to omit important information about the study. If a medical primary source is to be cited at all, the academic paper should be cited directly.

Secondly, media coverage of medical topics is often sensationalist. They tend to favor new, dramatic or interesting stories over predictable ones, even though studies that reflect the current scientific consensus tend to be predictable results. They tend to overemphasize the certainty of any result, such as reporting a study result as a conclusive "discovery" before it has been peer-reviewed or tested by other scientists. They may also exaggerate its significance; for instance, presenting a new and experimental treatment as "the cure" for a disease or an every-day substance as "the cause" of a disease. The sensationalism affects both which stories they choose to cover and the content of their coverage.

High-quality media outlets can be good sources of non-medical information in an article about a medical topic. Another acceptable use is using a popular press article to give a plain English summary of an academic paper (use the |laysummary= parameter of {{cite journal}} for this).

I have a source from PubMed, so that's reliable right?

Not necessarily. PubMed is merely a search engine and the majority of content it indexes is not WP:MEDRS. Searches on PUBMED may be narrowed to secondary sources (reviews, systematic reviews, meta-analyses, etc.) so it is a useful tool for source hunting.

It is a common misconception that because a source appears in PubMed it is published by, or has the approval of, the National Institutes of Health (NIH), National Center for Biotechnology Information (NCBI), or the US government. These organisations support the search engine but lend no particular weight to the content it indexes.

Can I use websites like Quackwatch?

Quackwatch is a self-published website by an author who is an expert in problems with complementary and alternative medicine. Whenever possible, you should use a scholarly source instead of Quackwatch. However, if no scholarly sources are available, and the subject is still notable, then it might be reasonable to cite Quackwatch with WP:INTEXT attribution to the POV.

Can I cite Chinese studies about Traditional Chinese Medicine?

As of 2014, there are concerns regarding positive bias in publications from China on Traditional Chinese Medicine.^[5]^[6] Such sources should be used with caution. The problem also includes issues with the academic system in China.^[7]

Can I cite NCCAM (now NCCIH)?

Yes, but again only with WP:DUE weight. Unlike other branches of the National Institutes of Health, which are generally accepted as authoritative in their fields, NCCAM has been the focus of significant criticism from within the scientific community.^[8] Whenever possible, you should cite the established literature directly.

What if I can’t find any MEDRS-compliant sources on a subject?

MEDRS contains a section about finding sources which may be helpful. Alternatively, a more experienced editor may be able to help you find them (or to confirm that they do not exist).

Neutrality

What is a fringe medical claim?

A fringe medical claim is one that differs significantly from the prevailing views or mainstream views in the scientific medical community. This is similar to Wikipedia's general definition of a fringe claim. A claim can still be a fringe medical claim even if it has a large following in other areas of public life (such as politics and the popular press).

How should fringe medical claims be described?

When fringe claims have been widely reported in the press, have a large popular following, and/or have a long history, it may be appropriate to describe them in terms of that reporting, popularity, or history. However, weight should be determined by MEDRS-compliant sources, and the context (or lack thereof) should not make implications about medical statements that are not supported by such sources. Guidance on the additional considerations relevant to fringe subjects can be found at WP:FRINGE, as well as at other places such as WP:WEIGHT and WP:EXCEPTIONAL.

In the case of alternative medicine, medical statements are often derived from an underlying belief system, which will include many propositions that are not subject to MEDRS. These propositions are subject to the usual sourcing requirements and the usual requirements for determining fringe status.

If a treatment hasn't been shown to work, can we say it doesn't work?

There are three possible situations:

No evidence exists, either became no studies for the treatment have been published, or because the studies published are too small or weak to draw any conclusions.
Evidence exists, and it shows no effect.
Evidence exists, and it shows an effect.

In the first case, we cannot say that it does not work, but we can say that there is no evidence to determine whether it works. After multiple, high-quality independent studies have been published, the understanding may transition from "no evidence" to "some evidence" of either an effect or no effect. You should follow the lead of review articles and other secondary sources for determining when this threshold has been crossed.

Reports may conflict with each other. For example, a clinical trial may produce no evidence of an effect, but the treatment's manufacturer might produce testimonials claiming a positive effect. You should follow the lead of review articles and other secondary sources for determining how to balance these claims.

Should medical content be attributed?

In other words, is it necessary to say in the article's text the source which supports a medical statement (with attribution)? Or can it simply be stated as an unchallenged fact, with the source only mentioned in the citation (without attribution)? A statement without attribution will come across as being a stronger claim than one with attribution.

A result or statement from a reliable secondary source should be included without attribution if it is not disputed by any other recent secondary sources. You should do a search to check that the secondary source you are citing is the most up-to-date assessment of the topic.

If there have been two recent secondary sources that contradict each other, then you should attribute the disputed findings. On the other hand, if the findings of one or more recent secondary sources are disputed by one or more secondary sources from many years ago, but not by any recent ones, the recent findings can be stated without attribution. You should also take into account the relative weight secondary sources have. For example, Cochrane Collaboration reviews provide stronger evidence than a regular secondary source.

In the rare cases where primary sources can be used, they should be attributed.

Why not say there is a call for more research?

It is common for scientific publications to say something like this, either directly or indirectly. There are several reasons for this. It could be argued that more research is always a bonus, even if the topic has already been thoroughly researched. Sometimes, these statements may be made partly because authors need to convince readers that the topic is important in order to secure future funding sources. As such, saying this does not communicate much information, and it may also mislead readers into thinking that the existing information on a topic is less reliable than it really is.

How can Quackwatch be considered a reliable source?

As noted above, Quackwatch does not meet the usual standard as a reliable source, but it can be used (with attribution) for information on a topic of alternative and complementary medicine if there are no scholarly sources available for the same purpose. The guidelines on fringe theories includes the concept of parity: if a notable fringe theory is primarily described by self-published sources, then verifiable and reliable criticism of the fringe theory does not need to be published in a peer-reviewed journal. It only needs to come from a better source.

Finding and using sources

How can I find good sources using PubMed?

National Library of Medicine (NLM), PubMed, NCBI, & MEDLINE help, tutorials, documentation, & support

Full, searchable list of all tutorials - training materials in HTML, PDF and Video formats

YouTube channel for the National Library of Medicine: Tutorial videos from the National Center for Biotechnology Information (NCBI), part of the U.S. National Library of Medicine. Includes presentations and tutorials about NCBI biomolecular and biomedical literature databases and tools.

PubMed FAQs

PubMed User Guide - FAQs

National Library of Medicine (NLM) Catalog

NLM Catalog Help - This book contains information on the NLM Catalog, a database which provides access to NLM bibliographic data for journals, books, audiovisuals, computer software, electronic resources, and other materials via the National Center for Biotechnology Information (NCBI) Entrez retrieval system. The NLM Catalog includes links to full text materials and the library's holdings in LocatorPlus, NLM's online public access catalog.

NLM Catalog (rev. December 19, 2019).

Overview

FAQs

Searching NLM Catalog

Finding journals that comply with WP:MEDRS standards

For full comprehensive instructions, go to: Searching for Journals in NLM Catalog

Determine if a specific journal is indexed in MEDLINE

If you know the full or abbreviated name for a journal, and you want to see if it is indexed in MEDLINE, see the instructions at searching by journal title, which I will also reproduce here:

If you know the journal’s exact title, enter it in the NLM Catalog search box followed by the field qualifier [jo].

Example:

The Journal of Supportive Oncology [jo]

Results = 1 record retrieved:

The Journal of Supportive Oncology

If you know the journal’s NLM Title Abbreviation, enter it in the NLM Catalog search box, followed by the field qualifier [ta].

Example:

n engl j med [ta]

Results = 1 record retrieved:

The New England journal of medicine

Review the list of Abridged Index Medicus journals

Via a search of the NLM Catalog: List of Abridged Index Medicus journals, also known as "Core clinical journals".

Stand alone list: List of current Abridged Index Medicus (AIM) journals (118 journals as of 5 May 2020)

Create a list of all Index Medicus journals

Search the NLM Catalog using jsubsetim[All Fields] to find all Index Medicus journals (5021 journals as of 29 May 2020); or go directly to the search results for all Index Medicus journals. (Note that immediately above "Search Results" on that page, you can change the default "20 per page" to as many as 200 results per page, and you can change how the results are "sorted", e.g., if you are looking for a specific journal, you can sort by Title, instead of the default.)

====Create a list of all journals indexed in MEDLINE}} Search the NLM Catalog using currentlyindexed[All] to find all journals indexed in MEDLINE (5266 journals as of 29 May 2020); or go directly to the search results for all journals indexed in MEDLINE. (Note that immediately above "Search Results" on that page, you can change the default "20 per page" to as many as 200 results per page, and you can change how the results are "sorted", e.g., if you are looking for a specific journal, you can sort by Title, instead of the default.)

MEDLINE, PubMed, and PMC (PubMed Central): How are they different?

Are there ways to find good sources other than PubMed?

Besides being a secondary source, what else indicates a source is of high quality?

I found what looks like a good source, but can't access the full text – what next?

Most scholarly journals are behind paywalls. Some options to access these articles include visiting a local university library, visiting The Wikipedia Library, and WikiProject Resource Requests.

Note that paywalled articles are frequently pirated and made available on the open web. When linking to a journal article, care must be taken not to link to such a pirate copy, as such a link would be a copyright violating link in contravention of Wikipedia's policy. In general if you find such a copy and it is not accompanied by text explicitly stating that it is made available with the permission of the copyright holder, assume that it is potentially infringing, and do not link to it. This holds for all edits in Wikipedia, not just in article space.

Google Scholar

Search for the title of the article on Google Scholar. On the results page, click on "All n versions" (where n = the number of available versions of that article) at the bottom of a listing. The resulting page might contain PDF or HTML versions of the article.

Unpaywall

Consult Unpaywall.org for journal articles available without a subscription. Install the UnPaywall extension for Chrome or Firefox to immediately identify articles with a free version. After you install the extension, look to the right side of the page (when you are on the website for an article) for either a grey locked symbol (no free version) or a green unlocked symbol (click on that symbol to access the full text version of the article).

Librarian's advice

An article by librarian John Mark Ockerbloom, titled, "Why Pay for What’s Free? Finding Open Access and Public Domain Articles" offers helpful suggestions.^[9]

How do I reference a medical article?

Almost all medical articles are indexed by the PubMed search engine and have a Digital object identifier (DOI) assigned to them. All articles included in PubMed are assigned an eight-digit PubMed identifier (PMID). These identifiers can be used to refer to articles, which is preferred to URLs as it makes a reliable link which is resilient to changes beyond our control – i.e. the publisher being acquired by another publisher and it's "normal" web URLs changing as a consequence.

Once you have the PMID, there are a number of tools such as this one which you can use to generate a full citation automatically.

In article references, the "doi" and "pmid" parameters are preferred to the "url" parameter for such reasons.

On Talk pages, when referring to journal articles, is it good practice to make any link using these types of identifier also:

Typing "[[PMID:dddddddd]]", where dddddddd is a PMID, will create a link to the indicated article.
Any DOI can be turned into a resolvable web address by prepending "https://doi.org/" to it (e.g. https://doi.org/10.1136/bmj.c6801).

Conflict of interest

Are there special rules on conflicts of interest for health content?

What if I am being paid to edit medical content?

References

^ Laurent, MR; Vickers, TJ (2009). "Seeking health information online: does Wikipedia matter?". J Am Med Inform Assoc. 16 (4): 471–9. doi:10.1197/jamia.M3059. PMC 2705249. PMID 19390105.
^ Schwitzer G (2008). "How do US journalists cover treatments, tests, products, and procedures? an evaluation of 500 stories". PLOS Med. 5 (5): e95. doi:10.1371/journal.pmed.0050095. PMC 2689661. PMID 18507496.
^ Goldacre, Ben (2008-06-21). "Why reading should not be believing". Guardian.
^ Dentzer S (2009). "Communicating medical news—pitfalls of health care journalism". N Engl J Med. 360 (1): 1–3. doi:10.1056/NEJMp0805753. PMID 19118299.
^ Li J, et al The quality of reports of randomized clinical trials on traditional Chinese medicine treatments: a systematic review of articles indexed in the China National Knowledge Infrastructure database from 2005 to 2012. BMC Complement Altern Med. 2014 Sep 26;14:362. PMID 25256890
^
Further information:
- "Some countries publish unusually high proportions of positive results. Publication bias is a possible explanation. Researchers undertaking systematic reviews should consider carefully how to manage data from these countries." Vickers, Andrew (April 1, 1998), "Do certain countries produce only positive results? A systematic review of controlled trials.", Controlled Clinical Trials, 19 (2), Control Clin Trials: 159–66, doi:10.1016/s0197-2456(97)00150-5, PMID 9551280
- Ernst, Edzard (2012). "Acupuncture: What Does the Most Reliable Evidence Tell Us? An Update". Journal of Pain and Symptom Management. 43 (2): e11–e13. doi:10.1016/j.jpainsymman.2011.11.001. ISSN 0885-3924. PMID 22248792.
^ Qiu, Jane (January 12, 2010), "Publish or perish in China", Nature, 463 (7278): 142–143, doi:10.1038/463142a, PMID 20075887, S2CID 205052380
^
Some examples:
- Nature Reviews Cancer: "the subject of rancorous scientific and political debate over its mission and even continued existence"
- Clinical Rheumatology: "The criticism repeatedly aimed at NCCAM seems justified, as far as their RCTs of chiropractic is concerned. It seems questionable whether such research is worthwhile."
- Nature News: "still draws fire from traditional scientists", "Many US researchers still say such funding is a waste of time and money."
- Science News: "[NCCAM] is a political creation"; "This kind of science isn't worth any time or money" (quoting Wallace Sampson)
- Science Policy Forum: "[NCCAM] was created by pressure from a few advocates in Congress"; "NCCAM funds proposals of dubious merit; its research agenda is shaped more by politics than by science; and it is structured by its charter in a manner that precludes an independent review of its performance"; "NCCAM is unable to implement a research agenda that addresses legitimate scientific opportunities or health-care needs"
^ Ockerbloom, John Mark. "Why Pay for What’s Free? Finding Open Access and Public Domain Articles." Everybody's Libraries (23 Oct 2018).

Other helpful resources

Medicine

This page is within the scope of WikiProject Medicine. Please visit the project page for details or ask questions at Wikipedia talk:WikiProject Medicine.MedicineWikipedia:WikiProject MedicineTemplate:WikiProject Medicinemedicine articles

Pharmacology

This page is within the scope of WikiProject Pharmacology, a collaborative effort to improve the coverage of Pharmacology on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.PharmacologyWikipedia:WikiProject PharmacologyTemplate:WikiProject Pharmacologypharmacology articles

Psychoactive and Recreational Drugs (defunct)

This article is within the scope of WikiProject Psychoactive and Recreational Drugs, a project which is currently considered to be defunct.Psychoactive and Recreational DrugsWikipedia:WikiProject Psychoactive and Recreational DrugsTemplate:WikiProject Psychoactive and Recreational DrugsPsychoactive and Recreational Drugs articles

Alternative medicine

This page is within the scope of WikiProject Alternative medicine, a collaborative effort to improve the coverage of Alternative medicine related articles on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.Alternative medicineWikipedia:WikiProject Alternative medicineTemplate:WikiProject Alternative medicineAlternative medicine articles

This page has been mentioned by multiple media organizations:

Noam Cohem (March 15, 2020). "How Wikipedia Prevents the Spread of Coronavirus Misinformation". Wired. Wikipedia shows, however, that extreme circumstances, especially when related to public health, require different, more stringent rules, not better application of existing rules. The stakes are simply too high.
Jackson Ryan (June 24, 2021). "Inside Wikipedia's endless war over the coronavirus lab leak theory". CNET. The exclusion of the lab leak theory from Wikipedia predominantly rests on established guidelines. Chief among them is one known by editors as WP:MEDRS. It refers to the referencing of "biomedical" information on Wikipedia, stating sources must be "reliable, third-party published secondary sources, and must accurately reflect current knowledge." It's a guideline that has launched a thousand Talk page disputes.

Simply bad

Bendegúz Ács asked a little while ago for more information about the sentence that "A source can also simply be bad, where biases in criteria make it less than ideal." Here's a short explanation, in case anyone's interested:

We have criteria that help editors identify sources that are generally better (e.g., peer-reviewed sources, secondary sources, recent sources). However, no single criterion definitely makes a source be good (WP:NOTGOODSOURCE). In addition to all the criteria, you have to use common sense and have a good understanding of the subject. Sometimes a source seems superficially good, but when you read it, you discover that it's just bad. In such cases, you should use your WP:Editorial discretion to avoid relying on a bad source.

Here are some simple examples of how real-world bias could lead an editor to decide that a source is just bad:

The subject is a disease that's very common in poor countries, but your source tells you about the disease in a wealthy country.
You need to write something about pregnancy, and the clinical trial involves only males. (That's how the world ended up with the Thalidomide scandal, by the way.)
The subject is pediatrics, and the data is all from elderly people.

The main reason that we don't explain is because the potential sources of bias are enormous. Editors who are uncertain about whether their source can support the claims they are making should be discussing the specifics on the article's talk page or at Wikipedia talk:WikiProject Medicine. WhatamIdoing (talk) 02:41, 23 February 2024 (UTC)[reply]

Thank you for the clarification and the examples, they definitely make this particular issue much easier for me to understand. However, I would still like to suggest some improvements to the sentence. I think here there are two separate potential issues that may make a source bad, and both are worth mentioning separately.

One is what your examples are about, and I would call this something along the lines of "subjects/subgroup analysis with low relevancy" or "irrelevant/overly restrictive subjects/subgroup analysis", since the main issue I can identify in your examples is that the source fails to take into account most of the people affected by the topic. I think this particular issue would be worth mentioning in a separate sentence because it does seem important.

The other one is about problems that are so particular about a study that it is not possible to easily fit into any well-defined criterion. I would not limit these problems to "biases", however, since not every problem possible to make a study bad is considered a bias. So maybe mentioning it in this way would be an improvement: "A source can also simply be bad, due to a bias, or other problem, too particular to be covered by a common editorial criterion. In cases like this, WP:Editorial discretion may be used."

What do you think? Bendegúz Ács (talk) 23:30, 23 February 2024 (UTC)[reply]

It might be better to re-write the whole sub-section from scratch. I think it is meant to cover things like pesticide manufacturers claiming that their products are utterly harmless. WhatamIdoing (talk) 22:55, 24 February 2024 (UTC)[reply]

I would support a rewrite, but I am not competent enough to come up with a suggestion for a complete new text for the sub-section. I would be happy to review it, though. Bendegúz Ács (talk) 15:01, 25 February 2024 (UTC)[reply]

I've considered just blanking it. Do we truly need WP:MEDBIAS, or is it WP:CREEPY and redundant with other pages? It's only been linked in discussions about a handful of articles.

@Bon courage, what do you think? Could we live without it? WhatamIdoing (talk) 07:11, 7 March 2024 (UTC)[reply]

Hah! Have never paid any attention to this. It arrived[1] in 2016. WP:DONTSHOUTBIAS eh!? Bon courage (talk) 07:26, 7 March 2024 (UTC)[reply]

Oh, that would have been around the time that we tried (and failed) to put a medicine-specific version of WP:DUE in this guideline. WhatamIdoing (talk) 22:58, 7 March 2024 (UTC)[reply]

I don't see anything there that needs to be (medicine) specific and the "Personal conflicts of interest" isn't about "reliable sources" either. I'd vote for blanking both. The concerns there about biased sources or biased editors belong in general guidelines. -- Colin°^Talk 08:55, 7 March 2024 (UTC)[reply]

MEDRS-related discussion at WP:RSN

There is a new discussion at RSN that MEDRS-interested editors may want to get involved in: Wikipedia:Reliable_sources/Noticeboard#MEDRS_sources_on_curcumin_supplementation Schazjmd (talk) 23:53, 25 February 2024 (UTC)[reply]

Meta-analyses need careful scrutiny, peer-reviewed or not.

This

Glenza, Jessica (28 April 2024). "Junk science is cited in abortion ban cases. Researchers are fighting the 'fatally flawed' work". The Guardian. Of the 22 studies cited by the meta-analysis, 11 were by the lone author of the paper itself. The meta-analysis "failed to meet any published methodological criteria for systematic reviews" and failed to follow recommendations to avoid statistical dependencies, according to a criticism published in the British Medical Journal (BMJ).

is a cautionary tale. 𝕁𝕄𝔽 (talk) 19:15, 28 April 2024 (UTC)[reply]

Thanks, that's interesting.

I have been wondering recently why peer review doesn't involve putting each article through an evidence-based checklist (e.g., the Newcastle–Ottawa scale for non-randomized studies; the most relevant checklist for the subject matter could be agreed upon by the reviewers and the editor) and the results published alongside the article. It might make it easier to discover junk science before publication. WhatamIdoing (talk) 21:47, 28 April 2024 (UTC)[reply]

[1] Laurent, MR; Vickers, TJ (2009). "Seeking health information online: does Wikipedia matter?". J Am Med Inform Assoc. 16 (4): 471–9. doi:10.1197/jamia.M3059. PMC 2705249. PMID 19390105.

[2] Schwitzer G (2008). "How do US journalists cover treatments, tests, products, and procedures? an evaluation of 500 stories". PLOS Med. 5 (5): e95. doi:10.1371/journal.pmed.0050095. PMC 2689661. PMID 18507496.

[3] Goldacre, Ben (2008-06-21). "Why reading should not be believing". Guardian.

[4] Dentzer S (2009). "Communicating medical news—pitfalls of health care journalism". N Engl J Med. 360 (1): 1–3. doi:10.1056/NEJMp0805753. PMID 19118299.

[5] Li J, et al The quality of reports of randomized clinical trials on traditional Chinese medicine treatments: a systematic review of articles indexed in the China National Knowledge Infrastructure database from 2005 to 2012. BMC Complement Altern Med. 2014 Sep 26;14:362. PMID 25256890

[Vickers-6] Further information:
"Some countries publish unusually high proportions of positive results. Publication bias is a possible explanation. Researchers undertaking systematic reviews should consider carefully how to manage data from these countries." Vickers, Andrew (April 1, 1998), "Do certain countries produce only positive results? A systematic review of controlled trials.", Controlled Clinical Trials, 19 (2), Control Clin Trials: 159–66, doi:10.1016/s0197-2456(97)00150-5, PMID 9551280

Ernst, Edzard (2012). "Acupuncture: What Does the Most Reliable Evidence Tell Us? An Update". Journal of Pain and Symptom Management. 43 (2): e11–e13. doi:10.1016/j.jpainsymman.2011.11.001. ISSN 0885-3924. PMID 22248792.

[7] "Some countries publish unusually high proportions of positive results. Publication bias is a possible explanation. Researchers undertaking systematic reviews should consider carefully how to manage data from these countries." Vickers, Andrew (April 1, 1998), "Do certain countries produce only positive results? A systematic review of controlled trials.", Controlled Clinical Trials, 19 (2), Control Clin Trials: 159–66, doi:10.1016/s0197-2456(97)00150-5, PMID 9551280

[8] Ernst, Edzard (2012). "Acupuncture: What Does the Most Reliable Evidence Tell Us? An Update". Journal of Pain and Symptom Management. 43 (2): e11–e13. doi:10.1016/j.jpainsymman.2011.11.001. ISSN 0885-3924. PMID 22248792.

[Qiu-7] Qiu, Jane (January 12, 2010), "Publish or perish in China", Nature, 463 (7278): 142–143, doi:10.1038/463142a, PMID 20075887, S2CID 205052380

[8] Some examples:
Nature Reviews Cancer: "the subject of rancorous scientific and political debate over its mission and even continued existence"

Clinical Rheumatology: "The criticism repeatedly aimed at NCCAM seems justified, as far as their RCTs of chiropractic is concerned. It seems questionable whether such research is worthwhile."

Nature News: "still draws fire from traditional scientists", "Many US researchers still say such funding is a waste of time and money."

Science News: "[NCCAM] is a political creation"; "This kind of science isn't worth any time or money" (quoting Wallace Sampson)

Science Policy Forum: "[NCCAM] was created by pressure from a few advocates in Congress"; "NCCAM funds proposals of dubious merit; its research agenda is shaped more by politics than by science; and it is structured by its charter in a manner that precludes an independent review of its performance"; "NCCAM is unable to implement a research agenda that addresses legitimate scientific opportunities or health-care needs"

[11] Nature Reviews Cancer: "the subject of rancorous scientific and political debate over its mission and even continued existence"

[12] Clinical Rheumatology: "The criticism repeatedly aimed at NCCAM seems justified, as far as their RCTs of chiropractic is concerned. It seems questionable whether such research is worthwhile."

[13] Nature News: "still draws fire from traditional scientists", "Many US researchers still say such funding is a waste of time and money."

[14] Science News: "[NCCAM] is a political creation"; "This kind of science isn't worth any time or money" (quoting Wallace Sampson)

[15] Science Policy Forum: "[NCCAM] was created by pressure from a few advocates in Congress"; "NCCAM funds proposals of dubious merit; its research agenda is shaped more by politics than by science; and it is structured by its charter in a manner that precludes an independent review of its performance"; "NCCAM is unable to implement a research agenda that addresses legitimate scientific opportunities or health-care needs"

[9] Ockerbloom, John Mark. "Why Pay for What’s Free? Finding Open Access and Public Domain Articles." Everybody's Libraries (23 Oct 2018).

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

Wikipedia talk:Identifying reliable sources (medicine)

Simply bad

MEDRS-related discussion at WP:RSN

Meta-analyses need careful scrutiny, peer-reviewed or not.

Navigation menu

Search