Wikipedia talk:Identifying reliable sources (medicine)/Archive 31

Page contents not supported in other languages.
From Wikipedia, the free encyclopedia
Archive 25 ← Archive 29 Archive 30 Archive 31

Medical and scientific organizations

A recent edit by CrescentRidge added a note about examining whether guidelines adhere to The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach. It got reverted, but I think we should review this section.

Eubulides first created the section and his text is essentially what is our second paragraph. What are now the first and third paragraphs were added CFCF and tweaked by two others. One problem is that Eubulides was talking about "Statements and information" generally not really about official "guidelines" aimed at medical professionals. The "guides" he mentions are aimed at the public: think NHS advice to the public about coronavirus, thus the qualification that they "have the advantage of being freely readable but are generally less authoritative than the underlying medical literature" is fair enough.

But the added material, which now surrounds this paragraph, is about "Guidelines and position statements provided by major medical and scientific organizations" and "Guidelines by major medical and scientific organizations" (repetition!) The guidelines published by "Medical and scientific organizations" are first mentioned in the "Summarize scientific consensus" section which notes we should present the "prevailing medical or scientific consensus, which can be found in recent, authoritative review articles, in statements and practice guidelines issued by major professional medical or scientific societies (for example, the European Society of Cardiology or the Infectious Disease Society of America) and widely respected governmental and quasi-governmental health authorities (for example, AHRQ, USPSTF, NICE, and WHO), in textbooks, or in scholarly monographs."

So a reader might confuse these "public guides" with "guidelines for professionals". I'm not in fact seeing much text of any merit in the first and third paragraphs.

I think the "Referencing a guideline" diagram is dubious. There are things on that line that are not clearly distinct or support being ranked. Why is health technology assessment (HTA) at one end. Surely any one of these guidelines could follow that standard. Where does GRADE fit in with that? What is an "Authoratative Editorial Board"? What makes something an "International guideline" other than the nationality of its authors? The diagram dismisses "Patient advocacy groups" as being below the "MEDRS threshold" but that's perhaps more prejudice than fact.

For example, the various guidelines concerning Tuberous sclerosis are published after an international consensus conference, which gets together professionals in the field from around the world. It is supported by TSC International (the Worldwide Organization of Tuberous Sclerosis Complex Associations) which are all fundamentally patient support/advocacy groups. These are the groups who raised the money that pay many of these scientists grants, and which finds the genes and therapies. With Ketogenic diet the best recent recommendations are from the "International Ketogenic Diet Study Group" PMID 29881797 and that group is really just an international collection of top doctors and researchers in the field, encouraged to get together by the Charlie Foundation and Matthews Friends, which are US and UK patient advocacy groups respectively.

I suggest we remove the first and third paragraphs, pending a rewrite, and ditch the diagram. Then reconsider how we discuss "statements and practice guidelines issued by major professional medical or scientific societies". We already encourage their use per the importance of representing consensus. What extra, if anything much, do we need to say? -- Colin°Talk 17:35, 3 August 2021 (UTC)

No comment on most of this, but there is a difference between experts who got together and wrote a MEDRS because of patient advocacy groups, and the advocacy groups themselves. Some medical conditions are highly politicized, and advocacy groups (claiming to represent whoever) can push for and claim things outside the scientific consensus. Examples of such conditions are gender dysphoria and Lyme disease (regarding the latter, see chronic Lyme disease). I see no reason to consider advocacy groups of any sort a MEDRS. Crossroads -talk- 18:36, 3 August 2021 (UTC)
I agree, but I think there is a danger with the current wording on the diagram. In the published recommendations I mention, the patient advocacy groups are included in the list of the authors of the paper, which some might interpret as tainting the source. In fact organisations like NICE and SIGN consult with patient advocacy groups as part of their decision making process. Their recommendations come from a panel that doens't just include doctors and scientists. Perhaps the clincical guidelines and recommendations that are worth using are generally published in respected journals, whereas groups pushing a non-mainstream agenda might publish more directly. It would be good if people have more examples of good and bad published guidelines so we could see if there is a pattern. Currenlty, I think my two examples of first-class non-controversial guidelines demonstrate the current text is unhelpful. -- Colin°Talk 09:39, 4 August 2021 (UTC)
I don't object to the proposed change. WhatamIdoing (talk) 00:17, 6 August 2021 (UTC)

Requesting 3rd opinion at Talk:Ginkgo biloba

Hello! I would like to invite additional opinions at Talk:Ginkgo biloba#Removing information about traditional uses of Ginkgo biloba. The discussion concerns how to present information about ginkgo's long use in traditional Chinese medicine without violating WP:MEDRS or misleading readers. Thanks. Nosferattus (talk) 16:37, 12 August 2021 (UTC)

Nosferattus, you are better to post at WT:MED for such requests. This page is more about discussing changes to the guideline. -- Colin°Talk 07:23, 13 August 2021 (UTC)

About serious methodological flaws

I tried to add following text: ″Original researches with serious methodological flaws shouldn't be used as sources for their results at all. Such methodological flaws include lack of control group (in case of efficiency if such a trial can be performed with control group without any problems), violations of randomization or blinding procedures and data falsification″ ([1]). My edit was cancelled with reasonable thoughts that editors can reject sources on their own thoughts about serious methodological flaws.

1. I think we could specify recommendations at least about uncontrolled studies in situations where trials with control group can be performed without any problems. Uncontrolled trials can be perfect for 100% case fatality rate disease, but in common diseases such studies can be very biased and cannot be used to determine efficiency.
2. We can add disclaimer about serious methodological flaws with remark that such flows must be specified in other reliable sources.
-- D6194c-1cc (talk) 05:17, 18 August 2021 (UTC)

I won't be able to give you a full response for a while, but thank you for bringing this to the talk page. Firefangledfeathers (talk) 05:26, 18 August 2021 (UTC)
For #1, I think the guideline already addresses the lower value of trials without control groups, e.g. the two hierarchy diagrams in WP:MEDASSESS and the line "Case reports and series are especially avoided, as they are uncontrolled." Perhaps we could strengthen the language in that section?
For #2, I think using reliable sources that cast doubt on the reliability of particular studies is already part of policy and practice, and probably doesn't require a line in the guideline. Firefangledfeathers (talk) 12:39, 18 August 2021 (UTC)
Thanks for your reply! For #2 what about restricted randomization with small number of participants (let's say 80) if publication doesn't have clear definitions about why some people were not randomized? It's still RCT. And what about retractions and retraction reasons? For #1, I' d like to have some information about uncontrolled trials in secondary sources section with reference to appropriate section (MEDASSESS), secondary sources section is some kind of introduction. -- D6194c-1cc (talk) 05:35, 19 August 2021 (UTC)
Re: restricted randomization, that's sort of an example of what I mean. Primary sources "should generally not be used" and part of the reason is because we shouldn't trust editors to do the work of evaluating the quality of primary sources. There may be very good reasons for restricted randomization. Secondary sources are the place to go to see whether the reasons were good enough for inclusion in a meta-analysis. I often see that retractions lead to article edits, as expected; I don't think we need to spell out that retracted articles are less reliable. I am not sure what you mean by your final sentence. Firefangledfeathers (talk) 05:34, 20 August 2021 (UTC)
Well I wanted to add text like this: "For example, uncontrolled trials of medical interventions are not suitable sources for their results (where there are no significant barriers for control group) because they do not allow assessing the effectiveness of the intervention", but Colin said about shortening rules rather than extending. May be a right way would be to create quality articles about uncontrolled trials, scoping reviews, etc around Wikipedia medical guidances (that's what I usually in in ruwiki). --D6194c-1cc (talk) 18:21, 21 August 2021 (UTC)
It seems to encourage original research which goes against policy. Editors could come to different conclusions than those generally accepted in secondary sources. We could for example find flaws in COVID-19 vaccine studies and come to different conclusions about their safety and effectiveness than those generally accepted in the literature. TFD (talk) 12:35, 20 August 2021 (UTC)
I was saying about serious methodological flaws that could affect conclusions such a way that they could not be used to determine efficiency. Vaccine trials are randomized and well designed, minor errors cannot affect conclusions. ( I've already been vaccinated with Sputnik V. :) ) Let's take this review as example: [2] (Zinc for preventing and treating the common cold). It was withdrawn, but it contains some background information and it can be cited in many other publications since its a Cochrane systematic review and it was withdrawn two years after publication. Can background information be used from this article? Can we сarefully cite conclusions with remarks that article was withdrawn? If we can do we need remarks to contain information about reasons of retraction ([3], plagiarism)? Can we use review sources that cite withdrawn article? By the way systematic review about covid-19 from beginning of pandemics are of low quality ([4], [5]). Does it means something for us, editors? -- D6194c-1cc (talk) 18:55, 20 August 2021 (UTC)
But in fact I was saying about primary researches, but not about review. Primary researches should not be used, but it's a recommendation rather than rule. I translated this rules to my native language after Wikipedia in my country was bombarded with primary sources about COVID-19 some time ago. So I want to strengthen as much as possible about primary sources to reduce possible conflicts. And I want to make it synchronous with English section of Wikipedia. (Thanks to authors of this guidance, it's of very good quality!) --D6194c-1cc (talk) 18:55, 20 August 2021 (UTC)
MEDRS needs to be shorter, not longer, and there is no end to the amount one could write to offer advice. We shouldn't generally be citing an original research paper about its findings, so judging whether that research was gold standard or poor is something we try to leave to the authors of the secondary literature we do cite. Awareness of why some research is good and some bad is useful, which is why MEDRS explains it to some degree (and links to further reading on the matter). But we aren't here to review the primary literature. Wikipedia works by selecting reliable sources of information, which is something anyone can do. Reviewing the scientific evidence for whether a treatment is efficacious, safe and cost-effective is not our job, and best left to those with specialist degrees and professional experience. -- Colin°Talk 16:13, 21 August 2021 (UTC)
Colin covered it all. SandyGeorgia (Talk) 17:01, 21 August 2021 (UTC)
I would note that the "original research" used to decide if something is a good or a bad source is not really the same as writing something which no reliable source supports (WP:OR). In fact, I'd argue that there is an awful lot of original research that goes into deciding what sources to use deciding what is WP:DUE and determining the WP:BESTSOURCE. Now, it may be that we don't want people to engage in this sort of research when deciding not to use sources, but I don't think because it violate WP:OR.
@D6194c-1cc, I wonder what situation prompted you to think this clarification would be helpful. WhatamIdoing (talk) 18:35, 22 August 2021 (UTC)
My reality is a lot of primary sources in open access journals in my native language and many of small traditional literature reviews which can be characterized as advertising (like mention some meditation absolutely effective without any links on any systematic reviews). Nevertheless I can't say anything about closed journals since I haven't access to them. I tried to make explanation of why some primary sources must not be mentioned without reviews (most uncontrolled studies, small randomized trials, randomized trials from doubtful journals with some exceptions) and why another (large-scale studies like RECOVERY) can be mentioned with remarks about trial (that is just a single trial but large and important). I gave emphasis to uncontrolled trials because they are most easily conducted researches. My native language is not English so my formulations can miss something (and that's why I don't write new articles in English at a time), for example my language doesn't differentiate medications and drugs. Why you're asking? Do you have any suggestions on possible formulations? -- D6194c-1cc (talk) 20:32, 22 August 2021 (UTC)
I can't think of any circumstances where it would be acceptable to use primary sources that have attracted no attention in secondary sources that established their weight and credibility. What concerns me is that the recommendation opens the door to editors determining the credibility of primary studies themselves. Editors could argue for example about the methodology of studies of vaccine effectiveness or ivermectin. But it is better to just report the opinions of experts in review studies. TFD (talk) 01:46, 23 August 2021 (UTC)
I think the overall goal in this proposal is to reduce the use of primary sources. WhatamIdoing (talk) 02:29, 23 August 2021 (UTC)
One case that seems interesting for me is extremely large epidemiological studies with little interpretation. E.g. "the prevalence of heart attack is 1%". You often won't get many such studies so they are not necessarily reviewed, and can almost constitute government data if say, the UK's NHS released their data for statistics. Talpedia (talk) 22:30, 4 September 2021 (UTC)
@D6194c-1cc, you should qualify for Wikipedia:The Wikipedia Library, which will give you free access to some closed/paywalled sources (mostly, but not only, in English). You can also ask at Wikipedia talk:WikiProject Medicine or Wikipedia:WikiProject Resource Exchange/Resource Request; often, another editor will have access and can give you a quick summary.
Is the Russian Wikipedia's equivalent of WP:FTN at w:ru:Википедия:К посредничеству/Неакадемичность, атеизм и религия/ЗКА active and useful? Sometimes what's needed isn't a big rule, but a couple of friends to help out, especially on subjects such as meditation.
At the English Wikipedia, we often find it simpler to just say "no primary sources" when we encounter a dubious primary source. The official rule is more complex (e.g., the RECOVERY Trial could be mentioned in some contexts; very old primary sources might be mentioned in a ==History== section), but we often claim that the rule is "no primary sources" and use selective enforcement to accept the good uses.
Also, for many disputed subjects, the question isn't really whether the source is "reliable"; the question is whether that source represents mainstream scientific opinions. Sometimes, it seems that if you have 10 sources about a subject, and nine say it doesn't work, but only one says that it works, it is the one outlier that gets added to Wikipedia. WhatamIdoing (talk) 02:28, 23 August 2021 (UTC)
Thanks for the tips! Especially for Wikipedia Library, I'll try it out. --D6194c-1cc (talk) 16:00, 23 August 2021 (UTC)
And about friends. In future I want to translate some articles I wrote to English. Where can I find help with grammar (I probably would need some explanations about my mistakes before publication)? --D6194c-1cc (talk) 20:25, 23 August 2021 (UTC)
@D6194c-1cc, I recommend using the Wikipedia:Content translation tool and posting the page in your user space (e.g., User:D6194c-1cc/Article name). Then you could ask at Wikipedia talk:WikiProject Medicine for help. WhatamIdoing (talk) 04:19, 24 August 2021 (UTC)
@D6194c-1cc: 2. We can add disclaimer [...] that such flows must be specified in other reliable sources. I disagree with this one. In most cases it's obvious that the study in question lacks certain methods or approaches. Additional references proving that are always welcome though. No need for specific provisions for that. AXONOV (talk) ⚑ 08:17, 4 September 2021 (UTC)

Epidemiology

Above, Talpedia writes One case that seems interesting for me is extremely large epidemiological studies with little interpretation. E.g. "the prevalence of heart attack is 1%". You often won't get many such studies so they are not necessarily reviewed, and can almost constitute government data if say, the UK's NHS released their data for statistics.. This reminded me of when I looked at the Epidemiology at Chronic obstructive pulmonary disease and wrote up some comments on the talk page. (still haven't found the time to revise it). Firstly that experience supports the "you often won't get many such studies". In that case there was the Global Burden of Disease study, the WHO data, and there was also some UK data. These are periodically/haphazardly updated. But I question the "studies with little interpretation". Perhaps heart attack is more clear-cut, but with COPD there seemed to be huge variation in how the disease is diagnosed (both methods used and criteria used) and variability from country-to-country about how reliably it was diagnosed. If a disease is significantly under-reported, the meaning of a % figure is questionable.

Then there's the matter that these huge studies don't actually perform a census and get everyone in the country to tick a box saying what dreadful diseases they have. And they don't examine every death certificate for cause of death. They sample, extrapolate and manipulate the figures, and out pops a number like 3.1 million deaths. That looks like a simple, easy to understand number. If I tell you that next decade there were 2.9 million deaths you might conclude the deaths were falling, and an editor might engage in original research and claim as much. But if the confidence interval on those estimates was 2.9-3.3 and 2.7-3.2 then we might more reasonably conclude we don't know, as the change is within expected sampling noise. (I'm no statistician, so forgive me if I'm making mistakes or explaining this wrongly). Even just telling the reader that the numbers were 3.1 million and then 2.9 million next decade is suggesting to them that there is a change, and that we do know the numbers to two significant figures. Perhaps the experts really believe the number is closer to 4 million if it weren't so under-diagnosed. Better perhaps to just write 3 million.

So what value do the reviews or other secondary sources provide? It is a mixed picture. One review I looked at made a mistake (IMO) with the numbers, and it would have been bad if we'd used those numbers. But likely both the Global Burden and the WHO numbers are fairly well respected and one could pick either. A reviewer may pick the larger number if they want to make it look like the disease was more important. Or they might pick numbers that demonstrate a trend they want to discuss. The GOLD report hinted that they thought the WHO data was poor, but didn't go into much detail as to why. Certainly I struggled to find the confidence intervals for the WHO figures. WHO's own fact sheets were variable in quality. An old one that we had used claimed a similarity between sexes that wasn't supported by the data IMO, and newer fact sheets don't make that claim.

I wished the GOLD reports had more detailed data, with which to say more in our article. But perhaps they know the data is weak and best not examined or discussed in detail? I feel that statistics are hard enough that both reviewers and Wikipedians can get them wrong, and that it is easy to mislead the reader with numbers that are too precise, or to claim the numbers are an actual count of people with a disease rather than crude estimates.

Anyone got any other experiences here, or opinions? -- Colin°Talk 09:28, 5 September 2021 (UTC)

Yep I definitely agree that in some cases interpretation can be an issue even with large epidemiological studies. One hopes that there are other (secondary) sources that address these definitional issues. I don't know if we think normal editorial discussion can deal with these questions, or whether we need to deal with it with policy. I might point at covid mortality rate, long covid, recovery rates of psychosis, and causes of driving accidents as examples where the definitions matter and are contentioius.
I'm suspicious in many case your choice will be a primary large epidemological study or nothing, perhaps with some editorials or reviews cursorarily mentioning and citing this study.
I wonder how often WP:BESTSOURCE solves problems that might otherwise be solved by WP:MEDRS. I would take BESTSOURCE to mean *if* there is a secondary source that contextualizes a primary source, then that secondary source should be included instead or as well as the primary source. The effect the difference between MEDRS and BESTSOURCE seem to be to prevent the inclusion of the information from primary sources until such time as they have been reviewed in a secondary source, and I guess MEDRS forces the people adding a piece of information to actually find the best source. Talpedia (talk) 11:10, 5 September 2021 (UTC)
Citing a secondary source that mentions it in passing has generally been the preferred approach. WhatamIdoing (talk) 16:32, 5 September 2021 (UTC)
I think we get too hung up on "reviews" and other articles in journals, perhaps because they are easily searched and some editors have easy access to them without getting off their backsides. There are other secondary sources too, such as websites and books. I don't read WP:BESTSOURCE the way you do. To me it is simply a link to Wikipedia:Reliable sources and a hint that Wikipedians should let published experts resolve the disputes rather than arguing over the primary sources/data on a talk page. WP:SCHOLARSHIP says Articles should rely on secondary sources whenever possible. For example, a paper reviewing existing research, a review article, monograph, or textbook is often better than a primary research paper. When relying on primary sources, extreme caution is advised. Wikipedians should never interpret the content of primary sources for themselves. The danger with easy access to so much primary data, whether disease statistics or drug prices, is that history tells us Wikipedians will interpret it, invent their own calculations, claim illusory precision and confidence, juxtapose incompatible figures, and neglect to mention important caveats. The "extreme caution" warning gets neutralised by illusory superiority, where editors think they are careful and clever enough. It is frustrating when the secondary literature doesn't mention interesting stuff, or facts we personally consider important, but I think that's just one of the things we have to accept on Wikipedia that makes it different to other publishing platforms. -- Colin°Talk 17:07, 5 September 2021 (UTC)
I agree with most of that. Although on a kind of tangential point, as an unreformed "opinion above my pay grade haver", I'm not sure that people are thinking much about superiority at all when they engage in interpretation, more thinking and filling in the gaps. The failure seems to be more, not engaging with the writing on the topic; and in the case of wikipedia "thinking by writing" as a form of WP:OR. I get that on wikipedia what you do is more "thinking by searching for sources and reading". Talpedia (talk) 09:06, 6 September 2021 (UTC)
I mentioned "illusory superiority" just because that's the article that seems to cover the issue. The "superior" word isn't meant to suggest the person has a superiority complex, or consciously thinks themselves better than everyone else. More that this is actually the default human behaviour. The classic example is when you ask people about their driving ability and over 90% of Americans think they are above average in ability. My guess is that if you ask editors "Are you careful when writing about numerical facts and statistics?", nearly every Wikipedian would reply that they were. Nobody thinks they are the careless clueless editors. So the caution becomes worthless. -- Colin°Talk 10:01, 6 September 2021 (UTC)
Maybe... but if you look at the example of driving. I may well think I'm better than 60% of drivers, but that doesn't mean I'm not scared when I drive on a freeway for the first time, or that I don't find clutch-control difficult and ask for tips. My attribution of my own relative competence doesn't *necessarily* affect how I carry out a task. Talpedia (talk) 11:06, 6 September 2021 (UTC)
Even if we stipulate that you are not normal, the normal Wikipedia editor likely is normal (in this respect). WhatamIdoing (talk) 21:18, 6 September 2021 (UTC)
Well... this isn't really about me. It's more about the mechanics of self-regard in people's behaviour and whether it's correct to attribute incaution to an abstract feeling of superiority, when it might just be the case that people are incautious and have an abstract feeling of superiority and whether this has implications. My personal take is that people are incautious, and then have injured egos if upbrided for being incautious... which is ever so slightly different, but often functionally equivalent particularly for questions of policy. But I'm getting miles off topic! Talpedia (talk) 11:36, 7 September 2021 (UTC)

Toxicology Reports

Elsevier journal Toxicology Reports appears to publish pseudoscience (via Sj). We have a dozen articles citing it. Nemo 06:49, 25 September 2021 (UTC)

Nemo, I'm not sure this post is relevant here. This talk page is for changes to the guideline, not a noticeboard of potential bad source usage in articles. WT:MED might be more relevant, though most of the linked usages seem to refer to non-human toxicity data, and all of them are dubious from the "citing primary research paper" point of view. The fact that the journal may once have published a dodgy covid paper isn't the most critical problem with those sources imo. -- Colin°Talk 11:05, 25 September 2021 (UTC)
This seems to be a connected to a special issue. See [6]. Headbomb {t · c · p · b} 12:10, 25 September 2021 (UTC)

More information: https://blogs.harvard.edu/sj/2021/09/30/journal-level-fraud-elsevier-fakes-peer-review-of-covid-click-bait/ Nemo 14:50, 1 October 2021 (UTC)

Does a report on the health of a person need a medical opinion?

At Talk:Julian_Assange#Mini-stroke the inclusion of a report by the fiancee of a person that he suffered a mini-stoke and had an MRI and is on medication is being opposed on the basis that it is a medical conditio,, to quote ""Any medical information should come from a qualified professional who has examined Assange." Is this true? NadVolum (talk) 12:19, 16 December 2021 (UTC)

WP:MEDRS doesn't require that. This is not a situation that MEDRS says anything about. MEDRS is not about the reporting of individual's health statuses. I think you need to go back to WP:RS on this one. Bondegezou (talk) 12:27, 16 December 2021 (UTC)
I would agree with Bondegezou that this isn’t a MEDRS issue, but I would also hesitate to state it as fact. Perhaps it could be included with in-text attribution (as in: “according to X, Assange suffered a stroke…”) but even then, it is more in the realm of unsubstantiated rumor than confirmed fact. Blueboar (talk) 12:36, 16 December 2021 (UTC)
It's not true. Cambial — foliar❧ 12:53, 16 December 2021 (UTC)
Probably best to comment over at the article talk page. The main purpose of this talk page is improving the guideline, not articles. -- Colin°Talk 12:55, 16 December 2021 (UTC)

Secondary/Review Scientific Sources

This text of mine is copied from a discussion on Talk:Cannabis_(drug)#"Inappropriate"_source. I'm tossing it out just because I'd be interested in what opinions or approaches people here might have on this general topic.

>>>>>>>>>>>>>>

Here, and in WP:BMI, I totally get now that the consensus is that text in Wikipedia stating "facts" about cannabis needs to have a WP:MEDRS quality reference. Although I don't totally agree, I have no trouble adhering to that.

However (and this is explicitly a *personal opinion*), a whole bunch of scientific secondary references (including ones meeting WP:MEDRS) are in fact shit. E.g., they may cite a lame and invalid primary study without any actual evaluation (thus propagating false material). They may take speculative conclusions ("possible relationship" or "potential link") in a primary source and state it as a fact. Now, as a semi-professional I can find stuff in WP:MEDRS secondary sources that are blatently inaccurate, but Wikipedia requires them to be taken as absolute encyclopedia truth (note that I am *not* arguing against WP:NOR; I totally approve of it).

Also: although "The Pot Book" is a lay-oriented book and doesn't technically meet WP:MEDRS, it was edited by an MD/psychiatrist, Julie Holland. Some of the articles are solid articles by academic writers ( e.g., the Kirkpatrick and Hart chapter I originally cited), and some are not "science" (e.g., Julie Holland's interview with Tommy Chong). However, the Kirkpatrick and Hart article (*read the entire article* before you criticize it) does meet WP:Verifiability by my standards, and, IMO, ought to qualify for WP:MEDRS (but doesn't). I believe there was a comment above that stated the the Kirkpatrick article was problematic because some sentences didn't cite references. That is also frequently true of WP:MEDRS qualifying secondary sources.

The above is not an attempt to argue against WP:MEDRS, but just a description of some of its weaknesses. And I'd be interested in people's opinions about whether there are any ways to address these problems.— Preceding unsigned comment added by Finney1234 (talk • contribs) 04:34, 4 December 2021 (UTC)

Could you give a few actual examples of such "shit" sources that you are arguing MEDRS encourages? There is nothing that "Wikipedia requires ... to be taken as absolute encyclopedia truth" stipulated by MEDRS (or indeed any WP:PAG). Use of sources requires a brain. The Pot Book is not a good examples of something valuable MEDRS wrongly excludes, for sure. Alexbrn (talk) 04:38, 4 December 2021 (UTC)
I believe evaluation of WP:MEDRS-qualifying sources as "good" or "bad" would qualify as WP:NOR. E.g., discussion of the issue could quickly revert into an edit war, with repetitive reversions. Is there some competent WikiLawyer here who can cite chapter and verse one way or the other (or, possibly, both, as sometimes occurs when consulting different Wikipedia guidelines)? Finney1234 (talk) 05:40, 4 December 2021 (UTC)
Evaluation of source quality is one of the cornerstones of editing Wikipedia. Or are you proposing to shut down WP:RSN? Alexbrn (talk) 05:43, 4 December 2021 (UTC)

Markworthen comment

I mostly agree with AXONOV's comment at 22:06, 3 December 2021 (UTC) on the other discussion page):

I've checked the book (The Pot Book). Generally speaking the source in question (added on [21:05, December 1, 2021]) in fact cites some primary studies (like [1]). However, the attributed text (worded a bit differently), which is mentioned on the page 28 under the SUBJECTIVE EFFECTS IN THE LABORATORY subtitle, does not contain any specific attributions... The same goes for the phrase rapid onset in a context of use of Sativex on the page 219 (NEUROPATHIC PAIN subtitle) - no attribution given. I conclude that claim is mostly speculative. On the other side I don't see how WP:BMI/WP:MEDRES applies here... It's certainly not a curative effect. But I would propose to bring a better source on that anyway...

My only quibble is that I see the psychoactive effects of cannabis falling under WP:BMI and WP:MEDRS.

"The Pot Book" was written for a lay audience, therefore it doesn't qualify as a reputable textbook, treatise, or scholarly book. However, that does not mean it is "junk". In fact, it contains book chapters written by biomedical researchers and scholars from a range of disciplines. Like most edited books, quality varies, but it's much more grounded in science than the many "wonders of weed" books on the market. I would perhaps use "The Pot Book" as a jumping off point in my search for reliable sources.

I wholeheartedly agree that in the United States (and elsewhere) we have cut off our nose to spite our face with regard to cannabis. While we reduced cannabis consumption in the 1980s, we continued to essentially ban research on marijuana and we lied about marijuana so much and for so long that Americans born in the 1990s onward have a hard time trusting anything our government says about cannabis, even if it comes from leading researchers who don't have a political agenda.

Finney1234 - You are a really good writer and have obviously worked long and hard to add a lot of valuable content to Wikipedia. And you seem to have an open mind and you seek to understand along with seeking to be understood. There are a couple of relatively minor aspects of MEDRS that I don't agree with, and there are a handful of editors who use the policy to bludgeon others into submission or to justify their authoritarian behavior (I am not referring to Alexbrn), but they are a very small minority. Over all, WP:MEDRS is a thoughtful, conscientious, and well-reasoned set of policies and guidelines. All the best, Mark D Worthen PsyD (talk) [he/his/him] 07:13, 4 December 2021 (UTC)

You're luring me off topic, but one the the worst things about cannabis is the US-centric politicisation of it - here in Europe things are generally a bit more sensible. The protracted former "war" on cannabis in the US has fostered the idea that it's the wonder drug THEY are not telling us about, and this has garnered the substance disproportionate attention - indeed in the Pot Book we get the usual hints of its time that cannabis is useful for treating AIDS, cancer, etc. Interestingly - or depressingly - the cannabis conspiracist problems prefigure more recent problems with COVID drugs and vaccination, and how when they get politicized it causes a lot of work for Wikipedians. In the context of considering it as a source for use on WP:BMI I would call The Pot Book "junk" - but I'm harsh . Alexbrn (talk) 07:30, 4 December 2021 (UTC)
Fair enough. And thank you for reminding us that many countries and regional bodies have adopted a more balanced and well-informed stance toward "get high" drugs. I also agree that the pendulum has swung way too far to the "it has no adverse effects and cures almost anything!" space. Mark D Worthen PsyD (talk) [he/his/him] 15:49, 4 December 2021 (UTC)
@Markworthen: […] psychoactive effects of cannabis falling under WP:BMI and WP:MEDRS
Is there a provision on that? One doesn't invoke WP:MEDRS/WP:BMI arbitrarily. AXONOV (talk) ⚑ 08:45, 4 December 2021 (UTC)
AXONOV See the discussion at Wikipedia_talk:Biomedical_information#Is_this_biomedical_information?_Clarification_requested. Current consensus there is that describing effects of cannabis requires WP:MEDRS sources. I'm OK working with that consensus.Finney1234 (talk) 09:06, 4 December 2021 (UTC)

Mark D Worthen PsyD Thanks! However, my personal (and semi-professional) opinion is that some Wikipedia articles (like the current version of Cannabis_(drug) ) have a huge non-factually-based negative bias. I'm just trying to work on making it more factually based (and WP:neutral) to match Wikipedia guidelines. Some editors apparently think this approach is WP:WL.Finney1234 (talk) 23:11, 4 December 2021 (UTC)

@Finney1234: If I may be so bold as to offer an unsolicited suggestion: Invest 80% of your talent and time into finding unassailably top-notch, MEDRS-compliant sources that provide the facts missing from an article, leaving the other 20% for prolonged brief battles with "opposing" Wikipedians. Mark D Worthen PsyD (talk) [he/his/him] 05:23, 5 December 2021 (UTC)
Wise words indeed: in my experience nearly all content disputes on Wikipedia can be resolved by raising sourcing quality. Alexbrn (talk) 05:27, 5 December 2021 (UTC)
Thank you. // It took me about 10 years to reach that conclusion for myself. But I'm much more relaxed and happy since then! Mark D Worthen PsyD (talk) [he/his/him] 05:43, 5 December 2021 (UTC)
I totally agree with that....how collegial! However, one of the main questions that I intended to raise in this discussion is how do you deal with text that other editors have put in that is based on poor sources that nonetheless can be argued to meet Wikipedia guidelines (e.g., Habboushe/Chocron)? Just trashing someone else's edit saying "I don't agree with this source" (unless there is an explicit applicable guideline, like WP:MEDRS) is a personal-opinion approach that is not appropriate, and can lead to hostile discussions, edit wars, etc. So there is really no way to deal with this (at least if there are no follow-up quality published sources that have addressed it...and even then, you now have two contradictory sources). Finney1234 (talk) 02:00, 6 December 2021 (UTC)
Also: I'd much prefer to have the battle ratio lowered to somewhere between 0 and 5% :-) Finney1234 (talk) 02:42, 6 December 2021 (UTC)
Any source can "be argued" to be in line with Wikipedia's guidelines, no matter how poor it is. We see it all the time. The Habboushe/Chocron sources you mention though will be rejected by all WP:CLUEful editors and continued argument for them would become tendentious. At that point it becomes a behaviour issue rather than a source issue, and continually disruptive editors will eventually have their editing privileges removed. That is how the issue is resolved. It's much better if sourcing quality is raised rather than lowered - which is what this thread has been saying. Alexbrn (talk) 05:40, 6 December 2021 (UTC)

Proposal: close this discussion

I propose to close this discussion and tag it as {{moved}} per duplicate (the same policy is being discussed): WP:BMI#Is this biomedical information. Objections? --AXONOV (talk) ⚑ 10:25, 4 December 2021 (UTC)

I strongly disagree. And please sign your talk posts. Finney1234 (talk) 09:48, 4 December 2021 (UTC)
Note that this is a general topic dealing with the WP:MEDRS criterion. It does not duplicate the WP:BMI discussion, which deals with a completely different topic: does describing Marijuana effects require a WP:MEDRS source? Consensus says "yes", so that's resolved for now.Finney1234 (talk) 09:54, 4 December 2021 (UTC)
@Finney1234: Spreading the same discussion across 3 talk pages isn't helpful. --AXONOV (talk) ⚑ 10:25, 4 December 2021 (UTC)
Hi AXONOV : it is unfortunate that there is some overlap between 3 talk pages due to how the discussion has developed, but the three sections are dealing with significantly different topics. Wikipedia_talk:Biomedical_information#Is_this_biomedical_information?_Clarification_requested. is primarily focussed on whether describing the effects of drugs such as cannabis requires a WP:MEDRS reference, and the current consensus seems to be "yes"; as far as I'm concerned, that discussion is closed. Talk:Cannabis_(drug)#"Inappropriate"_source does deal with the same topic, but specifically with details of the Cannabis_(drug) article. It also includes an interesting example of a revert being (IMO) inappropriately done to an article that had effectively reached WP:Consensus 2 weeks ago due to lack of objection. *This* talk page section is primarily focussed on the general issue of inaccurate WP:MEDRS resources, and is more of an general-issue discussion; I hope/assume more people will chip in to this general topic. There is some stuff I should have perhaps edited out (e.g., the discussion of "The Pot Book"), but the particular issue of how to deal with poor-quality WP:MEDRS approved sources (and how to decide if a WP:MEDRS source is "good" or "bad" without violating WP:NOR) is an important issue that is *not* being pursued in either of the other Talk pages.Finney1234 (talk) 12:03, 4 December 2021 (UTC)
It's a not-uncommon misconception that WP:NOR prevents editors using their brains to assess sources. It doesn't; in fact such brain-work is often necessary (and for medical topics, encouraged). WP:NOR prohibits the publication of original thought in article space. That's it. Alexbrn (talk) 12:11, 4 December 2021 (UTC)
What do you expect? The WP:MEDRS is highly controversial, contentious, and seriously flawed guideline that is often abused, misapplied, or misinterpret (see archives). Don't waste time, make proposals and WP:RFC instead. The talk pages are governed by WP:TPG, not by WP:NOR btw. All the best. AXONOV (talk) ⚑ 12:44, 4 December 2021 (UTC)
In my experience, MEDRS is one of the community's most respected and supported guidelines. It agree with you that it is sometimes abused, misapplied, and misunderstood, and to your list I will add that it is sometimes even invoked by people who apparently haven't read it (because Wikipedia:Nobody reads the directions; we just guess what the pages say based on hearsay and their shortcuts). But the mistakes don't invalidate the concept. WhatamIdoing (talk) 04:07, 5 December 2021 (UTC)

Example 1: Inaccurate WP:MEDRS-qualifying secondary sources

Warning: this section has turned into a long contentious discussion, but the key issue (providing a simple example of a (plausibly) WP:MEDRS review article that is factually flawed, which was all I was trying to demonstrate) is covered in the very first section.

_____________________

This first example is provided in response to User:Alexbrn's request above.This secondary source has been an ongoing issue at Cannabinoid hyperemesis syndrome (CHS). This is an example of what I expect is a valid WP:MEDRS secondary source (BMJ: British Medical Journal) that uncritically cites a lame primary study. See the CHS article and Talk page for discussion and fuller references (e.g., if I've mistyped the DOI's below. Please correct or expand the references if you can).

Background: CHS is a nasty, sometimes fatal (at least 5 reported deaths) syndrome that sometimes occurs in heavy cannabis users. The acute symptoms involve very serious nausea and vomiting that often requires emergency room (ER) care. Exceedingly hot showers or baths are reported to help relieve the symptoms.

In Habboushe et al, 2018 , Journal = "Basic and Clinical Pharmacology and Toxicology", (DOI 10:1111/bcpt.12962), 155 emergency room (ER) patients who admitted to cannabis use (but were *not* currently in the ER for nausea or vomiting problems) were asked to rate (on a 0-10 Likert scale) whether a hot shower or bath had been helpful when they had previously had cases of nausea or vomiting. 33% of the subjects rated the helpfulness level of hot showers at 5 or higher. Habboushe concludes that about 33% of daily marijuana users in the US (so: about 27,000,000 users) are likely to suffer from CHS (with the explicit caveat "...if this is extractable to the general population").

Chocron et al., 2019, Journal =BMJ (DOI 10.1135/bmj.14336) (a WP:MEDRS-qualifying secondary source, I believe) cites this result uncritically: "Extrapolating those results to the population of the United States, it is estimated thatcounty 2.1-3.3 million people might suffer from CHS annually".

Some issues with the (primary source) Habboushe study:

  • Subjects were not queried about their use of other drugs.
  • Subjects were not queried about the nature of their previous nausea/vomiting issues. There is *no evidence* that they had actually experienced CHS.
  • A >= 5 value is *less than 50%* on a 0-10 scale. This is way too low for a cutoff criterion (in my opinion).

Also, given the seriousness of CHS, 2.7 million sufferers would have overwhelmed the ER's in the US. But it obviously hasn't. So: this is a blatantly incorrect published claim.

However, because Chocron et al. is (I believe) a WP:MEDRS source (much more strictly defined than WP:Verifiability), it is virtually impossible to argue for its falseness, because that would be WP:NOR.

Again, I'm not trying to argue here that WP:MEDRS should be changed to reflect this issue, but I am interested in people's perspectives on it. I couldn't avoid dealing with the issue when editing the CHS article. Finney1234 (talk) 09:48, 4 December 2021 (UTC)

That would be pmid:31324702, "case reports" and so unreliable per WP:MEDRS. It does contain some notionally "secondary" background but such content is often suspect as it tends to serve the specific interests of the article rather than being a wider review of the evidence - so I would avoid it for anything WP:EXCEPTIONAL. There is a lot of poor sourcing in that article. Alexbrn (talk) 09:55, 4 December 2021 (UTC)
The article starts with a case study, but the main text of the article makes many general statements about CHS. See, for example, the "What you need to know" section that precedes the case study. This is a literature review in what appears to be a respected medical journal (assuming BMJ is considered to be one).Finney1234 (talk) 10:01, 4 December 2021 (UTC)
It's not a literature review. The publisher/PUBMED classification is "case reports". We generally treat the "secondary" parts of primary publications with caution as I said above. For MEDRS you typically want a (genuine) review, meta-analysis, systematic review, textbook, or the knowledge of some esteemed medical organisation. Alexbrn (talk) 10:04, 4 December 2021 (UTC)
If you actually look at the article (and the 33 references), it is clearly a review article (again, see the "What you need to know" section). If Pubmed classifies this as a "case study", that is an error. In fact, the "case study" paragraph at the beginning is later described as a fictitious vignette that did not involve an actual patient, so it's not a case study at all.
Is BMJ considered a respected peer-reviewed journal, or is it not? (I don't know. I ain't an MD.) Finney1234 (talk) 10:14, 4 December 2021 (UTC)
BMJ is a reputable journal, and they know how to classify their content. They classify pmid:31324702 as case reports and WP:MEDRS says "Case reports and series are especially avoided, as they are uncontrolled". It also appears to be a commissioned piece that was not peer-reviewed. Whatever problems there are here, I don't think they're with WP:MEDRS. Alexbrn (talk) 10:20, 4 December 2021 (UTC)
To anyone who actually reads the Chocron (2019) article, it is clearly a review article and not a case study (see notes above), so the classification is incorrect. *Read the article*. If it is a "commissioned piece that was not peer-reviewed", that might be a significant concern. How would one determine whether that was true or false? Does WP:MEDRS address this? How would an average editor determine that this was not WP:MEDRS when the article classification is incorrect? Finney1234 (talk) 12:10, 4 December 2021 (UTC)
The clasification is not "incorrect". The piece is published as part of BMJ's "Practice" / "Often missed?" series, and as such is a fairly informal piece that is categorized as "case reports". When BMJ publishes clinical reviews they make that plain too. All (recent) BMJ articles say whether they are internally or externally peer-reviewed, if they are, and the article provenance is published.[7] In this case, the piece is not peer-reviewed. As to how do editors determine this - the answer is by being careful. All this information is there in the publication. Alexbrn (talk) 12:18, 4 December 2021 (UTC)
@Finney1234, just in case it's a unclear, Peer review and Review article are different things. The ideal is a peer-reviewed review article. Neither a peer-reviewed non-review nor a non-peer reviewed review is the ideal source.
It's my understanding that the classification (visible at PubMed) is provided by the publisher (e.g., BMJ). It is possible to have a journal article be both a case study and a review article. This is generally seen only in (very) rare diseases. WhatamIdoing (talk) 04:14, 5 December 2021 (UTC)

From talk section higher up:

You are correct that the Chocron article, in an fairly obscure place near the end, states "Provenance and peer review: commisioned, based on an idea from the author". But an average editor reading a respected journal like BMJ might easily overlook or not understand this (in fact, I don't understand it). I believe something about "commissioned articles" needs to be placed in WP:MEDRS to clarify this (that sort of conclusion is in fact the motivating point for this discussion).
Also, from what you write, a classification of "case report" does not necessarily mean an article is actually a case report (the Chocron article is not). Rather,"case report" apparently can mean "informal piece" (in spite of being highly technical in both the text and the references). Perhaps something about this problematic issue should also be added to WP:MEDRS.Finney1234 (talk) 12:32, 4 December 2021 (UTC)
It is a case report. A fictional case report in this case (which, as evidence, is not much different from a "real" case report, which is equivalent to an anecdote). A whole lot of background is added to the case report, as often happens. As I said above we don't use generally use the "secondary" parts of primary source, which very often have a lot of background content. That something is commissioned does not in itself mean it is not peer-reviewed, but in this case that is so.
Anyway the outcome here is plain: WP:MEDRS is not the cause of the problem; rather diligently applying WP:MEDRS would have prevented the problem. Alexbrn (talk) 12:44, 4 December 2021 (UTC)
Nope, it is not a "case report". There is a one-paragraph description of a fictional patient to help illustrate the disease. *None* of the other text refers back to that initial description. The introductory highlighted list at the top of the article starts with "CHS..[accounted] for up to 6% of patients presenting to emergency departments...in one retrospective study", and is followed by two more similarly generic descriptions. This is a review article. Finney1234 (talk) 12:50, 4 December 2021 (UTC)
Facepalm Facepalm I give up. In your view a lot is wrong: MEDRS is wrong, the article authors are wrong, the BMJ don't know how to classify their own content, PUBMED is wrong, Alexbrn is wrong. The only thing that's right is Finney1234's view. There should be enough of a pattern there to give pause for reflection. Alexbrn (talk) 12:59, 4 December 2021 (UTC)
Nope, virtually everything I've stated is fact based, and I believe other editors will back that up. And I'm not the only person to consider Chocron to be a review article: Peter Wu, 2019, BMJ, "Cannabinoid hyperemesis syndrome: Caution with urine drugs screens for marijuana" (DOI=10.1136/bmj.l5621) write that "Chocron and colleagues provide a timely and important review on cannabinoid hyperemesis syndrome."
I have demonstrated that the Habboushe primary source is lame (something you haven't disputed), and that the Chocron review article in the respected BMJ journal quoted the invalid Habboushe results without any evaluation. I stated that in the first few paragraphs, and that was the example you wished me to provide. I have done so. The remainder of this very-long-winded discussion has been due to your objections (based, as far as I can tell, only on personal opinions, not Wikipedia guidelines). I believe anyone who actually *reads* the two page Chocron article (unfortunately behind a paywall, but see the Wikipedia article on Sci-hub if your ethics are flexible) would consider it a WP:MEDRS-qualifying source (with the single possible exception of the mysterious and unclear phrase "Provenance and peer review: commissioned, based on an idea from the author"). Finney1234 (talk) 13:10, 4 December 2021 (UTC)
It is quite possible (I haven't looked at Wu (2019), e.g, "commission" or "peer review") that Wu could have propagated the Habboushe BS into a inarguably qualifying WP:MEDRS secondary source. Alexbrn, how can we deal with this issue? Personal opinion is not a good starting point. Explicit Wikipedia guidelines (e.g., possibly useful WP:WL research) would be excellent. Finney1234 (talk) 22:48, 4 December 2021 (UTC)
That would be pmid:31548253 which is just a comment, so also not WP:MEDRS. WP:MEDSEARCH gives guidance on how to use PUBMED filters so that you don't even see these unsuitable sources that you are raising. I have added three WP:MEDRS sources to the "Further reading" section of Cannabinoid hyperemesis syndrome, which may be used to improve the article. Alexbrn (talk) 02:40, 5 December 2021 (UTC)
About the Habboushe: What you were doing with the bullet points above, about how bad Habboushe's paper was, is exactly the thing that WP:MEDASSESS says that Wikipedia editors must not do. See Respect the levels of evidence: Do not reject a high-level type of source (e.g., a meta-analysis) in favor of a source from lower levels of evidence (e.g., any primary source) because of personal objections to the inclusion criteria, references, funding sources, or conclusions in the higher-level source. Editors should not perform detailed academic peer review.
Because Wikipedia:Wikipedia is a volunteer service, you can personally choose (NB: I refer to you, a single human being that has no right to impose your choice on any other human being, including any other Wikipedia editor) to employ some Wikipedia:Editorial discretion when you encounter a source that does not satisfy you. This means that you, personally and individually, can refuse to use or cite a source that you object to in your own edits. It does not mean that you have the right to remove such a source if other editors decide that they want to use it. WhatamIdoing (talk) 04:23, 5 December 2021 (UTC)
Hi WhatamIdoing My bullet points about Habboushe's article were in a talk page discussion, so I think that's OK. I agree it would be inappropriate while editing an article.
However, I think it does illustrate an important problem (and one which currently, and may never, have a solution). I believe most scientific readers (e.g., people on this talk page) would likely agree with my personal opinion ("detailed academic peer review") that the Habboushe article is negatively-biased BS. And this "primary source" was cited uncritically in a secondary source, which I believe meets (or comes very close to meeting) WP:MEDRS standards. You are correct that I, as an editor, can choose what I think are reliable sources (User:Alexbrn cites this above as simply using your brain).
However, the question is, is there anything that can be done if another less conscientious editor chooses a (seemingly) reliable WP:MEDRS source that actually includes a bogus conclusion? If later WP:MEDRS sources have contradicted that, then there's a path you can take, but if not, the false (IMHO) information stays in the encyclopedia. I think that's a problem that could use some discussion, or even an RFC. I certainly am not claiming that an editor can take their personal opinion and use it to remove material created by other good-faith editors. My approach in Cannabinoid hyperemesis syndrome was to contribute to describing the Habboushe primary research in enough detail that readers could make their own conclusions, but I knew it was not correct to simply remove the invalid 2.75 million number. Finney1234 (talk) 01:39, 6 December 2021 (UTC)
What you were doing with the bullet points is something that MEDRS tells you not to do at all, not even on talk pages. We've had a problem with editors who pick apart details of studies, just like you did in those bullet points, when – and only when – they disagree. The actual process appears to be:
  • Find source.
  • Check source's conclusions.
  • If you agree with the conclusions: This is obviously a wonderful study. No further analysis is needed.
  • If you disagree with the conclusions: Let me check their methodology. Let's see: they didn't control for this, used the wrong metric for that, didn't use the right cutoff values – well, that's obviously a lousy study that nobody should rely on for anything.
Of course, on wiki, the only part we see is the part where the editor lists their personal opinions of the source's flaws, but we all recognize the pattern. MEDRS says you must not do this, full stop. If it's a good type of source (e.g., a review article in a reasonably reputable journal), then you must accept it on an equal footing with any other good type of source (e.g., a different review article in a reasonably reputable journal that says the opposite). WhatamIdoing (talk) 17:00, 7 December 2021 (UTC)
WhatamIdoing I totally get the difference between "peer review" and "review article". The Chocron article discussed extensively above is (IMO) a review article (User:Alexbrn disagrees, see above). It contains a paragraph describing a fictitious patient's symptoms, and then engages in general technical discussion about CHS (citing 33 source), and never refers back to the fictitious patient; it has been referred to as a "review article" in another BMJ publication. I would tend to assume that any article in a journal like BMJ was peer-reviewed, although there is a sentence at the end stating "Provenance and peer review: commisioned, based on an idea from the author" that I don't understand the details of at al. However, aside from that vague sentence, I believe the Chocron article meets the strict WP:MEDRS standards. The topic I was interested in is whether there is (or should be) some way of dealing with issues like this (that is, lame primary sources uncritically cited in a secondary source). Finney1234 (talk) 02:17, 6 December 2021 (UTC)
It is false that "another BMJ publication" refers to pmid:31324702 as a "review article". BMJ classify it as "Case reports" as can be seen on the PUBMED landing page. It is an informal commissioned piece and not peer-reviewed. I previously linked BMJ's methods for identifying peer review[8] and they are straightforward: you can read the article provenance and peer-review status for everything they publish, according to their wording formula. So why "assume" or "believe" that this is a peer-reviewed review article when the journal itself explicitly labels it otherwise? If you are going to continue to argue a point strongly on the basis of things you "don't understand the details of at al", then WP:CIR could be an issue.
More generally, if a truly exceptional claim is being made, then WP:EXCEPTIONAL says that even one strong source is insufficient. This year, a number of superficially fine WP:MEDRS secondary sources (e.g. PMID:34145166) were rejected by Wikipedia editors because of their WP:EXCEPTIONAL claims (rightly, as it turned out, because of the research fraud which later came to light). Alexbrn (talk) 05:52, 6 December 2021 (UTC)

Just as an addendum to this Finney1234, if you're interested in a WP:MEDRS-quality debunking of the Habboushe primary source you dislike so much, check out PMID:31241817, a systematic review. Once again, and as Markworthen observes, the way out of problems like this is to find better sources! Alexbrn (talk) 13:40, 7 December 2021 (UTC)

Thanks, Alexbrn, I'll check it out. But many conscientious and skilled Wikipedia editors would not have had the effort (or skill) to find this very useful citation (for instance, *no one* who has previously edited the CHS article)...but it's clearly a strength of yours. And it could easily have involved a serious editor war to try to remove the Habboushe/Chocron citations from the article (that is why I did not attempt to do so). In fact, how would you have dealt with an editor who claimed that Chrocron was a MEDRS source, given the discussion above? It would be difficult (and close to Wikilawyering) to argue against it. How would you justified removing Chocron, even with the new source? Serious question. Finney1234 (talk) 04:20, 8 December 2021 (UTC)
Wikipedia has WP:dispute resolution processes. For medical content, the first port of call should be WT:MED (not this page) where a lot of knowledgeable editors hang out. In the rare event discussion there doesn't resolve matters, a WP:RFC might eventually be necessary. In general, in any dispute it helps (1) to have the WP:BESTSOURCES, (2) to be arguing for the POV of those sources and (3) not to care personally about the subject matter. In the case of CHS the danger - and surprisingly this hasn't happened - is that the article would be attacked by editors with a strong POV either that CHS doesn't exist (pah! just reefer madness again!) or that it's a suppressed epidemic (3 million in hospital!). It's true that editing medical content is one of the more demanding activities on Wikipedia, but that's inherent in collaboratively writing good encyclopedic medical content, so probably not a soluble problem. Alexbrn (talk) 06:07, 8 December 2021 (UTC)
Thanks. I'm pretty sure some of the earlier verbose discussion on the CHS talk page *did* (contentiously) discuss the issue of the Habboushe/Chocron 2.75 million ER CHS case number. It looks like you removed it (thanks; as discussed, I didn't), but if the person who'd put it in was paying attention to the article it might have turned into a significant dispute. Timeshifter was involved in trying to deal with this problem and knows the history better than I do.Finney1234 (talk) 15:38, 8 December 2021 (UTC)

Example 2: WP:MEDRS-qualifying secondary source partly bases its conclusion about the cause of millions of deaths on an email from a suspect. Wikipedia accepts it. Furthermore, we report the conclusion without mentioning that particular reason.

An email from the suspect says she didn't do it. Therefore, she didn't do it. That's exactly how we do things on Wikipedia. And the community is fine with it. Discussion probably soon to be archived, but for now it's here. Talk:COVID-19 lab leak theory#RfC about how we should use the Frutos source. The underlying sources are these: [9] [10] Adoring nanny (talk) 19:50, 6 January 2022 (UTC)

RFC on the use of WP:MEDRS sources on cannabis-related fatalities

An RFC related to a conflict on the use of WP:MEDRS sources on this topic has been posted at Talk:Cannabis_(drug)#RFC:_Cannabis_overdose. Please feel free to chip in, whatever your opinion. Finney1234 (talk) 18:19, 7 January 2022 (UTC)

lay-url= etc

See also broader discussion of (related) problems at User talk:SandyGeorgia [11]

|lay-date=, |lay-format=, |lay-source=, and |lay-url= have been deprecated. If a 'lay' source is important to an article, it should be given its own citation with all of the appropriate bibliographic detail. |lay-url= is recommended at WP:MEDPOP:

"One possibility is to cite a higher-quality source along with a more-accessible popular source, for example, with the |lay-url= parameter of {{cite journal}}."

That recommendation should be reconsidered.

—Trappist the monk (talk) 16:55, 26 January 2022 (UTC)

Trappist the Monk would you please:

  • Point to the discussion where consensus for this change was formed
  • Explain why notifications of proposed changes that broadly affect at least 300 featured articles (and who knows how many more) aren't posted in advance to pages most affected (like this one, for lay-source)?
  • Explain your reasoning for removing this parameter?

The recommendation now being made Template:Cite journal#Lay summary is to separately cite the lay article, which encourages novice editors to breach WP:MEDRS by using lay sources. The lay source parameters were designed, and intended, to work with MEDRS by adding a lay source only to a journal citation which conforms to MEDRS, thereby allowing a more accessible explanation to readers. (I believe it may have been designed or added by User:Eubulides, who helped develop MEDRS and wrote several medical FAs, but has been gone for ten years now; I could be wrong on that.)

Examples of how the lay parameters are used can be seen in many medical articles (I know not how to go about finding all of its uses in medical content). Encouraging editors to directly cite the lay press in medical content is unwise; the parameter was specifically intended to provide for an add-on to a MEDRS-compliant source, while not replacing a MEDRS-compliant source. A useful parameter has been removed. Why? We should not be encouraging direct citations to the laypress in medical content; as can be seen in the examples below, the lay parameters allowed for an additional link to aid readers who may not have journal access, or who may find it difficult to digest journal writing.

SandyGeorgia (Talk) 18:06, 26 January 2022 (UTC)

@Graham Beards and Lukelahood: (for recent edits at Menstrual cycle and Multiple sclerosis) SandyGeorgia (Talk) 18:29, 26 January 2022 (UTC)
I agree with SandyGeorgia. In particular, I do not support adding lay summary articles as separate references. New editors to pages where that is done would be right to remove the non-MEDRS sources, citing the guideline or the lack of need for a less reliable source. I thought for a microsecond that SandyGeorgia was calling Graham Beards "Menstrual cycle" and Lukelahood "Multiple scleroris" as puzzling nicknames). Firefangledfeathers 18:57, 26 January 2022 (UTC)
Amended above :) SandyGeorgia (Talk) 19:02, 26 January 2022 (UTC)
FFF, the (recurring) problem with these citation issues is that they become a fait accompli; once they're done, the community has been unable to get them undone. So, that has appeared to be the normal operating mode with respect to citation changes (get it done, without notifying anywhere, knowing it will stick after the protest subsides). SandyGeorgia (Talk) 19:23, 26 January 2022 (UTC)
You are welcome to call me FFF, though I'd prefer to be called "Early onset dementia" . Yes, I am also concerned about how this was accompli-shed. Would it be best to take this discussion to Help talk:Citation Style 1, and notify some relevant talk pages? Firefangledfeathers 19:34, 26 January 2022 (UTC)
I dunno. The problem is, unless the community gets together to do something about this, it will just continue, as it has for ten years. We used to have admins who stood up to it. I am at a loss for how to make it stop. SandyGeorgia (Talk) 19:45, 26 January 2022 (UTC)

This wording has been written into this very widely used and cited guideline with

One possibility is to cite a higher-quality source along with a more-accessible popular source, for example, with the |lay-url= parameter of {{cite journal}}.

since at least the end of 2008 (that's a lot of years on a highly used guideline for a couple of editors to be undoing).WP:MEDRS has 389 page watchers, and has included the text about the lay source parameter, with text unchanged for all those years. The cite journal template page has 130 watchers, but editors there feel empowered to make such a change without notifying this page.

When deprecating a parameter with such wide and long-standing acceptance, why was this page not notified of the proposal? This de facto, fait accompli method of operating on citation templates should be addressed by the community more broadly. SandyGeorgia (Talk) 19:47, 26 January 2022 (UTC)

Hi, I am also interested in this but I read through this dicussion and got a little confused about what question is being asked- I am not familiar with the templates. Could someone please give me a super brief overview? JenOttawa (talk) 20:00, 26 January 2022 (UTC)
JenOttawa just look at the sample citation above from Menstrual cycle; that should explain. Our wording here, since 2008, has suggested and the parameter allowed for adding useful lay press sources to the citation journal template, along with the MEDRS-compliant source. By adding it to a journal citation, we aren't encouraging citation to the lay press; just adding it as additional information to interested readers who may not have journal access or want a much simpler explanation. Now, with this parameter removed on a "consensus" of about three editors, we have to jump through manual hoops to accomplish the same. The core issue here is that this is how the editors who maintain the citation templates are accustomed to operating as similar issues have come up for years and years: on extremely narrow consensus, of one to three editors, broad changes are made that affect hundreds to thousands of articles, leaving in this case, thousands of error messages in Good and Featured articles, but narry a heads up to the pages most affected. MEDRS is a widely used and quoted page, which enjoys broad consensus, and one back-room change just wiped out a useful option, leaving articles with error messages. SandyGeorgia (Talk) 21:19, 26 January 2022 (UTC)
From my very brief exposure to this feature, I liked it. It seems like it'd prevent someone from flagging the citation as "non-medical," despite it never being intended to serve as medical backing, but rather as a lay summary.Lukelahood (talk) 02:43, 27 January 2022 (UTC)
Over at Help talk:Citation Style 1/Archive 82#Lay-url no one is listening; once these things are pushed through (always on negligible consensus) it’s a fait accompli and good luck getting it reversed. SandyGeorgia (Talk) 02:48, 27 January 2022 (UTC)
Thanks for dubbing it down for me! I think that I understand. I can see your manual edit on Menstration. For WP:MEDPOP my vote would be to leave (reinstate) the option. Despite my support, I think that the popular press article(s)/summary-articles that are 'attached' to the secondary source would have to be checked carefully to make sure that the evidence from the MEDRS source is not lost in translation. I am very concerned if the new suggestion is to add in popular press citations on their own to support medical facts as I see mentioned in this discussion. Thanks again for flagging this! JenOttawa (talk) 03:13, 27 January 2022 (UTC)
That is precisely what they are advocating, and by challenging it, I was labeled passive-aggressive. The few folk who run the citation templates expect us to instead add bundled citations, citing the laypress. SandyGeorgia (Talk) 03:16, 27 January 2022 (UTC)

First, I think it's important to remember that this isn't an emergency. The parameters are still there, and they're still working. Also, almost none of our readers actually click on a ref, so even though the danger–danger-danger red text is there, it's mostly editors like us who will notice it.

Second, I think there are several ways to accomplish the goal.

  1. Add the same content just after the citation template.
  2. Separate the two sources. This is not my favorite idea, but it could be done.
  3. Bundle the two sources:

I think that any of these could work, and I think that the last has some advantages over the traditional approach in terms of WP:LINKROT risks. WhatamIdoing (talk) 04:20, 27 January 2022 (UTC)

Please don’t disregard “editors like us” who are sensitive to Featured articles being delisted if they fill up with unnecessary big red errors caused for unnecessary reasons. Please recognize that the “emergency” as you judge the reaction may be a weariness of seeing this same thing happen over and over again for more than a decade. And knowing we are powerless to stop it. And have fun writing up the explanation above at MEDRS for how to bundle citations in such a way that you can quickly and easily do exactly and precisely what the lay parameters did. In one sentence. The current sentence, which has stood since 2008, is

One possibility is to cite a higher-quality source along with a more-accessible popular source, for example, with the |lay-url= parameter of {{cite journal}}.

It worked. As designed. By a prolific content writer, who also spoke geek and helped write MEDRS. I look forward to seeing all of that above written in to MEDRS. SandyGeorgia (Talk) 04:32, 27 January 2022 (UTC)
I would be astonished if a featured article were delisted this week over citation templates. That would suggest a broken FAR system, run by people who don't know or care about what's going on.
We should choose one method to recommend. The IP, by the way, suggests that we take the old system, which I will abbreviate here as:
{{cite journal ... |lay-url= |lay-date= |lay-source=}}
and turn it into this:
{{cite journal and summary ... |lay-url= |lay-date= |lay-source=}}
with the new template being one that says "Cite journal, please, but stick the lay stuff onto the end." The only think that would differ for you is that you'd type "cite journal and summary" (or whatever we name the new template) instead of "cite journal" at the start. It is a change that a bot could make to all articles, or (probably faster) an editor could do in a single AWB run.
The rare parameters would stop putting any performance drain on the 99.99% instances that don't use it, but the net effect for the few uses would be nothing (from the readers' POV) and a changed template name (from the editors' POV). WhatamIdoing (talk) 15:46, 28 January 2022 (UTC)
I'm not really keen on creating a new template just for journal citations with lay summary. I think option 3 above is best as it gives freedom to cite the summary using whichever template is most appropriate. Sorry Sandy, but I don't think bundling the lay stuff in with the main citation was a great idea, though the way this change has been rolled out was very poor for reasons everyone has already stated. There should have been a migration plan before it got deprecated and before the warnings. Now, I fear these lay sources will just get deleted by a gnome to make the red go away. The Cite toolbar only offers me vanilla cite journal, not any new template. Btw, how does all this work with editors using the visual editor? Can they handle bundled citations? -- Colin°Talk 17:19, 28 January 2022 (UTC)
I strongly oppose any formalization of any method that cites lay press in separate citation templates. Slippery slope. Looks like an actual citation when it's not. If that's where we end up, I'll stop using lay sources in my own editing, as I don't want to find us on that slippery slope. SandyGeorgia (Talk) 17:42, 28 January 2022 (UTC)
I understand the frustration at having relied-upon features change/disappear all of a sudden. I personally haven't used the |lay-url= family of parameters before, but I see how they're useful. I like WAID's option #3 quite a bit, and if this is the new normal, I think we could formalize in WP:MEDRS that providing a separate lay source in this manner is acceptable. If we go with this, I assume a bot could convert citations to this new format to prevent anyone from "fixing" the citations by manually removing the lay sources. Ajpolino (talk) 18:07, 28 January 2022 (UTC)
At the end of the day a citation is just a way of describing a work (journal article, web page, news article, etc) that is referenced from our body text. The lay summary isn't that work. It is another work and it isn't one that necessarily meets all the requirements. I don't see how anyone could be confused about which is the actual source and which is clearly described as a lay summary. There are competing features we want from any solution, but I do think that using standard templates that many editors will be familiar with is a bonus, and using obscure templates will hinder others editing. -- Colin°Talk 18:19, 28 January 2022 (UTC)
(edit conflict)
Editor WhatamIdoing suggested a slightly different format at Help talk:Citation Style 1 (this edit). From that example, I created a wrapper template {{lay source}}. Rewriting the lay summary part of Editor WhatamIdoing's example 3 using {{lay source}} gives this:
{{lay source |template=cite news|last=Dasgupta |first=Shreya |date=20 April 2015 |title=Why do women have periods when most animals don't? |url=https://web.archive.org/web/20210801180159/http://www.bbc.com/earth/story/20150420-why-do-women-have-periods |work=BBC Earth}}
Lay summary in: Dasgupta, Shreya (20 April 2015). "Why do women have periods when most animals don't?". BBC Earth.
The lay summary template cannot be linked to from {{sfn}} and similar templates. |template= accepts the name of any of the standard cs1|2 templates so if a good lay source is in a book, it can be noted with this template; |url= not required.
—Trappist the monk (talk) 18:25, 28 January 2022 (UTC)
@Trappist the monk, I'm not sure what the point of the |template= parameter is. Does it matter? WhatamIdoing (talk) 01:15, 30 January 2022 (UTC)
Suppose that the lay source is a review in another journal with its own |doi=, |volume=, |issue=, etc; or suppose that the lay source is a chapter in a text book; |chapter= and possibly |chapter-url= might apply; suppose that the lay source is in a conference proceedings.... Telling {{lay source}} which of the cs1|2 templates to use ensures that the rendering is correct for the source you are !citing. Default when |template= is omitted is {{cite web}}.
—Trappist the monk (talk) 01:58, 30 January 2022 (UTC)
Colin, sorry for piecemeal iPad typing from car, but what do you mean by “creating a new template for lay summary”? We have had a parameter in cite journal for 15 years; it’s not new. SandyGeorgia (Talk) 18:28, 28 January 2022 (UTC)
It'd look like what Trappist posted. You'd type something like this:
<ref>{{cite journal |vauthors=Emera D, Romero R, Wagner G |title=The evolution of menstruation: a new model for genetic assimilation: explaining molecular origins of maternal responses to fetal invasiveness |journal=BioEssays |volume=34 |issue=1 |pages=26–35 |date=January 2012 |pmid=22057551 |pmc=3528014 |doi=10.1002/bies.201100099 |type=Journal article}} {{lay source |template=cite news|last=Dasgupta |first=Shreya |date=20 April 2015 |title=Why do women have periods when most animals don't? |url=https://web.archive.org/web/20210801180159/http://www.bbc.com/earth/story/20150420-why-do-women-have-periods |work=BBC Earth}}</ref>
and get this for when you're reading the page:
The "Lay summary in:" part is generated by a separate template, so you could add it after any citation template. WhatamIdoing (talk) 01:25, 30 January 2022 (UTC)
They won't be delisted because of a non-consensual backroom change. Nonetheless, many editors take pride in their work, value readers, and keep errors cleaned up asap. SandyGeorgia (Talk) 17:35, 28 January 2022 (UTC)

Having removed the MEDRS-compliant lay-url, over at the citation template discussion, the best I can decipher, an IP is rewriting MEDRS to their own interpretation of how to use laysources, with statements like “lay sources that pass RS should be preferable”. Help_talk:Citation_Style_1#A_proposal SandyGeorgia (Talk) 05:45, 28 January 2022 (UTC)

Sourcing discussion in village pump: "In general, are animal and in vitro studies acceptable sources to support toxicology statements?"

There is a medical sourcing discussion related to toxicology at Wikipedia:Village_pump_(policy)#In_general,_are_animal_and_in_vitro_studies_acceptable_sources_to_support_toxicology_statements?. MarshallKe (talk) 18:50, 8 March 2022 (UTC)

Hierarchies of evidence in medicine

Christopher J Blunt wrote in 2015 a PhD Thesis dedicated to evaluate the usefulness and limitations of hierarchies of evidence on the clinical practice. As part of his work, he freely published a systematic compilation of hierarchies of evidence up to 08/10/2020 [12]. This may be useful for MEDRS which only currently suggests 2 hierarchies (which may produce the illusion that there are only 2 competing hierarchies, when there are at least 195 from various institutions worldwide). Maybe just adding a link would be good so that the curious reader can dig further into the subject if they so wish. Best regards. --Lyrono (talk) 05:34, 6 April 2022 (UTC)

Is there a hierarchy of hierarchies? Alexbrn (talk) 05:39, 6 April 2022 (UTC)
I'm not sure that this subject is critical to the guideline. A brief reference to the Wikipedia article(s) on the subject might be enough. WhatamIdoing (talk) 18:46, 14 April 2022 (UTC)

Odd reference to Sokal affair in Summarize scientific consensus section

"Be careful of material published in disreputable journals or disreputable fields. (See: Sokal affair.)"

I'm not sure what the link to the Sokal affair article is doing here? The implication seems to be either that Social Text is a disreputable journal or Postmodernism a disreputable field. Whatever opinion one may have of Social Text, it remains a relatively popular and prominent journal in it's field. At bare minimum it is certainly considered reputable in the field. And postmodernism is... just not a "disreputable field", whatever that would mean? I'd love some clarification here, otherwise I think this should be removed. --Donna's Cyborg (talk) 19:47, 10 May 2022 (UTC)

It might make more sense to link to List of scholarly publishing stings or Paper generator instead. The Sokal affair was long enough ago that many editors won't recognize it. WhatamIdoing (talk) 04:14, 13 May 2022 (UTC)
Some fields are disreputable fields when it comes to medical claims. Social Text's field of cultural studies and critical theory is not reliable for medical claims, even though at times academics in such fields may seek to deconstruct modern medicine as "a social and linguistic construct" that "cannot assert a privileged epistemological status with respect to counterhegemonic narratives emanating from dissident or marginalized communities" as they so readily bought into for physics. The Science-Based Medicine site, which is an RS, discussed in-depth efforts by some in academia to reframe "Western" medicine as just one among many "other ways of knowing" and of equal validity to e.g. indigenous folklore and spiritual beliefs. Such people are indeed academics in their fields and may write secondary sources published by academic presses, but those are not valid medical sources. Crossroads -talk- 05:27, 13 May 2022 (UTC)
I don't think that the Sokal affair was mentioned here to say anything about philosophy, or to imply that people's cultural or religious should have no place in articles about medical subjects. Indeed, there is a rich literature in the medical field that talks about the importance of the patients' own beliefs. Chicken soup has no magical medical qualities, but if your culture taught you that it is a good food for sick people, then eating chicken soup really can make you feel better – a post-modernist idea (e.g., a subjective, personal, culture-based experience instead of a universal objective fact) that is completely compatible with conventional Western medicine.
Similarly, people who "just intuitively know" that they're going to die from a life-threatening illness are, in fact, more likely to die than people with the same diagnosis but no such belief. Intuition has been derided by some as one of those non-scientific "other ways of knowing", but it's sometimes right. Subsequent reappraisals suggest that intuition can be based on facts – the veteran firefighter who suddenly leaves the building just before it collapses, and who can only later explain that the "intuition" was not a flash of otherworldly knowledge, but the result of years of training and experience about what it means when a large fire sounds like that, or what the spongy feeling underfoot means – and that it is not at all incompatible with modern science.
As for why it was added here, @SandyGeorgia added it in April 2007 – it originally mentioned another example, about a student who thought he could estimate how many images in USENET were pornographic – and she could probably tell you whether it was intended as a slam on philosophy or as an example of research not always being perfect. WhatamIdoing (talk) 18:30, 13 May 2022 (UTC)
Sorry I can be of no help ... the fifteen-year-old memory can't come up with that, and ... as the edit summary says, I was just copying it from WP:RS. One would have to dig in to the archives at that page to discover what it was about. Sorry ! SandyGeorgia (Talk) 20:25, 13 May 2022 (UTC)

Aromaticum rosatum

How should we handle articles like Aromaticum rosatum, which only cite to one ancient source? BD2412 T 05:54, 5 August 2022 (UTC)

Probably a good start to make clear it's a Traditional medicine/Herbal medicine, rather than a mainstream modern medicinal powder. Definitely don't appear to be any MEDRS sources for this particular mixture, so we can't make any claims of efficacy. I made a first attempt to update this.
We have a List of plants used in herbalism, perhaps a similar list of ancient European traditional medicine formulations would be better than individual stubs. Bakkster Man (talk) 13:45, 5 August 2022 (UTC)
I added the date to give some context. –Novem Linguae (talk) 20:25, 5 August 2022 (UTC)
Thanks. FYI, there area a number of articles from the same source, and with the same tenor, scattered throughout the encyclopedia. I will drop a note here if I come across more. BD2412 T 20:42, 5 August 2022 (UTC)
@BD2412, when you are looking at 16th-century medical ideas, books.google.com often produces more information than the current scientific literature. WhatamIdoing (talk) 03:59, 7 August 2022 (UTC)
My more immediate concern is the fact that we have articles that may be presenting verbatim 18th century medical notions as current information. BD2412 T 19:54, 7 August 2022 (UTC)
I think the recent edits to that one have resolved that problem ("was a powder used in traditional medicine" does not even hint at this being current information). I have no idea how many others might need similar edits. WhatamIdoing (talk) 23:30, 11 August 2022 (UTC)
Any medical article linked from Cyclopædia, or an Universal Dictionary of Arts and Sciences would probably be worth scrutiny, since a number of entries were imported from that source. BD2412 T 02:36, 13 August 2022 (UTC)
I wish those search results could be narrowed down to articles with <100 edits, or smaller articles, etc. WhatamIdoing (talk) 18:01, 13 August 2022 (UTC)
Arcanum joviale, although at least that one describes the subject as "pre-modern medicine". BD2412 T 03:10, 13 August 2022 (UTC)
Category:Wikipedia articles incorporating a citation from the 1728 Cyclopaedia, for those of you that like rabbit holes. Probably similar types of articles needing similar types of cleanup. –Novem Linguae (talk) 18:59, 13 August 2022 (UTC)

COVID-19 vaccine

Eyes would be helpful at COVID-19 vaccine and Talk:COVID-19 vaccine, where some editors and IPs continue pressing non-MEDRS sources in support of theories about the vaccines. Cheers! BD2412 T 20:17, 26 November 2022 (UTC)

This would be better raised at WT:MED (where, incidentally, it has been raised before). Bon courage (talk) 20:20, 26 November 2022 (UTC)

The rise and fall of peer review

For your consideration, an essay I came across on social media today: The rise and fall of peer review

I have occassionally been bothered by an editor's excessive credulity for peer-reviewed papers based just on the mere fact of their having been reviewed, but this essay explores the topic more incisevely than I could hope to. Sennalen (talk) 17:09, 14 December 2022 (UTC)

For medical topics, peer-review has never been enough. That's why there's an emphasis on high-quality secondary sources. But even those are only as good as the underlying research. Ultimately Wikipedia is an encyclopedia reflecting accepted knowledge as published in good sources. If that "accepted knowledge" is wrong, Wikipedia will be too. By design. Bon courage (talk) 17:12, 14 December 2022 (UTC)
Agreed that peer review is no good by itself. This is one reason why MEDRS requires review articles. –Novem Linguae (talk) 02:16, 15 December 2022 (UTC)

Discussion at WP:MCQ § Proactive request for input

 You are invited to join the discussion at WP:MCQ § Proactive request for input. -- Marchjuly (talk) 19:44, 23 February 2023 (UTC)

Deprecated hierarchy of evidence

The inline text acknowledges that the figure on the left at WP:MEDASSESS in this article has been deprecated and that the source now presents a pyramid that differs in important ways from the one in the figure. Would it be possible for somebody to clean this up? (Deleting the left figure would be the simplest fix.) sbelknap (talk) 13:05, 28 December 2022 (UTC)

@Sbelknap, I have been wondering whether the whole "hierarchy of evidence" explanation in MEDASSESS could be (mostly) replaced with a link to a relevant Wikipedia articles. I'd be happy to see both pyramids removed. WhatamIdoing (talk) 03:32, 22 February 2023 (UTC)
Best would be to use the correct pyramid in MEDASSESS. sbelknap (talk) 17:09, 22 February 2023 (UTC)
I thought that the point that was being made is that there isn't just one correct way to evaluate evidence. It would be strange to say that there are different ways to rank levels of evidence in medicine, but they have some similarities, and then provide a picture of the One True™ Way to Rank Evidence. WhatamIdoing (talk) 06:35, 23 February 2023 (UTC)
This is a valid point. Using more than one pyramid to reflect the various ones in use seems OK to me. The text needs changes then to make this point more clearly. sbelknap (talk) 03:35, 1 March 2023 (UTC)
Well, maybe this would work better?

Assess evidence quality

When editors are writing about treatment efficacy, epidemiology, survival rates, and similar subjects, evidence quality helps editors distinguish between minor and major views, determine due weight, and identify accepted information. Even within the same journal, different types of papers should not be given equal weight. Research studies can be categorized into different levels of evidence.[1] There are multiple accepted models for ranking evidence levels, but they all rank sources that combine the results of multiple research studies higher than other types of sources.

Wikipedia editors should rely on sources that offer higher levels of evidence, such as systematic reviews, meta-analyses of randomized controlled trials, narrative reviews, medical guidelines, and position statements by internationally or nationally recognized expert bodies (see WP:MEDORG).

Procter & Gamble[2]
Canadian Association of Pharmacy in Oncology[3]
There are several ways to rank levels of evidence in medicine, but they all put review articles and practice guidelines at the top.

Sources that present lower levels of evidence or non-evidence should be avoided. Lower levels of evidence in medical research come from primary sources (see WP:MEDDEF). Roughly in descending order, these include: individual randomized controlled trials; quasi-experimental studies; prospective observational (non-experimental) studies, such as prospective cohort studies (one type of longitudinal study); case control studies; cross-sectional studies (surveys), and other correlation studies such as ecological studies; case reports, other retrospective analyses (including retrospective cohort studies); and non-evidence-based expert opinion, clinical experience, or conventional wisdom.

Speculative proposals and early-stage research should not be cited to imply wide acceptance. For example, results of an early-stage clinical trial would not be appropriate in the 'Treatment' section of a typical disease article, because possible future treatments have little bearing on current practice. The results might, in some cases, be appropriate for inclusion in an article specifically dedicated to an experimental drug or procedure in question, or to the researchers or businesses involved in it. Such information, particularly if analyzed by secondary sources, may be appropriate in research sections of disease articles. To prevent misunderstanding, the text should clearly identify the level of research cited (e.g., "first-in-human safety testing").

Note: Editors should not perform detailed academic peer review. "Assess evidence quality" means that Wikipedia editors should identify and use the appropriate type of source and quality of publication for the subject at hand. Do not reject a higher-level source (e.g., a meta-analysis) in favor of a lower one (e.g., any primary source) because of personal objections to the inclusion criteria, references, funding sources, or conclusions in the higher-level source.

I'm not sure that the paragraph about speculative proposals is strictly necessary.
I wonder if it needs a statement that the goal is to use the best evidence that exists (assuming evidence is a relevant quality; it's irrelevant for things like "What's this body part called?"), and if there aren't any review articles about some super-rare disease, then it's okay to use the best that you've got. WhatamIdoing (talk) 00:13, 23 March 2023 (UTC)

References

  1. ^ Wright JG (May 2007). "A practical guide to assigning levels of evidence". The Journal of Bone and Joint Surgery. American Volume. 89 (5): 1128–30. doi:10.2106/JBJS.F.01380. PMID 17473152.
  2. ^ "Evidence-Based Decision Making: Introduction and Formulating Good Clinical Questions | Continuing Education Course | dentalcare.com Course Pages | DentalCare.com". www.dentalcare.com. Retrieved 2015-09-03.
  3. ^ "The Journey of Research - Levels of Evidence | CAPhO". www.capho.org. Archived from the original on 21 February 2016. Retrieved 2015-09-03.

The Lancet

This is about [13]. It might sound funny, but The Lancet has less draconian editorial standards than Wikipedia. E.g. WP:PRIMARY medical studies are allowed to make medical claims. tgeorgescu (talk) 03:33, 23 April 2023 (UTC)

I don't think that's quite true. The Lancet has different standards, but not necessarily lower ones. WhatamIdoing (talk) 23:31, 23 April 2023 (UTC)

More tarnish on those holy systematic reviews. The need for open access

Reminds me of mortgage-backed securities helping cause the 2007-2009 Great Recession. Wrapping crap up in fancy paper with a bow. We need people with money to pay all medical journals to make everything open access. Including the mega-problematic anonymized individual participant data (IPD). Then peer review would consist of millions of readers, and Wikipedia editors who aren't easily fooled.

Medicine is plagued by untrustworthy clinical trials. How many studies are faked or flawed? By Richard Van Noorden. July 18, 2023. Nature. Subheading: Investigations suggest that, in some fields, at least one-quarter of clinical trials might be problematic or even entirely made up, warn some researchers. They urge stronger scrutiny.

Excerpts. Emphasis added.
But faked or unreliable RCTs are a particularly dangerous threat. They not only are about medical interventions, but also can be laundered into respectability by being included in meta-analyses and systematic reviews, which thoroughly comb the literature to assess evidence for clinical treatments. Medical guidelines often cite such assessments, and physicians look to them when deciding how to treat patients. ... His work has had a wide impact: researchers found that 27 of Sato’s retracted RCTs had been cited by 88 systematic reviews and clinical guidelines, ... “Untrustworthy work must be removed from systematic reviews,” says Stephanie Weibel, a biologist at the University of Wuerzberg in Germany, who co-authored the review.

--Timeshifter (talk) 06:31, 20 July 2023 (UTC)

Call me naive, but I thought people went into medicine and medical research to try to make people better. But it seems they are human just like everyone else. If truth doesn't matter any more to a large chunk of politicians, and businesses, is it any surprise that truth doesn't matter any more to a large chunk of medical researchers.
In terms of evidence, they are still our best source. And from the article you cite, it seems some groups like Cochrane are starting to include checks for untrustworthy trials. So that would be an extra level of scrutiny above the initial editorial and peer review a paper gets. I think the main issue I've complained about wrt systematic reviews is that they only address a very small aspect of a comprehensive article, and so they aren't great sources for building an article. But the other sources will cite those systematic reviews, so we can't escape them. -- Colin°Talk 08:42, 20 July 2023 (UTC)
So they say that "in some fields, at least one-quarter of clinical trials might be problematic or even entirely made up". In translation, that means we have a reliable source saying that about three-quarters of clinical trials are probably good in the worst fields (as low as 60% for ivermectin+COVID, but that's not a whole field), and that in most fields, a higher proportion of trials are non-problematic.
That's better than I had hoped for. When the dataset is large enough, having a minority of data faked will not necessarily throw off the overall result. In the Yoshihiro Sato example they give, excluding his work only changed some of the results in half of the reviews that cited his papers. In half the reviews citing the fake work, all of the results were unaffected; in the other half, most of the results were unaffected.
The bit about citing retracted studies is an unsurprising temporal problem: An RTC gets published one year, cited in a review the next year, and retracted in the third year. It would only be surprising if past reviews didn't cite papers that had been retracted later.
BTW, in terms of understanding the scope, it's useful to know that less than 50% of modern medicine is evidence-based to begin with. If you read this thinking that you get 100% evidence, and now it's really just 80% or 90% evidence overall, then you might feel like you had a serious loss. But you're starting somewhere below 50%, and the difference this makes, on average, is both small and expected. WhatamIdoing (talk) 19:36, 20 July 2023 (UTC)
I will always appreciate the oncology prof in grad school who had us read this under-cited article on the staggering failure of cancer prevention treatments (especially vitamin supplements) (by John D. Potter, who, at 129, has the highest h-index I've ever encountered for someone without a Wikipedia article). Intro:

Chemoprevention employs pharmaceutical agents to reduce the likelihood of disease progression, predicated on the assumptions that individuals at risk can be identified, we understand progression and we have effective agents.
In the case of cardiovascular disease (CVD), these assumptions largely hold but for cancer chemoprevention, although we can identify some high-risk individuals, most potential agents have been identified by inference from epidemiologic or laboratory studies; we have a multiplicity of theories of carcinogenesis that do not provide clear understanding of what is causal (1) and, thus, an incomplete understanding of the role these agents might play in cancer.
As a result, cancer chemoprevention is an almost universal failure.
There are enthusiasts for the possibilities of cancer chemoprevention (2) but they have been selective about their review of the evidence as I will show below.

Basically, only aspirin and tamoxifen seem to be at all beneficial in high-risk individuals, and even those drugs shouldn't be encouraged as preventative measures in non-high-risk people. JoelleJay (talk) 23:43, 20 July 2023 (UTC)
My favorite prof once said that the main result of taking a multivitamin (for an otherwise healthy person) was "expensive urine". That throwaway comment may have saved me more than a thousand dollars over the years. WhatamIdoing (talk) 00:18, 21 July 2023 (UTC)
I am surprised that an experienced editor such as yourself passes down such info as received wisdom. You have the right to believe what you want to believe. Like doctors who used to smoke in hospitals and said it was fine. Or that white bread was as good as whole-grain bread because the white bread had vitamins added to it. Or that food grown on poor soil is as healthy as food grown organically. I prefer this:
https://www.google.com/search?q=epidemiological+studies+of+natural+diets
https://www.google.com/search?q=epidemiological+studies+of+diets+with+many+synthetic+additives
https://www.google.com/search?q=human+vitamin+deficiencies+from+foods+grown+in+poor+soil
https://www.google.com/search?q=dietary+supplementation+for+people+eating+foods+grown+in+poor+soils
Dirt Poor: Have Fruits and Vegetables Become Less Nutritious? Scientific American. Subheading: Because of soil depletion, crops grown decades ago were much richer in vitamins and minerals than the varieties most of us get today.
I bet there were systematic reviews back in the day that said smoking in hospitals by doctors and patients was fine.
https://www.google.com/search?q=doctors+used+to+say+smoking+was+healthy
--Timeshifter (talk) 02:19, 21 July 2023 (UTC)
Timeshifter, WhatamIdoing was specifically referring to multivitamin tablets. None of the links you give address whether taking them does anyone any good (or even, causes harm, which seems to be the case for antioxidant supplements, particularly if you smoke as well). We know that a few specific vitamins and minerals are worth supplementing in our diets. So there is advice for example of folic acid, iodine and vitamin D. Some of these get added to our processed food or are encouraged as supplements (e.g., the UK advises vitamin D in winter months).
And, no, I don't think there were ever systematic reviews saying smoking was fine. The big pyramid pictures on this guideline page have systematic reviews at the top, wrt quality advice, and "background information / expert opinion" at the very bottom. Those doctors in the past, who smoked, who said it didn't cause me any harm and a good hearty cough each morning was a good thing, were very much demonstrating the low quality of opinion by so called "experts" who lack actual research data but were in the pocket of tobacco firms.
There is a danger in becoming too disillusioned about the science. That opens us up to the deliberate tactics used by anti-science organisations such as climate change deniers and the supplement industry that want folk to think the science is uncertain or that everyone is a crook so who can you trust (the bright orange politician, or the pretty actress of course). -- Colin°Talk 07:58, 21 July 2023 (UTC)
"There is a danger in becoming too disillusioned about the science". Fun fact, the discoverer of Prions and Nobel Prize winner was funded by the tobacco industry (https://www.aan.com/siteassets/home-page/footer/about-the-aan/history/prusiner-interview.pdf) in this sort of "anti mainstream" / "who knows anything" / "look at all the other dangerous things" play. Kind of conflicting isn't it! Talpedia 12:10, 21 July 2023 (UTC)
Cigarettes were once ‘physician’ tested, approved. March 10, 2009. In HemOnc today (Hematology/Oncology). Subheading: From the 1930s to the 1950s, ‘doctors’ once lit up the pages of cigarette advertisements. "The pages of The New England Journal of Medicine and The Journal of the American Medical Association were home to many tobacco advertisements throughout the 1930s, 1940s and beyond. ... finding researchers who could substantiate the medical claims that the company was making in advertisements. Now the company was able to refer to research findings in their advertisements, both to consumers and to physicians." --Timeshifter (talk) 15:38, 21 July 2023 (UTC)
"finding researchers who could substantiate" is not the same thing as finding "research" that can substantiate the medical claims, and a long way from a systematic review of high quality research (e.g. RCTs). All it means is that in those days you could pay anyone to say anything about tobacco. -- Colin°Talk 07:44, 24 July 2023 (UTC)
And decades later, there is still no scientific evidence supporting multivitamins for the general public: doi:10.1001/jama.2022.8970, doi:10.7326/0003-4819-159-12-201312170-00011, doi:10.3945/ajcn.112.049304. WhatamIdoing (talk) 02:48, 22 July 2023 (UTC)
The Evolving Role of Multivitamin/Multimineral Supplement Use among Adults in the Age of Personalized Nutrition. 2018 Feb 22. doi: 10.3390/nu10020248. Nutrients (journal). "Micronutrient deficiencies occur in segments of the adult population in the United States. Multivitamin/multimineral supplements (MVMS) are widely used by this population, which reduces inadequacies in micronutrient intake, ... The role of MVMS in health maintenance and chronic disease prevention remains controversial. Conducting studies in this area has been hampered by, among other factors, inconsistent definitions of MVMS, ranging from as few as three vitamins to broad-spectrum products containing more than two dozen vitamins and minerals. Results from some observational studies and large-scale, randomized, controlled trials suggest that MVMS may reduce the risk of some forms of cancer and, potentially, cardiovascular disease. The ongoing COcoa Supplement and Multivitamin Outcomes Study (COSMOS) is expected to build on this research and provide additional insights into these areas." --Timeshifter (talk) 03:24, 22 July 2023 (UTC)
Did you notice that the lead author of that study is "a consultant to companies that manufacture or market dietary supplements", and another is being paid by the COSMOS study, and all of them were paid to attend conferences and spout the party line by major manufacturers of vitamins and supplements? And even then, they didn't feel like they could make a statement stronger than "some" (cherry-picked?) students "may" have a positive effect. WhatamIdoing (talk) 03:47, 22 July 2023 (UTC)

Effects of vitamin and mineral supplementation during pregnancy on maternal, birth, child health and development outcomes in low- and middle-income countries: A systematic review. 26 June 2021. https://doi.org/10.1002/cl2.1127 - Campbell Systematic Reviews. "The findings suggest that MMN [multiple micronutrient] and vitamin supplementation improve maternal and child health outcomes, including maternal anaemia, LBW, preterm birth, SGA, stillbirths, micronutrient deficiencies, and morbidities, including pre-eclampsia/eclampsia and diarrhea among children. MMN supplementation demonstrated a beneficial impact on the most number of outcomes. In addition, MMN with >4 micronutrients appeared to be more impactful than MMN with only 3 or 4 micronutrients included in the tablet." --Timeshifter (talk) 04:19, 22 July 2023 (UTC)

Normal lifespan: 1000 months.
Amount of time most women in developed countries spend pregnant: 18 months.
I'm willing to admit that a Prenatal vitamin has some value. Are you willing to admit that 99% of the (modern) human life is spent in a non-pregnant state? WhatamIdoing (talk) 21:11, 22 July 2023 (UTC)
Also, I want to say that this whole exchange feels very weird. Unless you have some sort of massive conflict of interest (maybe you're involved in one of those MLM companies like Melaleuca or Shaklee?) and you think that my comment will somehow have a substantial effect on vitamin sales, I really can't imagine why you should care whether or not I buy multivitamins. WhatamIdoing (talk) 21:17, 22 July 2023 (UTC)

Believe it or not, my posts were not about you. And I have no conflicts of interest. I learned a lot from this exchange, and the web reading I did. I found various trials of multivitamins, multiminerals, etc.. that got positive results. But I didn't want to clutter up the page with single trials.

What I learned is that the best trials broke down the results by subgroup. Then there were some positive significant results. But if the results were measured and averaged out for the group as a whole, then then there may have still been a positive result. But not a significant one. Because of the averaging.

It makes sense. Especially if your group includes people of all incomes in a developed western country spread out across the country. But if you do studies of people with higher nutritional needs, like pregnancy, or poor people living in food deserts. Or people in poorer countries or regions. Or people getting their food from poor soils in densely populated regions. Or old people in nursing homes. Etc., etc.. Or people who are sick.

I could go on and on, but I don't want to waste the time of people on this forum. And people who take multivitamins, etc. are mostly taking them as insurance. The shotgun approach. You never know if your particular genetics may need them. Or if you are beginning to get sick, etc.. Or you are someone who needs the higher limits of the RDAs, but foods no longer provide as many nutrients as before.

It don't want to argue with people who don't see the value of organic food, and avoiding synthetic additives. More and more studies show doing both help. But I remember decades ago when few studies were done. Because you can't patent anything to make money from the knowledge. But the logic is obvious. Every year more additives are found to be dangerous. Every year more medical conditions are found to be improved in some way by some nutrients.

From my first post: "We need people with money to pay all medical journals to make everything open access." Then we all can provide constructive criticism. Like why are you not breaking down your huge group into subgroups. Why aren't you doing a systematic review of trials with those subgroups. --Timeshifter (talk) 03:38, 23 July 2023 (UTC)

You opened this topic by noting that dodgy research was making its was into systematic reviews and then conclude by citing dodgy research that supports the opinion that we should take multivitamins. Multivitamins are not "insurance". For one, while you cite your genetics being unknown, so is whether you currently have a cancer growing. Taking antioxidants if you have cancer[14] may actually make things worse or make you more likely to develop it ("Some of the largest clinical trials, in fact, had to be aborted because the patients receiving antioxidants had a higher incidence of cancer than patients who did not receive them."). The overwhelming advice for vitamins and minerals is that they do you good when packaged inside fruit and vegetables, but when you take them in a pill, only a few are recommended and even then not necessarily to take all the time by everybody. The second reason is the idea that one could continue to eat junk food or insufficiently healthy food, and just pop a multivitamin to make up for it. No evidence for that. And anyone concerned about eating "synthetic additives" should certainly avoid supplement pills, which are..... synthetic. They don't grow on trees. -- Colin°Talk 08:51, 24 July 2023 (UTC)
I did not cite any dodgy research saying that we should take multivitamins. To the contrary, the article I cited said: "The role of MVMS in health maintenance and chronic disease prevention remains controversial."
And it is well-known that taking vitamins and minerals in megadoses can be dodgy. There are many studies that break down what are dangerous levels. Vitamins may be synthetically made, but are usually identical to the versions found in nature. If not, they should be considered to be a pharmaceutical. Like the form of Vitamin E (dl-tocopherol) that is not found in nature. Substituting supplements for healthy foods is dodgy. But some people don't get enough sun, and have to take vitamin D.
Some people eat food from regions with low iodine in the soil. So they eat iodized salt. Some people have vitamin deficiencies created by their medications. So they have to take supplements. I have taken medications where I had to be tested for those vitamin deficiencies. I could go on and on. Nutraceuticals are another area. Many may not be essential to one's health. They have to pass muster with studies, too, as to safety, effectiveness, etc.. Just like pharmaceuticals. Of course, there is little incentive to do those necessary studies since there is no patentability and profit motive. I believe in studies. I wish more were done. --Timeshifter (talk) 18:22, 24 July 2023 (UTC)
WhatamIdoing already pointed out what is dodgy about the studies you cited. Lots of the additives in food are "found in nature" and aren't classed as pharmaceuticals either. Here's Tesco Multivitamin pills:
  • Bulking Agents (Microcrystalline Cellulose, Calcium Carbonate)
  • Vitamin C
  • Vitamin E
  • Modified Maize Starch
  • Niacin
  • Glazing Agents (Hydroxypropyl Methyl Cellulose, Talc)
  • Pantothenic Acid
  • Anti-caking Agents (Silicon Dioxide, Magnesium Salts of Fatty Acids, Fatty Acids)
  • Colours (Calcium Carbonate, Iron Oxides and Hydroxides)
  • Vitamin B6
  • Thiamin
  • Riboflavin
  • Vitamin A
  • Folic Acid
  • Antioxidants (Alpha-Tocopherol, Sodium Ascorbate)
  • Vitamin K
  • Biotin
  • Vitamin D
  • Vitamin B12
Nothing in this pill is a fruit or a vegetable or a cake your mum just made. It is even more artificial than a highly processed fried chicken, sausage roll, or ready meal lasagne, because at least those contained actual chicken or actual pork or actual beef, whereas this contains no actual food at all. This is 100% additive. -- Colin°Talk 10:46, 25 July 2023 (UTC)
"Lots of the additives in food are 'found in nature'". I don't have a problem eating foods with those additives.
And that multivitamin is not that bad. Most of the additives are found in nature. And I don't see any artificial colors or flavors.
There are multivitamins, multiminerals, and single vitamin/mineral supplements with all the additives being found in nature.
I am not trying to convince you or anybody else to take supplements.
I am looking at how systematic reviews function, or don't function. Something that has interested me for awhile. Systematic reviews don't say that you should or should not take supplements. They pretty much can only point out the dangers of particular supplements at certain levels. Same is true for food. Eating too much of almost anything can be unhealthy. Drinking too much water can kill you. --Timeshifter (talk) 11:13, 25 July 2023 (UTC)
One does want to be careful about the "nature is good" idea. Aside from the obvious – what, nobody fancies a nice salad of organic poison ivy today, with crunchy natural asbestos topping? – I see two "natural" inactive ingredients in that list above that are known carcinogens (though one is really only a concern if you inhale it, rather than swallowing it).
Systematic reviews function better than all of the available alternatives for their main purpose, which is determining whether there is an overall trend in the published global data. There are limitations (e.g., they cannot account for unpublished data or undetected fraud) but the limitations are less than the alternatives (e.g., relying on a single study, which might be 100% fraudulent rather than merely including some fraudulent data among the honest data). WhatamIdoing (talk) 16:21, 25 July 2023 (UTC)

Some multivitamin/multimineral studies with positive results are found here:

Scroll down to the section called: "Who May be at Risk for a Nutrient Deficiency?" it says: "Certain groups are at higher risk for a nutrient deficiency:" It lists them.

Scroll down to "Multivitamins and Health". It says: "There is no arguing that multivitamins are important when nutritional requirements are not met through diet alone."

Then scroll down to this paragraph near the end:

For many diseases, but especially for cancer, only long-term trials are informative. The following studies looked at the effect of multivitamins on specific diseases and included healthy people as well as those with chronic diseases at the start of the study:

Click the 4 dropdowns for: Cancer. Cardiovascular disease. Mortality. Cognitive Health. Positive results for 2:

Cancer:

A randomized double-blind placebo-controlled trial provided a multivitamin or placebo to more than 14,000 male physicians, some with a history of cancer. After 11 years, the men taking a MVI had a significant 8% reduction in total cancers, compared with men taking a placebo. The results did not differ among men who had a history of cancer at the start of the study and men who were healthy at baseline.

Cognitive Health:

The COcoa Supplement and Multivitamin Outcomes Study (COSMOS) is a large randomized controlled trial that followed more than 21,000 older men and women (≥60 and ≥65 years, respectively) ... Results showed that taking a daily multivitamin-mineral supplement improved memory and executive functioning (e.g., time management, planning, self-monitoring) more than a placebo. The benefit appeared strongest for participants with a history of cardiovascular disease (CVD);

There is some contradictory info above those positive results. The difference seems to be "only long-term trials are informative."

This is a very useful caveat as to whether a systematic review is the final word on something. Was it a systematic review of short-term studies?

Concerning systematic reviews the more transparency the better. We need funding to make all medical journals and studies open access. The Nature article in my original thread post said: They urge stronger scrutiny.

Billionaires and the government need to put money in this area. Researchers make many mistakes. More readers and reviewers allows more useful feedback. And it makes healthcare more effective. Especially, cost effective. --Timeshifter (talk) 22:39, 25 July 2023 (UTC)

I think we are getting off the track for what MEDMOS talk page is for. Consider two things. You aren't citing any systematic reviews, which is what this section is about, but citing individual trials. And secondly, take your cancer trial for example. If a trial in 2012 truly demonstrated that popping a simple multivitamin pill each day significantly prevented cancer, you'd think that in 2023 various government and cancer bodies would recommend we all do that. They don't. And no, there's no conspiracy. This is a hundred billion dollar industry. If they could push for those recommendations, they certainly would. This suggests to me there were other possibly better trials that found no effect. Which is what systematic reviews are designed to achieve. Any old "expert" (particularly if their shiny new supplement research lab was funded by industry) can cherry pick research that supports their opinions. That's one of the reasons we love systematic reviews for evidence on Wikipedia and don't want articles to directly cite primary research studies. -- Colin°Talk 08:29, 26 July 2023 (UTC)

How this applies to certain articles

I'm not sure if this is the right place to ask this. Here's an example, from Dandelion coffee:

Dandelion root has been linked to a possible treatment for cancer.

The source for this is strictly primary. There are many similar statements on other pages about, generally, plants. Does this not violate WP:MEDRS? ByVarying | talk 01:46, 12 August 2023 (UTC)

@ByVarying, saying that something cures cancer (or even that it might be kinda-sorta linked to a possible treatment for cancer – it doesn't matter how many weasel words are put in there), based on any single primary source, is generally not a desirable way to write an article. If you were going to keep the section, I might change the "It has been linked" approach to something that sounds more like "Someone did some research in 2016".
However, in this particular case, you might be able to do better than that. It's possible that it could be updated based on secondary sources such as PMID 33719631, PMID 33187200, PMID 32377288, or PMID 36207736. (Some of these may be lousy journals or may not prove to be useful in practice; caveat lector.) WhatamIdoing (talk) 04:39, 12 August 2023 (UTC)

MEDRS required for activities of an NGO?

I would like to seek clarification on whether MEDRS are required in a section on the activities of an environmental NGO working on, for example, toxic chemicals and drinking water contamination. My interpretation is that the second example in WP:MEDRS#Biomedical v. general information is pretty comparable, as long as the content is what they do, rather than what they have proven. Wikipedia:Biomedical information does not contradict this interpretation either. (The situation would obviously be different if an NGO's publications on such a topic were used in articles on chemicals as sources of information on, for example, their effects or human exposure.) --Leyo 08:55, 23 September 2023 (UTC)

I haven't looked at the article in question, but in general it would depend if there was any biomedical implication. A statement like "${CHARITY} is working to improve water quality in ${LOCATION}" is WP:NOTBMI, but "${CHARITY} is working to raise awareness of the cancer-causing contaminant ${SUBSTANCE}" would be. Bon courage (talk) 09:17, 23 September 2023 (UTC)
Thanks. Do you agree that if "cancer-causing" is deleted (and ${SUBSTANCE} is linked to its article), also your second example would be WP:NOTBMI? --Leyo 13:52, 23 September 2023 (UTC)
It would be difficult to do it in a way which either didn't imply there was something harmful at hand, or leaving out information to the extent it would be mysterious and baffling to the reader. Having now looked at the article, I see this is about Per- and polyfluoroalkyl substances. Surely there's some mainstream sourcing on what the risk of this is to contextualize any mention of advocacy around it? Relatedly, I notice the Per- and polyfluoroalkyl substances has some dodgy claims from non-MEDRS sources. Bon courage (talk) 14:56, 23 September 2023 (UTC)
I have seen more than one discussion where the question has been raised as to whether MEDRS is needed for the activities of a particular NGO. --Leyo 19:21, 23 September 2023 (UTC)
Like almost all hypothetical questions, it depends. If 'what they do' is make fringey medical claims, MEDRS is going to come into it very rapidly. If 'what they do' is block whaling vessels, then obviously it won't. It isn't going to be possible to get any kind of blanket answer on this. MrOllie (talk) 19:30, 23 September 2023 (UTC)
Are we talking about Environmental Working Group? It would be better to see what statement in which context we are talking about. It is often easier to generalize from one or more examples than the other way around. Polygnotus (talk) 20:44, 23 September 2023 (UTC)
Yes, that's indeed a recent example. See this RfC for details, including a text proposal. --Leyo 21:24, 23 September 2023 (UTC)
Yeah, the EWG page is where there's been opposition to MEDRS lately. The short of it is that the EWG is known for pushing a lot of fringe stuff between anti-GMO, vaccines=autism, etc., so we've been trying to stick to what MEDRS sources have to say about the groups advocacy to sort out what's really WP:DUE there for some time now. That's in part because WP:MEDPOP sources aren't reliable for the EWG since they frequently cite their reports uncritically (a problem addressed in MEDRS sources actually). Things like the Dirty Dozen list are one such example, but that did make the cut because MEDRS sources mention it (and point out the fringe nature of that activity).
Others like PFAS advocacy really haven't had MEDRS sources presented that significantly mention the EWG's work in this subject (as opposed to all the other research groups out there) outside of brief one-liners or name dropping at best. That's where the main rub has been. Their activity in this subject just doesn't come across as WP:DUE in MEDRS sources, much less whether EWG is even semi-reliable in this topic compared to all their other fringe stuff. In these cases, news reports about primary studies the EWG has put out have been the main sources being used to try to insert content, so there's always that tie to primary research without good secondary sources that runs into issues with MEDRS. The main questions for a group like this is if their research activity is noteworthy according to reliable sources (i.e., MEDRS, not newspapers) and also if that research is considered accurate. KoA (talk) 03:32, 24 September 2023 (UTC)
KoA is actually the user referred to above who is advocating the need for MEDRS even though the text only talks about what they do (e.g. their collaboration with Northeastern University to display measurements in drinking water on a MAP). The idea was to hear the views of other users and potentially to add some clarification to WP:MEDRS or WP:NOTBMI. I would suggest continuing the EWG-specific discussion in the RfC, where the views of more users would be helpful. --Leyo 06:15, 24 September 2023 (UTC)
  • The difficulty is less when discussing an NGO and more when discussing pseudoscience claims, whether pushed by an NGO or anyone else. Which does need MEDRS backing in order to put plainly in an article, because even if the claims are accurately being sourced to the NGO, the accuracy of the claims themselves are not being properly balanced with scientific sourcing. SilverserenC 22:01, 23 September 2023 (UTC)
Exactly. Who made the claim shouldn't affect how we deal with it imho. Polygnotus (talk) 22:58, 23 September 2023 (UTC)
The one caveat I'd say with that is it can matter when it's a fringe advocacy NGO on health subjects saying something. We're usually going to be more cautious and lean into MEDRS even more then, but the pathway in how we deal with it isn't substantially different. KoA (talk) 03:32, 24 September 2023 (UTC)
The (majority of) EWG's on PFASs is clearly not pseudoscience. They have even been collaborating with a university. And again, it's about text on their activities in their article, not in an article on a scientific or medical topic (e.g. about specific chemicals and their health effects). Furthermore, the text can be phrased in a way what makes it clear that their findings are not accepted by all stakeholders (if applicable).
KoA, may I ask you to refrain from using pluralis majestatis, when it's actually your own position? Thanks. --Leyo 05:58, 24 September 2023 (UTC)
EWG is a bit of a different matter, since they are a pseudoscience group in their entirety. They are well known for misrepresenting information on chemicals and their health impacts. So any topic they are involved in would be of concern on the pushing of pseudoscience. SilverserenC 06:06, 24 September 2023 (UTC)
Yes, there have been some issues with certain topics in the past, but I'm not aware of any major ones related to PFASs. In any case, things like media coverage of their activities, their presence in hearings of the Senate and the House, etc. are still worth a brief mention. --Leyo 06:26, 24 September 2023 (UTC)
https://www.acsh.org/news/2017/05/25/dear-ewg-why-real-scientists-think-poorly-you-11323 has a decent overview Polygnotus (talk) 20:01, 25 September 2023 (UTC)
Generally, American Council on Science and Health is avoided as a source if we're talking about article use due to its industry ties. It's not as bad as some other industry related sources sometimes if you take a careful eye while reading just for personal background though. KoA (talk) 21:16, 25 September 2023 (UTC)
Leyo, considering you were just in hot water for abuse of admin tools directed at me, that kind sniping isn't a good course of action. Personally, I'm trying to avoid you as it is.
Multiple editors have been working on cleaning up or watching the EWG page for some time now, so of course I'm not going to try to claim it all was just me. Likewise for folks who frequently deal with fringe subjects. For those of us that frequent fringe subjects, FTN, etc. it's pretty uncontroversial to say that the community gets really cautious about psuedoscience groups like this as Silver seren is mentioning. KoA (talk) 21:16, 25 September 2023 (UTC)
It should be possible to structure a brief statement about any notable org's area(s) of focus in a way that does not require MEDRS sourcing. An area of activity is not in itself a health claim. As an example, in our article about the clearly WP:FRINGE-promoting Autism Research Institute, statements about the organizations focus and advocacy are cited to mainstream news. If the Org is involved in promoting pseudoscience, the article should make this clear.Dialectric (talk) 21:30, 24 September 2023 (UTC)
Well, if it's significant enough to be DUE, and if it's actual pseudoscience and not something else. For example, one doesn't really need to write "Subject is a pseudoscientific teaching hospital", when the pseudoscience is offering massage to a tiny fraction of patients, or "Subject is a natural foods company", on the assumption that not producing food with synthetic additives is somehow a bad thing. For another example, Andrew Wakefield's paper on MMRs and autism was fraud rather than pseudoscience. I believe that the same sort of fraud, if used to tout a publicly traded stock he owned, or to get a loan from his bank, would have landed him in jail. (Of course, when you're applying for a loan at the bank, they usually make you sign an explicit statement that says you're not lying, "under penalty of perjury", and academic journals don't seem to do the same thing.)
But, yes, generally, the goal is to accurately say what they do and what their POV is, without saying that (or whether) everyone agrees with what they do or with their POV. For this, you normally just want decent/best available sources, rather than MEDRS. WhatamIdoing (talk) 14:55, 25 September 2023 (UTC)

HealthNewsReview.org

HealthNewsReview.org is now defunct. Are there any good alternatives?

In the meantime, I am removing from the article all mentions of this website. Thatsme314 (talk) 06:38, 10 November 2023 (UTC)

@Thatsme314, you mean "from the guideline"? If you were actually editing articles that linked to the HealthNewsReview website, then WP:DEADREF would apply, but guidelines don't have to follow DEADREF. WhatamIdoing (talk) 18:05, 10 November 2023 (UTC)
Yes, I mean "from the guideline". Thanks for the two links; I haven't seen them before. Thatsme314 (talk) 01:12, 13 November 2023 (UTC)

SUNY Downstate picture: "needs update"

Why does the SUNY Downstate picture need an update? Thatsme314 (talk) 08:25, 10 November 2023 (UTC)

@François Robere Thatsme314 (talk) 08:26, 10 November 2023 (UTC)
I see that the current picture at the SUNY Downstate webpage differs from the one in the Wikipedia article. Will there be any copyright issues if we just copy the current picture at the SUNY Downstate webpage? Thatsme314 (talk) 08:27, 10 November 2023 (UTC)
@Thatsme314: I believe that was the problem: the image differs not only visually, but also content-wise. Perhaps it was used as inspiration for the graphic?
@CFCF: Can you tell us what part did the SUNY graphic play in the creation of this one? François Robere (talk) 16:45, 10 November 2023 (UTC)
These evidence pyramids exists in hundreds of different iterations, I do not recall looking at that one specifically - it may be a rest of a previous pyramid used here, and there is even a reference to the one that I used, that is archived at https://web.archive.org/web/20160304091218/http://media.dentalcare.com/images/en-US/education/ce311/fig02.jpg . Also, there is commons:Template:PD-shape for the design and there is no copyrightability of the concept of any evidence pyramid.
I'm not sure the update is worth the time investment, the point has throughout been to illustrate that some pyramids place clinical practice guidelines at the top, and others place meta-analysis at the top. CFCF (talk) 11:39, 14 November 2023 (UTC)
Excellent. Thanks. Thatsme314, I've made the changes on both pages, and removed the tag. François Robere (talk) 17:41, 14 November 2023 (UTC)

Biomedical information

Does MEDRS apply to the content in this section?

Does Wikipedia:Biomedical information sufficiently address a scenario such as this? SandyGeorgia (Talk) 10:28, 17 November 2023 (UTC)

I'll reply over at the article but my answer is no. -- Colin°Talk 10:38, 17 November 2023 (UTC)
Discussion at Talk:Sanctions during the Venezuelan crisis#The Lancet; perhaps more work is needed on Wikipedia:Biomedical information to clarify a situation like this one. SandyGeorgia (Talk) 11:09, 17 November 2023 (UTC)
Reading the policy and per my statement in the main discussion, it doesn't. --NoonIcarus (talk) 11:26, 17 November 2023 (UTC)

There is a worrying discussion at WP:RSN#Can Cureus be trusted enough to be used anywhere?

It also discusses a junk article published by Sage Publishing. Doug Weller talk 14:02, 30 January 2024 (UTC)

Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia_talk:Identifying_reliable_sources_(medicine)/Archive_31&oldid=1221501407"