Wikipedia talk:Evaluating sources

A review of this proposed guideline

Nutshell: This is an essay and should be tagged as such. Marking it as a potential guideline or policy is not helpful. It should be renamed and the scope reviewed because the current name/shortcut is too grand.

Summary:

I understand this was originally an attempt to replace PSTS and provide further guidance. But the article is too long and the nutshell too condensed. Some sections should be excised and the appropriate encyclopaedia articles improved if required. Some sections repeat policy too much, which is not ideal. Much of the text, I feel, shows the author(s) opinions framed round the problems they have seen in their field, but when viewed by folk outside that field, appear unbalanced. Generally, the article appears to dislike secondary sources, which is odd and counter to the emphasis in the original PSTS.

The original idea of expanding PSTS had merit, but the grand title awarded to this page has caused the scope to become unclear. I can't see how such a long and problematic piece of text can become official any time soon. I suggest it be tagged as an essay and the scope re-examined. "Evaluating sources" is the scope of a book, and probably too ambitious. As an essay, it may be that the addition of editors has occurred too early and it might benefit from being userfied for a while. An alternative would be to instigate a few wikiproject-specific essays since the concepts and problems surrounding PSTS seem sometimes to be unique within disciplines.

Here's my review of the text at this version (2007-12-31). BTW: I've only skimmed the above talk-page discussions. There are many editors on WP who evaluate sources and write articles as part of their day job. I'm a complete amateur and my main interest is medical articles, as a lay editor. Given the talent there is here, we can do better. Colin°^Talk 19:46, 31 December 2007 (UTC)

Nutshell

The current nutshell "Sources should be critically evaluated within the context of the articles in which they are used. When using primary sources, editors should stick to quoting from them and describing their contents. Any analysis of a primary source requires a secondary source." The first sentence generates the "tell me something I don't know" response. It appears to be a counter argument to the idea that evaluating sources is simple, but as a nutshell statement it is wasting space. The second is similar to what PSTS said but has two problems. (1) It demands ("should") people quote from primary sources, when quoting is not always the best technique and selective quoting from a primary source can be terrible. (2) "describing their contents" is worded all wrong. I don't want someone to "describe" a scientific paper to me, for example. I want to know what results it contained. The final nutshell sentence has "analysis" rather than "interpretive claims, analyses, or synthetic claims" from the original PSTS and the longer "analytic, synthetic, interpretive, explanatory, or evaluative" from further up PSTS. Is "analysis" a sufficient replacement for all these?

Types of sources

This section attempts to teach the editor about the different PST source types. In making no reference to Wikipedia policy or guidelines, this section is merely encyclopaedic text that should be in, em, an encyclopaedia. The reader looking for guidance has to plough through all this, and a couple of bulky tables, before getting to any actual guidance. It doesn't belong here.

Evaluation

This section then looks at each of our content policies from the framework of PST source typing. This is just one way of looking at the problem, and not necessarily the most obvious way. To have such a grand WP:EVALUATE title, this article should either follow the primary method used by professional writers (whatever that is, if there is one) or discuss several methods. From the discussions above, it appears I'm not alone in thinking PST is an odd choice to pick when starting to evaluate a source. Why elevate it above the other criteria?

While looking at the content policies, the text drops into essay style by repeating policy and providing an opinionated discussion based on the author(s) POV. In order for this to become a consensus document it needs to be seriously tightened up, trimmed and criticised.

NPOV

"Often, secondary sources introduce a distinct viewpoint" is this true? I think the writer here is thinking about just one kind of secondary source (not sure which) but I can't see how such a wide sweeping remark is helpful. I suspect the section is only considering a secondary source that is itself discussing a single primary source (or a very few) when in fact many secondary sources collate and evaluate primary source material in a manner that is invaluable to us as WP editors (we aren't allowed to do it). The overall tone of this NPOV section seems to hint that secondary sources are somehow more likely to be corrupt wrt POV than primary. For example, "secondary sources have an opportunity to introduce further bias, or to introduce bias where none existed in the primary source". There's absolutely no counter arguments in this section about the opportunities for good professional secondary sources to remove bias. In the field of medicine, for example, a systematic review, provides an objective framework specifically designed to eliminate bias and separate the wheat from the chaff.

The "Secondary sources may also be used a primary sources." is confusing and I think the "used as" is wrong. I think what the author is saying is that some secondary source additionally contain primary material (usually the results of a synthesis of primary sources generating a novel result). It is also true that many primary sources contain secondary material. For example, many medical research papers have their research results as their primary information but begin with background material (description of disease, history, epidemiology, etc) that may build on both cited sources as well as the author's own experience. Shades of grey. This is all very interesting but not completely concerned with the NOV the section is about.

OR/V

There's some careless text here: "Descriptions of primary sources must stay close to the original." We don't describe primary sources, well not unless they are particularly interesting in and of themselves. I have a problem with "Anyone, even a non-specialist, who reads the primary source should be able to verify that the Wikipedia passage simply reflects the content of the primary source. In highly-technical articles, content need simply be verifiable by the intended audience, which may require technical knowledge and expertise." The original from NOR was always a bit vague. Do we really expect "anyone" to be able to read a scientific paper or a scholarly literary discourse? I think the original intention is that one doesn't need a specific subject specialist (e.g., an epileptologist in an article on epilepsy) but some level of education in the general area may be required. If we have the room to expand this here, lets use it. The second sentence is troublesome as WP article are written for the general reader. The "intended audience" is always the "general reader". The audience of an article on epilepsy is neither an epileptologist, a neurologist, a general practitioner or a nurse. It is you and me. Again, I think I know what the author means here, but it needs work.

There's some odd stuff about "interpretations" in primary source material needing to be treated differently to secondary material.

BLP

Can you give an example of third-party primary source material?

Application

In academic writing

"Much has been written about the academic theory of source classification." what is this puff doing in a guideline? I couldn't find any text here concerning "application". This belongs in the "types of source" section, which belongs in the encyclopaedia.

In the sciences

"The general rule is to always consider the source." empty stuff. Considering the source is why the editor has come to this page.

The opening statement about someone else evaluating the text is true of all material. We wouldn't use any other kind of source because that's called "self-published sources". Editors and publishers are one kind of reviewer. The peer-review system is another. There are yet more involving committees and sending drafts out for public/selective consultation, etc, etc. This section doesn't explain that in a so-called "peer-reviewed journal" there may be articles that had no peer review and some (e.g., the editorial) that are distinctly opinion pieces. What is lacking in this section in an article that claims to help one "evaluate sources" is any idea of the vastly different kinds of scientific articles there are and their relative worth as sources in an encyclopaedia. How should one compare a telephone questionnaire of 10 children with a multi-centre randomised controlled trial involving thousands of individuals? Should a careful meta-analysis by the FDA be treated with the same strength as the case notes of a single physician? All these might appear in a "peer-reviewed journal".

The paragraph on lay literature and newspapers is quite misguided. Our quality scientific and medical article do not rely on pop science books nor the newspapers for their facts (newspapers are for news). I could go on at length about why newspapers and magazines (even good ones) make poor sources for such facts. The section on using research reported in the press prior to proper scientific scrutiny is also very weak in its warnings. Such "research" should almost never be be used. The only time such unsound scientific practice should be reported in an encyclopaedia (which isn't in a hurry, remember) is when the controversy stirred up is noteworthy in itself. Newspapers tend to print the gushing press releases of research establishments and rarely take the care to interview the researchers. The combination of press-office and journalism leads to an article of negative merit.

The section on review articles is quite misguided and I'm not sure where the idea comes from that they are more likely to escape peer-review than some others. A review in a scientific journal is a secondary source that should form the core from which to base a WP article. Lets not encourage editors to choose primary sources here when the secondary source is often superior. There are some cases when the primary source is valuable (for extra detail, it is seminal in some way) but this isn't the rule. For example, an up-to-date review of the epidemiology of epilepsy will examine numerous studies over the years and might choose to pick one best study to-date as authoritative wrt to some aspect (it was big, well conducted, had few flaws, etc). I should cite the review, not the study. My source is stronger because it is based not only on the "best study" but also an expert's opinion that this is the "best study", and not merely me (as a WP editor) possibly cherry picking the primary material that suits my case.

The section does not mention monographs, nor other forms of scientific and medical literature such as clinical guidelines or government and industry standards. National clinical guidelines, for example, are some of the most scrutinised documents you may find.

In the humanities

I've no experience here but the final sentence seems to discuss editor behaviour and the need to avoid outdated ideas that are common to all disciplines.

Colin°^Talk 19:46, 31 December 2007 (UTC)

responses

An excellent analysis, Colin, thank you. SlimVirgin ^{(talk)(contribs)} 14:41, 2 January 2008 (UTC)

Great work. The science material was put together as a quick, first draft, and has scarcely been edited since then. The main point was to try to ensure that the language didn't unnecessarily bar the use of primary sources, and the perspective (my perspective) was primarily biology, not medicine. Your edits would be welcome. --Lquilter (talk) 16:39, 2 January 2008 (UTC)

Hi Colin, tx for your analysis.

Side-remark: the "Application" section is a reworking of Wikipedia:Reliable source examples#Advice by subject area. That section was once a part of Wikipedia:Reliable sources, but thrown out of there mainly for excessive bloat (if my memory doesn't fail me). It still isn't regarded very highly, e.g. "[...] poorly written and not that useful, and probably not needed anymore" as someone commented about a month ago. Yes, I agree with you: needs more rewriting if someone wants to prove it generally useful.

Just a question, without wanting to drain too much time from you, could you have a look at WP:WITS? Would welcome any comments! tx again. --Francis Schonken (talk) 20:25, 31 December 2007 (UTC)

Thanks Carl Colin, but this is still a live proposal. If it becomes a guideline or not, time and more discussions are needed. ≈ jossi ≈ (talk) 01:43, 1 January 2008 (UTC)

And you are welcome to add/edit/expand/delete from it ... this is a wiki. ≈ jossi ≈ (talk) 01:44, 1 January 2008 (UTC)

Who is "Carl"? Do you mean me? I don't see the point in editing a proposed guideline that is flawed in scope and premise. I'm interested in knowing how near readiness you think this is? You haven't responded to any of the criticism. Is this because you think it entirely without merit? Francis agreed with me that this should be an essay but you reverted. I think you need to justify to us, in the face of the described weaknesses, why you still consider this potential guideline material. Colin°^Talk 12:28, 1 January 2008 (UTC)

Nope, Colin. On the bases of your opinion and the opinion of Francis, that don't see the point in editing a proposed guideline that is flawed in scope and premise, then don't edit. But please respect all other editors that have contributed to this proposal. There is no rush, it would either be edited further, used, promoted or demoted. Time will tell. ≈ jossi ≈ (talk) 17:46, 1 January 2008 (UTC)

I have respect for the small handful of editors who have worked on this and am rather surprised, given their experience, at its current state. I hope others (some of whom may have worked on this) will agree that this approach needs a rethink. If I'm wrong about the approach, then I'm happy to be proven wrong. However, this proposal also contains "guidelines" that are just plain wrong and is unbalanced in its handling of some types of source material. The latter issues mean that ultimately, if those are not resolved, then I will strongly oppose its promotion. I suspect I won't be the only one. Colin°^Talk 19:55, 1 January 2008 (UTC)

Colin, the discussion about primary sources and PSTS is and will remain a highly debated issue for a while. It is popping out in many pages, and this is one of them. ≈ jossi ≈ (talk) 20:08, 1 January 2008 (UTC)

Jossi, I agree that, as written, this is an essay. It's full of personal opinion, and it's anyway hard to see how it could be promoted. It couldn't be policy, because it's too idiosyncratic, but the main part of it (the primary/secondary thing) is already policy, so that part can't be demoted to a guideline. It seems that the only sensible thing is to leave it as an essay. SlimVirgin ^{(talk)(contribs)} 15:31, 2 January 2008 (UTC)

I think the problen with it is that it lost its way. It started as a subpage to NOR to explain the primary/secondary thing, but then it spread out and started giving general advice about sourcing, often doing nothing but state the obvious (e.g. "sources should be evaluated in context," which doesn't really say anything, because how could they be evaluated out of context?). It seemed to be turning into a re-run of Reliable sources. SlimVirgin ^{(talk)(contribs)} 18:14, 2 January 2008 (UTC)

I believe that I and others had hoped that it could evolve and replace the PSTS section in the current policy, allowing that policy to clean up. As for idiosyncratic, this is, sure, but so is the current PSTS material. However, if others have changed their minds and now agree with SV that its unlikely to go anywhere, then I imagine we'll all just go back to edit-warring on WP:NOR and sparring on Talk:NOR. --Lquilter (talk) 22:13, 2 January 2008 (UTC)

If it were to replace any part of NOR, it would need to be written as policy, but that's not the language it uses, and it includes passages that are completely impenetrable, so it would need a serious clean-up before it could be proposed as policy. Also, it didn't stick to the primary/secondary sources thing, but included all kinds of general advice that shouldn't be on a policy page, so those parts would have to be removed. But then it's not clear there would be much left. SlimVirgin ^{(talk)(contribs)} 22:19, 2 January 2008 (UTC)

NPOV section

It's very hard to see what this section is saying, so I'm moving it here for discussion. SlimVirgin ^{(talk)(contribs)} 14:55, 2 January 2008 (UTC)

Begins:

Neutral point of view

Main page: Wikipedia:Neutral point of view

Maintaining a neutral point of view is one of Wikipedia's most fundamental policies. It is important to ensure that all relevant major and significant-minority viewpoints are presented, and in a neutral way.

The secondary source may dispute the primary source, question its validity or authenticity, or "spin" the source according to a particular ideology or agenda. Therefore, while primary sources are often biased, secondary sources have an opportunity to introduce further bias, or to introduce bias where none existed in the primary source. In addition, the meaning of primary sources might change based on the historical context. Thus, secondary sources will often discuss the historical context of the primary source, and such information is often invaluable, and frequently there is a consensus as to the context within the academic field. Such views as to the context are almost always relevant to the discussion.

Therefore, it is important when using secondary sources to ensure that all important viewpoints about the primary source are represented in a balanced way, taking care to avoid undue weight. For example, if a primary source is the Bible, and experts in the field disagree about the meaning of a particular passage, make sure that a sufficient number of secondary sources are presented to accurately reflect the variety of opinions within that field of study. Also, it would be important to ensure that the Biblical text presented as "authoritative" is the primary source itself, and not an interpretive translation or paraphrase from an opinionated secondary source. In many cases, of course, where the meaning or significance of the primary source is clear, and it is not disputed within the field, secondary sources are not required.

Secondary sources may also be used a primary sources. Such sources serving a dual role are typically important and relevant to the topic, and represent points of view that should be presented. For example, if a secondary source is available that itself summarizes all the important points of view on an issue, and does so in a neutral manner, that secondary source may be cited as a primary source for the current state of academic thought within the field about the original source. If a secondary source was written by a historical contemporary of the primary source, that source may have significance beyond its role as a secondary source, and may also be primary, and often represents an important point of view as to the original meaning of the author and the relevant historical context.

In some cases, secondary sources may be interpreted or "spun" by contemporary sources that are even more secondary; if such doubly-secondary sources are cited, a sufficient number of them should be presented to ensure that the balance is neutral.

Ends

Main problems:

• The issue of primary and secondary sources has nothing to do with bias. A secondary source may introduce bias, or it may reduce it. A primary source may be highly biased, or not. Bias is a completely separate issue.

• The bit about secondary sources being primary sources if they summarize all the important points of an issue is not correct.

• The part about "doubly secondary sources" is impenetrable.

SlimVirgin ^{(talk)(contribs)} 14:59, 2 January 2008 (UTC)

Personal opinions in this proposal

Some of the material on this page seems idiosyncratic at best. For example: "In the humanities, materials that are potential objects of academic research but do not themselves constitute academic research are considered 'primary sources'."

It's not clear what this is trying to say, but at face value, it sounds wrong. Can someone provide a source for it, please? SlimVirgin ^{(talk)(contribs)} 15:27, 2 January 2008 (UTC)

I believe what it's trying to say is that, for instance, a novel would be a primary source. It's the relative use issue again. --Lquilter (talk) 22:08, 2 January 2008 (UTC)

A primary source on itself and its subject matter, yes. But how do you deduce that from "In the humanities, materials that are potential objects of academic research but do not themselves constitute academic research are considered 'primary sources'"? :-) SlimVirgin ^{(talk)(contribs)} 22:12, 2 January 2008 (UTC)

My sense is that the key point is that the relativity. A novel is a primary source for literary criticism about the novel. A novel is an example of an object of academic research (e.g., literary criticism, literary history, literary biography); it is not itself academic research; its study and production are in "the humanities". So that's just how I read the sentence. It doesn't seem confusing to me (although it doesn't seem like the most helpfully worded way to express that point--I hope I didn't write it). What does it look like to you? --Lquilter (talk) 22:17, 2 January 2008 (UTC)

It doesn't really mean anything to me. I think it's either meaningless or almost tautological (when studying primary sources, academics are studying primary sources). I would rewrite it if I knew what it was trying to say, but alas ... SlimVirgin ^{(talk)(contribs)} 22:22, 2 January 2008 (UTC)

I don't disagree .... --Lquilter (talk) 05:33, 3 January 2008 (UTC)

As that entire section seems problematic, I'm moving it here:

In the humanities, materials that are potential objects of academic research but do not themselves constitute academic research are considered "primary sources." In turn, the academic research that evaluates those primary sources are the "secondary sources."

Typically, text books and articles in reliable reference works that are known to have been written by experts on the subject are often the only means to determine whether a statement in a secondary source has merit or not, or if it has not long since been superseded by another hypothesis that has since been overwhelmingly accepted. However, all reference works have a limited life-span, beyond which their articles are only of limited practical use. Some encyclopedias, even highly respected ones, have notoriously long update cycles with the result that they might contain articles that may be as much as 40 years old.

The first sentence is meaningless. Second sentence is fine, but it's just repeating what's said elsewhere on the page. Third sentence is also unclear and/or false, and that paragraph isn't the kind of thing that should appear in a guideline — it's either not clear what it means, or it's stating the obvious. SlimVirgin ^{(talk)(contribs)} 21:32, 3 January 2008 (UTC)

I think it is an attempt to express the "disconnect" between the library science definition of primary sources and the humanities usage. For example, in a lot of PSTS definitions Bede's work would be considered a secondary source, though in it's own field (humanities) it's a historical (and therefore primary) source. Certainly, there are better ways to express this, I'm sure. Vassyana (talk) 09:21, 4 January 2008 (UTC)

SWOT

Added a new section to the essay: Wikipedia:Evaluating sources#SWOT analysis on primary, secondary and tertiary sources

I'm open to constructive comments. --Francis Schonken (talk) 18:54, 5 January 2008 (UTC)

SWOT analysis

Moved to Wikipedia:Sources - SWOT analysis. This section moves this essay in adifferent direction than it was originally created (This essay examines how to evaluate sources within the context of Wikipedia's content policies., as per the lead). As a separate essay, it can be developed further. ≈ jossi ≈ (talk) 03:12, 6 January 2008 (UTC)

Also note that SWOT analysis are commonly performed on business ventures and/or projects. Using that model to the analysis of sources, pushes OR to a new dimension. ≈ jossi ≈ (talk)

• Essay that is exclusively on evaluating sources within the context of Wikipedia's content policies has been moved to Wikipedia:Evaluating sources within the context of Wikipedia's content policies
• An appropriate link to that essay has been placed under the Wikipedia:Evaluating sources#Evaluation per content policies and guidelines section header (note that, for instance, Wikipedia:Notability is a guideline, not a policy).
• First sentence of Wikipedia:Evaluating sources has been changed from "This essay examines how to evaluate sources within the context of Wikipedia's content policies." to "This essay examines several methods for the evaluation of sources." (title and content of a page are not separable)
• SWOT analysis is an evaluation method, and is present in the essay.
• An appropriate link to Wikipedia:Sources - SWOT analysis has been provide under the Wikipedia:Evaluating sources#SWOT analysis on primary, secondary and tertiary sources section header. --Francis Schonken (talk) 09:14, 6 January 2008 (UTC)

As I wrote in the previous section (#SWOT) I'm open to constructive criticism about the new SWOT section. --Francis Schonken (talk) 09:14, 6 January 2008 (UTC)

You are incapable of accepting constructive criticism. I am undoing your WP:POINT. ≈ jossi ≈ (talk) 16:11, 6 January 2008 (UTC)

What is wrong with you? This is an essay that was being worked on with a specific aim. You come along and add massive section on some idea about SWOT analysis and sources without even considering asking editors here if this is useful or noit. I reverted and moved your material to a separate essay, so that you can work on it. And then you come and do this? Please do not disrupt to make a point. ≈ jossi ≈ (talk) 16:17, 6 January 2008 (UTC)

Sorry, but I do not have any constructive criticism about that material. In my opinion it is a brainfart. ≈ jossi ≈ (talk) 16:37, 6 January 2008 (UTC)

Whether or not it is a brainfart is irrelevant, isn't it? I think it is a positive step to get people out of their trenches and stop talking next to each other regarding the PSTS issue. That's how it worked for Dhaluza,[1] and I'm sure many more to follow. --Francis Schonken (talk) 18:37, 6 January 2008 (UTC)

This is an essay, not a policy page, guideline or main page article. This is an essay; it contains the advice and/or opinions of one or more Wikipedia contributors. The drive behind this page was an attempt to clarify the uses of sources within the context of our content policies. You are welcome to explore other ideas such as the SWOT one, in a separate essay as that does not fit here as per the lead on the essay, with Dhaluza or any other editors interested in that idea of yours. ≈ jossi ≈ (talk) 20:25, 6 January 2008 (UTC)

Re. This is an essay - you mean: this is an exception to WP:OWN? --Francis Schonken (talk) 20:36, 6 January 2008 (UTC)

Re. "OR about OR" argument [2]: This is not really an argument is it? For example, "[...] it is easy to misuse [primary sources]" (WP:PSTS) is as much OR about OR. Dismissing argument, much of Wikipedia's guidance is ultimately based on OR. Other example, WP:V#Sources wouldn't be thinkable without OR. --Francis Schonken (talk) 10:08, 8 January 2008 (UTC)