I have noticed that a lot of student papers cite material from the internet without asking “Who wrote this?” or “What was the source of this information?”  Here are some tips to help you choose and evaluate sources.

I.  There are two sections.
A.  Types of sources.
B.  How to evaluate suspicious sources.
 
II.  Types of sources.  Let’s consider five types.
A.  Peer-reviewed journals.  When an author submits an article to one of these journals that article is subjected to a thorough review.
1.  Here are the steps the article goes through.
a.  The editor selects an action editor (a person with general expertise in the area).
b.  The action editor will choose reviewers (experts in the specific area covered by the article).
c.  The reviewers will critically evaluate the article.  They try to balance these concerns:
i.  Does the article make a contribution?
ii.  Is that contribution big enough to warrant publication?
iii.  Does the article fit the journal?
iv.  Is the methodology of the experiment(s) sound?
v.  Have the right statistical procedures been selected?
vi.  Have the statistics been used properly?
vii.  Can the conclusions be drawn from the data?
viii.  Are there alternative explanations?
ix.  Is it well written?
d.  The action editor collects the reviews.  This person also generally reads and evaluates the article.  The possible outcomes:
i.  Reject the article.
ii.  Revise and resubmit.  The article has merit, but the current form is unacceptable.  Sometimes a new analysis or a brief rewrite will suffice (like including something in the literature review).  Sometimes more experiments will need to be conducted.
iii.  Accept as is.  This is rare.
e.  The author makes revisions and repeats the steps.
2.  Top journals have rejection rates in the 90% range.  Most peer reviewed journals have high rejection rates.  This does not guarantee that something published in a peer reviewed journal is “good,” it just lets you have confidence that the work has been carefully reviewed by experts.  On the flip side, “good” work can sometimes be rejected.
3.  Publishing in peer-reviewed journals takes time.  The research could be years old by the time the latest issue of the journal appears.  Most journals publish the article’s history at the end of the article.
4.  Evaluation.  You can find sites that rank journals by prestige/quality.  You can also uncover rejection rates, although this information is not usually published in the journal itself.  My advice is this:  If the journal is peer reviewed you still need to be skeptical, but you can have strong a priori confidence in the source.
B.  Non peer-reviewed journals/ultra-low rejection rate journals.  These sources generally charge authors to publish, but do not critically review articles.  (They may send the article to reviewers but the situation is usually “This article will be published, help the author fix it up.”)  Some notes:
1.  Page charges do not necessarily indicate easy acceptance.
2.  “Good” research can still appear in these journals.
3.  Evaluation.  When using these journals you should know that the quality can be more variable than in peer-reviewed journals.  Approach the articles with a higher level of skepticism and consider the issues for reviewers in IAc. above.
C.  Conference presentations/proceedings.  You might get a copy of a paper derived from a talk or poster presentation at a conference.  Some conferences allow all presenters to publish a brief paper in conference proceedings.
1.  Some conferences use peer review.  However, the process is usually less rigorous than for a journal because conferences exist to disseminate newer, less polished research.  Conferences also get a large number of submissions and only have a short amount of time to make a decision.  Finally, conference proposals are usually shorter than papers, so some of the information needed for a full review is not sent to reviewers.
2.  Evaluation.  If possible, try to find a published version of information presented at a conference.  If you rely on a report from a conference, be sure to subject the information to the skeptical reading suggested in IAc. above.
D.  Books and book chapters.  Usually a person with a reputation in an area will ask experts in the field to contribute chapters for an edited volume.  Many edited books cover recent research or special topics.  Some gather reviews into a handbook.  One person may also write an entire book.
1.  Books and chapters are frequently based on peer-reviewed research, but the books and chapters themselves are frequently not peer-reviewed.  For edited books, the editor usually decides whether or not a chapter is “good” enough.
2.  When original research is reported in books or chapters, be skeptical.  It could be that the research was new and the author wanted to tell people about it.  On the other hand, the research might not have been able to withstand scrutiny.
3.  Journal editors usually try to remove slanted language or reject clearly biased articles.  Book authors and editors might not.  If you turn to a handbook to learn about a particular kind of research, know that there may be another side (and a large number of research articles) that has not been included.
4.  Evaluation.  Again, there is a lot to learn from books and chapters, but skeptical reading is wise.
E.  Internet pages.  Somebody puts up a web page.
1.  Books cost money to produce, so publishers generally check them for merit before allowing them to be published.  Journals cost money and have reviewers.  Web pages are relatively cheap.  Most of the time, nobody checks the content of web pages.
2.  You should not take information presented on the internet at face value.  If research results appear only on the internet, be suspicious.  If a web site includes a summary of other people’s research, get some of that.  Compare their summary of the research to the original research.  If the summary appears unbiased and accurate, you can gain confidence in the rest of the site.
3.  Consider the warning signs in II. below when evaluating web sites.
4.  Some sites are peer-reviewed, provide discussion forums for experts, or are the web-based portion of more reputable sources.  You may place more confidence in these types of sites.  For discussion forums, be sure to read enough to find out both sides of issues being debated.
5.  Sites claiming to be skeptical are not necessarily above suspicion.  Unbridled skepticism is also unscientific.  Evaluate the claims of sites designed to debunk in the same way you would evaluate sites designed to promote.
6.  Evaluation.  When you consider internet sources, be very careful.  Assume the role of reviewer.  In general, use the internet to find a direction, turn to stronger sources for the foundation of your research.
 
Top
 
III.  How to evaluate suspicious sources.  You are cautioned to be skeptical.  The list in IAc. above is a good start.  Here are some other things to look for:
A.  Pathological science (see my notes on the topic).
B.  In psychology, something that relies on a pseudo-psychoanalytic view of the magical unconscious that can do anything to make the theory work out.  (Note:  This is not a criticism of Freud, it’s a criticism of people who invoke his theories without knowing anything about them.)
C.  It takes a lot of practice/training to think/see like the person proposing the great new finding.
D.  The person claims that they can’t get a fair hearing because:
1.  Their ideas are too dangerous to be allowed to see the light of day.
2.  Science is hostile to new ideas.
3.  The government won’t let their ideas be presented.
4.  These ideas will turn the world on its head.
E.  The person presenting a phenomenon has to tell you what you’re going to see before you can see it.  Alternatively, they might not let you see it because you won’t be able to understand it.
F.  The person emphasizes their own experience with the phenomenon over data.  For example “You put in 20 years studying this and then you might be qualified to criticize me.”
G.  The person spends a lot of their time detailing the ways they’ve been persecuted for their beliefs.
H.  The person and the phenomenon are inextricably intertwined.  The person is described as the discoverer of a previously unimagined phenomenon.
 
Remember:  Being skeptical doesn’t mean you don’t get to believe in anything interesting.  It just means you think about what you believe in.  When you switch your brain off, you invite trouble. 
 
Top
 
This document was written by me and wasn’t reviewed by anyone.  I encourage comments and suggestions.

Back to Langston's Research Methods Page