Dr. Scott Comments on Research Design

Dr. Timothy Scott is an astute researcher and analyst, and America Fooled is filled with "amazing scholarship," according to one reviewer.   Dr. Scott made the following comments about critiquing research:

 “In every class I have ever taught I would seek to heighten students’ appreciation of how essential good research design is.  Psychology textbooks are filled with errors because there is simply a failure to uphold adequate research design standards.  The results of correlational studies are commonly reported without any consideration of critical third factors.  Yet if third factors are not controlled, it is absolutely impossible to know if the research results have any validity at all.  Experimental studies are sometimes reported which do not utilize a control group, thus making meaningful conclusions impossible.  The fact that JAMA (Journal of the American Medical Association), the New England Journal of Medicine, Pediatrics and other leading medical journals regularly print articles with these design flaws makes it clear that a large number of medical editors, perhaps the majority, simply are not adequately versed in research design.

 “I have been forced to conclude that this is a skill that is not wholly a matter of education.  Year in and year out I would drill classes on research design issues, asking students to analyze a study and tell me where the weaknesses were.  Typically I would have one or two students who were fairly capable of this in any given class.  In some classes, no one had this ability.  But then I would occasionally have a student who could easily and immediately see the design flaws in every study I shared.  I remember when I was a graduate student at Baylor and the professor had those of us in the class do the same thing.  We were to find research design errors.  Though these fellow students were all at the least college graduates, I sat in amazement at how many of these remarkably bright individuals simply could not see very obvious flaws in the research.  These experiences have convinced me that this is a skill that is a combination of natural analytical ability as well as training.  Unfortunately, no skill is more important and yet most college graduates and most MDs lack that skill.  Until large numbers of Americans can read original research and see the flaws in the design of poor or deceptive research studies, Americans will continue to be fooled.”