W3006 '01 Physiology    Some questions to ask when critiquing a research paper

Here are some questions to guide you in writing your critique.  The comments below each question refer to examples where we discussed related issues in class. 

THE SAMPLE

How was the sample selected? Does it reflect the population of interest?

Framingham study - Are those who volunteer representative of all people?
Political polls - Telephone sample - who's home? - elderly, moms w/kids, unemployed. Not random sample of all voters.
Did glucosamine studies look at just sickest people?

Is the sample size big enough?

RU486 was 100% successful in initial studies...till they increased sample size. Large sample had lower rate.

Are appropriate controls used?

Selye injected putative hormone vs. saline vehicle

Are subjects in sample homogeneous?

Hard to draw generalizations if subjects differ in age, fitness, ethnicity, nutrition
BUT may want heterogeneous group, if trying to draw conclusions re: entire population.

Are controls and experimental subjects matched on all variables?

Heart disease in Japanese: San Francisco vs. Nagasaki. But immigrants may be different.

Is sample influenced by place where it's taken?

                Smokers indoors or outdoors; WBC's in blood vs. tissues
                Sample of substance in urine not necessarily same as in blood.
                Sample in blood doesn't necessarily reflect what's in brain, if can't penetrate BBB.


Is sample influenced by when it's taken? Were measurements taken at most appropriate time, or just when convenient?

                Some substances may have been there, but disintegrate quickly (NO, PGs)
                Many parameters different during day vs. night. (GH high at night during puberty)
                Many hormones are released in pulsatile fashion; concentration varies in periodic fashion

If one system is used as a model for another, does it seem appropriate?

Animal model of human condition: Is this an appropriate species?
Cell/tissue culture model of physiological process: does it approximate the in vivo condition?

THE METHODS

How accurate is the measuring technique?

Elections: hand count vs. machine count
Studies relying on memory may be inaccurate

Does the technique that's used measure what you really want to know?

RIA vs. bioassay
Immune system studies in vivo vs. in vitro; cell number vs. cell function

What parameter of a biologically-active substance was measured?

Easier to measure in urine or saliva, but don't necessarily reflect what's in blood
Measure blood levels of some substance -- What does a change reflect? Change in synthesis? Degradation? How much in an inactive form, or bound to protein? How much has left the blood for interstitial spaces?
Measure biological response to hormone -- What does a change reflect? Change in amount of hormone? Or in number of receptors?

What endpoint was used?

# of sweat glands? amount of sweat?  number of functional glands?
Want to see if cigarette smoking affects disability from osteoporosis. Endpoint might be number of broken bones; time takes for break to heal; bone density;
Want to see if stress affects occurrence of disease. Endpoint might be occurrence of disease...or decrease in immune response....or increase in stress hormones.
Is the endpoint selected meaningful with respect to what we really want to know?

How long does study last?

Cross-sectional vs. longitudinal
Could be change found is short-term, would disappear in the long run.
Could be no effect in short-term study, but would see one if looked for longer (stress)

What type of experimental design used?

Correlational, Observational, Interventional....Would an alternative design better provide the kind of information that's desired?

INTERPRETATION

Correlation doesn't imply causation.

Shoe size of children correlates with spelling ability, but that doesn't mean we spell with our feet; rather, both are correlated with age. If the researchers concluded that X causes Y, is it possible that there is some intermediate factor that is really responsible for Y?

Effect may be statistically significant, but is it also biologically (or clinically) significant?

                Visual acuity between males and females

Have conclusions been drawn beyond what the data supports?

Dietary fat didn't correlate with CVD, but conclusion applied only to narrow range of fat.
Results found in animals, but don't necessarily apply to humans.

Does the situation studied reflect real-life scenario for this species?

                Dietary fat and atherosclerosis in rabbits...who don't naturally eat much fat.

 Are there individual differences in response, that may be obscured by the group differences that are reported?