General Course Information: SYSTEMATIC REVIEW & META-ANALYSIS METHODS W      02:00P-03:50P SCHOOL OF SO 924 Instructor Information: Edward Joseph  Mullen E-mail: ejm3@columbia.edu Prerequisites This course is offered as a doctoral tutorial in the Columbia University School of Social Work. It qualifies as an advanced research methods elective and as one of the three required research methods courses for students in the doctoral program. The course is open to second year graduate students in the School of Social Work's master's degree program as well as graduate students in other University divisions with the instructor's permission. Students are required to have satisfactorily completed an introductory graduate level research methods and statistics course or their equivalent. Students should be motivated to either learn how to critically read systematic reviews and/or to learn the methods necessary to conduct systematic reviews. Course Objectives This course introduces students to methods for conducting systematic reviews in social work and the social & behavioral sciences. Specifically the course introduces students to seven steps for carrying out systematic reviews: Defining the question that the review is designed to answer as well as the hypotheses (if any) that the review is aiming to test Specifying the types of studies needed to answer the review question and test the review hypotheses Designing and implementing a comprehensive evidence review Screening the results of the evidence review using predetermined inclusion and exclusion criteria Critically appraising the included studies Synthesizing the studies & assessing the robustness of the results Disseminating the findings of the systematic review Students will be introduced to methods for conducting systematic reviews to answer questions about: Effectiveness of a policy, program or practice Screening & diagnostic properties of measures Risk or protective factors Observational associations between interventions and outcomes Prevalence of social conditions or problems Implementation outcomes or meanings and processes associated with interventions Methodological questions Economic questions such as cost-effectiveness or cost-benefit of policies, programs and practices Rationale for Systematic Review Methods: Evidence-based policy, administration, and direct practice are now required by governmental bodies, funders, licensing and accreditation organizations, professional societies, and the public. It is now expected that professional decision-making be informed by the best evidence, especially the best research evidence. However, past practice has been to rely on findings from only one or only a few studies which happen to be brought to the decision-makers attention. Alternatively in some instances literature reviews are conducted but often these literature reviews do not have methodological safeguards built into the review process that would control for biases in evidence searchers or in evidence interpretations. These methods of gathering and using evidence can result in biased decision-making. The scientific approach requires that conclusions be based on the entire body of pertinent studies identified through well-designed and executed evidence searches clearly focused on the decision requirements. In recent years social scientists have developed methods for conducting comprehensive evidence reviews that seek to: (1) minimize bias; (2)include a critical assessment of the methodological quality of studies found; and, (3) that increase the likelihood that an appropriate wide-net evidence search of relevance to the decisions to be made, is conducted. These scientific approaches are commonly called systematic reviews. Alternatively these approaches are called research syntheses or evidence syntheses. There are a number of types of systematic reviews including both qualitative and quantitative reviews. When a sufficiently large number of quantitative studies are available that examine similar outcomes it is sometimes possible to combine these data and to conduct a meta-analysis of all of the data provided by the set of studies to arrive at robust and reliable conclusions not possible with any one study. Method of Instruction The course will be conducted as a tutorial organized around key readings and class discussion of the readings. Readings for each topic will be identified and serve as a resource for class discussion and problems. The tutorial will include a discussion of the readings for clarification. In addition each class will include a relevant assignment to be completed either prior to the class or during class. The class sessions will be used to discuss the assignment as completed by each student. These assignments will follow the seven steps for carrying out systematic reviews outlined above. Students will be introduced to internet sources for locating systematic reviews. Published reviews will be critically discussed with reference to each of the seven steps. Since students will be expected to write a protocol for a Campbell or Cochrane Collaboration systematic review the concluding classes will provide students with the opportunity to present these protocols. The primary course texts are: Littell, J.H., Corcoran, J. & Pillai, V. (2008). Systematic reviews and meta-analysis. New York: Oxford Univeristy Press. Petticrew, M., & Roberts, H. (2005). Systematic reviews in the social sciences: A practical guide. Oxford: Blackwell. The primary course software: Review Manager (RevMan) is the software used for preparing and maintaining Cochrane reviews. You can use RevMan for protocols and full reviews. It is most useful when you have formulated the question for the review, and allows you to prepare the text, build the tables showing the characteristics of studies and the comparisons in the review, and add study data. It can perform meta-analyses and present the results graphically. RevMan 5 was released on 14 March 2008 (updated to 5.0.14 on 2 July 2008). Download RevMan 5 RevMan is accessible with documentation at no charge from: http://www.cc-ims.net/RevMan .Instuctions and training materials designed to help those planning Cochrane/Campbell systematic reviews can be found at http://www.cochrane.org/resources/training.htm . Also training materials can be found on the Open Learning site at: http://www.cochrane-net.org/openlearning/ . Borenstein, M., & Rothstein, H. (2006). Comprehensive metaAnalysis: A computer program for research synthesis. Englewood, NJ: . Biostat, Inc. http://www.meta-analysis.com/ A free trial download is available. A lite copy may be purchased for $95 . Download the order form as a Word document .Download the order form as a PDF file Additional references include: Cooper, H. & Hedges, L.V. (Eds). (1994). The handbook of research synthesis. New York: Russell Sage Foundation. Centre for Reviews and Dissemination. Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews. CRD Report 4 (2nd edition). March 2001. Available online at: http://www.york.ac.uk/inst/crd/report4.htm Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0 [updated February 2008]. The Cochrane Collaboration, 2008. Available at: http://www.cochrane.org/resources/handbook/ . The Cochrane Handbook for Systematic Reviews of Interventions is the official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions. The current version of the Handbook is 5.0.0 (available in full in browseable format only). A Glossary is available to accompany the Handbook. The current version of the Glossary is 4.2.5. The full Handbook is scheduled to be published in book form by Wiley in late 2008. Hunter, J., & Schmidt, F. (2004). Methods of Meta-Analysis (2 ed.): Sage. Lipsey, M. W. & Wilson, D. B. (2001) Practical Meta-Analysis. Thousand Oaks: Sage. Mullen, E. J. (2006). Choosing outcome measures in systematic reviews: Critical challenges. Research on Social Work Practice, 16(1) National Health Service Centre for Reviews and Dissemination. (2000). Undertaking systematic reviews of research on effectiveness. York, England: University of York. (Available at: http://www.york.ac.uk/inst/crd/report4.htm). William Shadish web site for links to psychotherapy meta-analyses, single subject meta-analyses, meta-anaysis software, effect size calculator http://faculty.ucmerced.edu/wshadish/Meta-Analysis%20Links.htm . Method of Evaluation The course grade will be based on completion of the mid-term and final assignments.
	Session	Topic
	Session 1	Introduction to the Course & to Systematic Review Methods
	Session 2	Locating Systematic Reviews: Key Resources
	Session 3	Starting the Review: Forming a Systematic Review Question
	Session 4	Planning the Search Strategy for Locating Relevant Studies
	Session 5	Defining the Review's Inclusion & Exclusion Criteria: What Types of Studies will be Examined?
	Session 6	Assessing Study Quality: Quantitative Studies
	Session 7	Assessing Study Quality: Qualitative Studies
	Session 8	Synthesizing the Evidence
	Session 9	Synthesizing the Evidence: Meta-analysis
	Session 10	Synthesizing the Evidence: Meta-analysis (continued)
	Session 11	Meta-analysis: Special Topics & Issues
	Session 12	Reporting Findings & Disseminating the Review
	Session 13	Meta-analysis of Assessment & Diagnostic Studies
	Session 14	Critical Review of Student Protocols Developed During the Semester
	Session 15	Critical Review of Student Protocols Developed During the Semester & Special Topics
Session 1
Topic: Introduction to the Course & to Systematic Review Methods Class Plan: Review of course materials and class plan. Discussion of assignments. Overview of systematic review methods. Reschedule session 4. Required Readings: · Petticrew & Roberts. Systematic Reviews in the Social Sciences, Chapter 1 : · Littell, Corcoran & Pillai. Systematic Reviews and Meta-Analysis, Chapter Introduction : · Appraising the quality of systematic reviews : Schlosser, R. W. (2007). Appraising the quality of systematic reviews, Focus (Vol. Brief No. 17). Recommended Readings: · Cochrane Handbook for Systematic Reviews of Interventions, chapters forthcoming : · Cooper, H. & Hedges, L.V. , Chapter 1. .Research Synthesis as a Scientific Enterprise : · Hunter, J., & Schmidt, F. (2004). Methods of Meta-Analysis (2 ed.): Sage, chapter 1 : Files: · Overview PowerPoint

Session 2

Topic:
Locating Systematic Reviews: Key Resources

Class Plan:

This class will provide opportunity for:

• Examination of internet sites that provide access to systematic reviews
• Student reports of a systematic review located on the internet or in a journal
• Student reports describing possible systematic review topics considered for the mid-term and final assignments

Assignments Due:

1. Think about a systematic review topic that you would like to work on for the course mid-term and final assignments. Prepare a brief (5 minute) report about this topic for presentation at today's class.
2. Locate a systematic review or a protocol for a systematic review either on the internet or in a journal. Prepare a brief description of this review and include this in your brief presention at today's class. This review can be used for problems addressed in subsequent classes (or you may subsequently find another review that you would prefer to use).

Required Readings:

· Center for Reviews and Dissemination Web Site :

Center for Reviews and Dissemination Web Site

· Cochrane Collaboration Library Web Site : Cochrane Collaboration Library Web Site

· Campbell Collaboration Web Site : Campbell Collaboration Web Site

· What Works Clearinghouse Web Site : What Works Clearinghouse Web Site

· Petticrew & Roberts. Chapter 4. :

· Littell, Corcoran & Pillai, Chapter 3 :

Recommended Readings:

· Cooper & Hedges. Part III Searching the Literature : The four chapters in this section can be scanned to get ideas about searching the literature for locating systematic reviews.

Session 3

Topic:
Starting the Review: Forming a Systematic Review Question

Class Plan:

• Discuss the topic - forming a systematic review question and hypotheses.
• Student reports describing the research question and hypotheses (if there are hypotheses) in the systematic review located for the last class (or a new systematic review if a new one is preferred).
• Student reports describing the systematic review question and hypotheses that will be used for the mid-term and final assignments.

Assignments Due:

1. Bring to this class a copy of the systematic review that you plan to use this semester for in-class analysis. Prepare a brief report describing the question and hypotheses (if included) in this systematic review. Provide a critique of the question based on class readings.
2. Prepare a brief statement describing the question and hypotheses that you propose to work on for the mid-term and final class assignments.

Required Readings:

· Petticrew & Roberts: Chapter 2 :

· Cochrane Handbook for Systematic Reviews of Interventions, chapter 5 :

· Littell, Corcoran & Pillai, chapter 2 :

· Cochrane Handbook for Diagnostic Test Accuracy, chapters 4.1 through 4.5 :

Recommended Readings:

· Cooper & Hedges. Part II: Formulating a Problem for a Research Synthesis : The two chapters in this section of the Handbook are excellent introductions to this topic. Larry Hedges chapter on statistical considerations is a fine overview of some key statistical concepts.

Files:

· Review Questions, Objectives PowerPoint

· Setting Review Objectives PowerPoint

Session 4

Topic:
Planning the Search Strategy for Locating Relevant Studies

Class Plan:

• Discuss the topic - . planning the search strategy
• Student reports describing the search strategy in the systematic review selected for use in the in-class assignments
• Student reports describing the proposed search strategy that will be used for the mid-term and final assignments.

Assignments Due:

1. Prepare a brief report describing the search strategy in the systematic review selected for use in the in-class assignments
2. Prepare a brief report describing the search strategy that will be used for the mid-term and final assignments

Required Readings:

· Petticrew & Roberts. Chapter 4 :

· Littell, Corcoran & Pillai. Chapter 3 & Appendix D :

· Cochrane Handbook, Chapter 6 :

Recommended Readings:

· Cochrane Handbook, Chapter 6 :

Files:

· Examples of Search Strategies

Session 5

Topic:
Defining the Review's Inclusion & Exclusion Criteria: What Types of Studies will be Examined?

Class Plan:

• Discuss the topic - . defining the reviews inclusion & exclusion criteria: what types of studies will be examined
• Student reports describing the inclusion & exclusion criteria in the systematic review selected for use in the in-class assignments
• Student reports describing the inclusion & exclusion criteria that will be used for the mid-term and final assignments.

Assignments Due:

1. Prepare a brief report describing the inclusion & exclusion criteria in the systematic review selected for use in the in-class assignments
2. Prepare a brief report describing the inclusion & exclusion criteria that will be used for the mid-term and final assignments

Required Readings:

· Petticrew & Roberts: Chapter 3 :

· Littell, Corcoran & Pillai, Chapter 3 :

· Cochrane Handbook, Chapter 5 :

Session 6

Topic:
Assessing Study Quality: Quantitative Studies

Class Plan:

• Discuss the topic - . assessing study quality in quantitative reviews 
• Student reports describing how study quality was assessed in the systematic review selected for use in the in-class assignments
• Student reports describing the proposed methods for assessing study quality to be used for the mid-term and final assignments.

Assignments Due:

1. Prepare a brief report describing the methods used to assess study quality in the systematic review selected for use in the in-class assignments
2. Prepare a brief report describing the proposed methods for assessing study quality to be used for the mid-term and final assignments

Required Readings:

· Petticrew & Roberts: Chapter 5 :

· Littell, Corcoran & Pillai: Chapter 4 :

Recommended Readings:

· Cooper & Hedges, Part IV. Evaluating the Literature : Part IV of the Cooper & Hedges text includes six excellent chapters dealing with advanced, specialized topics about evaluating the literature.

· Cochrane Handbook: Chapters 7 & 8 : This is an excellent description of the topic.

Files:

· Assessing Study Quality: Quantitative & Qualitative Studies

Session 7

Topic:
Assessing Study Quality: Qualitative Studies

Class Plan:

• Discuss the topic - . assessing study quality in qualitative reviews 
• Student reports describing how study quality was assessed in the systematic review selected for use in the in-class assignments
• Student reports describing the proposed methods for assessing study quality to be used for the mid-term and final assignments.

Assignments Due:

1. Prepare a brief report describing the methods used to assess study quality in the systematic review selected for use in the in-class assignments. If the review does not include qualitative studies locate a systematic review that does include qualitative studies for use in this assignment
2. Prepare a brief report describing the proposed methods for assessing study quality to be used for the mid-term and final assignments. Develop a plan for assessing qualitative studies that could be used (even if you decide not to include qualitative studies in the mid-term and final assignments).

Required Readings:

· Petticrew & Roberts: Chapter 5 :

· Cochrane Qualitative Research Methods Group Web Site : Review the resources at this web site. Especially check out the tab for "Tools to Assist Qualitative Researchers". For today's topic #3 "Critical Appraisal of Qualitative Studies" tab includes several relevant readings that can be downloaded.

Recommended Readings:

· Sandelowski, M., & Barroso, J. (2006). Handbook for Synthesizing Qualitative Research: Springer Publishing Company. : Chapters 4 & 5.

Assignments Due:

· Title Registration Form

Files:

· Joanna Briggs Institute QARI - Assessment Criteria

Session 8

Topic:
Synthesizing the Evidence

Class Plan:

• Discuss the topic - . synthesizing the evidence 
• Student reports describing what synthesis methods were use in the systematic review selected for use in the in-class assignments
• Student reports describing the proposed synthesis methods to be used for the mid-term and final assignments.

Assignments Due:

1. Prepare a brief report describing the synthesis methods used in the systematic review selected for use in the in-class assignments
2. Prepare a brief report describing the proposed synthesis methods to be used for the mid-term and final assignments

Required Readings:

· Petticrew & Roberts: Chapter 6 :

· Littell, Corcoran & Pillai: Chapter 5 :

Recommended Readings:

· Cooper & Hedges, Part V : The five chapters in section V of Cooper and Hedges provide in-depth discussions of various methods for statistically describing and combining studies. These chapters are relevant to this and the following 2 classes.

· Cochrane Handbook: Chapters 9, 10, 11 :

Files:

· PowerPoint on Synthesizing Evidence in Systematic Reviews

Session 9

Topic:
Synthesizing the Evidence: Meta-analysis

Class Plan:

• Discuss the topic - . meta-analysis
• Student reports describing any meta-analytic methods used in the systematic review selected for use in the in-class assignments
• Student reports describing any meta-analytic methods to be used for the mid-term and final assignments

Assignments Due:

1. Prepare a brief report describing the meta-analytic methods used in the systematic review selected for use in the in-class assignments. If none were used locate a synthesis that employed meta-analysis for use in today's class discussion.
2. Prepare a brief report describing any meta-analytic methods to be used for the mid-term and final assignments

Required Readings:

· Petticrew & Roberts: Chapter 6 :

· Borenstein, M., & Rothstein, H. (2006). Comprehensive MetaAnalysis: A Computer Program for Research Synthesis. Englewood, NJ: . Biostat, Inc., Tutorial :

· RevMan 5, chapters 9, 10, 11, 12 : 9.1.7 specifies what should be included in your protocol pertaining to a meta-analysis. Study this section and address these aspects in your final protocol if you propose to include a meta-analysis.

· Littell, Corcoran & Pillai: Chapters 5 & 6, Appendices A & B :

Recommended Readings:

· Hunter, J., & Schmidt, F. (2004). Methods of Meta-Analysis (2 ed.): Sage. :

· Cooper & Hedges: Part V & Chapter 29 : The five chapters in section V of Cooper and Hedges provide in-depth discussions of various methods for statistically describing and combining studies. These chapters are relevant to this and the following class.

· Borenstein, M., et al. Meta-analysis: Fixed effect versus Random Effect Models : Attached as a file below.

Files:

· Fixed & Random Effects Models - Borenstein, et al. 2007

· PowerPoint on Meta-analysis Introduction

Session 10

Topic:
Synthesizing the Evidence: Meta-analysis (continued)

Class Plan:

 Continuation of last classes discussion

• Discuss the topic - . meta-analysis
• Student reports describing any meta-analytic methods used in the systematic review selected for use in the in-class assignments
• Student reports describing any meta-analytic methods to be used for the mid-term and final assignments.

Assignments Due:

Same assignment as for the last class

1. Prepare a brief report describing the meta-analytic methods used in the systematic review selected for use in the in-class assignments. If none were used locate a synthesis that employed meta-analysis for use in today's class discussion.
2. Prepare a brief report describing any meta-analytic methods to be used for the mid-term and final assignments

Required Readings:

· Same Readings & Software as Required for Class 9 :

Files:

· Guides for Assessing Quality of Systematic Reviews & Meta-analysis

· PowerPoint on Introduction to Meta-analysis Methods

Session 11

Topic:
Meta-analysis: Special Topics & Issues

Class Plan:

• Discuss the topic - . special topics and issues in meta-analysis 
• Student reports describing how special issues examined in this class (e.g., heterogeneity and publication bias) were examined and controlled for in the systematic review selected for use in the in-class assignments
• Student reports describing how special issues and topics examined in this class (e.g., heterogeneity and publication bias) will be examined and controlled for in the mid-term and final assignments.

Assignments Due:

1. Prepare a brief report describing how some of the special issues and topics discussed in today's readings (e.g., heterogeneity and publication bias) were examined and controlled for in the systematic review selected for use in the in-class assignments.
2. Prepare a brief report describing how these special issues and topics may apply and be examined and controlled for in the mid-term and final assignments

Required Readings:

· Petticrew & Roberts: Chapters 6 & 7 :

· Cochrane Handbook for Systematic Reviews of Interventions: Chapters 9 & 10 :

· Littell, Corcoran & Pillai: Chapter 6 :

Recommended Readings:

· Cooper & Hedges, Part VI, Chapters 24 & 25 :

· Hunter & Schmidt, Part IV :

· The Handbook of Research Synthesis, chapters 15, 24, 25 :

· Practical Meta-analysis, chapter 6, 7, appendix B, appendix E :

Files:

· PowerPoint for Today's Class on Meta-Analysis Special Topics

Session 12
Wednesday - November 19, 2008

Topic:
Reporting Findings & Disseminating the Review

Class Plan:

• Class will discuss the topic - . disseminating the review and getting it used 
• Michael will comment on interpretation of tau-square
• Joe will present a summary of the Dobbins, et al. (2004) study (file below).
• Kennith will present a report on how the U.S. Center for Disease Control (CDC) facilitates reporting, dissemination, and implementation of research findings and findings of systematic reviews.
• Review of reporting guidelines: EQUATOR Network - collection of reporting guidelines
• Review of reporting results of systematic reviews, dissemination, and implementation strategies 

Required Readings:

· Petticrew & Roberts: Chapter 8 :

· Littell, Corcoran & Pillai: Appendix C :

Recommended Readings:

· Cooper & Hedges, Part VIII :

· Cochrane Handbook for Systematic Reviews of Interventions (2008), chapter 4 :

Files:

· Dobbins, et al. (2004)

· Reporting, Disseminating, Implementing SRs

Session 13
Wednesday - November 26, 2008

Topic:
Meta-analysis of Assessment & Diagnostic Studies

Class Plan:

• We have focused this semester primarily on systematic reviews and meta-analysis for intervention studies. Systematic reviews and meta-analysis can be used as well to assess prognostic and diagnostic assessment instruments. This class will examine such applications.
• Professor Craig Schwalbe will present his recent meta-analysis of the predictive validity of prognostic assessessment instruments for juvenile offenders invovled with juvenile courts. His two articles reporting the results of this work are attached as files.

Assignments Due:

• None

Required Readings:

· Altman, 2001 : Attached as a file

· Schwalbe, 2007 : Attached as a file.

· Schwalbe, in press : Attached as a file.

Recommended Readings:

· Cochrane Handbook for Diagnostic Test Accuracy Reviews :

This reading is available in RevMan under the Help tab.

Files:

· Altman, 2001

· Altman_PDF

· Schwabe, 2007

· Schwalbe, in press

· Aron Shlonsky Campbell Collaboration Protocol

· Aron Shlonsky Campbell CollaborationTitle Registration

Session 14

Topic:
Critical Review of Student Protocols Developed During the Semester

Class Plan:

• Students will present their draft protocols for critical discussion. It is expected that the presentation will be a systematic (formal) PowerPoint presentation. Your Identification of issues for class discussion and feedback to help you finalize your protocols is encouraged. Presentations should be approximately 15 minutes in duration including discussion.

Assignments Due:

• Students should prepare a PowerPoint presentations which cover all aspects of the draft protocol.

Session 15

Topic:
Critical Review of Student Protocols Developed During the Semester & Special Topics

Class Plan:

• If additional time is needed students will continue to present their protocols for critical discussion.
• As time permits additional topics may be reviewed.

Assignments Due:

• Students will present a PowerPoint covering all of the protocol areas.