BEST PRACTICES: HOW TO REVIEW THE LITERATURE Karen E E. Luh Luh, PhD PhD, CQE Corporate Director, Clinical Information Analysis p Health System y Saint Joseph
Objectives Understand definition and importance of EBPs Know how to identify two good sources for EBP
information Explain how quality and strength of evidence are determined Know how to apply EBP methods to IP issues Understand how to evaluate studies
EVIDENCE-BASED PRACTICE
Evidence-Based Practice What is it? What is driving EBP? What are the barriers? What are the steps?
Evidence-Based Practice - Definition Treatment decisions based on Best available, current evidence Clinical expertise Patient values (Source: IOM, 2001)
In the absence of EBP, physicians and nurses rely on: What they learned in school Experience (what we’ve always done) Advice from co-workers
EBP helps us avoid this:
Evidence-Based Practice - Drivers IOM report Crossing the Quality Chasm (2001).
Ten rules to cross the chasm from the care patients receive to the care patients should receive receive, including: Evidence-based decision making
Demand e a d for o greater g eate accou accountability tab ty Evidence that clinicians often not aware
of most current information Large variations in the care delivered IOM report: “illogical”
EBP – Barriers/ Strategies ENORMOUS increase in available evidence Strategies: Synopses/ Reviews
Complexity of evaluating quality of
evidence Strategies: Synopses/ Reviews
Knowledge gaps – no methods to
identify and link clinical research priorities to research being p g done Strategies: AHRQ supports research into quality of care,
guidelines, EBPs Strategies: g “Smart” EMRs p push info ((but must be kept p up-to-date) p )
EBP – Barriers/ Strategies Peer group - Practice patterns based on local standards Strategies: P4P, public reporting
Knowledge/ skills Strategies: Continuing education Strategies: “Expert” caregiver leaders
Attitudes Attit d - Resistance R i t tto “cookbook” “ kb k” medicine di i Strategies: EBP balances evidence with expertise Strategies: Clinician expertise is key to applying evidence to individual
Patients’ preferences Strategies: EBP explicitly includes patient in decision-making
Evidence-Based Practice – Steps
Formulate the question - PICO What is the Patient group or Problem? What is the Intervention? What is the Comparison? What are the Outcomes?
Formulate the question - P Patient group or Problem How would I describe a group of patients similar to mine? To identify inpatients colonized with MRSA . . .
Formulate the question - I Intervention What is the intervention I am considering? is it cost-effective to do 100% screening at admission . . .
Formulate the question - C Comparison What alternative might I consider? as compared to using risk factors to screen selected patients . . .
Formulate the question - O Outcome What am I hoping to learn/ accomplish? to identify need for extra precautions/ isolation?
PICO P – To identify patients colonized with MRSA I - is i it cost-effective t ff ti to t do d 100% screening i att admission C - as compared to using risk factors to screen selected patients y need for extra p precautions/ isolation? O - to identify
Formulate the question – HH Compliance Problem –Among healthcare workers in acute care faclities Intervention – what is best strategy Comparison – as compared to current actions Outcomes – to improve p hand hygiene yg compliance? p
SOURCES OF EVIDENCE
Hierarchy of Evidence
EBM P Pyramid id and d EBM P Page G Generator. t (c) Copyright 2006-2011. Trustees of Dartmouth College and Yale University.
TOP OF EVIDENCE PYRAMID (FILTERED)
Systematic Reviews Best source – pre-appraised/ prefiltered Performed by subject matter experts Overview O i off existing i ti lit literature t Unbiased search of literature Explicit rules for selecting/ evaluating studies Evidence is synthesized– may have meta-analysis Conclusion/ recommended action Not narrative reviews
Rigor of research method applied to review
Systematic Reviews - examples AHRQ National Guideline Clearinghouse http://www.guideline.gov/about/
Cochrane Database of Systematic Reviews Cochrane.org Abstracts free – your library may have subscription
Clinical Evidence ClinicalEvidence.bmj.com
FIRSTConsult Database with summaries and reviews, including systematic reviews
Centre C t ffor E Evidence-Based id B d Medicine M di i Bandolier PubMed: Find Systematic y Reviews
Appraise Systematic Reviews Well-defined W ll d fi d ttopic; i h homogenous sett off studies? t di ? Thorough search strategy? Inclusion/ exclusion criteria clear/ unbiased? Reviewers blind to sources/ independent? Review included follow-up follow up with study authors to fill in
missing information? Robustness of findings evaluated; possible publication bi considered? bias id d? Conclusions firmly linked to evidence?
Systematic Review - HH Compliance Interventions to improve p hand hygiene yg compliance p in patient care (Review) (2011). Gould, D.J., Moralejo D., Drey, N., & Chudleigh, J.H. Key conclusions: 1 Quality of intervention studies disappointing; 1. Only four studies met criteria 2. Alcohol-based hand rubs plus education insufficient 3. Multiple strategies, involvement of staff in planning campaigns may be helpful 4 More 4. M research h needed d d
Other Syntheses Databases TRIP Database (Turning Research into Practice) UpToDate DynaMed Journals Evidence-Based Nursing International Journal of Evidence-Based Healthcare Bandolier
These are just a sampling pp y Same evaluation rules apply
BOTTOM OF EVIDENCE PYRAMID (UNFILTERED)
RCTs Gold standard Random R d assignment i t removes bi bias b by randomly d l
distributing known and unknown variables Control group means we can be sure that outcome is related to intervention Possible to make causal inferences
Hand Hygiene Compliance Huang (2002) from Cochrane review 100 nurses randomly d l assigned i d to t educational d ti l vs control t l
groups Direct observation of hand hygiene both before and after intervention Findings – significant increase in compliance post education d ti compared d tto b baseline li and d control t l group
Cohort Studies Observational longitudinal (over time) Compare C ttwo groups (cohorts) ( h t ) with ith diff differing i condition/ diti /
treatment/ exposure Retrospective or prospective No random assignment – other variables related to the group could have an effect on the outcome Not possible to make causal inferences
Case – Control Studies
Compare two groups one with vs without outcome of
interest Retrospective – often rely on medical records, patient recall No random assignment – other variables ariables related to the group could have an effect on the outcome Not p possible to make causal inferences
Descriptive, Qualitative Studies or Expert Descriptive Opinion
Case reports or series Based on interviews, observations, focus groups No comparison groups, no hypothesis testing No N generalizability li bilit Good for generating hypotheses (ideas)
Worst case scenario Worst-case
FINDING THE EVIDENCE
Finding the evidence - Medline Start with y your reference librarian! Start with your reference librarian!! Database searches Boolean logic - and, or, not Truncation – use word stems (control$ vs controlled) MeSH (Medical Subject Headings) – key index words Standard filters – built into search engines
Hand Hygiene Compliance Search From Cochrane review:
EVALUATING THE EVIDENCE
Evaluating the evidence Source – differentiate scientific evidence,, opinion, p , and
marketing Strength – hierarchy of evidence pyramid Quality – Is the study valid? Well done?
Even in peer-reviewed journal quality not guaranteed Applicability to your population
Strength Hierarchy y All things being equal, consider evidence hierarchy
Design g AND execution Well-done observational study trumps poorly executed RCT
Replication or consistency Has this been found more than once?
Quality Randomization – best if offsite Attrition – are patients lost to follow-up? This could
undermine randomization Double blinding – group assignment unknown to both
patient and individual(s) assessing outcome Internal validity – can we be sure that intervention under
study is causing outcome and not some other variable? Were groups treated identically EXCEPT for intervention? External validity – can these results be generalized
beyond y studyy sample? p
Integrating evidence with clinical expertise, patient preference Efficacy versus effectiveness Experts working in controlled environment vs “real-world” application
Do these results apply to my patient? Generalizability plus patient choice
Does this intervention benefit this patient? Clinical expertise needed
Is this intervention in keeping with this patient’s values
and preferences? Patient-centered care mandates centrality of patient choice
Summary EBP is shift: From reliance on experts To integration of evidence WITH clinical expertise WITH patient’s circumstances and values
EBP Resources Systematic reviews EBP journals R f Reference librarians lib i
NOT CONVINCED WE NEED EBP?
Semelweis (1847) Post-childbirth infection and handwashing
Current Practice in Infection Prevention – What does the literature tell us? Surveillance and public reporting of central line-
associated bloodstream infections Surveillance vs administrative coding data Nursing and physician attire as possible source of h hospital it l – acquired i d iinfections f ti Other examples
Quality of Surveillance for CLA-BSI Quality of traditional surveillance for public reporting of nosocomial bloodstream infection rates. Lin MY, Hota B, Khan YM, Woeltje KF, Borlawsky TB et al. (2010). JAMA, 304(18): 2035-2041.
Introduction I t d ti lays out rationale closes with clear statement of research questions: measure relationship between traditional surveillance and computer algorithm for identifying CLA-BSI determine if there is heterogeneity in this relationship among 4 institutions studied
Research R h questions ti and d rationale ti l ffor th those questions ti very clear.
Quality of Surveillance for CLA-BSI Methods Sample clearly described – 20 ICUs in 4 academic medical centers Time frame, methods for obtaining sample described IPs IP blinded bli d d to participation i i i Computer algorithm described; reference to publication Discloses power analysis and statistical tools and methods
Good clear description, it would be possible to replicate this studyy based on what we’ve learned in the methods section
WHAT KIND OF STUDY IS THIS?
Cohort Studies Observational longitudinal (over time) Compare C ttwo groups (cohorts) ( h t ) with ith diff differing i condition/ diti /
treatment/ exposure Retrospective or prospective No random assignment – other variables related to the group could have an effect on the outcome Not possible to make causal inferences
Quality of Surveillance for CLA-BSI Results statistics with measures of spread analyses clearly relate to research questions:
measure relationship l ti hi b between t ttraditional diti l surveillance ill and d computer algorithm for identifying CLA-BSI: weak relationship Spearman ρ (rho) = 0.34 determine if there is heterogeneity in this relationship among 4 institutions studied: ρ ranged from 0.83 to nonsignificant 0.10.
Results sufficiently detailed to allow us to determine whether we agree with the analyses and the findings; analyses address research questions.
Quality of Surveillance for CLA-BSI Discussion or Comment Relates results to original research questions Relates findings to other research Discusses Di possible ibl b bases ffor fifindings di
(Remember they cannot make causal statements as this isn’t RCT) Discusses limitations and ideas for future research
Authors avoid over-generalizing their results – stay within bounds of their methods.
Administrative data compared to NHSN criteria for identifying HAIs Administrative coding data, compared with CDC/NHSN criteria, are poor indicators of health care-associated infections. Stevenson KB, Khan YM, Dickman J, Gillenwater T, Kulich P, et al. (2008). Amer J Infect Control, 36: 155-164.
Introduction Lays out rationale – embeds issue in current P4P/ public reporting
requirements Closes with clear statement of research question: “compare directly the accuracy of HAIs identified by ICD-9-CM . . . codes to those identified by traditional epidemiologic methods . . . .
Clearly lays out concern that coding data are overly – sensitive; does NOT
address issue of whether NHSN criteria are under-sensitive
Doesn’tt adequately address arguments for use of coding data Doesn No star
Administrative data compared to NHSN Methods
Sample clearly described – 1 academic complex Time frame, methods for obtaining sample described IPs blinded for initial surveillance since study done retrospectively Experience/ reliability of surveillance personnel not addressed Used codes identified by independent sources – avoids investigator bias in code selection Methods for review of discordant cases unclear – IPs were NOT blinded during this review – possible source of bias
Might be possible to replicate this study based on what we’ve we ve learned in the methods section with some reservations No star
Administrative data compared to NHSN Results Statistics with measures of spread Analyses relate to research question: Discordant cases with coding-identified coding identified HAIs reviewed by IPs to determine if infections also matched NHSN definitions Subsample of discordant CLA-BSIs reviewed (50/ 485) but results extrapolated to total sample Accuracy computed AFTER adjustment Results poorly described – difficult to determine actual findings and
therefore difficult to evaluate
Results somewhat hard to follow – actual numbers of cases buried in text – tables with counts laid out as described in methods would have been helpful. No star
Administrative data compared to NHSN Discussion or Comment Relates results to original research questions Questionable evaluation of finding that ICD-9 codes flagged 59 HAIs
missed by IPs “only a small number of these discordant cases (59/879, 7%) were subsequently found to meet CDC/NHSN criteria, suggesting that the case-finding methods outlined by the CDC/NHSN are reasonably robust.” (p. 161) BUT initial surveillance identified 318 HAIs and 59/ (318 + 59) = 15.7%
Relates findings to other research Discusses possible bases for findings
(Remember (R b they th cannott make k causall statements t t t as thi this iisn’t ’t RCT) Discusses limitations and ideas for future research
Some appearance of bias in interpretation of results No star
QUESTIONS? Contact information: Karen E. Luh @ j g
[email protected]
