Document 21772

Refer to: Sanazaro PJ: Medical audit, continuing medical education and quality assurance. West J Med 125:241-252, Sep 1976

Special Article

Medical Audit, Continuing Medical Education and Quality Assurance PAUL J. SANAZARO, MD, Berkeley

Medical audit and continuing medical education (CME) are now the mainstays of quality assurance in hospitals. Audits should address problems that have serious consequences for patients if proper treatment is not given. The single most important step is the selection of, essential or scientific criteria that relate process to outcomes. CME does less than commonly believed to improve care. Today, quality assurance increasingly means a near-guarantee to every patient of appropriate treatment and fewest possible complications. Maintenance of the public trust rests on a firm commitment of the medical staff and board to this principle, implemented through an organized program of quality assurance. Under these conditions, medical audit and CME can effectively improve care by improving physician performance. TWENTY YEARS AGO, Lembcke systematically described medical auditing by scientific methods.' Today these methods are widely used and abused in assessing the technical quality of care. Medical auditing is done in most hospitals because the Joint Commission on Accreditation of Hospitals (JCAH) and the Professional Standards Review Organization program (PSRO) both require it.2-4 Along with this, continuing medical education (CME) is universally endorsed as the principal vehicle for maintaining medical competence.5 Considering the enormous amount of time and effort devoted to these activities nationally and the public's expectations of benefit from the results, we should critically examine the effectiveness of audit and CME in improving patient care Dr. Sanazaro is a private consultant in health services research and development and Clinical Professor of Medicine at the University of California, San Francisco. Reprint requests to: Paul J. Sanazaro, MD, 1126 Grizzly Peak Blvd., Berkeley, CA 94708.

by improving physician performance. At stake is the medical profession's continuing autonomy in assuring the quality of care.6

Medical Audit The purpose of auditing is to assure that patients with specified conditions are receiving the full benefit of medical care with the least possible number of complications. The techniques for doing this are imperfect and are not standardized, despite the seemingly clear-cut methods described in official publications.278 Being retrospective and dependent entirely on information contained in the record, auditing can only assess limited aspects of the technical quality of care. Of central importance are the procedures for selecting a topic and adopting objective criteria. The proper approach to these two steps is a necessary condition for effective auditing directed to improving physician performance. THE WESTERN JOURNAL OF MEDICINE

241

ASSURING QUALITY IN HOSPITALS ABBREVIATIONS USED IN TEXT CME=continuing medical education JCAH=Joint Commission on Accreditation of Hospitals PAS=Professional Activities Study PEP=Professional Evaluation Procedure PSRO = Professional Standards Review Organization QAM=Quality Assurance Monitor RCT=randomized controlled trials

Selection of Topic Medical audits should only examine important areas of care as originally suggested by Williamson.9"10 The disease or condition chosen should be curable, controllable or preventable. That is, an effective treatment or preventive measure exists for that condition which produces predictable clinical results or outcomes. Also it should be an established fact that patients suffer serious consequences complications of the disease or of the treatment, or both-if that treatment is not given properly. Priorities for possible audit subjects are decided on the basis of these considerations, as well as knowledge or suspicion of a particular problem. Selecting audit topics in this way assures the validity of the results and increases the likelihood of medical staff commitment to taking any needed corrective action. Secondary requirements in selecting topics are precision of diagnosis and frequency. Audits can be best applied to precisely defined primary diagnoses-for example, bacterial pneumonia of specified bacterial origin rather than all types and causes of pijeumonitis, or, diabetes mellitus with ketoacidosis. The presence and severity of each diagnosis, condition or complication should be capable of objective confirmation, preferably by quantitative data. For example, one audit committee specified a blood glucose value of 200 mg per 100 ml or more plus a serum acetone value greater than 1:4 or blood pH less than 7.32, or both, as substantiating the diagnosis of diabetes mellitus plus ketoacidosis. Subdividing a diagnosis or condition into objectively specifiable manifestations, stages of severity or complications facilitates auditing and eliminates some of the drawbacks in using criteria.'1 The trade-off between importance and frequency in selecting a topic is straightforward. The more serious the consequences for any patient receiving inadequate or inappropriate treatment, the fewer the cases needed for a worthwhile audit. 242

SEPTEMBER 1976

*

125

*

3

Adoption of Criteria More than anything else, it is the type of criteria and the method of adopting them that determine the effectiveness of an audit in documenting whether medical care in the hospital meets high contemporary standards. Some physicians still quesdion the necessity of objective, written criteria, claiming they can readily and accurately judge the quality of care by reading their colleagues' charts. To a certain extent this is true. But the implicit criteria of quality which each physician carries with him may not be widely shared. Even physicians within the same specialty can differ surprisingly in their judgments regarding the quality of care as reflected in a particular record. Richardson found that as many as 16 to 28 physicians would have to read and judge each record to be 95 percent certain that care for that patient was or was not adequate.'2 It is clearly impossible for physicians to devote this much effort to such a task, and the alternative now in general use is to specify explicit, written criteria. These criteria enable nonphysician personnel to screen large numbers of records to identify potential instances of substandard care. Only those records so identified are then subjected to peer review.

Whether the audit can be relied upon for evaluating physician performance and identifying important problems in patient care depends entirely on the method of choosing criteria. As shown in Table 1, there are only three basic types of criteria: statistical, ntormative and scientific. An understanding of their source and significance for purposes of quality assurance is necessary for making effective use of the audit. TABLE 1.-The Basic Types and Sources of Criteria for Medical Audits Type of Criteria

Statistical (empirical)

Source

Regional or national statistics on length of stay, current practices, complications, mortality

Normative (consensus) Consensus of physicians on proOptimal care cedures that constitute good medi(general cal care for a particular condition consensus) Essential (critical) Consensus of experts in a particular disease or condition on efficacious treatment and achievable clinical results for that condition Scientific (validated) Clinical research that objectively establishes the efficacy of treatment and its clinical results in specific conditions

ASSURING QUALITY IN HOSPITALS

Statistical criteria (also called empirical criteria) are derived from statistics on actual practice. They define what physicians presently do in the care of their patients. These statistics may come from the individual hospital's records or, more commonly, from hospital data abstracting systems, like Professional Activities Study (PAS) or California Health Data Corporation. The most widely used are the PAS regional statistics on length of stay.13 According to these, the average length of stay is longest in the East (8.3 days) and shortest in the West (6.2 days). Because of this, the PSRO program permits regional variations in setting criteria for length of stay. These differences have never been adequately explained, but they apparently do not relate to variations in quality of care. When national statistics on hospital care are based on a large and reasonably representative sample of hospitals, they may be taken as representing average physician performance. Examples of these are the 25th to 75th percentiles for a number of procedures as reported by the Quality Assurance Monitor (QAM) of PAS.14 According to the QAM reports, culture and sensitivity determinations are done in 14 to 39 percent of adult patients with pneumonia; in adult patients with acute pyelonephritis, the figures are 43 to 66 percent. If audit committees accept these statistics as reference points for setting their own criteria, they are equating the existing average level of practice TABLE 2.-Optimal Care Criteria for Acute Urinary Tract Infection"' Criterion

Percent Observed

History: Urination frequency ......... ............. Obstructive symptoms ......... ............ Pain . ................................... Hematuria .............................. Pattern of incontinence ........ ............

Chronology

..............................

Previous urologic disease ....... ........... Physical Examination: Digital rectal and/or pelvic ....... .......... Bladder examination ......... ............. Kidney area examination ........ ........... Laboratory: Urinalysis with stain sediment or culture ..... Urine culture ............................. Sensitivity ............................... Complete blood count ......... ............ Renal function test ........... ............. Intravenous pyelogram unless prostatitis ...... Antibacterial therapy within one hour ..... ... Average Percent Observed

14 26 71 29 12 86 56 37 27 59 74 70 49 87 42 36 68 56

with the desired level of quality. Adopting such statistics amounts to endorsing the status quo. It is doubtful that any specialty or professional organization would endorse the national averages as characterizing quality of care. In short, statistical criteria may be useful in initially assigning lengths of stay, but they are not suitable for auditing the technical quality of care. Normative criteria (or corsensual criteria) represent the judgment of physicians regarding what ought to be done in the care of patients with certain diagnoses. There are two varieties of these. Optimal care criteria (or general consensus) incorporate the consensus of judgments by physicians regarding the elements of good or optimal medical care for a given condition. Essential criteria are developed by experts in the diagnosis and management of the particular diseases or conditions being considered. The most widely used normative criteria are optimal care criteria, first developed by Payne in Michigan.15 These are the ones referred to as "cookbook medicine" or "laundry lists"; they cannot be used to assess the technical quality of care. For example, in Table 2 are shown optimal care criteria for acute urinary tract infection as proposed and agreed upon by a committee of physicians.'6 The items represent local consensus on the best care for that condition. The figure to the right of each item in the table is the percent of charts in which that criterion was actually observed. Not a single criterion was met in 100 percent of charts, and the average for the study was 56 percent. Similar results were found when optimal care criteria were applied to 20 other diagnoses; an overall average of 71 percent was observed. Similar discrepancies between what physicians propose as elements of good care and what those same physicians actually do in the care of patients were reported by the American Society of Internal Medicine and by the American Academy of Pediatrics.17"18 This seems to pose a serious dilemma: If committees of physicians develop criteria for good care, and if these same physicians actually abide by fewer than 100 percent of their own criteria, then, by definition, does that not mean that their care is substandard? The fact is, the dilemma is a purely semantic one: there is no way of knowing whether the care of individual patients is optimal when the audit uses so-called optimal care criteria. There are two specific reasons for this. The first is that this type of criteria evolved from a study THE WESTERN JOURNAL OF MEDICINE

243

ASSURING QUALITY IN HOSPITALS

whose primary concern was the effective use of hospitals.'5"l'9 They were actually utilization criteria, intended to make certain that third parties would not deny payment for any procedures listed for particular diagnoses. Consequently, the original sets of criteria from Michigan contain all procedures that might be necessary in diagnosing or treating all patients with a particular diagnosis. The criteria are useful to fiscal intermediaries in deciding whether or not to pay for a procedure because their only consideration is that the procedure be consistent with the diagnosis. But when such lists as shown in Table 2 are used in a medical audit, there is no way of knowing which of the listed procedures was essential for the appropriate management of a particular patient. That is why Brook found that only 1 to 2 percent of the records in his study contained all the optimal care criteria listed by faculty members as assuring quality

care.20 The second reason that these criteria are not suitable for evaluating the care given individual patients is the tendency to use the audit as an educational vehicle to promote better workups and writeups. The criteria often include a number of symptoms and signs whose presence or absence "should be recorded." But because no two physicians arrive at a diagnosis in the same way, audits which include such educationallyoriented items only show what everyone already knows: There is less in the record than should be there ideally. The fundamental shortcoming of optimal care criteria is their lack of relationship to outcomes.20 No matter how many or how few of the criteria are observed, there is little demonstrable relationship to the clinical results. In fact, the correlations can be negative, meaning that the larger the number of criteria met in the care of patients, the less favorable is the result.16 When adherence to a set of criteria cannot be shown to produce good results in individual patients, audits based on such criteria do little to promote quality of patient care. In contrast, essential criteria are indispensable to an effective audit. Essential criteria (also called critical criteria) apply to almost every patient with a specified condition because they stipulate elements of care known to produce the desired clinical results in patients with that condition. For this reason, essential criteria enable the medical staff to determine whether care of individual patients, and the results of that care, conform to contemporary high standards. The distinction be244

SEPTEMBER 1976 * 125 * 3

tween essential criteria and optimal care criteria was clarified by the efforts of Experimental Medical Care Review Organizations, originally those in Albemarle County (Virginia), Hawaii and Utah and, more recently, in Southern California."112123 The concept is promoted by the JCAH through the "critical management criteria" of its own audit format.2 Private Initiative in PSRO, a national project supported entirely by the W. K. Kellogg Foundation, is currently testing the applicability of essential criteria to concurrent monitoring of care.6 When used in a retrospective audit, essential criteria specify the objective data or information needed to: * substantiate the diagnosis and the presence or absence of complications or other conditions which influence treatment and prognosis; * document that each patient received treatment of established efficacy, given properly or to the proper end point; * document that each patient did not receive contraindicated treatment; * document that the expected clinical results were achieved in each patient. Essential criteria apply to "almost every patient" because they permit precise specification of a condition and incorporate only those elements of treatment known to be effective in producing the desired results for that condition. Essential criteria are based on the best available scientific evidence of efficacy in diagnosis and treatment. They may also be derived from the application of a basic principle. For example, the prerequisite for rational antibiotic therapy of a severe infection is identification of the causative organism, or, in urgent situations, taking the appropriate specimens before starting or modifying antibiotic treatment. The ideal criteria for an audit are purely scientific criteria derived from results of randomized clinical trials (RCT), but these are few and far between.24 One example is the RCT of length of stay for patients with uncomplicated myocardial infarction.25 Findings in this study showed that the clinical and functional results in patients in hospital for only 14 days are no different from those of patients in hospital for 21 days. Other examples of scientific criteria are those based on the results reported by the Veterans Administration Study Group on Antihypertensive Agents.26 Lowering the diastolic blood pressure of certain patients to specified levels significantly reduced

ASSURING QUALITY IN HOSPITALS TABLE 3.-Professional Standards Review Organization Definitions of the Three Basic Types of Criteria'0 PSRO Ternm

Type of Criteria

Norms ... Statistical Standards . Normative, type unspecified Criteria .... Normative: general consensus or essential Scientific

the incidence of serious or fatal complications in those patients. The data on reduced morbidity and mortality stand as scientific evidence of efficacy of lowering the blood pressure. Efficacy is what criteria for audits are all about. Scientific study establishes the degree of efficacy or effectiveness of drugs, treatments or operations in reducing mortality, preventing complications or objectively improving the patient's condition. Unfortunately, all this information is not assemblcd or published in a form that permits audit committees to pick out prespecified "scientific criteria." Instead, clinical experts must be relied on to identify the relatively few items that constitute essential criteria for therapy and the results of therapy.'; 10,2 Experts do this fairly quickly because they limit the criteria in their area of expertise to those that can be supported by scientific evidence. Many commonly used surgical and medical treatments have not been shown to be effective, and there is a growing challenge to the profession to submit them to clinical trials.24'28 As this is done, and biomedical and clinical research continue to yield more proven therapies, the scientific bases for criteria will grow. Medical auditing will then expedite the incorporation into hospital practice of the effective innovations and modifications derived from sound clinical research. One other consideration now enters discussions of criteria-namely, recent rulings on standards of reference in malpractice suits. Historically, courts have admitted expert testimony under the principle that a physician's performance should be compared with that of his peers in his own community. But in 1968 the Brune-Belinkoff decision established the precedent that competence be judged by national standards.29 The same standards apply to physicians in San Francisco and Baltimore alike. The basis for this ruling is the recognition that scientific criteria of care are generally applicable, without geographic variation. For example drug dosages do not show a regional variation, nor do the results of reducing diastolic blood pressure.

The adoption of essential or scientific criteria makes it mandatory that the audit committee define every instance of nonadherence as an important deviation. As proposed by the JCAH, anything other than 100 percent adherence to essential treatment, in the absence of an adequate justification, calls for peer review.2 It is inappropriate to set arbitrary expected performance levels when using essential criteria. However, when criteria are selected by general consensus, the threshold for corrective action has to be set at some mutually agreed upon level below 100 percent because there is usually no objectively established relationship between the processes incorporated in the criteria and actual patient results. PSRO Definitions and Guidelines for Criteria The National Professional Standards Review Council has adopted different terms for the three types of criteria, and the accompanying definitions are potentially confusing.30 As shown in Table 3, the Council refers to statistical criteria as "norms." The QAM Report of PAS has adopted this definition and refers to the 25th to 75th percentiles of its national statistics as "norms."'14 But, as discussed above, statistical criteria are only averages. Although useful to some extent in analyzing length of stay, they cannot be construed as acceptable "norms" for purposes of evaluating quality of care. PSRO refers to normative criteria as "standards" but the manual also states that "crit_ria" are derived from professional expertise and professional literature. If the criteria are derived from expert professional judgment, they can be essential criteria, based on the best available scientific evidence of efficacy. However, "professional judgment" can also be the basis of optimal care criteria, which are not suitable for evaluating the technical quality of care. Similarly, "professional literature" may refer to results of soundly conducted clinical research (scientific criteria), or to expert opinion (essential criteria) or to general consensus. These overlapping meanings can mislead some PSRO's into adopting statistical or general consensus criteria for their audits. If so, these PSRO'S would inadvertently be maintaining and reinforcing the status quo of care even though their original intent might have been to raise the quality of care. It is therefore fortunate for the PSRO program THE WESTERN JOURNAL OF MEDICINE

245

ASSURING QUALITY IN HOSPITALS

and for medical auditing in general that the American Medical Association's Task Force on Guidelines of Care for PSRO's realized the importance of the fundamental distinction between optimal care (general consensus) and essential criteria. Its first report was illustrated by several lists of the former but it offered no clear statement on their proper use. Subsequently, the Task Force did an about-face: it endorsed critical or essential criteria and cogently defined their rationale and proper use.32 Everyone concerned with medical auditing should set aside time to study these two reports side by side. This comparison will make clear the necessity to use essential or critical criteria in medical care evaluation studies directed to quality assurance. "Process" and "Outcome" Critics of "process" criteria still argue that the only valid basis of assessing medical care is the "outcome." Three things need to be said about this. First, just about everything a physician does in the care of his patients (process) can have either the desired positive effect (that is, a beneficial outcome) or a negative effect (that is, a detrimental outcome).33 Second, essential, scientific or critical management criteria are process criteria which are predictive of outcomes that may be immediately observable or long-term and not apparent for one or more years. Examples of the former are the rapid clinical recovery from properly managed nonmalignant intestinal obstruction or from diabetes with ketoacidosis. Examples of the latter are control of diastolic blood pressure in severe hypertension and internal fixation of a hip fracture. Third, and most pertinent to this discussion, outcomes cannot properly be included in an audit unless they are directly attributable to medical care (process) received in the hospital. Viewed in this light, the tiresome arguments over "process versus outcome" are irrelevant. Any process included in an audit must be related to predictable and objectively definable outcomes, and any outcomes that are examined must be directly caused by specified procedures. Therefore, essential criteria for both process and outcome must be specifiable for any audit. In practical terms, most outcome audits address preventable or treatable complications of the disease or of its surgical or medical treatment. If such are found in a higher proportion of patients than reported in the best available clinical studies, the causes of the unacceptably high rates can then be

246

SEPTEMBER 1976 *

125 * 3

discovered by applying essential (critical management) criteria to the steps taken for the prevention and management of those complications. This approach was proposed by Williamson and subsequently adopted by the JCAH Professional Evaluation Procedure (PEP) program.2'0 Even though touted as an outcome audit, PEP must limit itself to immediate outcomes caused by efficacious medical care received in the hospital. If clinical experts cannot specify essential process criteria (that is, no effective treatment exists), the topic is not suitable for an audit whose purpose is to evaluate physician performance, no matter what the analysis of outcomes shows. With the growing awareness of patient compliance as a determinant of proper management following discharge from hospital, results of patient education are now being proposed as legitimate, immediate outcomes of hospital care. These would define how well the patient should know and understand his or her own condition and his or her own responsibilities for continuing selfcare after discharge. At present, the methods for specifying and collecting such information in a dependable, useful and acceptable manner are not yet sufficiently refined for general adoption. The importance of documenting this information in selected situations seems clear.34

Continuing Medical Education In the past 20 years, the profession's own initiatives and governmental and societal pressures have combined to firmly establish CME as the third major segment of medical education for the purpose of improving medical competence and medical care.' The State of California has joined other states in a growing trend to enact legislation which requires participation in CME as a condition for the reregistration of the medical license.3' All of the 22 specialty boards of the American Board of Medical Specialties have endorsed the principle of periodic recertification. Ten have set target dates and two already offer examinations. This is powerful voluntary peer pressure to engage in self-education in order to maintain certification. The overall situation amounts to mandatory continuing education, stemming from the belief of organized medicine and state legislatures that CME assures better medical care. This belief has an obvious justification in the fact that it is impossible to remain abreast of new knowledge and techniques without an organized effort in continuing self-education. And the litera-

ASSURING QUALITY IN HOSPITALS

ture does contain many reports of improvement in medical care attributable to CME or simply feedback of information indicating substandard performance. --10.1516.l 4.36.37 Donabedian has compiled some of the previously unpublished reports to this effect.