Berapa harga 25 reminders functionality monitoring

J Am Med Inform Assoc. 2008 Jul-Aug; 15(4): 424–429.

Michael E. Matheny, MD, MS, MPH, a , b Thomas D. Sequist, MD, MPH, a , c Andrew C. Seger, PharmD, a , d Julie M. Fiskio, a , e Michael Sperling, e Don Bugbee, e David W. Bates, MD, MPH, a and Tejal K. Gandhi, MD, MPH a , ∗

Abstract

Objective

Recommendations for routine laboratory monitoring to reduce the risk of adverse medication events are not consistently followed. We evaluated the impact of electronic reminders delivered to primary care physicians on rates of appropriate routine medication laboratory monitoring.

Design

We enrolled 303 primary care physicians caring for 1,922 patients across 20 ambulatory clinics that had at least one overdue routine laboratory test for a given medication between January and June 2004. Clinics were randomized so that physicians received either usual care or electronic reminders at the time of office visits focused on potassium, creatinine, liver function, thyroid function, and therapeutic drug levels.

Measurements

Primary outcomes were the receipt of recommended laboratory monitoring within 14 days following an outpatient clinic visit. The effect of the intervention was assessed for each reminder after adjusting for clustering within clinics, as well as patient and provider characteristics.

Results

Medication-laboratory monitoring non-compliance ranged from 1.6% (potassium monitoring with potassium-supplement use) to 6.3% (liver function monitoring with HMG CoA Reductase Inhibitor use). Rates of appropriate laboratory monitoring following an outpatient visit ranged from 14% (therapeutic drug levels) to 64% (potassium monitoring with potassium-sparing diuretic use). Reminders for appropriate laboratory monitoring had no impact on rates of receiving appropriate testing for creatinine, potassium, liver function, renal function, or therapeutic drug level monitoring.

Conclusion

We identified high rates of appropriate laboratory monitoring, and electronic reminders did not significantly improve these monitoring rates. Future studies should focus on settings with lower baseline adherence rates and alternate drug-laboratory combinations.

Background

Many medications commonly used by clinicians have recommendations for routine laboratory monitoring intended to ameliorate the risk of developing an organ toxicity or electrolyte imbalance. These recommendations are put in place by the FDA in the form of black box warnings included in the product inserts, 1 or suggested by various published guidelines as a result of adverse events found in phase 3 trials or post-marketing studies. However, these lab monitoring recommendations are not routinely followed. 2–12 Approximately one-third of patients started on a new medication fail to receive appropriate baseline laboratory assessements, 13,14 with failure rates above 50% for medications such as amiodarone that require more extensive testing. 15 Laboratory monitoring for maintenance therapy represents an even larger problem than baseline monitoring, with 20% of patients not receiving renal function monitoring upon initiation of metformin therapy and over 50% failing to receive annual follow-up testing. 16 The failure to perform appropriate laboratory monitoring in the outpatient setting has been shown to translate into inpatient admissions and significant morbidity. Approximately one-quarter of preventable medication-related admissions can be attributed to failures of appropriate monitoring. 17

Electronic health records improve health care delivery by providing superior access to patient information and enhancing clinical decision-making. Point-of-care reminder systems have been shown to improve care for both chronic disease management and preventive services. 18–22 These systems might also be useful for laboratory monitoring if delivered when busy clinicians are reviewing or renewing medications in the office setting. 23,24 We conducted a prospective, randomized trial to evaluate the impact of electronic reminders delivered to primary care physicians within an electronic health record on rates of appropriate routine medication laboratory monitoring.

Methods

Study Setting

The study was conducted within the Partners HealthCare System, which includes two academic teaching hospitals (Brigham and Women's Hospital and Massachusetts General Hospital) and a number of community hospitals and outpatient clinics. Primary care physicians practicing at 20 outpatient clinics were recruited to participate, including 4 community health centers, 9 hospital-based clinics, and 7 off-site practices. All participating clinics used an ambulatory electronic health record (Longitudinal Medical Record) that allowed for maintenance of patient problem, medication, and allergy lists as well as for viewing test results and electronic medication prescribing. 25–27 The Partners Institutional Review Board approved a waiver of individual informed consent for physicians and patients in this study, as the goal of the intervention was to promote receipt of services that are widely accepted as the standard of care.

Intervention

Evidence-based guidelines were reviewed for routine medication laboratory monitoring, 28–32 and medications were selected for inclusion in the reminder system based on 1) prevalence of use, and 2) potential morbidity associated with the failure to perform appropriate laboratory monitoring. We focused on laboratory monitoring for maintenance therapy rather than medication initiation based on prior studies indicating a larger performance gap in this area. 16 This selection process was also performed to restrict information overload and resulting physician desensitization to the presented reminders. 33 Annual intervals for laboratory monitoring were chosen for all of the reminders to standardize the recommendations. We focused on laboratory monitoring of potassium, creatinine, liver function, thyroid function, and therapeutic drug levels for appropriate medications (▶). The clinical content of the electronic reminders was developed by a team of practicing general internists and pharmacists. An EHR clinical content committee did not exist at the time of this study, but has since been formed.

Table 1

Table 1 Summary of Drug-laboratory Monitoring Recommendations that were Evaluated

∗ All of the medications that required therapeutic drug monitoring were aggregated.

Medication use was based on coded information entered in the patient's electronic medication list. The dates of all laboratory tests were extracted from the Partners central data repository. Evaluation of accuracy among outpatient medication use in the LMR revealed that chronic medication use was fairly accurate, while non-prescription medication use and antibiotic use was relatively insensitive (many medications actually prescribed or taken in these categories were missing). 34,35 All of the medications included in this study were prescription medications used on a chronic basis. The decision support algorithm for the reminders was run each time a clinician opened a patient's chart and reminders were generated if a patient was on the medication for at least 365 days and there was no relevant laboratory test for that medication within the prior 365 days. For physicians in the intervention group, the reminders were displayed within the patient summary screen of the LMR with other information such as the medication list, diagnosis list, and allergies (▶). The reminders were suppressed for physicians in the control group.

Electronic reminders are displayed within the main patient summary screen alongside the patient problem and medication lists.

Randomization

We performed a stratified randomization of the 20 primary care sites based on clinic characteristics to balance the distribution of gender (women's health centers) and socioeconomic factors (academic versus community clinic) between intervention and control groups. We randomly assigned ten clinical sites to receive electronic reminders for overdue laboratory monitoring, and ten clinical sites to serve as control sites.

Subjects

Patients and physicians were enrolled during a six month period from January 01, 2004 to June 30, 2004 on the first occasion that a physician opened a patient chart during an outpatient clinic visit and the reminder algorithm identified the patient as being overdue for recommended testing. We also identified all patients with outpatient visits during the study period that were being treated with the targeted medications and were being appropriately monitored to calculate the incidence of appropriate monitoring in the overall clinic population.

Data Collection

Baseline patient and physician characteristics were obtained from administrative databases. Data on reminders were stored at the time of generation, including the date of the reminder and the treating physician. Medication therapy was identified from the coded medication list in the electronic health record. Dates and values of laboratory tests were obtained from the Partners central data repository, which included both inpatient and outpatient values, and could have been ordered by any provider.

Statistical Analysis

We defined the unit of analysis as a single outpatient clinic visit. The primary outcome was the patient's receipt of the appropriate laboratory testing within 14 days of the clinical encounter. In our institution, the laboratory tests included in this study are routinely completed within 12 to 24 hours of the time of blood specimen receipt.

All reminders for therapeutic drug level monitoring were aggregated into one category for analysis as these recommendations are conceptually distinct from recommendations that monitor drug toxicity indirectly. Multi-variable logistic regression models were used to assess the impact of the reminder system on rates of appropriate laboratory monitoring after adjusting for patient age, sex, race, and insurance status as well as provider age and sex. The GENMOD procedure within the SAS (Version 9,1, Cary, NC) statistical software package was used to fit models that also accounted for clustering of patients within clinical sites. 36 We estimated 90% power to detect a 10% absolute increase (80% to 90%) in compliance rates in the intervention arm based on a sample size of 266 clinic visits and a two sided α of 0.05.

Results

For the population-based adherence analysis, there were 21,083 patients seen by 464 physicians in 45,662 clinic visits who were on at least one of the monitored medications for one year at the time of the visit. For the randomized intervention, there were 1,922 patients seen by 303 physicians in 2,507 clinic visits during which a patient was overdue for recommended laboratory monitoring. Among those enrolled in the randomized intervention, patient race and insurance status were significantly different by intervention status, and there were no differences in physician characteristics (▶).

Table 2

Table 2 Baseline Patient and Physician Characteristics of Patients Overdue for Routine Laboratory Monitoring

The prevalence of targeted medication use varied widely among the total clinic population. The most commonly used medications were hydroxymethylglutaryl Co A reductase inhibitors (statins) (20,376 visits) and non-steroidal anti-inflammatory drugs (NSAIDs), (17,794 visits,) and the least commonly used were the drugs requiring therapeutic monitoring (1,269 visits) and potassium supplementation (1,330 visits). Medication-laboratory monitoring non-compliance (defined as no appropriate laboratory testing in the previous year) ranged from 1.6% (21/1330) in potassium supplementation to 6.3% (1287/20376) for statin use.

Rates of appropriate laboratory monitoring within 14 days of an office visit ranged from 14% for therapeutic drug levels to 64% for potassium levels in the presence of potassium-sparing diuretics. In adjusted analyses, reminders for appropriate laboratory monitoring had no impact on rates of receiving these tests for creatinine, potassium, liver function, renal function, or therapeutic drug level monitoring (▶).

Table 3

Table 3 Population Guideline Adherence and Impact of Electronic Reminders on Medication Laboratory Monitoring

Discussion

There is a substantial body of evidence documenting performance gaps related to appropriate monitoring of laboratory tests in the presence of medication use, as well as the poor outcomes associated with such shortcomings. Our study found only a relatively small proportion (approximately 5%) of patients on targeted medications were overdue for appropriate laboratory monitoring at the time of office visits with primary care physicians, and that nearly half of these patients had appropriate testing within two weeks of the office visit. Electronic reminders to physicians were not effective in increasing the rates of appropriate laboratory monitoring in this setting.

The prevalence of overdue routine laboratory monitoring among patients in this study was low compared to previous work in this area, where other investigators have documented much higher rates of overdue creatinine testing with non-steroidal anti-inflammatory medication use (30%) and metformin use (52%). 15,16 These low rates of overdue testing may relate to the fact that our health system employs a unified clinical data repository across all outpatient and inpatient settings, and thus captures lab values from episodes of clinical care that may not be available in settings that lack such a comprehensive system. Alternatively, the low overdue rate might reflect the benefits of an electronic medical record system that increases access to patient information such as up to date medication lists. While we did not collect additional data on redundant testing, it is important to note that these high rates of testing likely come at the expense of over-utilization, as prior studies have documented rates of redundant inpatient laboratory testing within our system of nearly 10%. 37

While a number of studies have shown that computerized reminders are efficacious in the inpatient setting, their use in the outpatient setting is less well described. A few randomized trials have shown improvements in ordering appropriate laboratory tests upon medication initiation with the use of outpatient computerized reminders. 38–40 However, only one previous randomized trial has evaluated the effect of outpatient computerized reminders on recommended routine monitoring. In an Israeli healthcare provider network, approximately 20% of patients taking diuretics were overdue for an annual potassium check, and reminders were effective in increasing monitoring compliance. 41

Barriers to provider utilization of clinical reminders are limitations of our study that might have contributed to the negative findings. For example, our previous work indicates that only one-third of providers in our system notice reminders during clinical encounters. 25 The reminders are visible from the patient summary screen, which is a common portal for physicians. However, since our reminders are passive, it is possible for a clinician to not use this screen and thereby circumvent the reminder system. Physicians may also be less likely to utilize reminder systems as workload increases. 42 While inaccuracy of recommendations is often cited as a barrier to effective clinical decision support tools, 43 this was unlikely in our system as we were able to capture both laboratory results and medication prescriptions in a relatively complete manner. Other studies have noted decreases in the use of reminders over the course of a study, attributed to reminder fatigue. 44,45 Clinical content could also have limited the utility of the reminders if physicians felt that the recommendations were unnecessary, or if the interval was too frequent. In order to address these issues in our evaluation, we chose a conservative monitoring interval of one year and drug-lab recommendations in which there were clear FDA concerns or significant adverse effects from a lack of monitoring. Laboratory test values obtained outside our health system were not electronically available, but could have been known by providers. This could falsely increase overdue rates and contribute to apparent reminder non-compliance by displaying reminders for labs appropriately performed elsewhere. However, it is important to note that the vast majority of our patients' labs are performed within the Partners system, and would therefore show up in our Central Data Repository. In addition, patient non-compliance is included in reminder non-compliance, since only ordering a laboratory test is insufficient to fulfill guideline compliance. Finally, clinicians who viewed the reminders still needed to order the lab tests in a separate step using a paper lab ordering form. This additional step could have been an obstacle to taking action based on the reminder.

A final explanatory mechanism for our failure to demonstrate an effect with the reminder system is the ceiling effect associated with the high baseline rates of adherence. However, a post-hoc power calculation using higher rates of compliance (90% baseline rate, detecting 5% absolute difference) required 582 clinic visits per arm to achieve 90% power, suggesting adequate power among the reminders in this study to detect such a treatment effect.

In summary, we found relatively high rates of appropriate laboratory monitoring in our study population, and an electronic reminder system did not significantly improve rates of such testing. Future studies should focus on laboratory monitoring for different classes of medications, laboratory monitoring at medication initiation, and settings with lower rates of baseline adherence to monitoring recommendations.

Acknowledgments

The authors thank the study participants, including the patients and primary care physicians within the Partners HealthCare System. We also thank Anne Fladger and her staff for their assistance. This work was funded in part by grant 5-U18-HS-011046 from the Agency for Healthcare Research and Quality and by grant 1-T15-LM-07092 from the National Library of Medicine.

References

1. Beach JE, Faich GA, Bormel FG, Sasinowski FJ. Black box warnings in prescription drug labeling: results of a survey of 206 drugs Food & Drug Law Journal 1998;53(3):403-411. [PubMed] [Google Scholar]

2. Schoenenberger RA, Tanasijevic MJ, Jha A, Bates DW. Appropriateness of antiepileptic drug level monitoring[see comment] JAMA 1995;274(20):1622-1626Nov 22–29. [PubMed] [Google Scholar]

3. Graham DJ, Drinkard CR, Shatin D, Tsong Y, Burgess MJ. Liver enzyme monitoring in patients treated with troglitazone JAMA 2001;286(7):831-833Aug 15. [PubMed] [Google Scholar]

4. Calabrese AT, Coley KC, DaPos SV, Swanson D, Rao RH. Evaluation of prescribing practices: risk of lactic acidosis with metformin therapy Arch Intern Med 2002;162(4):434-437Feb 25. [PubMed] [Google Scholar]

5. Roblin DW, Nielsen DM. Assessment of quality in adult primary care: developing process measures from administrative data Clin Perform Qual Health Care 1994;2:200-208. [Google Scholar]

6. Emslie-Smith AM, Boyle DI, Evans JM, Sullivan F, Morris AD, Collaboration DM Contraindications to metformin therapy in patients with Type 2 diabetes—a population-based study of adherence to prescribing guidelines Diabetic Medicine 2001;18(6):483-488Jun. [PubMed] [Google Scholar]

7. Abookire SA, Karson AS, Fiskio J, Bates DW. Use and monitoring of “statin” lipid-lowering drugs compared with guidelines[see comment] Arch Intern Med 2001;161(1):53-58Jan 8. [PubMed] [Google Scholar]

8. Tegeder I, Levy M, Muth-Selbach U, et al. Retrospective analysis of the frequency and recognition of adverse drug reactions by means of automatically recorded laboratory signals Br J Clin Pharmacol 1999;47(5):557-564May. [PMC free article] [PubMed] [Google Scholar]

9. Selby JV, Ettinger B, Swain BE, Brown JB. First 20 months' experience with use of metformin for type 2 diabetes in a large health maintenance organization Diabetes Care 1999;22(1):38-44Jan. [PubMed] [Google Scholar]

10. Stelfox HT, Ahmed SB, Fiskio J, Bates DW. Monitoring amiodarone's toxicities: recommendations, evidence, and clinical practice Clinical Pharmacology & Therapeutics 2004;75(1):110-122Jan. [PubMed] [Google Scholar]

11. Lasser KE, Seger DL, Yu DT, Karson AS, Fiskio JM, Seger AC, et al. Adherence to black box warnings for prescription medications in outpatients Arch Intern Med 2006;166(3):338-344Feb 13. [PubMed] [Google Scholar]

12. Gurwitz JH, Field TS, Harrold LR, Rothschild J, Debellis K, Seger AC, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting JAMA 2003;289(9):1107-1116Mar 5. [PubMed] [Google Scholar]

13. Simon SR, Andrade SE, Ellis JL, et al. Baseline Laboratory Monitoring of Cardiovascular Medications in Elderly Health Maintenance Organization Enrollees J Am Ger Soc 2005;53(12):2165-2169. [PubMed] [Google Scholar]

14. Raebel MA, Lyons EE, Andrade SE, Chan KA, Chester EA, Davis RL, et al. Laboratory Monitoring of Drugs at Initiation of Therapy in Ambulatory Care J Gen Intern Med 2005;20(12):1120-1126Dec. [PMC free article] [PubMed] [Google Scholar]

15. Patino FG, Olivieri J, Allison JJ, Mikuls TR, Moreland L, Kovac SH, et al. Nonsteroidal antiinflammatory drug toxicity monitoring and safety practices Journal of Rheumatology 2003;30(12):2680-2688Dec. [PubMed] [Google Scholar]

16. Choe HM, Cornish L, Townsend K, Jobe J, Mitrovich S. Monitoring safety and effectiveness in patients receiving metformin Am J Health-Syst Pharm 2004;61(15):1550-1551Aug 1. [PubMed] [Google Scholar]

17. Howard RL, Avery AJ, Howard PD, Partridge M. Investigation into the reasons for preventable drug related admissions to a medical admissions unit: observational study Qual Saf Health Care 2003;12(4):280-285August 1. [PMC free article] [PubMed] [Google Scholar]

18. Bates DW, Ebell M, Gotlieb E, Zapp J, Mullins HC. A proposal for electronic medical records in U.S. primary care J Am Med Inform Assoc 2003;10(1):1-10Jan-Feb. [PMC free article] [PubMed] [Google Scholar]

19. Bodenheimer T, Grumbach K. Electronic technology: a spark to revitalize primary care? JAMA 2003;290(2):259-264Jul 9. [PubMed] [Google Scholar]

20. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review[see comment] JAMA 1998;280(15):1339-1346Oct 21. [PubMed] [Google Scholar]

21. Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology[see comment] J Am Med Inform Assoc 2001;8(4):299-308Jul-Aug. [PMC free article] [PubMed] [Google Scholar]

22. Schiff GD, Klass D, Peterson J, Shah G, Bates DW. Linking laboratory and pharmacy: opportunities for reducing errors and improving care Arch Intern Med 2003;163(8):893-900Apr 28. [PubMed] [Google Scholar]

23. Boohaker EA, Ward RE, Uman JE, McCarthy BD. Patient notification and follow-up of abnormal test results. A physician survey. [see comment] Arch Intern Med 1996;156(3):327-331Feb 12. [PubMed] [Google Scholar]

24. Poon EG, Gandhi TK, Sequist TD, Murff HJ, Karson AS, Bates DW. “I wish I had seen this test result earlier!” Dissatisfaction with test result management systems in primary care Arch Intern Med 2004;164(20):2223-2228Nov 8. [PubMed] [Google Scholar]

25. Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease J Am Med Inform Assoc 2005;12(4):431-437Jul-Aug. [PMC free article] [PubMed] [Google Scholar]

26. Poon EG, Wald J, Bates DW, Middleton B, Kuperman GJ, Gandhi TK. Supporting patient care beyond the clinical encounter: three informatics innovations from partners health care AMIA Annu Symp Proc 2003:1072. [PMC free article] [PubMed]

27. Maviglia SM, Teich JM, Fiskio J, Bates DW. Using an electronic medical record to identify opportunities to improve compliance with cholesterol guidelines J Gen Intern Med 2001;16(8):531-537Aug. [PMC free article] [PubMed] [Google Scholar]

28. American Diabetes A Standards of Medical Care in Diabetes Diabetes Care 2005;28(suppl_1):S4-S36Jan 1. [PubMed] [Google Scholar]

29. Henrich WL, Agodoa LE, Barrett B, Bennett WM, Blantz RC, Buckalew Jr. VM, et al. Analgesics and the kidney: summary and recommendations to the Scientific Advisory Board of the National Kidney Foundation from an Ad Hoc Committee of the National Kidney Foundation[see comment] Am J Kidney Dis 1996;27(1):162-165Jan. [PubMed] [Google Scholar]

30. Chobanian AV, Bakris GL, Black HR, Cushman WC, Green LA, Izzo Jr. JL, et al. The Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure: the JNC 7 report[see comment] [erratum appears in JAMA. 2003 Jul 9;290(2):197] JAMA 2003;289(19):2560-2572May 21. [PubMed] [Google Scholar]

31. Warner A, TA TA, eds. Guidelines for Therapeutic Drug Monitoring Services. 1999.

32. Expert consensus guideline series. Optimizing pharmacologic treatment of psychotic disorders. J Clin Psych 2003;64(Suppl 12):2-9798–100. [PubMed] [Google Scholar]

33. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry J Am Med Inform Assoc 2006;13(2):138-147Mar-Apr. [PMC free article] [PubMed] [Google Scholar]

34. Linder JA, Bates DW, Williams DH, Connolly MA, Middleton B. Acute infections in primary care: accuracy of electronic diagnoses and electronic antibiotic prescribing J Am Med Inform Assoc 2006;13(1):61-66Jan-Feb. Epub 2005 Oct 2012. [PMC free article] [PubMed] [Google Scholar]

35. Staroselsky M, Volk LA, Tsurikova R, Newmark LP, Lippincott M, Litvak I, et al. An effort to improve electronic health record medication list accuracy between visits: Patients' and physicians' response Int J Med Inform 2008;77(3):153-160Mar. Epub 2007 Apr 2016. [PubMed] [Google Scholar]

36. Localio AR, Berlin JA, Ten Have TR, Kimmel SE. Adjustments for center in multicenter studies: an overview Ann Intern Med 2001;135(2):112-123Jul 17. [PubMed] [Google Scholar]

37. Bates DW, Boyle DL, Rittenberg E, Kuperman GJ, Ma'Luf N, Menkin V, et al. What proportion of common diagnostic tests appear redundant? Am J Med 1998;104(4):361-368Apr. [PubMed] [Google Scholar]

38. Raebel MA, Lyons EE, Chester EA, Bodily MA, Kelleher JA, Long CL, et al. Improving Laboratory Monitoring at Initiation of Drug Therapy in Ambulatory Care Arch Intern Med 2005;165:2395-2401November 14. [PubMed] [Google Scholar]

39. Feldstein AC, Smith DH, Perrin N, Yang X, Rix M, Raebel MA, et al. Improved therapeutic monitoring with several interventions: a randomized trial Arch Intern Med 2006;166(17):1848-1854Sep 25. [PubMed] [Google Scholar]

40. Palen TE, Raebel M, Lyons E, Magid DM. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders[see comment] Am J Manag Care 2006;12(7):389-395Jul. [PubMed] [Google Scholar]

41. Hoch I, Heymann AD, Kurman I, Valinsky LJ, Chodick G, Shalev V. Countrywide computer alerts to community physicians improve potassium testing in patients receiving diuretics J Am Med Inform Assoc 2003;10(6):541-546Nov-Dec. Epub 2003 Aug 2004. [PMC free article] [PubMed] [Google Scholar]

42. Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM. Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods J Biomed Inform 2005;38(3):189-199Jun. [PubMed] [Google Scholar]

43. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality J Am Med Inform Assoc 2003;10(6):523-530Nov-Dec. Epub 2003 Aug 2004. [PMC free article] [PubMed] [Google Scholar]

44. Bindels R, Hasman A, van Wersch JW, Talmon J, Winkens RA. Evaluation of an automated test ordering and feedback system for general practitioners in daily practice Int J Med Inform 2004;73(9–10):705-712Sep. [PubMed] [Google Scholar]

45. Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, et al. Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders JAMA 2000;284(11):1411-1416Sep 20. [PubMed] [Google Scholar]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press