Wagner, S. (2020): Development and application of comprehensive evaluation criteria for efficient hospital information systems.
Highlights
- Evaluation of comprehensive practical requirements of hospital information systems
- Previous catalogues for screening due to smaller scope and lower level of detail
- Precise and specific analysis to derive concrete improvement potentials
- Criteria for improvement of existing implementations and selection of software
Introduction
Materials and methods
Results
Discussion
Conclusions
Summary points
Abstract
Background: Data processing is a significant productivity and quality factor for each hospital. Inefficient processes with a lack of structured documentation, transfer, reuse of data and automation lead to delays, loss of time, cost explosion and reduced quality despite improvements of the last years. Doctors spend several hours a day on documentation and still find it difficult to obtain complete information. Integrated hospital information systems can overcome these problems by accelerating access, increasing quality of communication, ensuring timely availability and positive effects on process and result quality. Evaluations of information systems are the prerequisite for development, high-quality as well as efficient care. Available criteria are often too general and not comprehensive enough to be applied directly to a hospital with the goal of global efficiency improvement or concentrate only on single aspects, institutions or user groups.
Objective: Evaluation criteria for efficient hospital information systems including practical requirements of all departments and user groups are the key for achieving an efficient daily working routine emphasizing interdisciplinary cooperation and data reuse.
Methods: Development of the evaluation criteria was based on a combination of intensive literature analysis, hidden external observation for inclusion of practical requirements of doctors and nurses from all departments of a big German hospital, derivation from system goals and a profound product analysis of a market-leading hospital information system including strong and weak points. The creativity method brainstorming was used for collection and structuring of practical requirements. Available third-party systems, paper documentation and extensions were also analyzed in depth. The local implementation at a typical German hospital was compared to the most comprehensive one including all available features and the intended state of maximum efficiency.
Results: A total of 1,102 criteria were developed and tested successfully at a German hospital. Criteria were divided into 959 task-related and 143 cross-functional requirements, structured in tasks and subtasks. Development of criteria and evaluation of the hospital information system covered a master’s thesis in the Master of Health Business Administration (MHBA) study program comprising 347 pages with 240 illustrations, 28 tables, 92 annexes and 1,085 footnotes. With regard to task-related requirements local implementation achieved 2,285 points (79 %), the maximum possible functionality 2,843 points (99 %). For the cross-functional criteria the on-site implementation achieved 379 points (88 %), the most comprehensive one 426 points (99 %). Compared to the overall amount of functions, only a small number of modules was licensed. At a glance, it became clear how efficiency increases could be achieved through further modules, addons and adjustments. Lack of a medical knowledge database for decision support in addition to media discontinuities of documentation in anesthesia and intensive care was of particular importance. Solutions for rising efficiency and elimination of discovered weaknesses were pointed out.
Conclusion: Previous catalogues for evaluation criteria can be used as a first screening solution due to smaller scope and lower level of detail. Comprehensive criteria can be used in a subsequent step for a more precise and specific analysis to derive concrete improvement potentials contributing to an increase in efficiency of everyday hospital life and interdisciplinary collaboration. Unique are the focus on a primary consideration of comprehensive practical requirements and reduction of cross-functional requirements to a minimum. Covering all practical requirements of stakeholders at a hospital is possible by shifting the focus to the specific needs of clinical end users. Specialized systems for departments are no longer necessary if integrated functions of the information system or addons are capable of replacing these at least equivalently. Parameterization and personalization remain important. After all, systems offer numerous functions by default. Many hospital information systems also allow constantly expanding with addons, thereby increasing efficiency. Evaluation criteria depend on use case, objectives and evaluators. Nevertheless, with minor modifications, criteria can be transferred to other institutions, both for evaluation of existing implementations and acquisition of new software. Doctors, nursing staff and patients benefit from documentation support and workload reduction.
Keywords
Hospital information system, evaluation, assessment, evaluation criteria, efficiency
Literatur
[1] S. Wagner, M.W. Beckmann, B. Wullich, C. Seggewies, M. Ries, T. Bürkle, H.-U. Prokosch, Analysis and classification of oncology activities on the way to workflow based single source documentation in clinical information systems, BioMed Central (BMC) medical informatics and decision making 15 (2015) 107. https://doi.org/10.1186/s12911-015-0231-x.
[2] L. Ahmadian, S.S. Nejad, R. Khajouei, Evaluation methods used on health information systems (HISs) in Iran and the effects of HISs on Iranian healthcare: a systematic review, Int. J. Med. Inform. 84 (2015) 444–453. https://doi.org/10.1016/j.ijmedinf.2015.02.002.
[3] A. Winter, K. Takabayashi, F. Jahn, E. Kimura, R. Engelbrecht, R. Haux, M. Honda, U.H. Hübner, S. Inoue, C.D. Kohl, T. Matsumoto, Y. Matsumura, K. Miyo, N. Nakashima, H.-U. Prokosch, M. Staemmler, Quality Requirements for Electronic Health Record Systems. A Japanese-German Information Management Perspective, Methods Inf. Med. 56 (2017) e92-e104. https://doi.org/10.3414/ME17-05-0002.
[4] L.C. Roman, J.S. Ancker, S.B. Johnson, Y. Senathirajah, Navigation in the electronic health record: A review of the safety and usability literature, J. Biomed. Inform. 67 (2017) 69–79. https://doi.org/10.1016/j.jbi.2017.01.005.
[5] F. Sultan, M.T. Aziz, I. Khokhar, H. Qadri, M. Abbas, A. Mukhtar, W. Manzoor, M.A. Yusuf, Development of an in-house hospital information system in a hospital in Pakistan, Int. J. Med. Inform. 83 (2014) 180–188. https://doi.org/10.1016/j.ijmedinf.2013.12.004.
[6] V.P. Aggelidis, P.D. Chatzoglou, Hospital information systems: measuring end user computing satisfaction (EUCS), J. Biomed. Inform. 45 (2012) 566–579. https://doi.org/10.1016/j.jbi.2012.02.009.
[7] E. Ammenwerth, F. Kaiser, I. Wilhelmy, S. Höfer, Evaluation of user acceptance of information systems in health care – the value of questionnaires, Stud. Health Technol. Inform. 95 (2003) 643–648.
[8] B. Gugerty, M. Maranda, D. Rook, The clinical information system implementation evaluation scale, Stud. Health Technol. Inform. 122 (2006) 621–625.
[9] J.D. Liebe, U. Hübner, M.C. Straede, J. Thye, Developing a Workflow Composite Score to Measure Clinical Information Logistics. A Top-down Approach, Methods Inf. Med. 54 (2015) 424–433. https://doi.org/10.3414/ME14-02-0025.
[10] B. Aldosari, Supportive care pathway functionalities of EHR system in a Saudi Arabian hospital, Comput. Biol. Med. 89 (2017) 190–196. https://doi.org/10.1016/j.compbiomed.2017.08.012.
[11] E. Ammenwerth, S. Gräber, G. Herrmann, T. Bürkle, J. König, Evaluation of health information systems-problems and challenges, Int. J. Med. Inform. 71 (2003) 125–135.
[12] H.K. Selbmann, B. Pietsch-Breitfeld, Hospital information systems and quality assurance, Quality assurance in health care 2 (1990) 335–344.
[13] E. Ammenwerth, F. Ehlers, B. Hirsch, G. Gratl, HIS-Monitor: an approach to assess the quality of information processing in hospitals, Int. J. Med. Inform. 76 (2007) 216–225. https://doi.org/10.1016/j.ijmedinf.2006.05.004.
[14] F. Ehlers, E. Ammenwerth, B. Hirsch, Design and development of a monitoring system to assess the quality of hospital information systems: concept and structure, Stud. Health Technol. Inform. 116 (2005) 575–580.
[15] U. Hübner, J.-D. Liebe, M. Esdar, J. Hüsers, J. Rauch, J. Thye, J.-P. Weiß, Stand der Digitalisierung und des Technologieeinsatzes in deutschen Krankenhäusern, in: J. Klauber, M. Geraedts, J. Friedrich, J. Wasem (Eds.), Krankenhaus-Report 2019: Das digitale Krankenhaus, Springer, Berlin, 2019, pp. 33–48.
[16] Wissenschaftliches Institut der AOK, Krankenhaus-Report 2019. Digitalisierungs-Rückstand in deutschen Krankenhäusern, 2019. https://www.wido.de/fileadmin/Dateien/Dokumente/News/Pressemitteilungen/2019/wido_kra_pm_krankenhaus-report_2019.pdf (accessed 4 April 2019).
[17] T. Maybaum, Krankenhäuser: Deutlicher Rückstand bei der Digitalisierung, Dtsch. Arztebl. Int. 116 (2019) A 666.
[18] M. Wallenfels, Dokumentation: Zeitfresser für Klinikärzte und Pflegepersonal, Ärzte Zeitung (2015) 19.
[19] K. Blum, U. Müller, Krankenhausärzte: Enormer Dokumentationsaufwand. Ergebnisse einer Untersuchung des Deutschen Krankenhausinstituts, Dtsch. Arztebl. Int. 100 (2003) A 1581.
[20] J. Wolff, G. Auber, T. Schober, F. Schwär, K. Hoffmann, M. Metzger, A. Heinzmann, M. Krüger, C. Normann, G. Gitsch, N. Südkamp, T. Reinhard, M. Berger, Work-Time Distribution of Physicians at a German University Hospital, Dtsch. Arztebl. Int. 114 (2017) A 705-A 711. https://doi.org/10.3238/arztebl.2017.0705.
[21] J. Oswald, K. Goedereis, Voraussetzungen und Potenziale des digitalen Krankenhauses, in: J. Klauber, M. Geraedts, J. Friedrich, J. Wasem (Eds.), Krankenhaus-Report 2019: Das digitale Krankenhaus, Springer, Berlin, 2019, pp. 49–66.
[22] R.-F. Chen, J.-L. Hsiao, An investigation on physicians’ acceptance of hospital information systems: a case study, Int. J. Med. Inform. 81 (2012) 810–820. https://doi.org/10.1016/j.ijmedinf.2012.05.003.
[23] D. Garcia-Smith, J.A. Effken, Development and initial evaluation of the Clinical Information Systems Success Model (CISSM), Int. J. Med. Inform. 82 (2013) 539–552. https://doi.org/10.1016/j.ijmedinf.2013.01.011.
[24] P. Carayon, T.B. Wetterneck, B. Alyousef, R.L. Brown, R.S. Cartmill, K. McGuire, P.L.T. Hoonakker, J. Slagle, K.S. van Roy, J.M. Walker, M.B. Weinger, A. Xie, K.E. Wood, Impact of electronic health record technology on the work and workflow of physicians in the intensive care unit, Int. J. Med. Inform. 84 (2015) 578–594. https://doi.org/10.1016/j.ijmedinf.2015.04.002.
[25] M.J. Ball, R.I. O’Desky, J.V. Douglas, Status and progress of hospital information systems (HIS), Int. J. Biomed. Comput. 29 (1991) 161–168.
[26] R. Lenz, P. Schmücker, Stete Weiterentwicklung nötig – Integration heterogener IT-Systeme im Krankenhaus, Der Klinikarzt 32 (2003) 385–390. https://doi.org/10.1055/s-2003-44532.
[27] L. Nguyen, E. Bellucci, L.T. Nguyen, Electronic health records implementation: an evaluation of information system impact and contingency factors, Int. J. Med. Inform. 83 (2014) 779–796. https://doi.org/10.1016/j.ijmedinf.2014.06.011.
[28] H. Mosser, M. Urban, M. Dürr, W. Rüger, W. Hruby, Integration of radiology and hospital information systems (RIS, HIS) with PACS: requirements of the radiologist, Eur. J. Radiol. 16 (1992) 69–73.
[29] N. Clynch, J. Kellett, Medical documentation: part of the solution, or part of the problem? A narrative review of the literature on the time spent on and value of medical documentation, Int. J. Med. Inform. 84 (2015) 221–228. https://doi.org/10.1016/j.ijmedinf.2014.12.001.
[30] P. Bobbert, Digitalisierung im Gesundheitswesen aus Sicht des Marburger Bundes. Mehr Chancen als Risiken, 2018. https://www.dmea.de/media/cit/cit_dl_vortraege/archiv_vortraege_2018/Bobbert_Peter_-_Digitalisierung_im__Gesundheitswesen_aus_Sicht_des_Marburger_Bundes.pdf (accessed 11 May 2019).
[31] J. Kaipio, T. Lääveri, H. Hyppönen, S. Vainiomäki, J. Reponen, A. Kushniruk, E. Borycki, J. Vänskä, Usability problems do not heal by themselves: National survey on physicians’ experiences with EHRs in Finland, Int. J. Med. Inform. 97 (2017) 266–281. https://doi.org/10.1016/j.ijmedinf.2016.10.010.
[32] M.M. Yusof, A. Papazafeiropoulou, R.J. Paul, L.K. Stergioulas, Investigating evaluation frameworks for health information systems, Int. J. Med. Inform. 77 (2008) 377–385. https://doi.org/10.1016/j.ijmedinf.2007.08.004.
[33] J.R. Krobock, A taxonomy: hospital information systems evaluation methodologies, J. Med. Syst. 8 (1984) 419–429.
[34] T. Bürkle, E. Ammenwerth, H.U. Prokosch, J. Dudeck, Evaluation of clinical information systems. What can be evaluated and what cannot?, J. Eval. Clin. Pract. 7 (2001) 373–385.
[35] C. Bräutigam, Auswirkung der Digitalisierung auf die Arbeitswelt: Mitarbeiter unzureichend beteiligt, Klinik Management aktuell (kma) – Das Gesundheitswirtschaftsmagazin 24 (2019) 74–75. https://doi.org/10.1055/s-0039-1692810.
[36] E. Ammenwerth, A. Buchauer, R. Haux, A requirements index for information processing in hospitals, Methods Inf. Med. 41 (2002) 282–288.
[37] G. Hübner-Bloder, E. Ammenwerth, Key performance indicators to benchmark hospital information systems – a delphi study, Methods Inf. Med. 48 (2009) 508–518. https://doi.org/10.3414/ME09-01-0044.
[38] B. Brigl, G. Hübner-Bloder, T. Wendt, R. Haux, A. Winter, Architectural quality criteria for hospital information systems, Proc. AMIA. Symp. (2005) 81–85.
[39] L. Lapointe, M. Mignerat, I. Vedel, The IT productivity paradox in health: a stakeholder’s perspective, Int. J. Med. Inform. 80 (2011) 102–115. https://doi.org/10.1016/j.ijmedinf.2010.11.004.
[40] A.W. Kushniruk, V.L. Patel, Cognitive and usability engineering methods for the evaluation of clinical information systems, J. Biomed. Inform. 37 (2004) 56–76. https://doi.org/10.1016/j.jbi.2004.01.003.
[41] A. Kushniruk, J. Kaipio, M. Nieminen, H. Hyppönen, T. Lääveri, C. Nohr, A.M. Kanstrup, M. Berg Christiansen, M.-H. Kuo, E. Borycki, Human Factors in the Large: Experiences from Denmark, Finland and Canada in Moving Towards Regional and National Evaluations of Health Information System Usability. Contribution of the IMIA Human Factors Working Group, Yearb. Med. Inform. 9 (2014) 67–81. https://doi.org/10.15265/IY-2014-0023.
[42] M.A. Ellsworth, M. Dziadzko, J.C. O’Horo, A.M. Farrell, J. Zhang, V. Herasevich, An appraisal of published usability evaluations of electronic health records via systematic review, J. Am. Med. Inform. Assoc. 24 (2017) 218–226. https://doi.org/10.1093/jamia/ocw046.
[43] A. Eslami Andargoli, H. Scheepers, D. Rajendran, A. Sohal, Health information systems evaluation frameworks: A systematic review, Int. J. Med. Inform. 97 (2017) 195–209. https://doi.org/10.1016/j.ijmedinf.2016.10.008.
[44] C.R. Dias, M.R. Pereira, A.P. Freire, Qualitative review of usability problems in health information systems for radiology, J. Biomed. Inform. 76 (2017) 19–33. https://doi.org/10.1016/j.jbi.2017.10.004.
[45] M. Farzandipour, Z. Meidani, M.S. Jabali, R.D. Bnadaki, Designing and evaluating functional laboratory information system requirements integrated to hospital information systems, J. Eval. Clin. Pract. (2018) 1–12. https://doi.org/10.1111/jep.13074.
[46] E. Ammenwerth, F. Rauchegger, F. Ehlers, B. Hirsch, C. Schaubmayr, Effect of a nursing information system on the quality of information processing in nursing: An evaluation study using the HIS-monitor instrument, Int. J. Med. Inform. 80 (2011) 25–38. https://doi.org/10.1016/j.ijmedinf.2010.10.010.
[47] K.M. Baumlin, J.S. Shapiro, C. Weiner, B. Gottlieb, N. Chawla, L.D. Richardson, Clinical information system and process redesign improves emergency department efficiency, Jt. Comm. J. Qual. Patient Saf. 36 (2010) 179–185.
[48] M.C. Beuscart-Zéphir, L. Watbled, A.M. Carpentier, M. Degroisse, O. Alao, A rapid usability assessment methodology to support the choice of clinical information systems: a case study, Proc. AMIA. Symp. (2002) 46–50.
[49] M.-C. Beuscart-Zéphir, H. Menu, F. Evrard, S. Guerlinger, L. Watbled, F. Anceaux, Multidimensional evaluation of a Clinical Information System for anaesthesiology: quality management, usability, and performances, Stud. Health Technol. Inform. 95 (2003) 649–654.
[50] F. Woernle, Software in der Geburtshilfe. Anforderungen aus Sicht des Klinikers, Der Gynäkologe 43 (2010) 411–415. https://doi.org/10.1007/s00129-009-2494-8.
[51] A. Sheikhtaheri, K. Kimiafar, M. Sarbaz, Evaluation of system quality of hospital information system: a case study on nurses’ experiences, Stud. Health Technol. Inform. 205 (2014) 960–964.
[52] C. Oroviogoicoechea, B. Elliott, R. Watson, Review: evaluating information systems in nursing, J. Clin. Nurs. 17 (2008) 567–575. https://doi.org/10.1111/j.1365-2702.2007.01985.x.
[53] C. Oroviogoicoechea, R. Watson, A quantitative analysis of the impact of a computerised information system on nurses’ clinical practice using a realistic evaluation framework, Int. J. Med. Inform. 78 (2009) 839–849. https://doi.org/10.1016/j.ijmedinf.2009.08.008.
[54] S. Lee, A multimethod approach for quality evaluation of clinical information systems, Computers, informatics, nursing (CIN) 30 (2012) 19–28. https://doi.org/10.1097/NCN.0b013e31822b8783.
[55] Y.-W. Fang, C.-P. Li, M.-H. Wang, The development and evaluation of a nursing information system for caring clinical in-patient, Technology and health care 24 Suppl 1 (2015) S401-6. https://doi.org/10.3233/THC1106.
[56] J. Mau, kma Online. Marktübersicht. Die Top 7 KIS-Anbieter 2017, 2017. https://www.kma-online.de/themenwelten/dmea/artikel/detail/die-top-7-kis-anbieter-2017-a-34535 (accessed 4 April 2019).
[57] R. Haux, E. Ammenwerth, A. Buchauer, Anforderungskatalog für die Informationsverarbeitung im Krankenhaus. Version 1.0, 2001. https://iig.umit.at/images/projects/documents/anforderungskatalog.pdf (accessed 4 April 2019).
[58] J. Horsky, K. McColgan, J.E. Pang, A.J. Melnikas, J.A. Linder, J.L. Schnipper, B. Middleton, Complementary methods of system usability evaluation: surveys and observations during software design and development cycles, J. Biomed. Inform. 43 (2010) 782–790. https://doi.org/10.1016/j.jbi.2010.05.010.
[59] E. Nabovati, H. Vakili-Arki, S. Eslami, R. Khajouei, Usability evaluation of Laboratory and Radiology Information Systems integrated into a hospital information system, J. Med. Syst. 38 (2014) 35. https://doi.org/10.1007/s10916-014-0035-z.
[60] S. Wagner, Bewertungskriterien für Krankenhausinformationssysteme (KIS): Freie wissenschaftliche Arbeit zur Erlangung des akademischen Grades „Master of Health Business Administration (MHBA)“, Friedrich-Alexander-Universität Erlangen-Nürnberg, Nürnberg, 2019.
[61] Marburger Bund Bundesverband, Check IT, 2019. https://mb-checkit.de (accessed 31 May 2019).
[62] Marburger Bund Bundesverband, Check IT. Das Marburger Bund-Analysetool zum digitalen Krankenhaus, 2019. https://www.marburger-bund.de/sites/default/files/files/2019-05/MB_flyer_CheckIT_Web.pdf (accessed 31 May 2019).
[63] Bundesverband Gesundheits-IT, Marburger Bund und bvitg erarbeiten eine gemeinsame Checkliste zum Digitalisierungsgrad von Krankenhäusern, 2018. https://www.bvitg.de/marburger-bund-und-bvitg-erarbeiten-checkliste-zum-digitalen-reifegrad-der-krankenhaeuser (accessed 12 May 2019).