Before the next investment
What evidence would help your team decide whether to continue, adapt, or scale?
HIMSS BOOK SERIES · SECOND EDITION
Health Information Technology Evaluation Handbook
From Meaningful Use
to Meaningful Outcomes
A practical book for evaluating health information technology, including artificial intelligence (AI) and machine learning (ML)—with foundations relevant to today’s large language model (LLM) tools.
A Productivity Press book · Routledge / Taylor & Francis
Second edition · © 2022
Hardcover ISBN: 978-0-367-48821-5
A HIMSS BOOK SERIES TITLE
The second edition
11 chapters · Print & digital formats
HARDCOVER ISBN978-0-367-48821-5
PAPERBACK ISBN978-1-032-13307-2
eBOOK ISBN978-1-003-04296-9
HIMSS Book Series · Second edition · DOI: 10.4324/9781003042969 ↗
STUDY DESIGNSAFETY & COSTUSABILITYAI & MACHINE LEARNING
BEYOND ADOPTION
Technology is an investment.
Impact is the question.
A new system is live. What should you ask next?
Start with the decisions you need to make: what to measure, whose experience matters, and what evidence would justify the next step.
A practical reference for setting evaluation questions, choosing study designs and measures, interpreting results, and communicating findings. Explore safety, cost, effectiveness, usability, and AI/ML through a clinical lens. These evaluation questions also provide a starting point for assessing newer generative AI and LLM applications.
AI, MACHINE LEARNING & LARGE LANGUAGE MODELS
New technologies.
The need for evidence endures.
Healthcare AI is changing the tools clinicians use. The evaluation questions remain concrete: What problem does a tool solve? What evidence supports its claims? Does it work for the intended patients and clinical setting? What happens when people use it in practice?
The handbook’s dedicated AI/ML chapter examines clinical decision support, validation, data quality and bias, implementation, and safety. Its ABC checklist connects the purpose of a tool with the evidence behind it and the conditions for its use.
Explore Chapter 11 ↓Connecting the framework to LLMs
For teams considering generative AI and large language models, the book offers a foundation for framing evaluation questions and measuring clinical impact. LLM-specific testing adds questions about factual grounding, consistency, sensitivity to prompts, and bias.
The second edition directly covers AI and machine learning. The authors’ 2025 article extends the discussion to LLMs and next-generation clinical decision support.
Read the authors’ AI & LLM article ↗A LOOK INSIDE
Find your starting point.
Explore topics by the question
you’re working on. Select one to read more.
11 chapter topics shown
01Why and how to evaluate
+
CHAPTER 1 OVERVIEW
What should an evaluation help you decide?
Start with the clinical purpose of a technology and the reasons for evaluating it. This chapter introduces the background of health IT assessment and the steps involved in a meaningful evaluation.
Topics covered: The need for evaluation; assessment concepts; the U.S. regulatory setting at publication; planning the evaluation.
Find the book & read the full chapter ↗02Building an evaluation study
+
CHAPTER 2 OVERVIEW
How do you turn a question into a study?
Define the technology, identify the stakeholders, and prioritize the questions your evaluation needs to answer. Explore study approaches and how to develop an evaluation protocol.
Topics covered: Research methodologies; study design choices; randomized trials; diagnostic performance studies; protocols.
Find the book & read the full chapter ↗03Designing reliable measurements
+
CHAPTER 3 OVERVIEW
What should you measure, and how reliably?
Examine the foundations of sound measurement, from selecting a sample to recognizing bias and confounding. Consider clinical, process, financial, and patient-reported outcomes alongside data collection and quality.
Topics covered: Validity; accuracy and precision; sampling; outcome measures; data quality.
Find the book & read the full chapter ↗04Making sense of the data
+
CHAPTER 4 OVERVIEW
How should you interpret the results?
Connect the type of data and the study question with an appropriate analytical approach. The chapter introduces statistical reasoning, common tests, and considerations that affect interpretation.
Topics covered: Descriptive statistics; confidence intervals; hypothesis testing; regression; diagnostic accuracy; sample size.
Find the book & read the full chapter ↗05Communicating your findings
+
CHAPTER 5 OVERVIEW
How do you make findings useful to others?
Plan how to present an evaluation to its intended audience. Explore ways to structure a report, use reporting guidelines, and communicate findings through different channels.
Topics covered: Audience selection; dissemination; scientific reporting; reporting standards; presentations.
Find the book & read the full chapter ↗06Assessing safety
+
CHAPTER 6 OVERVIEW
Could the technology introduce harm?
Consider how to identify safety problems and choose relevant measures. Explore passive methods of capturing safety events and active approaches, including simulation and testing in a safe environment.
Topics covered: Safety metrics; event reporting; regulatory context at publication; simulation; active and passive evaluation.
Find the book & read the full chapter ↗07Understanding costs
+
CHAPTER 7 OVERVIEW
What does the investment cost and deliver?
Introduce an economic perspective to the evaluation. Compare methods for examining costs and benefits and consider how to report an economic evaluation.
Topics covered: Health economics; cost-benefit analysis; cost-effectiveness; cost-minimization; return on investment.
Find the book & read the full chapter ↗08Testing whether it works
+
CHAPTER 8 OVERVIEW
Does the technology achieve its intended effects?
Organize an evaluation around the outcomes, setting, methods, and timing. Work through examples of survey, reference-standard, and before-and-after studies, with discussion of security evaluation.
Topics covered: Clinical outcomes; evaluation settings; study timing; practical study examples; security.
Find the book & read the full chapter ↗09Understanding usability
+
CHAPTER 9 OVERVIEW
How well does the system work for its users?
Explore how to examine efficiency, errors, and user satisfaction. The chapter describes practical usability methods and their application to electronic health records.
Topics covered: Cognitive walkthroughs; keystroke-level modeling; heuristic evaluation; the System Usability Scale.
Find the book & read the full chapter ↗10Learning from case studies
+
CHAPTER 10 OVERVIEW
What changes when a tool reaches clinical practice?
Follow two case studies: the SWIFT risk score and the AWARE dashboard. Examine their development, implementation, evaluation, and the lessons drawn from clinical use.
Topics covered: Clinical workflow; adoption; human-centered design; testing; implementation lessons.
Find the book & read the full chapter ↗11Evaluating AI & machine learning
+
CHAPTER 11 OVERVIEW
How should you assess a healthcare AI tool?
Explore evaluation considerations specific to AI and machine learning in clinical decision support. The ABC checklist connects organizational purpose and vendor claims with evidence, intended use, implementation, and safety.
Topics covered: Alignment with the clinical problem; evidence for claims; validation; data quality and bias; implementation; safety evaluation.
Find the book & read the full chapter ↗Chapter overviews summarize the second edition; full chapters are available in the book. View full contents & publisher preview ↗
YOUR WORK. YOUR QUESTIONS.
Bring evaluation into the conversation.
Beyond the launch date
How will you examine a system’s effects on care and the people who use it?
From pilot to decision
What would you need to learn before bringing a tool into everyday clinical work?
METHODS MEET CLINICAL PRACTICE
Look beyond the promise.
Examine the evidence.
Three ways into the handbook
for your next technology decision.
Make the cost question explicit.
Compare approaches to economic evaluation, including cost-benefit, cost-effectiveness, and return on investment. Consider what you need to measure before choosing an analysis.
Explore the chapter guide ↓Learn from implementation.
Two case studies examine the SWIFT risk score and AWARE dashboard. Follow the discussion from development to clinical use, including adoption, workflow, and lessons from evaluation.
Explore the chapter guide ↓Put AI and ML claims under scrutiny.
Chapter 11 applies health IT evaluation to artificial intelligence and machine learning, including validation, bias, and implementation risks. The ABC checklist brings these questions back to the clinical problem and the evidence for a tool’s claims.
Explore the chapter guide ↓Topic summaries from the second edition. Read the book for the full methods, examples, and references.
THE AUTHORS
Clinical experience.
Evaluation in practice.
Physicians and informatics researchers
at Mayo Clinic, Rochester, Minnesota.
Vitaly Herasevich
MD, PhD, MSc
Professor of Medicine and Anesthesiology, Mayo Clinic
Dr. Herasevich studies how clinical information systems support care in intensive care units and perioperative settings. His work spans patient-centered EHR interfaces, surveillance, decision support, virtual care, and health IT evaluation.
He is a co-principal investigator of the Acute Care Informatics Laboratory and works on methods for validating new technologies in clinical practice.
Mayo Clinic faculty profile ↗ — Vitaly Herasevich
Brian W. Pickering
MD, MSc
Professor of Anesthesiology, Mayo Clinic
Dr. Pickering is an anesthesiologist and critical care physician studying how digital systems can make acute care more reliable. His research brings together clinician-centered design, predictive analytics, implementation science, and evaluation in working clinical environments.
He is a co-principal investigator of the Acute Care Informatics Laboratory, with interests in hospitalwide surveillance and responsible AI deployment.
Mayo Clinic faculty profile ↗ — Brian W. PickeringMAKE YOUR NEXT QUESTION COUNT
Move from adoption
to understanding.
Choose a retailer for the second edition of the handbook.
More places to find your copy
Barnes & NoblePrint & eBook — view book listing (opens in a new tab)Apple BookseBook — view book listing (opens in a new tab)VitalSourceDigital textbook — view book listing (opens in a new tab)AbeBooksHardcover · new & used listings — view book listing (opens in a new tab)Better World BooksHardcover — view book listing (opens in a new tab)
Check each retailer for current prices, stock, shipping, and regional availability. Formats and digital access options vary by seller.
Looking for a teaching or institutional copy?
Visit the publisher’s listing for available formats and institutional purchase information.
Explore publisher options ↗