ELECTRONIC HEALTH RECORDS

VA Making Incremental Improvements in New System but Needs Updated Cost Estimate and Schedule

Report to Congressional Committees

March 2025

GAO-25-106874

United States Government Accountability Office

 

View GAO‑25‑106874. For more information, contact Carol Harris at (202) 512-4456 or HarrisCC@gao.gov

Highlights of GAO‑25‑106874, a report to congressional committees

March 2025

Electronic Health RecordS

VA Making Incremental Improvements in New System but Needs Updated Cost Estimate and Schedule

Why GAO Did This Study

After three unsuccessful attempts since 2001, VA initiated its fourth effort—the EHR modernization program—to replace its legacy system. In April 2023, after deploying the new system to five of its medical centers, VA paused deployments due to user concerns. On December 20, 2024, VA announced resumption of planned deployments to four Michigan facilities.

Congressional report language includes provisions for GAO to review VA’s EHR deployment. GAO’s objectives were to (1) determine the improvements VA has made to its new system at initial deployment sites, (2) describe user feedback on the new EHR system, and (3) determine the extent to which VA is measuring modernization progress.

GAO reviewed VA’s approach and goals for improving the new EHR system and compared actions taken to leading practices for program management. In addition, GAO conducted structured interviews with users from the five sites and reviewed survey results. GAO also compared the program’s efforts for measuring performance to leading practices.

What GAO Recommends

GAO is making three new recommendations to VA, including updating the EHR’s modernization life cycle cost estimate and schedule. VA concurred but its planned actions on updating cost and schedule do not encompass the modernization’s life cycle. GAO has previously made a total of 15 EHR recommendations. One has been implemented. The remaining 14 are critical to reducing EHR risks and delivering a quality system.

What GAO Found

The Department of Veterans Affairs (VA) is making incremental improvements to the new electronic health record (EHR) system but much more remains to be done. For example, as of June 2024, VA implemented over 1,500 configuration changes to the system. However, as of February 2025, it had not addressed about 1,800 configuration change requests. Among its other improvements at the initial five sites, VA has delivered patient safety and pharmacy enhancements, addressed system trouble ticket resolution, and increased system performance.

During the 20-month deployment pause, VA made numerous changes to the system at the initial five locations, initiated additional needed complex projects to address challenges identified via user feedback, and delayed planned enhancements. These many changes impact estimated costs and schedule. Regarding costs, in 2022 the Institute for Defense Analyses estimated that EHR modernization life cycle costs would total $49.8 billion—$32.7 billion for 13 years of implementation and $17.1 billion for 15 years of sustainment. Updating that estimate to reflect events over the last 2 years, such as the pause, is imperative to understanding the full magnitude of VA’s investment. Similarly, it is critically important that VA update its schedule to informing decision-making.

Regarding user feedback, while VA’s 2024 surveys show improvement since 2022, they continue to reflect general dissatisfaction with the new system. For example, as of September 2024, 75 percent (1,247 of 1,670) of users disagreed or strongly disagreed that the new system made them as efficient as possible.

Consistent with federal guidance, VA identified nine metrics to measure modernization progress, including improved end user experience. It also established and implemented baselines and performance targets for eight of the nine metrics. However, VA has not yet set a baseline and target for one metric on resolution of configuration change requests due to recent process changes.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Abbreviations
DOD	Department of Defense
EHR	electronic health record
EHRM	Electronic Health Record Modernization
IT	information technology
MHS	Military Health System
OIT	Office of Information and Technology
OMB	Office of Management and Budget
VA	Department of Veterans Affairs
VHA	Veterans Health Administration
VISN	Veterans Integrated Services Network
VistA	Veterans Health Information Systems and Technology Architecture

March 12, 2025

Congressional Committees

The Department of Veterans Affairs (VA) relies on its electronic health record (EHR) system to manage health care for its patients.[1] Since 2017, the department’s Electronic Health Record Modernization (EHRM) program has undertaken an effort to replace its legacy EHR system with a modernized, commercial system. VA first deployed the new EHR system in October 2020 and followed up with further deployments to four additional sites in 2022.

In April 2023, VA announced that it planned to halt future deployments to additional sites, with the exception of the Captain James A. Lovell Federal Health Care Center in North Chicago in March 2024.[2] VA explained that it was halting deployment due to feedback from veterans and clinicians that the new system was not meeting expectations. Further, VA noted that it would focus on making improvements at the five sites where the new system had been deployed before restarting deployments to the remaining VA facilities. On December 20, 2024, VA announced that it was resuming planned deployments, starting with four facilities in Michigan.

We have previously designated VA health care as a high-risk area for the federal government, in part due to its IT challenges and implementation of EHRM.[3] In our 2023 high-risk update, we noted that VA had a number of major modernization initiatives, including the EHR, that faced challenges.

Congressional report language associated with the Military Construction, Veterans Affairs, and Related Agencies Appropriations Act and the Consolidated Appropriations Act for Fiscal Years 2023 and 2024 includes provisions for GAO to review the VA EHRM implementation.[4]

Our objectives were to (1) determine the improvements VA has made to its new EHR system at initial deployment sites, (2) describe user feedback on the new EHR system, and (3) determine the extent to which VA is measuring modernization progress. The scope of our review included actions taken by VA since announcing the program implementation pause in April 2023. Specifically, our focus was on the original five deployed sites and did not include a review of the sixth deployment site, the Captain James A. Lovell Federal Health Care Center.

To address the first objective, we reviewed VA documentation describing the program’s planned incremental approach and progress made toward improving the EHR system. For example, we examined VA briefing slides that described the assessment of issues to be fixed, the program’s approach to configuration changes, and enhancements made to address the highest priority patient safety issues.[5]

Further, we reviewed the performance work statement for the VA EHRM program contract and related service level agreements identifying pharmacy enhancements and establishing trouble ticket resolution and system performance thresholds. In addition, we reviewed VA’s reports on the contractor’s performance and related measures to monitor progress on resolving trouble tickets and system performance. We also determined the implementation status of the recommendations we made in our prior reports on the EHRM program.[6] Given the magnitude of the EHRM effort, we also identified the extent to which VA had updated the EHRM life cycle cost estimate and schedule.[7]

To address the second objective, we obtained and reviewed results of surveys that VA conducted to determine users’ satisfaction with the new system. We also conducted structured interviews with selected users from the five locations where the new EHR was initially deployed.

In reviewing the results of VA’s user satisfaction surveys, we compared results from four surveys to determine trends in satisfaction over time.[8] We also assessed the reliability of the data from the surveys by reviewing documentation and written responses regarding the department’s administration of its user satisfaction surveys. We determined that the user experience survey data were sufficiently reliable for our purposes.

For our structured interviews, we interviewed 71 users from the first five sites where the new system was deployed: Mann-Grandstaff VA Medical Center (Spokane, Washington), Jonathan M. Wainwright Memorial VA Medical Center (Walla Walla, Washington), VA Central Ohio Health Care System (Columbus, Ohio), Roseburg VA Medical Center (Roseburg, Oregon), and the White City VA Medical Center (White City, Oregon). We selected a judgmental sample of users that was supplemented by random selections to help mitigate potential selection bias. We judgmentally selected users that were service line chiefs who represented users within their departments. These users embodied a variety of roles (e.g., doctors, nurses, pharmacists, dentists, mental health professionals, and administrative functions). We conducted interviews about user satisfaction with users between October 2023 to May 2024. While users’ responses cannot be generalized to the entire population of EHR users at the initial deployment sites, they provide useful insights from a broad range of user roles and clinical areas at the sites.

To address the third objective, we reviewed federal guidance and leading practices for performance measures.[9] We selected applicable leading practices for measuring the program’s progress on improvements such as establishing goals, metrics, baselines, and targets, as well as measuring and reporting progress. We reviewed program briefing slides and compared the program’s efforts for establishing goals, metrics, baselines, and targets, as well as measuring and reporting progress to the leading practices.

For all three objectives, we supplemented our review and analysis of documentation with interviews with VA officials from the EHRM Integration Office and the Veterans Health Administration (VHA). We also interviewed leadership from the five initial deployment sites. Appendix I provides additional details on our objectives, scope, and methodology.

We conducted this performance audit from June 2023 to March 2025 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Background

Within VA, the VHA operates one of the nation’s largest health care systems. The administration relies on its health information system—the Veterans Health Information Systems and Technology Architecture (VistA)—to deliver health care to veterans and to document this care. This technically complex system has been in operation for more than 30 years, is costly to maintain, and does not optimally support VA’s need to electronically exchange health records with other organizations, such as the Department of Defense (DOD) and community providers (who are non-VA providers that provide care to veterans and are reimbursed by VA).

The department has undertaken, and we have reported on, a number of initiatives to improve interoperability (i.e., the ability to exchange and use electronic health information) and modernize electronic health records across the department.[10] These initiatives have included four efforts over two decades to modernize VistA. The first three efforts—HealtheVet, the integrated Electronic Health Record, and VistA Evolution—reflect varying approaches that the department had taken since 2001 to achieve a modernized EHR system. However, these approaches were abandoned due to concerns about project planning, high costs, and undelivered capabilities.

VA’s current approach, EHRM, began in June 2017. At that time, the former VA Secretary announced that the department planned to acquire the Oracle Health EHR system—the same commercial system that DOD was implementing across the military health system—and configure it for VA.[11] According to the Secretary of VA, the department decided to acquire the same system as DOD because it would allow all of VA’s and DOD’s patient data to reside in one system. A single system is intended to enable seamless care between VA and DOD without the manual and electronic exchange and reconciliation of data between two separate systems.[12]

VA’s EHRM Integration Office manages the EHRM program and coordinates with stakeholders (e.g., VHA subject matter experts and site-specific staff) at the site, regional, and national levels on the transition to a new EHR system. According to the department, EHRM is designed to improve veterans’ experience by establishing a modernized, seamless, and secure EHR for VA. This modernization approach is intended to improve VA health care providers’ ability to deliver care by standardizing clinical practice workflows, enabling interoperability between VA and DOD, and increasing interoperability with community care partners.

The EHRM program originally planned to implement the new EHR system across VA’s health care medical facilities in phases over the span of a decade. VA deployed the new EHR system at its first location, the Mann-Grandstaff VA Medical Center in Spokane, Washington, in October 2020. In 2021, VA performed a strategic review of the program and decided to pause new deployments and focus on fixing initial deployment issues.

In fiscal year 2022, VA moved forward with implementation of the system at four additional locations:[13]

·         Jonathan M. Wainwright Memorial VA Medical Center in Walla Walla, Washington, in March 2022.

·         VA Central Ohio Health Care System in Columbus, Ohio, in April 2022.

·         Roseburg VA Health Care System in Roseburg, Oregon, in June 2022.

·         VA Southern Oregon Rehabilitation Center and Clinics in White City, Oregon, in June 2022.

Following the Roseburg and White City deployments, VA decided to delay upcoming deployments to address feedback from users at the initial sites who identified patient safety and system reliability issues. In addition, it performed an assessment to diagnose and address problems with the governance and processes for deployments.

Around the same time, the department modified a task order performance work statement issued under the EHRM contract to include seven priority enhancements intended to address related feedback from pharmacists and providers.[14] In March 2023, VA released a report from a multidisciplinary, enterprise-wide team that assessed concerns and recommended solutions for the most impactful and critical patient safety issues to be fixed before future deployments (referred to as “Go-live”). As part of the review, VA identified 14 high-priority enhancements requiring development work to address patient safety issues.

In April 2023, VA announced that feedback from veterans and clinicians continued to indicate that the new system was not meeting expectations at the five deployed sites. Consequently, the department halted future deployments, with the exception of the Captain James A. Lovell Federal Health Care Center in North Chicago, to focus on making improvements at the five initial sites. VA refers to its current improvement efforts as a program “reset.”[15] Figure 1 depicts a timeline of key program events.

Since the program reset began in April 2023, the department has focused resources on fixing issues identified by users and optimizing the system at the deployed sites. Using an incremental approach, the EHRM Integration Office and VHA have identified goals for planning and executing incremental changes aimed at making improvements rapidly at the live sites. Specifically, VA established the following outcome-oriented reset goals:

·         ensure that the new system is working as promised for its end users at the first five sites,

·         prepare for and deploy the system at the Captain James A. Lovell Federal Health Care Center in North Chicago, and

·         invest in foundational work for successful future deployment of the EHR across the enterprise (e.g., improving processes to manage issues for an enterprise system and workforce development).

While the reset efforts continued, VA deployed the new EHR system at the Captain James A. Lovell Federal Health Care Center in North Chicago, Illinois, in March 2024. On December 20, 2024, VA announced that it was beginning early-stage planning for restarting deployments to four sites in Michigan in mid-2026.

For the period of fiscal year 2018 through the third quarter of fiscal year 2024, VA reported that it obligated a total of about $12.71 billion on EHRM. This total is comprised of the following elements:

·         EHR contract ($5.42 billion)

·         IT infrastructure ($3.11 billion)

·         program management ($1.33 billion)

·         VHA ($2.58 billion)

·         Office of Information and Technology (OIT) ($273 million).

GAO Has Previously Reported on VA’s Health IT and EHRM Challenges

Since 2020, we have issued four reports on EHRM. Our June 2020 report found that VA’s decision-making procedures for configuring the EHR system were generally effective but did not always ensure key stakeholder involvement.[16] We recommended that the department ensure the involvement of all relevant medical facility stakeholders in the EHR system configuration decision process. VA concurred with our recommendation and stated that it intended to refine local workshop agendas and descriptions to facilitate subject matter expert identification and participation. However, the recommendation has not yet been implemented.

In February 2021, we reported that VA had made progress toward implementing its new system but needed to postpone further deployment until it had addressed all critical- and high-severity test findings.[17] We made two recommendations that it do so, and VA concurred. As of December 2024, the department has made progress toward implementing the recommendations as it had no critical- or high-severity test findings at Go-live at the four subsequent locations. There were also no critical- or high-severity test findings at the Captain James A. Lovell Federal Health Care Center Go-live. We will plan to consider the recommendations implemented when we have observed sustained resolution of significant test findings.

Our February 2022 report on VA’s data management plans discussed migrating data to the new EHR system and supporting the continuity of reporting.[18] We noted that the department had made progress towards implementing planned data management activities, but clinicians faced challenges with the quality of migrated data. In addition, VA had not established performance measures and goals for data quality and had not used a stakeholder register to identify and engage all stakeholders. Accordingly, we made two recommendations to the department to (1) establish performance measures and (2) use a stakeholder register to meet reporting needs. VA concurred and took action to fully implement the second recommendation. However, because the program had paused system deployments and lacked a path to migrating data at the next sites, there is not sufficient evidence to demonstrate that the first recommendation has been implemented.

In May 2023, we reported that the organizational change management activities for the EHRM program were partially consistent with seven leading practices and not consistent with one leading practice.[19] We also reported that users expressed dissatisfaction with the new system and VA did not adequately identify and address system issues. We made 10 high priority recommendations to address change management, user satisfaction, system trouble ticket, and independent operational assessment deficiencies. VA concurred with the recommendations. As of February 2025, it had not yet implemented the recommendations. For example:

·         The department has not yet developed and implemented a VA-specific change management strategy to formalize its approach to drive user adoption. According to the department, it worked with its contractor to update the contractor’s change management plan to be more VA-specific and stated that VHA and the EHRM program office were collaborating to develop a strategy that was targeted for completion in December 2025.

·         VA has not yet adequately demonstrated that it addressed users’ barriers to change by ensuring planned completion of actions identified in a strategic review of the program. VA had plans in place to address the open action items through reset initiatives and key priority projects, but the underlying user barriers have not yet been fully addressed.

·         VA has not reinstituted plans to conduct an independent operational assessment to evaluate the suitability and effectiveness of the new EHR system for users in the operational environment. As of November 2024, according to department officials, funding for the project was not approved and VA could not move forward with the action. In addition, EHRM program officials said that VA conducted a variety of independent assessments such as independent end user testing.

Our prior reports included 15 recommendations to assist the department in its efforts. While the department has taken steps to address our recommendations such as initiating key priority projects to address user concerns, as of February 2025, VA had fully implemented one of our 15 recommendations. As VA resumes planning activities for future deployments, we will continue to monitor its progress.

VA Has Made Improvements at Initial Sites, but Needs an Updated Cost Estimate and Schedule

Since the EHR deployments were paused and the program reset was announced in April 2023, VA has taken an incremental approach to deliver system improvements. Specifically, VA has made configuration changes to the new EHR system, delivered patient safety and pharmacy enhancements, and improved system trouble ticket resolution and system performance for the initial sites. However, much more work remains for VA to implement additional configuration changes and enhancements. Moving forward, VA has prioritized a number of complex projects that require additional stakeholder input and coordination. Further, the program lacks an updated total life cycle cost estimate and integrated master schedule that reflects the many EHRM changes and delays.

VA Is Incrementally Delivering Improvements at the Initial Sites, but Much More Work Remains

VA is making incremental improvements as it continues to address additional configuration changes, key patient safety and pharmacy enhancements, trouble ticket resolution, and system performance.

Configuration Changes

A significant part of VA’s improvement efforts has focused on making configuration changes to the EHR baseline.[20] Specifically, the department reported that it implemented more than 1,500 configuration changes that impacted users at the enterprise level, through its standard configuration change process as of June 2024. These changes addressed user needs (i.e., a requested change) or supported enterprise standards (i.e., a change to support a standardized workflow) and were critical for improving performance and usability for staff at the five initial sites.

In addition, during the reset period, VA focused on developing a structured approach to address more challenging configuration changes. To do so, VA reviewed the existing backlog of requests to identify, investigate, and prioritize manageable lists of configuration changes to be addressed in increments. The incremental work culminated thus far into two prioritized lists of configuration changes:

·         VA has implemented its first prioritized list of configuration changes through this incremental approach. The first list included approximately 200 configuration change requests and was completed by January 2024.

·         The second list consisted of approximately 200 requested configuration changes. However, implementation of this list was delayed as resources were diverted to the deployment of the new EHR to the Captain James A. Lovell Federal Health Care Center or challenged due to a lack of detail provided on some submitted change requests.

In May 2024, VA subsequently identified 29 system configuration change requests from its second list of prioritized configuration changes. According to VA, these additional requests affected a broad number of users and required additional contracted resources. As of September 2024, VA had completed 19 of these configuration changes. For the remaining changes, VA officials noted that two were in process and eight were cancelled because they were either not approved or they would be considered at a later time.

While VA has made configuration changes to improve the system, more work remains for the department to validate and address additional change requests. As of September 2024, there were about 2,200 complex configuration change requests that had not been resolved. Instead of developing a third list of configuration changes to be implemented, VA developed a revised process for addressing configuration change requests with a tiered escalation process for timely and collaborative problem-solving across facilities. As of February 2025, VA had not addressed approximately 1,800 complex configuration change requests.[21]

Patient Safety and Pharmacy Enhancements

VA had closed many, but not all, of its highest-priority patient safety enhancement requests as identified in the department’s March 2023 assessment of what needed to be fixed before the future deployments resumed.[22] As of July 2024, according to EHRM program officials, VA had closed nine of 14 highest-priority patient safety related enhancement requests. These enhancements were intended to address items most impactful and critical for patient safety. For example, VA reported that it had addressed the patient safety concern that DOD’s system was overwriting VA identity and demographics information.

As of July 2024, the program was continuing to address the remaining five patient safety enhancements, such as improving medication system alerts for clarity. VA estimated that the remaining patient safety enhancement requests would be closed by December 2024. However, as of December 2024, two of the five issues are pending closure and three others have yet to be fully resolved.

Regarding pharmacy enhancements, VA delivered six of seven pharmacy enhancements identified by pharmacists and providers. For example, the EHR was enhanced to allow pharmacy technicians to initiate prescription refills. Another enhancement allowed pharmacy users to discontinue duplicate prescription orders in the medication drug interaction warning window within the pharmacy module.

However, as of June 2024, VA had not deployed the remaining pharmacy enhancement for synchronization between the system module used by pharmacists and the module used by other health care team members. VA had planned to release enhanced functionality in February 2024, but the functionality failed testing at that time. As of June 2024, VA continued to delay release and estimated that the pharmacy capability would be delivered with the planned February 2025 system update.

Ticket Resolution

Users at the initial five sites report issues or ask for help with the new EHR system by entering a trouble ticket through a support help desk. Based on impact and urgency, each ticket is assigned a priority of critical, high, moderate, or low.[23] The EHRM contract service level agreement defined expectations for the time it takes to resolve these issues and to close the trouble tickets.[24] If performance thresholds are not met monthly, financial consequences (i.e., financial credits from the contractor to the department in the form of an invoice offset) may apply. Specifically, the service level agreement performance thresholds agreed upon between VA and Oracle Health are:

·         Critical - 100 percent of trouble tickets are to be resolved within 5 hours and closed within 24 hours.

·         High - 90 percent of trouble tickets are to be resolved within 16 hours and no single ticket is to exceed 64 hours.

·         Moderate - 80 percent of trouble tickets are to be resolved within 4 business days and no single ticket is to exceed 60 calendar days.

·         Low - 80 percent of trouble tickets are to be resolved within 6 business days and no single ticket is to exceed 60 calendar days.

VA’s data on trouble ticket resolution showed that the percentage of tickets that were resolved within service level agreement performance thresholds for timeliness were met since the implementation of financial consequences in September 2023. This was an improvement over earlier data that showed consistent challenges with resolving trouble tickets prior to July 2023, as shown in figure 2.

Note: Financial consequences associated with these service level agreement performance thresholds were financial credits from the contractor to the department in the form of an invoice offset.

However, the number of tickets that were not resolved within agreed upon not-to-exceed thresholds has continued to fall short of expectations, as shown in figure 3. According to VA EHRM program officials, the time periods were not met because they were still working to address outlier tickets that did not meet resolution performance thresholds. There were no financial consequences associated with the service level agreement performance threshold for the number of tickets that were not resolved within agreed upon not-to-exceed thresholds because VA was focused on the most critical service level agreements such as system performance and outages metrics.

Note: There were no financial consequences associated with these service level agreement performance thresholds.

System Outages and Incidents

VA established performance thresholds as part of its service level agreement with Oracle Health, which defined expectations that the EHR system operate free from incidents or outages for specific amounts of time. If performance thresholds are not met monthly, financial consequences may apply.[25] The outage-free time (system operates greater than 99.95 percent of the time without reported outages) was met for 15 of 18 months from April 2023 to September 2024. Since the program reset in April 2023, the outage-free time was not met in March and April 2024 due to database bugs. Incident-free time (system operates 95 percent of time without any reported incidents) was met for 13 of 18 months and not met for 5 of the 18 months.

Figure 4 shows the degree to which outage-free and incident-free performance thresholds were met over time. According to VA officials, the incidents were caused by a variety of factors such as database bugs, code defects, processing backlogs, and a ransomware attack of a data exchange partner.

Note: Financial consequences associated with these service level agreement thresholds were financial credits from the contractor to the department in the form of an invoice offset. Performance thresholds and financial consequences were modified as of June 2023.

According to the department, the EHRM program is focused on improving testing rigor for system changes to identify potential issues earlier that might cause incidents and improve overall system performance. EHRM program officials said that VA has made improvements to reduce major incidents and improve system performance. In September 2024, VA’s Office of Inspector General reported that the program needed to strengthen controls to address EHR system major performance incidents and made nine recommendations in this area.[26]

VA Identified Complex Projects Critical to Site Improvements

While VA executed a number of incremental improvement projects over the course of a year since site deployments were paused, in April 2024, the department identified additional complex projects deemed necessary to address challenges identified through user feedback from the live sites. According to EHRM program officials, these complex projects require significant stakeholder coordination and collaboration with the live sites beyond configuration changes and are critical to the effort to restart deployment of the new EHR.

Among other things, these projects are intended to provide improvements related to:

·         medication clinical decision support and alerts,

·         quick orders,

·         position standardization,

·         the data model for clinical events,

·         templates for displaying patient information and documenting care,

·         referral management,

·         synchronization between the system module used by pharmacists and the module used by other health care team members,

·         improved training, and

·         message center optimization.

For example, according to the EHRM program, referral management within the new EHR system lacks functionality and intuitiveness for users. Further, data exchanges related to referrals experience issues that may cause delays in care and frustration among staff. The related priority project is intended to identify improvements by simplifying existing processes and standardizing referrals across the enterprise, by specialty, into streamlined workflows and tools.

As another example, according to EHRM program officials, VA does not have a consistent strategy for standardizing and streamlining positions (i.e., roles) within the EHR, resulting in numerous unused, redundant, or overlapping positions. The priority project related to position standardization is intended to consolidate and reduce the number of standardized user positions in the system which could, among other things, relieve users from having to log in and log out to perform different position-specific functions.

VA Has Not Updated Life Cycle Cost Estimates and Schedule

In January 2019, VA estimated that its life cycle cost for EHRM would total about $16.1 billion, consisting of the following:

·         The EHRM program contract, awarded in May 2018, was for a maximum of nearly $10 billion over 10 years.

·         Expected costs for technology infrastructure readiness and program management support were estimated at $6.1 billion.

In October 2022, the Institute for Defense Analyses issued a final report on an independent cost estimate for the EHRM program. The Institute estimated a total life cycle cost of $49.8 billion—$32.7 billion for a 13-year implementation phase and $17.1 billion for 15 years of sustainment costs.[27]

Regarding the program’s schedule, the Deputy VA Secretary testified in September 2022 before the Senate Appropriations Committee that the full EHR deployment schedule through 2028 was currently under development and would be ready in the fall of 2022. However, this schedule was not released, due in part to the deployment pause announced in April 2023. VA’s December 2024 announcement pertained to the four locations in Michigan. A deployment schedule of the other approximately 160 VA medical centers and associated clinics was not provided.

Following cost estimating and schedule best practices helps minimize the risk of cost overruns and schedule delays for government acquisitions.

·         A high-quality, reliable cost estimate is a key tool for budgeting, planning, and managing federal programs. According to the Office of Management and Budget (OMB), programs must maintain current and well-documented estimates of program costs, and these estimates must encompass the full life cycle of the program.[28] Among other things, OMB policy states that generating reliable program cost estimates is a critical function necessary to support the capital programming process. Without this capability, agencies are at risk of experiencing program cost overruns, missed deadlines, and performance shortfalls.

GAO’s Cost Estimating and Assessment Guide describes best practices for developing reliable cost estimates that management can use to make informed decisions.[29] The development of a life cycle cost estimate entails identifying and estimating all cost elements that pertain to the program from initial concept all the way through each phase in the program’s duration.

Further, according to the guide, it is important to revise cost estimates to reflect program changes and maintain reliable estimates throughout the life of the program. To properly mitigate overly optimistic cost estimates that may not allow for changes in scope, schedule delays, or other elements of risks, it is also important to have an independent view of the program to help an organization better understand the risks its program faces and better manage them as they are realized.

·         The success of a project also depends, in part, on having an integrated and reliable master schedule that defines when and how long work will occur, and how each activity is related to the others. Integrated master schedules are a consolidation of lower-level project (i.e., subproject) schedules. A well-planned schedule is a fundamental management tool that can help government programs use public funds effectively by specifying when work will be performed in the future and measuring program performance against an approved plan. Moreover, an integrated and reliable schedule can show when major events are expected as well as the completion dates for all activities leading up to them, which can help determine if the program’s parameters are realistic and achievable.

We have previously reported in our Schedule Assessment Guide that a reliable schedule can provide a road map for the systematic execution of a program and the means by which to gauge progress, identify and address potential problems, and promote accountability.[30] Following changes in a program, the schedule is used to forecast the effects of delayed, deleted, and added effort, as well as possible avenues for time and cost recovery. In this respect, schedules can be used to verify and validate proposed adjustments to the planned time to complete the program.

On December 20, 2024, VA announced the decision to restart planning for deploying the new EHR system at four locations in Michigan beginning in mid-2026. In addition, work would continue to make improvements at the initial sites. However, the program did not commit to when it would update the EHRM life cycle cost estimate and integrated master schedule to reflect the changes and lessons learned from the initial deployments and the 20-month deployment pause.

Without updated cost estimates, management and oversight bodies must continue to rely on either the January 2019 VA total life cycle cost estimate of $16.1 billion, or an October 2022 independent cost estimate of $49.8 billion. However, these prior estimates do not reflect the full magnitude and impact of delays, system changes, and updated processes.[31] For example:

·         VA’s life cycle cost estimate of $16.1 billion is severely outdated and incomplete. Specifically, it does not encompass the full life cycle of the program through each phase in the program as it estimates only for a 10-year period. It also does not reflect changes that have occurred over the last 6 years.

·         The independent cost estimate represents a more realistic assessment of expected costs, but it also does not reflect the pause and other significant changes over the last 2 years. It includes a 15-year sustainment period reflecting continuing operations and maintenance costs and uses a more conservative 13-year implementation time frame rather than 10 years.

VA’s announcement that system deployment will occur at four additional sites in mid-2026 means that 8 years after contract award, the new system will be deployed at a total of 10 locations. With 2 years remaining on the contract in mid-2026, VA will have approximately 160 remaining medical centers and associated clinics (94 percent of its total number of medical centers) that will still be using the legacy system.

According to EHRM program officials, VA has learned from the program reset. Specifically, the department intends to address the long-term acquisition and program management work to support the rest of the program, including selecting the next sites for deployment and updating the life cycle cost estimate.

Developing an independent and reliable updated cost estimate is crucial for realistic planning, budgeting, and management. Given the pause, the volume and complexity of user-driven priority projects, other delays, and continuing issues, an updated cost estimate is essential to understanding the magnitude of VA’s investment.

Similarly, a realistic and updated integrated schedule can show when major events are expected as well as the completion dates for all activities leading up to them, which can help determine if the program’s parameters are realistic and achievable. Without a reliable schedule, management will face increased risks not being able to effectively gauge progress and identify and address potential problems.

Feedback Indicated Improvement, but Dissatisfaction with EHR at Initial Sites Persists

VA’s user satisfaction surveys indicated improvement since 2022, but about half of respondents continued to disagree or strongly disagree that the system enabled them to deliver high-quality care.[32] For example, in 2022, about 78 percent (1,640 of 2,090) of responding users disagreed or strongly disagreed that the system enabled the delivery of high-quality care. In September 2024, about 47 percent (786 of 1,669) disagreed or strongly disagreed that the new EHR enabled them to deliver high-quality care[33] (see figure 5).

Note: The user surveys were conducted in September 2022, August-September 2023, March-April 2024, and September 2024.

In addition, in 2022, 5 percent (112 of 2,094) of users responding to the survey agreed or strongly agreed that the system made them as efficient as possible. In September 2024, 13 percent (221 of 1,670) agreed or strongly agreed that the system made them as efficient as possible. However, 75 percent (1,247 of 1,670) disagreed or strongly disagreed that the EHR made them as efficient as possible (see figure 6).

Note: The user surveys were conducted in September 2022, August-September 2023, March-April 2024, and September 2024.

Further, users were dissatisfied with the new system in the most recent survey. Specifically, in April 2024, overall satisfaction was low: 72 percent (1,376 of 1,908) disagreed or strongly disagreed that they were satisfied with the EHR. In September 2024, 69 percent (1,142 of 1,648) disagreed or strongly disagreed that they were satisfied with the EHR (see figure 7).

Note: This survey item was included in user surveys conducted in March-April 2024 and September 2024.

In addition, in April 2024, most users (66 percent or 1,250 of 1,900) disagreed or strongly disagreed that problems with the EHR were resolved in a timely manner (see figure 8).[34]

Note: This survey item was included in user surveys conducted in March-April 2024. In the spring 2024 survey, 2 percent (43 of 1,900) responded that this question did not apply or they did not know.

Many Users Agreed the EHR Was Available When Needed, but Most Continued to Be Dissatisfied with It

Similar to VA’s survey results, many users we interviewed agreed that the system was available when they needed it. Specifically, in our interviews from 2023-2024, 30 of 71 users said they agreed that the system was available when needed, 16 of 71 neither agreed nor disagreed, and 23 of 71 said they disagreed or strongly disagreed (see table 1).[35]

	The new EHR system is available when I need it.
Strongly agree	0
Agree	30
Neither agree nor disagree	16
Disagree	15
Strongly disagree	8
No basis to judge	2
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

However, for the remaining questions, users were generally neutral or did not agree that they were satisfied with the new EHR. For example, according to user feedback, 21 of 71 users said that their experiences in using the system had slightly improved or greatly improved compared to their experiences using the system in April 2023; however, 34 of 71 users said that their experiences did not improve or worsen, and 11 of 71 users said that their experiences had slightly or greatly worsened.[36] See table 2.

	At this point in time, how does your experience in using the new EHR system compare to your experience using the new system in April 2023?
Greatly improved	2
Slightly improved	19
Did not improve or worsen	34
Slightly worsened	8
Greatly worsened	3
No basis to judge	5
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

In addition, 60 of 71 disagreed or strongly disagreed that they were satisfied with the system, five of 71 neither agreed nor disagreed, and five of 71 users agreed or strongly agreed.[37] See table 3.

	Overall, I am satisfied with the new EHR system.
Strongly agree	1
Agree	4
Neither agree nor disagree	5
Disagree	28
Strongly disagree	32
No basis to judge	1
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

Further, most users said that the system generally decreased productivity. Specifically, 54 of 71 reported that the EHR decreased or greatly decreased productivity, 14 of 71 reported that it neither increased nor decreased productivity, and one of 71 respondents said that it had increased productivity (see table 4).[38]

	How has the new EHR system affected your productivity?
Greatly increased productivity	0
Increased productivity	1
Neither increased nor decreased productivity	14
Decreased productivity	30
Greatly decreased productivity	24
No basis to judge	2
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

With respect to how the new EHR system had affected their productivity, the users we interviewed also provided examples of their concerns. These examples discussed anecdotal experiences with reductions in the amounts of patients that could be seen daily, needs for additional staff, and a backlog of referrals. On several occasions, users discussed their frustration with the increase in the number of steps required to complete actions in the system.

Finally, users were also dissatisfied with having their problems resolved when they contacted the helpdesk. Specifically, 49 of 71 were dissatisfied or very dissatisfied, eight of 71 were neither satisfied nor dissatisfied, and three of 71 users were satisfied or very satisfied with the resolutions to problems.[39] See table 5.

	In general, how satisfied have you been in obtaining resolution to problems with the  new EHR system when you have  contacted the helpdesk?
Very satisfied	0
Satisfied	3
Neither satisfied nor dissatisfied	8
Dissatisfied	23
Very dissatisfied	26
I have never contacted the helpdesk for assistance with the EHR system	11
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

See appendix II for additional user feedback results from our structured interviews.

VA Has Begun Monitoring Performance Measures but Has Not Fully Defined Targets

According to federal guidance, performance measures allow the program the ability to measure and communicate the benefits intended and achieved by the program.[40] Leading practices for performance measures include:

·         developing measurable outcome-based goals;

·         identifying and documenting metrics to measure outcomes of the program against goals; a range of indicators is important because most program activities require managers to balance their priorities and programs may seek to measure areas such as organizational efficiency and customer (user) satisfaction;

·         establishing baseline measures to serve as a comparison for current performance metrics and define success targets expected to be achieved by the program; and

·         monitoring performance and report objective results to inform decisions, identify areas for process improvement, and take appropriate corrective action.

Consistent with federal guidance, VA identified performance measures for the new EHR reset. Specifically, VA developed four categories for outcome-based goals for reset success. They are to:

·         improve end user and veteran experience,

·         improve health system operations,

·         sustain high performance and high reliability of the system, and

·         mature program processes.

Further, VA identified and documented nine focus metrics aligned to the goal categories to measure and track progress over time. These metrics were selected from existing data. These metrics are:

·         percentage of staff who agreed that the EHR provides them with the ability to provide quality care,

·         time in EHR per patient seen for primary care/family medicine specialties,

·         sites projected patient workload as a percentage of the veterans integrated services network patient weighted workload for performance based funding allotments,[41]

·         percent total collections to expected results,

·         productivity work relative value units for clinical full-time equivalents,

·         incident-free time,

·         number of crashes a user experiences,

·         time to user-acknowledged incident ticket resolution in days, and

·         time to user-acknowledged change request ticket resolution in days.

The EHRM program identified baselines and targets for eight of nine metrics to measure the impact of the new system at the live sites. However, VA has not established baselines and targets for one metric: the time to user-acknowledged ticket resolution for change requests. According to EHRM program officials, VA had not fully defined the baselines and targets for its metrics because it recently implemented a revised process for addressing configuration change requests at the end of August 2024. (Appendix III provides more detailed information about VA’s EHR baselines and targets for its metrics.)

In September 2024, VA began to measure and report on progress toward targets. Specifically, the EHRM program reported meeting its targets for four of the eight metrics for which it had established targets. For example, VA had met its target for the time spent in the EHR per patient seen for primary care or family medicine, patient weighted workload for performance based funding allotments, incident-free time, and user interruptions.

However, VA had not yet met its targets for percentage of staff who agree that the EHR provides them with the ability to provide quality care, collected income, productivity, and time to user-acknowledged ticket resolution for tickets and change requests. Metrics are an important set of tools for the department to determine program and system improvements. Establishing all nine of the baselines and targets will provide VA management with complete information on program performance.

Conclusions

VA is continuing to work to improve the new EHR system and address concerns from the five initial sites as it emerges from a 20-month deployment pause. Implementing our prior recommendations is critical to VA reducing program risks and delivering a quality system.

VA has not updated its total life cycle cost estimate and integrated master schedule to reflect the reality of the program and the impact of delays, system changes, and updated processes. Developing an independent and reliable updated cost estimate is crucial for realistic planning, budgeting, and management. Given the pause, the volume and complexity of user-driven priority projects, and other delays and continuing issues, an updated cost estimate is essential to understand and oversee the magnitude of VA’s investment. Similarly, without a reliable schedule, management will face increased risks of not being able to effectively gauge progress and quickly identify and address problems.

User satisfaction has improved over the last 3 years, but users at the five initial sites continue to be generally dissatisfied with the new system. To provide insights on system progress, VA has finalized baselines and targets for most of the identified relevant metrics. Finalizing the lone remaining metric’s baseline and target can provide department leadership with important information of program performance.

Recommendations for Executive Action

We are making three recommendations to VA:

The Secretary of VA should direct the EHRM Integration Office to obtain an updated and independent total life cycle cost estimate using best practices described in GAO’s Cost Estimating and Assessment Guide. (Recommendation 1)

The Secretary of VA should direct the EHRM Integration Office to expeditiously and reliably update its integrated master schedule using best practices described in GAO’s Schedule Assessment Guide. (Recommendation 2)

The Secretary of VA should direct the EHRM Integration Office and the VHA to identify baselines and performance targets for all nine identified metrics intended to measure program and system performance. (Recommendation 3)

Agency Comments and Our Evaluation

We provided a draft of this report to VA for review and comment. In its written comments, reproduced in appendix IV, VA concurred in principle with one recommendation and concurred with the remaining two recommendations. Specifically:

·         The department concurred in principle with the recommendation to obtain an updated and independent total life cycle cost estimate. VA noted that it intends to employ best practices to develop a cost estimate for site deployments through May 2028, which is when the current contract with Oracle Health concludes, assuming all options are exercised. However, the department acknowledged that not all VA medical centers would have the modernized EHR system by the end of the contract; therefore, VA stated that the life cycle cost estimate would be dependent on a future acquisition.

·         VA concurred with the recommendation to expeditiously and reliably update its integrated master schedule. The department noted that it intends to employ best practices to develop the implementation schedule for site deployments through May 2028, which, as previously noted, is when the current contract with Oracle Health concludes. Further, the department acknowledged that the full schedule would be dependent on a future contract acquisition because not all VA medical centers would have the modernized EHR system by the end May 2028.

Although VA states that it agrees with our first two recommendations, the planned actions described by the department in its written comments fall short of addressing the issues identified in this report. Specifically, VA’s plan to update its life cycle cost estimate and integrated master schedule only encompasses deployments planned through the end of the current contract in May 2028. Accordingly, the department and Congress will be missing vital information needed for determining the magnitude and length of this modernization effort.

As such, we reiterate that it is important for VA to:

·         Update the total life cycle cost estimate for the EHRM program as recommended. A well-documented cost estimate must encompass the full life cycle of a program and includes, among other things, identifying and estimating all cost elements through each phase throughout the life of a program. It comprises all work, not just the work through the end of a limited time frame. Further, to mitigate overly optimistic cost estimates, it is also important to have an independent view of the program to help better understand risks and better manage them as they are realized. With this management tool, the department and Congress can access critical information they need to forecast the effects of delays or added effort and validate adjustments or quantify impacts of performance shortfalls. As such, it is important that VA update the total life cycle cost estimate for the EHRM program as recommended.

·         Update the EHRM integrated master schedule to include all deployments (e.g., the four Michigan sites as well as the remaining approximately 160 sites). A well-planned and integrated schedule defines when and how long work will take in a consolidated view of subprojects in relation to each other. As such, it is important that VA update the EHRM integrated master schedule to include all deployments. When delivered, the updated schedule can be used to help determine if the program’s parameters are realistic and achievable and can serve as a measure by which the program can be held accountable.

·         VA also concurred with the recommendation to identify baselines and targets for all nine identified metrics to measure program and system performance. The department also stated that the metrics and operational monitoring processes were implemented in August 2024 and requested closure of the recommendation. However, the baseline and target for one of the nine metrics has yet to be determined. The program planned to establish the baseline and target for all nine identified metrics after it had operationalized its process for addressing configuration change requests. These actions, if implemented as described, should address our recommendation.

VA also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to interested congressional committees and the Secretary of Veterans Affairs. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-4456 or at harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Carol C. Harris
Director, Information Technology Management Issues

List of Congressional Committees

The Honorable Jerry Moran

Chairman

The Honorable Richard Blumenthal

Ranking Member

Committee on Veterans’ Affairs

United States Senate

The Honorable John Boozman

Chair

The Honorable Jon Ossoff

Ranking Member

Subcommittee on Military Construction, Veterans Affairs, and Related Agencies

Committee on Appropriations

United States Senate

The Honorable Mike Bost

Chairman

The Honorable Mark Takano

Ranking Member

Committee on Veterans’ Affairs

House of Representatives

The Honorable John Carter

Chairman

The Honorable Debbie Wasserman Schultz

Ranking Member

Subcommittee on Military Construction, Veterans Affairs, and Related Agencies

Committee on Appropriations

House of Representatives

The objectives of this report were to (1) determine the improvements the Department of Veterans Affairs (VA) has made to its new electronic health record (EHR) system at initial deployment sites, (2) describe user feedback on the new EHR system, and (3) determine the extent to which VA is measuring modernization progress.

The scope of our review examined actions taken by VA since announcing a program reset in April 2023. Specifically, our focus was on the original five deployed sites and did not include a review of the sixth site of deployment, the Captain James A. Lovell Federal Health Care Center.

To address the first objective, we reviewed VA documentation describing the program’s planned incremental approach and progress made toward improving the EHR system. For example, we reviewed:

·         VA briefing slides describing the program’s planned incremental approach to rapidly improve the EHR system through configuration changes;

·         the department’s assessment of what needed to be fixed before the next deployment, including the highest priority enhancements to address critical patient safety issues;[42] and

·         the performance work statement for the VA Electronic Health Record Modernization (EHRM) system contract and related service level agreements identifying priority pharmacy enhancements and establishing trouble ticket resolution and system performance thresholds.

We also reviewed program briefing slides and reset committee meeting presentations to monitor the reported incremental progress of configuration changes, patient safety enhancements, pharmacy enhancements, and other high-priority projects. We reviewed VA’s report on the contractor’s performance and service level agreement measures to monitor progress on resolving trouble tickets and system performance. Further, based on this program documentation and discussions with program officials, we determined the implementation status of the recommendations we made in our prior reports regarding the EHRM program.[43]

Given the magnitude of the EHRM effort, we also determined the extent to which VA had updated the EHRM life cycle cost estimate and schedule. We identified having a total life cycle cost and an integrated master schedule as best practices for project planning.[44] We evaluated whether VA had an updated cost estimate and schedule. Specifically, we discussed with program officials the status of these planning documents as they reflected actions taken during the deployment pause. We also examined an EHRM cost estimate from January 2019 and an independent cost estimate from October 2022.

We supplemented our documentation review and analysis with interviews with VA officials from the EHRM Integration Office and the Veterans Health Administration (VHA). We also interviewed leadership from the five initial sites where the new system was deployed: Mann-Grandstaff VA Medical Center (Spokane, Washington), Jonathan M. Wainwright Memorial VA Medical Center (Walla Walla, Washington), VA Central Ohio Health Care System (Columbus, Ohio), Roseburg VA Medical Center (Roseburg, Oregon), and the White City VA Medical Center (White City, Oregon).

To address the second objective, we (1) obtained and reviewed results of surveys that VA conducted to determine users’ satisfaction with the new system, and (2) conducted structured interviews with selected users from the five locations where the new EHR was first deployed. In reviewing the results of VA’s user satisfaction surveys, we compared results from four surveys to determine trends in satisfaction over time.[45] We also assessed the reliability of the data from the user satisfaction surveys by reviewing documentation and written responses regarding the department’s administration of its user satisfaction surveys. We determined that the user experience survey data were sufficiently reliable for our purposes.

For our structured interviews, we interviewed 71 users from the first five sites between October 2023 to May 2024.

·         We selected a judgmental sample of users that was supplemented by random selections to help mitigate potential selection bias. We conducted structured interviews with 71 users: 61 judgmentally selected (10 users from Roseburg, 10 users from White City, 12 users from Columbus, 14 users from Walla Walla, and 15 users from Spokane), and 10 randomly selected (two from each of the five sites).

·         We judgmentally selected users that were service line chiefs who represented users within their departments. These users embodied a variety of roles (e.g., doctors, nurses, pharmacists, dentists, mental health professionals, and administrative functions) and could provide perspectives on areas that are unique to VA or have been of particular concern.

·         For the random selection of users, we obtained a list of users of the EHR system for each deployed site and associated data fields as of September 2023. We filtered the list of users by location, assigned a random number to each user record, sorted the list by random number from largest to smallest and selected the first two users from each location. We excluded users that had view-only rights to the system and then selected the next user from the list if participants declined or did not respond to our request for an interview.

While users’ responses cannot be generalized to the entire population of EHR users at the initial deployment sites, they represent a broad range of user roles and clinical areas at the sites and provide useful insight into user perspectives.

Where applicable, we compared responses to similar structured interview responses that we obtained in 2022 from user feedback on the new EHR.[46] We conducted the structured interviews in 2022 with 63 users from Mann-Grandstaff VA Medical Center, Jonathan M. Wainwright Memorial VA Medical Center, and VA Central Ohio Healthcare System (the first three locations where the new EHR was deployed). We conducted these interviews between April and August 2022 using substantively identical questions in both series of interviews. To compare responses from our 2022 and our 2023-2024 interviews, we conducted categorical tests of association to determine whether a relationship exists between the rating levels of the nongeneralizable survey respondents for select questions asked in both 2022 and 2023-2024.[47]

To address the third objective, we reviewed federal guidance and leading practices for performance measures.[48] Based on that review, we selected applicable leading practices for measuring the program’s progress on improvements such as establishing goals, metrics, baselines, and targets, as well as measuring and reporting progress. We reviewed program briefing slides and compared the program’s efforts for establishing goals, metrics, baselines, and targets, and measuring and reporting progress to the applicable leading practices.

We determined that the control environment component of internal controls was significant to our review of the EHRM control activities, along with the underlying principle that management should design control activities to achieve objectives and respond to risks and design the information system to achieve objectives.[49] We assessed the extent to which the program established objectives and improved the new EHR system to achieve those objectives. We supplemented our analysis with interviews with VA officials from the EHRM Integration Office and the VHA.

We conducted this performance audit from June 2023 to March 2025 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

To gain user perspective on the extent to which the Department of Veterans Affairs (VA) has made progress toward improving the new electronic health record (EHR) system, we conducted structured interviews with 71 users from the Mann-Grandstaff VA Medical Center, Jonathan M. Wainwright Memorial VA Medical Center, VA Central Ohio Healthcare System, Roseburg, and White City, the first five locations where the new EHR was deployed. We conducted these interviews between October 2023 to May 2024. While users’ responses cannot be generalized to the entire population of EHR users at the initial deployment sites, they represent a broad range of user roles and clinical areas at the sites.

Each structured interview comprised closed- and open-ended questions. In this appendix, aggregated results of responses to the closed-ended questions we asked in our structured interviews with users of VA’s new EHR system are shown below.

	2023-2024
	At this point in time, how does your experience in using the new EHR system compare to your experience using the new system in April 2023?
Greatly improved	2
Slightly improved	19
Did not improve or worsen	34
Slightly worsened	8
Greatly worsened	3
No basis to judge	5
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	Compared to April 2023, how much time are you spending using the new EHR system to perform your job tasks?
Much more time	9
More time	8
About the same amount of time	43
Less time	4
Much less time	2
No basis to judge/don’t know	5
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	The new EHR system allows me to  effectively perform the duties  of my position.
Strongly agree	3
Agree	12
Neither agree nor disagree	9
Disagree	26
Strongly disagree	17
No basis to judge	4
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	The new EHR system meets my expectations.
Strongly agree	1
Agree	3
Neither agree nor disagree	4
Disagree	26
Strongly disagree	36
No basis to judge	1
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	The new EHR system is available when I need it.
Strongly agree	0
Agree	30
Neither agree nor disagree	16
Disagree	15
Strongly disagree	8
No basis to judge	2
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	The new EHR system’s performance is timely (e.g., minimal time to load pages and respond to commands).
Strongly agree	1
Agree	10
Neither agree nor disagree	8
Disagree	35
Strongly disagree	15
No basis to judge	2
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	Health data in the new EHR system is accurate.
Strongly agree	1
Agree	18
Neither agree nor disagree	9
Disagree	29
Strongly disagree	8
No basis to judge	6
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	Information in the new EHR system is presented in a logical manner.
Strongly agree	1
Agree	14
Neither agree nor disagree	11
Disagree	28
Strongly disagree	16
No basis to judge	1
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	I do not rely on workarounds to perform the duties of my position.
Strongly agree	1
Agree	4
Neither agree nor disagree	3
Disagree	24
Strongly disagree	34
No basis to judge	5
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	Compared to April 2023, the new EHR system requires fewer steps to accomplish what I need to do.
Strongly agree	0
Agree	4
Neither agree nor disagree	15
Disagree	24
Strongly disagree	24
No basis to judge	4
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  | GAO‑25‑106874

	2023-2024
	I have been provided effective training on how to use the new EHR system.
Strongly agree	0
Agree	3
Neither agree nor disagree	7
Disagree	27
Strongly disagree	33
No basis to judge	1
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	The helpdesk provides adequate resolution to problems I encounter with the new EHR system.
Strongly agree	0
Agree	6
Neither agree nor disagree	10
Disagree	30
Strongly disagree	21
No basis to judge	4
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	Overall, I am satisfied with the new EHR system.
Strongly agree	1
Agree	4
Neither agree nor disagree	5
Disagree	28
Strongly disagree	32
No basis to judge	1
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	Overall, in what way has the new EHR system affected your ability to perform the duties of your position?
Very positive impact	1
Positive impact	1
Neither positive nor negative impact	12
Negative impact	37
Very negative impact	19
No basis to judge	1
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	How has the new EHR system affected your productivity?
Greatly increased productivity	0
Increased productivity	1
Neither increased nor decreased productivity	14
Decreased productivity	30
Greatly decreased productivity	24
No basis to judge	2
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	How has the new EHR system affected veterans’ quality of care at this facility?
Greatly improved the quality of care	1
Improved the quality of care	1
Neither improved nor reduced the quality of care	16
Reduced the quality of care	36
Greatly reduced the quality of care	13
No basis to judge	4
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	From your perspective, how has the new EHR system affected patient safety?
Greatly increased patient safety risks	16
Increased patient safety risks	42
Neither increased nor decreased patient safety risks	9
Decreased patient safety risks	1
Greatly decreased patient safety risks	0
No basis to judge	3
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	To what extent has system performance (e.g., minimal time to load pages  and respond to commands)  improved or worsened?
Greatly improved	0
Slightly improved	15
Did not improve or worsen	38
Slightly worsened	9
Greatly worsened	5
No basis to judge	4
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.   |  GAO‑25‑106874

	2023-2024
	How satisfied have you been with communications about changes  to the new system?
Very satisfied	2
Satisfied	14
Neither satisfied nor dissatisfied	19
Dissatisfied	19
Very dissatisfied	14
No basis to judge	3
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

	2023-2024
	In general, how satisfied have you been in obtaining resolution to problems with the new EHR system when you have contacted the helpdesk?
Very satisfied	0
Satisfied	3
Neither satisfied nor dissatisfied	8
Dissatisfied	23
Very dissatisfied	26
I have never contacted the helpdesk for assistance with the EHR system	11
Total	71

Source: GAO analysis of interviews with Department of Veterans Affairs officials.  |  GAO‑25‑106874

We conducted structured interviews regarding user feedback on the new EHR with similar questions in 2022.[50] Of the users we interviewed in 2022 and 2023-2024, we did not detect a statistically significant association (either in a positive or negative direction) between rating levels across the two time periods for most questions.[51] However, we detected a significant association with rating levels and interview year for one question about experience with the system. Specifically, users responded more negatively when asked “At this point in time, how does your experience in using the new EHR system compare to your experience using the new system in April 2023” indicating that experiences had worsened.[52]

Goal category	Goal objective	Reset result focus metric	Baseline	Target	Desired trend direction	Measure (as of date)	Does measure meet target?
Improve end user and veteran experience	Improved overall user experience	Percentage of staff who agree that this EHR provides them with the ability to provide quality care	Initial survey rating–In 2022, 7 percent of respondents said they agreed or strongly agreed that the EHR enabled them to deliver high-quality care	20+ points from the initial survey rating, since the initial response was 7 percent, the target is 27 percent of users agreeing or strongly agreeing	Increase	As of September 2024, 23 percent of respondents agreed or strongly agreed that the EHR enabled them to deliver high-quality care	No
	Use of EHR is efficient	Time in EHR per patient seen for primary care	No comparable data from legacy system, go-live data varied between sites from 40-275 minutes	35-45 minutes	Decrease	37.22 minutes (as of August 2024)	Yes
Improved health system operations	Performance based funding allotments to veterans integrated services networks (VISN) and facilities result in acceptable operating income	Projected patient weighted work over baseline for facilities	2019 patient weighted work percentage of VISN 13.5 percent	Above the 2019 patient weighted work percentage of VISN	Increase	14.56 percent (as of September 2022)	Yes
	Collected income is restored	Percent total collections to expected results–monthly	Varies from site to site	100 or above percent of expected collections amount	Increase	72.35 percent (as of September 2024)	Noa
	Productivity is restored	Productivity (work relative value units) filtered for clinical full-time equivalents	Site pre-go-live baseline	100 percent of facility pre-go-live baseline compared to site itself	Increase	Roseburg – 90.11 Spokane – 106.87 Walla Walla – 77.68 White City – 98.78 Columbus – 90.52 (as of August 2024)	Yes, for one initial site, no for four initial sites
Mature product	EHR is available	Incident-free time of Oracle Health owned incidents measure: percentage of time solution or functions are working as intended	Greater than 95 percent	Three or more consecutive months of incident-free time being greater than 95 percent	Increase	97.56 percent (as of August 2024)	Yes
	Users experience few lags and hangs	User interruptions: number of crashes 10 percent of users experience	Less than 10	Three or more consecutive months of less than 10 interruptions per day for 10 percent of users	Decrease	2.52 (as of August 2024)	Yes
Mature program processes	Tickets processing satisfies users and facilities	Time to user acknowledged ticket resolution for incidents (in days)	5.73 days (as of June 2023)	Decreasing and stable backlog less than or equal to 5 days	Decrease	6.11 days (as of August 2024)	No
	Tickets processing satisfies users and facilities	Time to acknowledged ticket resolution for change requests (in days)	To be determined	To be determined	Decrease	142 days (as of August 2024)	N/A

Source: GAO analysis of VA documentation.  |  GAO‑25‑106874

^aAccording to VA, annual collection goals are established the year prior based on historical number of patients treated. Prior to EHRM transition, staffing levels and productivity declined due to training and education provided to the sites, the impact from COVID-19, and a data breach to a partner system. This decline impacted collections months prior to transition.

GAO Contact

Carol C. Harris, (202) 512-4456 or harriscc@gao.gov

Staff Acknowledgments

In addition to the individual named above, the following staff made key contributions to this report: Mark Bird (Assistant Director), Jennifer Stavros-Turner (Assistant Director), Merry Woo (Analyst in Charge), John Bornmann, Chris Businsky, Quintin Dorsey, Rebecca Eyler, Anthony Gray, Jacqueline Mai, Sachin Mirajkar, Monica Perez-Nelson, Frances Tirado, Walter Vance, and Geoffrey Zhang.

GAO’s Mission

The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through our website. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. You can also subscribe to GAO’s email updates to receive notification of newly posted products.

Order by Phone

The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, https://www.gao.gov/ordering.htm.

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO

Connect with GAO on Facebook, Flickr, X, and YouTube.
Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts.
Visit GAO on the web at https://www.gao.gov.

To Report Fraud, Waste, and Abuse in Federal Programs

Contact FraudNet:

Website: https://www.gao.gov/about/what-gao-does/fraudnet

Automated answering system: (800) 424-5454 or (202) 512-7700

Congressional Relations

A. Nicole Clowers, Managing Director, ClowersA@gao.gov, (202) 512-4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548

Public Affairs

Sarah Kaczmarek, Managing Director, KaczmarekS@gao.gov, (202) 512-4800, U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, DC 20548

Strategic Planning and External Liaison

Stephen J. Sanford, Managing Director, spel@gao.gov, (202) 512-4707
U.S. Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548