Printer Friendly Page
VA'S NORTHERN CALIFORNIA COMPUTER SYSTEM
SUFFERED EPIC FAILURE -- A blow-by-blow account
of the VistA system failure detailed a cascade
of events,
including two backup system failures and a lack
of
communication that exacerbated the problem.
For more on the VA's VistA medical records
system, use the VA Watchdog search engine...click here...
http://www.yourvabenefits.org/ses
search.php?q=VistA+CPRS&op=or
Story here...
http://www.modernhealthcare
.com/apps/pbcs.dll/article?AID=/2007100
1/FREE/310010001/0/FRONTPAGE
Story below:
-------------------------
Calif. system faced epic VistA failures: report
By: Joseph Conn
HITS staff writer
The Veterans Health Administration’s VistA clinical information
technology system and its key Computerized Patient Record System, or
CPRS, went on a historic fritz Aug. 31 at the VA’s Northern California
Healthcare System—an integrated healthcare delivery network serving
377,000 veterans with close to 2,000 patient visits a day—disrupting
service for more than eight hours, a pair of VA physician IT leaders
told a congressional oversight panel last week.
In the opinion of one VA physician-informaticist who testified before
the House Veterans Affairs Committee on Wednesday, part of the failure
can be attributed to changes in the management structure of information
technology services at the VA that were designed to improve system
security and strengthen IT project oversight. The House committee held a
hearing on the status of the IT restructuring program.
The overhaul of IT management at the VA came in the wake of the 2004
fiasco involving an unusable computer system for materials and financial
management. The Core Financial and Logistics System, or CoreFLS, was
developed under contract with consultant BearingPoint. VA officials
scrapped the failed system after five years of development and a cost of
nearly $250 million.
A blow-by-blow account of the VistA system failure in Northern
California detailed a cascade of events, including two backup system
failures and a lack of communication that exacerbated the problem. The
report was given by physician Bryan Volpp, associate chief of staff,
clinical informatics, for the Northern California system. The system is
composed of 17 patient-care sites, including a 50-bed acute- and
critical-care hospital in Sacramento and a 115-bed inpatient nursing
home and subacute-care facility in Martinez, Calif. A copy of Volpp’s
written testimony is posted on the House committee's Web site. Volpp
described a “major disruption” that began as a log-on failure at 7:30
a.m., as physicians and clinicians prepared for their day of patient
visits scheduled to begin at 8 a.m. Volpp said the problem was traced to
the Sacramento Regional Data Processing Center, which provides a
centralized database for the 17 Northern California care sites. The
sites tried to implement their backup procedures, but two of those—a
transfer to the database at the Denver data center and use of a system
that affords read-only access to existing data—both failed. Timely
reporting of the cause of the failure and estimates of the projected
downtime, typical of previous outages, also did not occur, Volpp said.
One saving grace was a third and final backup procedure—a switch to
printing out care-record summaries hosted on local personal
computers—that was activated successfully, albeit not quickly enough
that morning to prevent some clinicians from seeing patients scheduled
for the earliest appointments without having access to any of their
medical records.
A host of other problems ensued, including:
# The medical staff was forced to write discharge instructions and notes
on paper.
# The electronic lists of instructions and of medications were not
available for the patients being discharged.
# Patients being discharged could not be given follow-up appointments at
the time of discharge. The appointments had to be made later and the
patient notified by phone.
# There were delays in obtaining discharge medications and patients
remained on the wards longer than would normally be required.
# The nurses administered medications to the patients and used the paper
Medication Administration Record, or MAR, to record the administration
events. Initial medication passes were interrupted and delayed until the
paper copies of the (MAR) could be printed.
In fact, it took VA staff almost a week to get medication administration
records in the restored computerized system fully up to date, Volpp
said. The total effects of the system outage will last much longer,
according to Volpp.
“Administrative staff worked for over two weeks to complete the
checkouts on all the patients who were seen that day,” he said.
“However, entering checkout data on all these patients many days after
the fact is potentially inaccurate,” he said. “Many providers have gone
back into CPRS and tried to reconstruct notes that summarize the paper
notes that they wrote in order to mitigate the risk of missing
information. This work to recover the integrity of the medical record
will continue for many months since so much information was recorded on
paper that day. When you consider that hundreds of screening exams for
PTSD, depression, alcohol use and smoking, and entry of educational
interventions, records of outside results, discharge instructions and
assessments are all now on paper and are not in a format that is easily
found in the electronic record, the burden of this one failure will
persist for a long time.”
J. Ben Davoren, the physician director of clinical informatics at the
San Francisco VA Medical Center, said the broader VA management
realignment is causing its own problems, and the Aug. 31 system failure
is but one serious example.
Davoren called the Northern California system failure, “the most
significant technological threat to patient safety VA has ever had.”
Some of these potential problems with the management realignment were
foreseen and communicated to top VA officials as far back as 2005,
Davoren said. Disconnecting local facility officials from control over
needed IT projects has caused delays in development and implementation,
according to Davoren. Local innovation and multiple iterations of
software development facilitated by the close proximity of clinicians
and programmers has been a hallmark of the VA’s clinical system
development effort over the years.
According to his written testimony, Davoren said that in response to
then-VA Secretary Anthony Principi’s proposals for IT realignment, “I
believe that employees at some medical centers expressed a number of
concerns about the details of the plan. In particular, I believe they
felt that the regionalization of IT resources would create new points of
failure that could not be controlled by the sites experiencing the
impact, and that the system redundancy required to prevent this was
never listed as a prerequisite to centralization of critical
patient-care IT resources. From my point of view as the director of
clinical informatics, it was clear to me that the focus of
reorganization/realignment was on technical relationships and not on how
the missions of VHA would be communicated to the new (Office of
Information and Technology) structure. For example, realignment success
metrics were focused on (regional data processing center) deliverables
rather than facility needs. Finally, key facility-based IT staff had
been tightly integrated into local committees and planning groups as
subject matter experts, but could no longer be tasked directly by the
facility director to participate, and had no clear OI&T-driven incentive
to continue. Ultimately, the concern was that in trying to create a new
structure in the name of ‘standardization,’ support would wane to a
‘lowest common denominator’ for all facilities, no matter how diverse
their actual needs were.”
“In my view, there remains a tremendous uncertainty about how to work
with our long-standing IT colleagues to address local or regional
clinical care, research or educational needs,” Davoren said. As a
result, he said, “There is a sense of great inertia that overrides the
anticipation of great opportunities in the new OI&T structure.”
Davoren said there has been a welcome consciousness-raising within the
VA about privacy and security issues, but heightened security measures
also have had drawbacks, including difficulty scheduling teleconferences
and other snafus. “For example, to fully comply with security
requirements on our examination-room PCs, we must log out of both a
clinical application such as our Computerized Patient Record System and
the Microsoft Windows operating system each time we leave the room even
for a moment, yet it may take as long as 12 minutes to log back on when
we return. Given a 20- or 30-minute visit with their veteran patient,
the clinician is thus forced to choose to “do the right thing” for
either the patient or the system, but cannot do both, “the bad news is
that centralization of physical IT resources to the (regional approach)
has directly led to more system downtime for individual medical centers
than they have ever had before, resulting in hundreds of simultaneous
threats to the safety of our veteran patients.”
-------------------------
Larry Scott --
