Table of Contents | Appendix C-33 | Appendix C-35
The Post-Implementation Review is used to evaluate the effectiveness of the system development after the system has been in production for a period of time (normally 6 months). The objectives are to determine if the system does what it is designed to do : Does it support the user as required in an effective and efficient manner? The review should assess how successful the system is in terms of functionality, performance, and cost versus benefits, as well as assess the effectiveness of the life-cycle development activities that produced the system. The review results can be used to strengthen the system as well as system development procedures.
The review is scheduled to follow the release of a system or system revision by an appropriate amount of time to allow determination of the effectiveness of the system. A representative from the functional development group or other member of the major user organization participates in the review. The System Proponent ensures that all documentation and all personnel needed to participate in the review are accessible.
The reviewer and an assigned team collect the information needed for the Post-Implementation Review by interviewing end users and their managers, system administrators, and computer operations personnel. The report is then prepared and provided to the user organization that requested it and the information systems organization, which may jointly use the findings to initiate other actions.
The Post-Implementation Review is a free-form report, and not all sections are relevant or necessary to the final product. A description of the Post-Implementation Review Report is attached
1.0 INTRODUCTION
1.1 Project Identification
Provide the identifying information associated with the project, including the applicable project control code, system acronym, and system title.
1.2 System Proponent
Provide the name of the System Proponent.
1.3 History of the System
Briefly describe the system's history and predecessor, if any. State the mission needs and information requirements, including how the system is expected to help users.
1.4 Functional System Description and Data Usage
Briefly describe what the system does functionally and how the data are used by the system.
2.0 EVALUATION SUMMARY
The purpose of this section is to provide a summary of the overall adequacy and acceptance of the system.
2.1 General Satisfaction With the System
Describe the users' experience with the implemented system. Comments should address the following:
• The level of user
satisfaction
• The strengths of
the system, including specific areas of success
• Any problems
• Frequently used
features
• Infrequently used
features
• Features not used
at all
• Suggested improvements
2.2 Current Cost-Benefit Justification
Assess if the system is paying for itself. Base the assessment on the anticipated benefits and costs projected during the System Concept Development phase and revised during the subsequent phases of the systems development life cycle. This section is intended merely to review the costs and benefits and to provide details of costs and benefits in other sections. Comments should address the following:
• The extent of the
benefits and if they are reported to be less or greater than those
projected in the development
analysis and functional requirements report
• If any difference
is permanent or will change over time
• If the system is
or will be cost-justifiable.
2.3 Needed Changes or Enhancements
Gauge the magnitude of effort needed to change or improve the system. Describe the nature and priority of the suggested changes; more detail will be provided in other sections. Comments should address the following:
• The suggested changes
• The scope of the
changes
• The resource requirements
to effect the changes
3.0 ANALYSIS AND IMPLEMENTATION
The purpose of this section is to gauge the completeness of the functional requirements and implementation according to the study.
3.1 Purpose and Objectives
Evaluate the adequacy of the original definition of purpose and objectives presented in the functional requirements document and if the objectives were achieved during implementation. Evaluate if any objectives have changed or should have changed. Comments should address the following:
• Extent to which
goals were met
• The level of the
objective definition
• Extent to which
objectives were met
• Possible changes
to the objectives
3.2 Scope
Analyze if proper limits were established in the feasibility study and if they were maintained during implementation. Comments should address the following:
• Variations from
the scope definition as agreed to in the concept development
• The extent to which
the scope was followed
• Any possible future
changes to the scope
3.3 Benefits
Analyze if the benefits anticipated in the concept development and requirements definition analyses were realized. Detail all benefits, quantifiable or nonquantifiable, and any quantifiable resources associated with each. Comments should address the following:
• The adequacy of
the benefit definition
• The level of benefits
realized
• The anticipated
benefits that can be realized
• The reason for the
variance between planned and realized benefits
3.4 Development Cost
Determine the adequacy of the development cost estimated and any deviation between the estimated and actual development costs. Comments should address the following:
• The adequacy of
the original and subsequent cost estimates
• The actual costs,
by type
• The reasons for
any difference between estimated and actual costs
3.5 Operating Cost
Analyze the adequacy of the operating cost estimates and any deviation between the estimate and the actual operating costs. Summarize the resources required to operate the system. Comments should address the following:
• The adequacy of
the operating estimates
• The actual operating
costs
• The difference
3.6 Training
Evaluate if all levels of user training were adequate and timely. Comments should address the following:
• The timeliness
of the training provided
• The adequacy of
the training
• The appropriateness
of the training
• Identification of
additional training needs by job category
• The ability of the
personnel to use the training provided
4.0 OUTPUTS
The purpose of this section is to evaluate the adequacy and usefulness of the outputs from the system. Care must be taken to ensure that all reports are evaluated.
4.1 Usefulness
Measure the extent to which the users need the output of the system. Comments should address identification of the level of need, such as the following:
• Utility
-
Absolutely essential
- Important and highly desirable
- Interesting-- proves what is already known
- Incomplete-- does not provide all the necessary information
- Unnecessary
• Identification of
information/reports needed but not currently generated by the system or
unable to be obtained
• Demonstration of
the ability to do without the reports
• Alternatives for
obtaining the information where improvements can be achieved
4.2 Timeliness
Determine if output production performance meets user needs. Comments should address the frequency with which output arrives on time, early, and late; and the amount of follow up needed to obtain the output.
4.3 Data Quality
Assess the need to provide for effective use of shared data to enhance performance and system interoperability. Comments should address data accuracy and data reliability.
5.0 SECURITY
The purpose of this section is to determine if the system provides adequate security of data and programs. In addition to access security, procedures for backup, recovery, and restart should be reviewed.
5.1 Data Protection
Determine if the security, backup, recovery, and restart capabilities adequately safeguard data, including master, transaction and source. Online systems naturally require special techniques (such as, transaction logging). Comments should address the following:
• The adequacy of
the security, backup, recovery, and restart procedures
• The suggested changes
• The effort required
to make the changes
5.2 Disaster Recovery
Determine if appropriate files, programs, and procedures are established to enable recovery from a disaster resulting in the loss of data. Comments should address the following:
• The adequacy and
currency of off site storage procedures
• The extent that
procedures cover the following:
- Master data
- Transaction data
- Source programs
- Object programs
- Documentation (such as, systems, operations, user manuals)
• The results of any
adequacy-of-recovery test
5.3 Controls
Evaluate the adequacy of the controls on the database, source documents, transactions, and outputs of the system. Review each area thoroughly for financial controls and file control counts. Comments should address the following:
• The level of controls
present in the entire system and on each component (such as,
transaction and batch,
and file)
• The adequacy of
the controls, including the strengths and possible areas for improvement
• The amount of resources
required, if any, to obtain improvements
5.4 Audit Trails
Review the ability to trace transactions through the system and the tie-in of the system to itself. Comments should address the following:
• The thoroughness
of the audit trails
• The level of improvements
necessary, if any
• The requirements
of audit trails as outlined in the trusted criteria-- such as, C2
requirements--if any
5.5 Allowed Access
Evaluate the adherence to restriction of access to data. State desired privacy criteria for the system and then evaluate how the criteria have been followed up to this point. Comments should address the following:
• Established privacy
criteria
• Recommended privacy
criteria
• Adherence to and
violations of privacy
• The cost of providing
this level of privacy
• The potential effect
on individuals if the privacy criteria are not followed
6.0 COMPUTER OPERATIONS
The purpose of this section is to ascertain the current level of operational activities. Although the user point of view is primary to the Post-Implementation Review Report, the computer operations view is also important to investigate.
6.1 Control of Work Flow
Evaluate the user interface with the data processing organization. Investigate the submittal of source material, the receipt of outputs, and any problems getting work in, through, and out of computer operations. Comments should address the following:
• Any problems in
accomplishing the work
• The frequency and
extent of the problems
• Suggested changes
• The effort required
to make the changes
6.2 Scheduling
Determine the ability of computer operations to schedule according to user needs and to complete scheduled tasks. Comments should address the following:
• Any problems in
accomplishing the work
• The frequency and
extent of the problems
• Suggested changes
• The effort required
to make changes
6.3 User Interface
Analyze the usability of the system. The transaction throughput and error rate are included in this analysis. Comments should address the following:
• Volume of data
processed (number of transactions)
• Number of errors
made
• Frequency of problems
with the interface
• Suggested changes
• Effort required
to make the changes
6.4 Computer Processing
Analyze computer processing issues and problems. Some areas to review are as follows:
• The correct or
incorrect use of forms and off line files
• The adequacy of
instructions (such as, forms lineup and proper responses on the console)
• The extent of reruns,
if any
6.5 Peak Loads
Assess the ability of the system to handle peak loads and to resolve backlogs when they occur. Any offloading that could be helpful should be investigated. Comments should address the following:
• The level of user
satisfaction
• The adequacy of
the response time (for online systems)
• The effect of delays
on online and/or batch systems
• Suggested changes
• The effort required
to make the changes
7.0 MAINTENANCE ACTIVITIES
The purpose of this section is to evaluate maintenance activity involving the system.
7.1 Activity Summary
Provide a summary of maintenance activity to date. Provide type, number of actions, and scope of changes required. Estimate a projected maintenance workload based on the findings of the review. Discuss the adequacy of maintenance efforts or if major enhancement/revision is required.
7.2 Maintenance Review
Review completed and pending changes to the system. Provide conclusions regarding the benefits to be achieved by completing recommended changes. Provide conclusions about the amount of maintenance required based on activity that has occurred to date.
7.3 System Maintenance
Discuss the system maintenance based on the design, types of changes required, documentation, and knowledge about the system (both user and technical personnel).
Post-Implementation Review Outline
Cover Page
Table of Contents
1.0 Introduction
1.1 Project
Identification
1.2 Requesting
Organization
1.3 History
of the System
1.4 Functional
System Description and Data Usage
2.0 Evaluation Summary
2.1 General
Satisfaction with the System
2.2 Current
Cost-Benefit Justification
2.3 Needed
Changes or Enhancements
3.0 Analysis and Implementation
3.1 Purpose
and Objectives
3.2 Scope
3.3 Benefits
3.4 Development
Cost
3.5 Operating
Cost
3.6 Training
4.0 Outputs
4.1 Usefulness
4.2 Timeliness
4.3 Data
Quality
5.0 Security
5.1 Data
Protection
5.2 Disaster
Recovery
5.3 Controls
5.4 Audit
Trails
5.5 Allowed
Access
6.0 Computer Operations
6.1 Control
of Work Flow
6.2 Scheduling
6.3 User
Interface
6.4 Computer
Processing
6.5 Peak
Loads
7.0 Maintenance Activities
7.1 Activity
Summary
7.2 Maintenance
Review
7.3 System
Maintenance