Evaluate Solution Performance - Business Analyst

Purpose

Evaluate functioning solutions to understand the value they deliver and identify opportunities for improvement.

Description

Solution evaluation involves investigating how a solution is actually used after it is deployed, and assessing the effect it has had, both positive and negative. It may also be referred to as post - implementation assessment when performed immediately following the completion of a project.

Solutions may be adapted and modified directly by end users, including use of manual work arounds, recording of additional information, and adoption of informal policies and procedures in order to resolve problems that have occurred or to allow new uses of the solution. In order to properly evaluate the solution, it is also necessary to understand when, where and why this has occurred and assess the benefit that these changes have brought to the organization.

Input

Business Requirements: The performance of the solution will be measured against the business requirements. Without clear business requirements it is impossible to assess the solution’s performance effectively, since there are no defined goals that it is supposed to meet.

Identified Defects: Any known defects must be considered in assessing the quality of a solution.

Solution Performance

Solution Performance Metrics: These represent the criteria by which the performance of the solution is to be assessed. They may be quantitative (measures of time, volume, revenue, errors found, or other information for which hard numbers are available) or qualitative (user or customer satisfaction, recommendations, or other measures which summarize the opinions of stakeholders).

Solution [Deployed]: This task cannot be performed until the solution is in use.

Elements

1. Understand Value Delivered By Solution

Gather the actual metrics that describe the performance of the solution. Applications may automatically report on some or all of the defined metrics, but where they do not, it will be necessary to gather qualitative and quantitative performance information. Significant over or under-performance against targets may be investigated to identify a root cause or determine an appropriate response.

If the root cause for under-performance is a factor that is potentially under the control of the enterprise, addressing it may become a business need.

Significant over - performance may indicate that resources devoted to the solution can be used elsewhere, or that the value of the solution to the business was underestimated. It is likely that there are lessons that can be learned and applied elsewhere.

2. Validate Solution Metrics

In some cases, the performance of a solution will be considered excellent, based on the defined performance metrics for that solution, but the business goals and objectives that those metrics are supposed to be aligned with are not being met. An analysis effort to identify and define more appropriate metrics, including modification of the solution to collect and report on those metrics, may be required.

3. Solution Replacement or Elimination

Eventually, it will be necessary to consider the replacement of a solution or solution component. This may occur because an IT system or other technology component has reached the end of its useful life, services are being insourced or outsourced, the solution is not fulfilling the business goals set for it, or for any number of other reasons. Issues that may influence the replacement or elimination decision may include:

  • Ongoing Cost versus Initial Investment:It is common for the existing solution to have increasing costs over time, while alternatives have a higher investment cost up front but lower maintenance costs.
  • Opportunity Cost:Opportunity cost represents the potential value that could be realized by pursuing alternative courses of action. Replacement of an existing solution is unlikely to produce high initial returns on investment (as it will likely replicate existing capabilities, at least initially, rather than create many new ones). As the effort to develop a replacement will pull resources away from other initiatives the organization may be considering, the potential benefits from those initiatives need to be considered to determine if they are greater than the benefit of replacement (this is generally not a consideration when considering elimination).
  • Necessity: Most solution components have a limited lifespan (due to obsolescence, changing market conditions, and other causes). After a certain point in the lifecycle it will become impossible to maintain the existing component.
  • Sunk Cost: Sunk cost describes the money and effort already committed to an initiative. The psychological impact of sunk costs may make it difficult for stakeholders to objectively assess the rationale for replacement or elimination, as they may feel reluctant to “waste” the effort or money already invested. As this investment cannot be recovered, it is effectively irrelevant when considering future action. Decisions should be based on the future investment required and the future benefits that can be gained.

Techniques

Decision Analysis: A cost / benefit analysis is typically used to determine the financial impact of the solution on the organization. While critical, it is important to ensure that non - financial costs (including opportunity cost) and benefits are evaluated.

Focus Groups: Useful to gain a detailed qualitative understanding of the value of a solution to a group of stakeholders. It can be used to uncover new information beyond the scope of previously defined metrics.

Observation: May reveal uses or problems that are not being reported.

Survey / Questionnaire: Enables gathering quantitative and qualitative information from large numbers of stakeholders. If a survey is properly designed, and is responded to by a statistically significant and representative sample of the stakeholder population, it can accurately reflect the opinions of the entire population. Surveys are not especially effective at eliciting unexpected information.

Stakeholders

Customer, Domain SME, and Supplier: May provide recommendations for improvements.

End User: responsible for the day-to-day operation of the solution and a major source of information on problems or defects.

Operational Support: Will be involved in monitoring the performance and effectiveness of a solution or its components.

Regulator: May have requirements regarding the performance of a solution that must be met on an ongoing basis.

Sponsor: The person responsible for the operation of the solution from a business perspective will be responsible for deciding if the solution evaluation warrants the initiation of a change initiative.

Output

Solution Performance Assessment: Describes how the solution is performing in relation to business goals and objectives.

All rights reserved © 2018 Wisdom IT Services India Pvt. Ltd DMCA.com Protection Status

Business Analyst Topics