A Novel Methodology for Project Assessment and Evaluation

Technical and managerial viewpoints are important elements in the assessment and evaluation of project success. Project success can be measured by the level of achieving project objectives. A by-product of this process will lead to providing an independent and, unbiased opinion on the gains and losses attributed to the project. This may be interpreted as a way of performing a rough benefit/cost analysis. The process is done through addressing key issues that will lead to an action plan which serves in optimizing the sustainable benefits of the project and mitigate the negative aspects, or failures, of the project. The assessment and evaluation process must focus on providing the management and stakeholders involved in any project with solutions and directions that will serve in project success to achieve its objectives. The proposed methodology was used in the final evaluation of an energy efficiency program in a developing country. The overall objective achievement for the project reached about 80%, which is a good result. Moreover, the overall calculated B/C ratio for the project is 1.31, which indicates the wellbeing of that project.


Introduction
Projects represent the driving force for success of societies, companies and individuals.
Once project targets are set out, and as the project commences, then it must be monitored and evaluated to check for its success and efficiency.This is not an easy task, and many measures are implemented to keep track of project performance and consistency thereof.In the advanced stages, i.e. upon project completion, a final assessment and evaluation should be carried out which will lead to determine the effectiveness of project implementation, assessing the impact and sustainability of results, highlighting issues requiring decisions and actions, and Presenting initial lessons concerning the project design, implementation and management.
The project being evaluated in this paper is an energy efficiency project at the national level in one of the developing countries.One of the authors was contracted to perform a final evaluation process of the project.The project at the time of the final evaluation was practically complete after two and a half years since inception.The terms of reference and scope of work for the final evaluation was prepared by the contracting entity.
The process of project evaluation involves imple-menting different tasks including: 1) Review of project concept, design, and performance in terms of: timeliness, quality of inputs, cost-effectiveness, efficient utilization of inputs available, quality and quantity of outputs produced, achievement of immediate objectives, efficiency of project management and institutional framework, quality and effectiveness of technical support, and quality and timeliness of monitoring.
2) Review of project impact in terms of economic, technological, and social effects.
3) Systematic dissemination of project results, utilization of the personnel trained under the project.
4) Sustainability of project achievements, and environmental considerations.
5) Negative effects, and the lessons learned represented by the major problems in project implementation, and the impacts on the development of the project owner.
Energy sector related projects are very important in modern world, as they touch the daily needs of individuals, companies, utilities, and governments.Measuring the success of any energy-related project is very important due to its impact on the stakeholders.
In this paper a proposed methodology is laid out that is properly designed to assess the degree of success of a given energy oriented project.This is done through the analysis of surveys and queries which produce project progress related indicators.The validity of this methodology was tested as it was implemented in a developing country and results reflected the project situation to a high degree of reality.
The people in charge of that project did not want the authors to cite the name of the project or the country, for reasons the authors do not understand.This looks peculiar in light of the good results of the project.
This paper is organized as follows: in Section 2 project monitoring, evaluation and control are discussed.In Section 3 the assessment and evaluation tools are presented and outlined.Section 4 demonstrates energy sector problems and challenges, and Section 5 describes the implementation process and Section 6 the results.Finally Section 7 presents the findings and conclusions.

Project Monitoring, Evaluation and Control
Project monitoring is concerned with collecting information related to answering questions about the project.It serves in keeping the project on target and alert management when something wrong happens, and aims at improving the efficiency and effectiveness of a project.Moreover, monitoring provides a useful base for evaluation.It enables management to judge if the resources are sufficient, being well used, and the plan is met [1].
Evaluation is a comparison of actual impacts of the project against the planned outcomes and has two main purposes: a) learning and development; b) accountability (displaying effectiveness of the project).In general, evaluation is conducted to gain information before project development, as in a needs assessment.This will serve to improve and refine the project; or to make judgments about its future.There are three main types of evaluation [2,3]: 1) Developmental evaluation: here, the evaluator plays a part in the project design and development.Such information helps in providing feedback to members of the design team and management so as to enhance the project design.
2) Formative evaluation is conducted for improvement purposes.
3) Summative evaluation: is usually conducted after a project's completion.It results in determining the success level, or value of the project under evaluation so as to arrive at a final evaluative judgment.
In this paper all three above-mentioned types of evaluation are incorporated, however, more emphasis is given to the third type.
The purpose of any evaluation is to perform an indepth analysis from technical and managerial viewpoints to see whether the project has successfully accomplished its objectives and to evaluate the impact and sustainability of the project outcomes.The evaluation will provide an independent and, hopefully unbiased opinion on the gains and losses attributed to the project in an attempt to perform a rough benefit/cost analysis.Furthermore, the evaluation process will address key issues as to the necessary action plan to optimize the sustainable benefits of the project and mitigate the negative aspects, or failures, of the project [1,2].This will help all stakeholders in reorienting and re-prioritizing their specific goals and objectives to mold with the national goals and objectives with respect to energy efficiency and to decide on a preferred course of action; i.e. business plan.
In summary, various reasons prove the importance of project evaluation including, inter alia, [2,4]: 1) Understand, verify or increase the impact of output on clients.
2) Improve delivery mechanisms to be more efficient and less costly.
3) Verify that the plan is running as originally planned in terms of time and budget.
4) Facilitate management's thinking in terms of the degree of meeting objectives.
5) Produce data or verify results that can be used in future aspects.
6) Produce valid comparisons that will aid decisions related to continuing or retaining the project.
7) Examine and describe effective programs and avoid mistakes encountered.
Project Control deals with four main resources: money, manpower, materials, and equipment [5].It comes into action based on monitoring the performance, cost and time factors related to the project [5][6][7][8].
Project control may come into action when more resources or less resources have been used than planned, or when activities are taking longer than planned, or if the cost of an activity (or of project as a whole) is higher than expected.The project control process involves one or more of the following actions: 1) adjust the resources; 2) increase the number of labor; 3) employ more qualified labor; 4) provide better supervision; 5) provide more and/or better supplies; 6) provide more and/or better equipment; 7) inject more money; 8) adjust the project characteristics: size, scope, etc.; 9) reduce project size, and/or scope; and 10) terminate the project.

Risk Management
Risk Management (RM) refers to the logical and systematic method of: identifying, analyzing, treating and monitoring the risks involved in any activity, process, or project.RM is usually considered an integral part of any business plan of any organization or project.This implies that it is important to identify as many risks to the project as possible so that a contingency plan(s) can be prepared.Some of the common project risks are [7]: 1) too optimistic time and cost estimates; 2) very slow customer review and feedback cycle; 3) unexpected budget cuts; 4) unclear roles and responsibilities; 5) stakeholder related risks; 6) poor communication resulting in misunderstandings, quality problems and rework; 7) lack of resource commitment.
The adopted RM plan must take care of the following issues: methodology, roles and responsibilities, budgeting, timing, risk ranking, risks and impacts defining, stakeholder tolerances, reporting formats, and tracking [8].
Various methods are available and used to identifying risk including [7,8]: interviewing, assumptions analysis, brainstorming, Delphi technique, SWOT analysis, root cause analysis, checklist analysis , diagramming methods (e.g.Cause-&-Effect diagram, System process flowcharts, Influence diagrams).The simplest technique to track risks is to use a risk log, and then add each identified risk to this log along with action taken.The resulting risk list must be followed up regularly bearing in mind the fact that ignoring risks doesn't mean that they have gone away.

Risk/Impact Analysis
All projects must deliver certain positive results which in tern have positive impacts on the associated surrounding environments.For example, an energy efficiency project is expected to have impacts on consumers, power utilities, and the market.Each of the project stakeholders is concerned with specific benefits which may be common with others or unique.On the other hand, some of these benefits may be in conflict with other benefits.This is illustrated, as an example, by the desire of the power utility to maximize its profit while consumers seek to reduce their bills.
Positive impacts or consequences of any project are usually challenged and faced with certain risks.The interaction between risks and impacts is what determines overall project feasibility.If risks that are not monitored, not identified or not mitigated exceed the positive impacts of the project then there is a big chance that the project is a failure.If, on the other hand, impacts exceed risks and their consequences then the project is likely to be a success.
Since risk/impact analysis is subjective in nature, a quantification methodology is required to translate the risks/impacts into measurable quantities.The results can then be used to judge on the validity and effectiveness or success of the project.
In the project under investigation, the chosen risk/im-pact weights (scores) were selected to reflect the degree of their effect such that the associated score increases as the risk/impact increases.This relationship can be depicted as shown in Figure 1, where this matrix can be used to classify risks/impacts associated with a given project, the likelihood of occurrence, and the appropriate action to be done.It is easily seen that high risks/impacts and high probability of occurrence have high scores, while low risks/ impacts and low probability of occurrence have lowest scores.Naturally risks have a negative consequence while impacts have positive consequences.This means that a high risk item associated with a high probability of occurrence means a disaster, for example.On the other hand a low risk item with low probability of occurrence can be tolerated.When risks are mitigated their negative consequences are reduced.Similarly, positive impacts need to be monitored and realized in order to benefit from them; otherwise if their likelihood of occurrence is low their score will be low.Practically speaking, risk/ impact is usually quantified by assigning weight factors or scores to the risk/impact levels.The mitigated risks are also assigned specific weights.

Cost/Benefit Ratio (CBR)
Cost/benefit analysis has useful applications especially in large governmental projects as it quantifies the project components in terms of monetary equivalents.Usually the assessment is done by adding all cost and benefit elements.This represents a common sense and attractive approach, however, it has two main drawbacks: a) the difficulty of quantifying all benefit and cost elements into monetary equivalents; and b) misleading conclusions that may be made when aggregate benefits exceed aggregate cost and vice versa.The benefit cost ratio is a method for analyzing the desirability of public works projects, or any other project where benefits and costs can be quantified [9].
In the cost benefit analysis, the costs (C) are the anticipated expenditures for construction, operation and maintenance, etc. while the benefits (B) are related to the advantages to the owner (The owner may be the public).On the other hand the disbenefits (D) represent the disadvantages to the owner when the project under consideration is implemented.
The BCR is computed using the absolute values and is given as: B, C and D must all be in same dollar units (i.e.present, or annual, or future worth values).If BCR is greater than unity then the project is advantageous.
For projects involving operation and maintenance costs and a disposal value, then the modified BCR must be used to judge for the project viability.This is given as:

Proxy Analysis
In this paper the authors use what is called the proxy analysis, whereby impacts are converted into monetary values and equated to benefits.On the other hand, risks are converted into monetary values and equated to costs.The proxy analysis is a tool that is used when certain project parameters are not readily available.Other similar or equivalent factors, which are easily calculated, are used in place of the original factors.As such, proxy analysis represents a procedure that is utilized in determining the value of the impacts/risks of a given project.Due to the lack of data, this form of analysis examines how the project partners evaluate the project impacts/risks.Once an acceptable number of the partners' representatives have been have been identified, analyses on various project impacts/risks can be performed.Upon completion of the statistical analyses of the data the findings can be utilized in comparing the achievements incurred in the objectives against the target.The results are then utilized in determining the appropriate actions to be implemented to enhance the outputs of objectives.

Energy Sector Problems and Challenges
The potential problems and challenges that face the en-ergy sector are huge in any country.This will define the scope of energy related projects and serve in putting realistic solutions.The related problems and challenges in the energy sector in the country of the project are classified, but not limited, to the following categories:

Policy
1) Low generation capacity reserve margin during the hot summer season or the cold winter season.
2) Lack of explicit national policy at end-use levels.
3) Environmental costs are not reflected in energy tariffs.
4) Non existence of a central institution to adopt and implement a national energy efficiency strategy.
5) Little incentives to save energy due to subsidized energy prices.This leads to low energy prices which may contribute to a wasteful behavior with respect to energy consumption.

Market Environment
1) Rapidly increasing energy demand driven by economic and social development.
2) Enormous capital required to expand the supply system in order to meet demand, specifically the cost of electricity generation/transmission/distribution systems.
3) Large seasonal variation in electricity consumption due to the changes in weather.

Technology
1) Lack of awareness of energy saving opportunities, technologies, practices, and benefits, at both the utility and customer levels.
2) Lack of energy efficiency culture among consumers.
3) Lack of skilled cadre of managers, engineers and technicians in the various fields and disciplines of energy efficiency.
4) Risk aversion towards investing in new technologies and consequently, reluctance to commit capital to energy efficiency projects offering 2 -3 year payback periods.
All of the above factors lead to the reluctance of financial institutions to invest in energy efficiency projects.
The project under investigation, which is the focus of assessment and evaluation presented in the paper, came into existence to alleviate some of these problems and challenges.This project aims to establish energy efficiency activities within the relevant country.The project included eight objectives, which if achieved, would help the country solve some of the problems and challenges facing the energy sector.These objectives are listed in Table 1.

Methodologies
The techniques and methodologies adopted in the assessment and evaluation of the project involved various tools and procedures.These are divided into the following main categories: 1) Thorough review of the project document (PD), mid-term evaluation report, which was performed by another expert, and other relevant documents.
2) Questionnaires: These were dispatched before the evaluation mission.The questionnaires are related to the abovementioned project objectives.
3) Interviews: These were done on a one-on-one basis with steering committee (SC) members, project management, and stakeholders.
4) Discussions with decision makers and stakeholders, and beneficiaries from project outputs.
5) Logical evaluation sheets to indicate subjective opinions of certain SC members and project lead engineers.
6) Thorough review of project files and supporting materials such as minutes of meetings (MOMs), correspondences, and survey/field results.
7) Quantification of project risks as costs, impacts as benefits, turnover ratios.
8) Calculation of objective achievement scores (weighted average).9) Assessment and evaluation of lessons learned.10) Performing a benefit/cost analysis.In summary, the evaluation process was implemented in a transparent and participative manner and expert unbiased opinion was used in deciding if the project was all-in-all a success or not.

Impact/Risk (Benefit/Cost) Analysis
The last item is performed through a novel risk/impact analysis methodology, which is based on giving the level of impact or risk a certain score defined in Table 2.
The risks identified were due to various factors that are: 2) Reluctance to adopt energy efficiency policies: building codes, tariffs, custom duties, etc.

1) Inadequacy of qualified local engineers/technicians/
3) Program is heavily dependant on technology push rather than market pull.
4) Exceeding allocated budget, and time frame.

External
1) Lack of: a) government support; b) coordination among stakeholders; c) key players cooperation either in participation or provision of information; d) data (consumption records, energy efficiency, etc.); and e) clear incentives for private sector participation/financing.
2) Week project management and week coordination between stakeholders and international bodies.
3) Improper review system of consultants' work.4) Inability to gain sustainable government support for the utility.
5) Electricity tariff is heavily subsidized such that no energy efficiency or DSM program would be feasible.
While, on the other hand, the impacts were similarly identified as: 1) Supporting efficiency improvement and loss reduction in generation, transmission and distribution of electric power.
2) Facilitating adoption and implementation of energy conservation measures in residential, commercial and industrial sectors through education, promotion, financing and standard settings.
3) Stimulating and guiding the private sector in the development of a capability for end-use energy efficiency service, planning, feasibility analysis, conceptual design, and project implementation including the manufacture of energy efficient products.
4) Assisting technology and experience transfer (international and regional) which is instrumental in energy consumption reduction.
5) Promoting public and private sector investments in energy efficient and environmental friendly projects.
6) Cultural build-up among the decision makers and technical personnel involved in electricity or energy sectors; hopefully leading to capacity building, and reaching some critical mass of skilled personnel in certain aspects of the project.7) Creating a snow ball effect starting from the kickoff event through project activities and finally follow-up activities.
8) Evidence of changes in policies (decline of consumption growth rates, institutional changes, new laws and regulations, etc. 9) Economic development: increased productivity (electricity intensity), job creation, create market for energy efficiency equipment and consultancy.10) Offset supply side investment generation/transmission/distribution by reducing peak demand.
The benefit/cost ratio assessment is carried out using the proxy analysis.In other words the benefit is calculated from the positive impact factors, which the project had.On the other hand the cost is calculated from the risks associated with the project.The analysis of impacts (benefits) and risks (costs) yields the sought benefit/cost ratio.This "benefit/cost" ratio is a good indicator of project success in creating the required and desired impacts with respect to the associated risks.

Objective Achievement Scores
The project has achieved about 80% of its objectives.This was calculated through a subjective assessment of the degree of fulfilling the objectives with some consultations and discussions with key project management and key persons involved in the project.The scoring was based on assessing the fulfillment of the tasks of each output for each objective.This resulted in output achievement scores, which were used in calculating a weighted average for each objective.In other words in calculating objective achievement scores the weight of the tasks of each output were factored in the calculation, i.e. outputs with more tasks had more weight in the objective achievement score.However, all objectives were given the same equal weight in calculating the overall project achievement score.
It is worth mentioning that certain "smart" objectives had already objective measures.The objective measure included success criterion.However, some objectives were not.This obliged the use of subjective judgment of project personnel, stakeholders, and the opinion of the reviewer.Table 3 summarizes the output and objective achievement scores for the project.The following points can be derived: 1) The overall score of 80% achievement is satisfactory for a project of this size and duration.
2) The high turnover which characterized the project and the absence of sustainability made it even more difficult to have higher achievement score.
3) The objective with lowest achievement score is understandably objective 2 which faced difficulties in all phases of work and was delayed for quite some time.
4) Objective 3 ranks the best among all objectives with a score of 100%.Then objective 4 follows with a score of 89% and objective 8 with 86%.
It is also worth noting that the project, in addition to achieving almost 80% of its stated objectives as mentioned above, has created some general positive impacts.For one thing it is the first national program on this very large scale.Moreover, it addressed several important issues for the first time.
It also has focused attention on hot issues which were not previously brought up; such as introducing new tariff and new building codes.The project has also addressed the partnership between government and private sector.The dialogue among the key players of the electricity sector has been put to test through this project.The results of the interaction, cooperation, and coordination efforts are very much encouraging.

Impact/Risk and Benefit/Cost Analysis
This may be regarded as an initial estimate of the benefit/cost ratio, and is implemented in the quantification of the assessment of the project.This step was needed to get the indicators as the project did not have a target objective achievement goal or key performance indicators to gauge the success or failure of the project.Therefore, the results were assessed in a proxy manner.The overall results of the project risks and impacts are given as shown in Table 4.
The ratio of 1.31 is an indication of a "feasible" project; although this is already known as the overall objective achievement is close to 80%.As a matter of fact, assuming that the energy efficiency issue is sustained, then this ratio will even be higher due to the rippling and cascading effects.Although the defined risks (costs) were more than the impacts (benefits), which looks to be biased, analysis results show that the BCR is in favor of impacts over risks.This means that the project is really feasible.

Summary of Main Results
The project in hand has shown that it is really difficult to assess the degree and level of success in achieving the objectives of the project.Although the success criteria of some of the objectives are considered objective measures it is difficult to quantify the criteria for other objectives.
The achievement scores for all eight objectives of the project and the overall score for the program as a whole are all acceptable.In fact, the performance indicator, as a measure of objective achievement, showed that about 80% of project objectives were achieved.The project showed that two critical issues: a) capacity building and sustainability; and b) exit strategy, were not incorporated in the project document and, therefore, need to be addressed in the "Business Plan".These are considered, in our viewpoint, key objectives that are critical to the success of the project.Moreover, the project has also created unannounced positive impacts, by-benefits, in many disciplines such as creating a culture of cooperation, coordination and working for the national interests, which cut across the boundaries of entities, and establishing a first time consciousness in tariff adjustments, adoption of building codes, encouraging private sector participation and attempting to change consumer buying and consumption habits.The analysis of the project turnover reveals that the project really suffered from a rather high turnover.The major drawback related to the concept and design of the project was seen in the lack of inter-linkages among objectives.Finally, a novel benefit/cost ratio concept was adopted and implemented.This ratio was assessed using the proxy analysis where the benefit is calculated from the positive impact factors of the project while the cost is calculated from the risks associated with the project.The assessment showed that the project has a

Conclusions
This paper presents a novel approach for assessing and evaluating a large project in the energy management field.
The project is at the national level involving stakeholders and beneficiaries from different institutions and entities, each with different objectives and motives.The project scope is very wide and ambitious.It is further a longterm project covering several years.The methodology incorporates two metrics; a) objective achievement scoring; and b) impact/risk ratio.The first metric is simply to calculate the achievement score for each project output.
The overall project achievement score is the weighted average of all project outputs.The assessment was done through subjective evaluation with consultations with key players of project stakeholders, key players, and staff.

Table 4 . Project risk/impact (benefit/cost) analysis matrix.
/C ratio of 1.31, which is considered feasible in all measures. B