Evaluating Project Management Software Packages Using a Scoring Model—A Comparison between MS Project and Primavera ()
1. Introduction
Computer systems are known to have a role in all aspects of the engineering business. The revolution in the field of computer software in the last decade has been enormous. Now, engineers and managers working in the engi- neering industry can no longer ignore the facilities and tools in computer technology if they wish to become successful.
In the engineering industry, several software packages have been developed to enhance the various project stages. Starting with the design stage, the marketing of architectural, engineering, and construction software is very impressive, as it is exemplified by colors, 3-D drawings animation, audio video effects, and much more. Now there are even other software systems that help in cost estimation, construction documents preparation, quantity surveying, and of course, project scheduling and management [1] .
In the world of project management, computer applications play an increasing role in the delivery of projects. Software programs are involved in just about every aspect of project work from the discovery of a problem or an opportunity, to the commissioning and start-up of the new facility. Computer tools are now involved with every aspect in the project, providing solutions that will help in faster project completion, lower costs as well as improving the productivity and accuracy of work. Projects require participation and support from all levels within an organization, and so teams are the foundation of a successful project. Modern software packages provide comprehensive information on all the projects in the business, from executive-level summaries to detailed work assignments, so each team member can confidently contribute to the project’s success.
Growth for most businesses is a result of successfully developing a project that generates new products and services. This day-to-day explosion of new products and services is driving the challenges that face most com- panies. Limited skilled resources, constantly changing project priorities and shortened development cycles magnify these challenges. Thus, it is no longer business as usual. The ability to make decisions, choose the right projects, will determine how effectively projects are deployed and how quickly the company’s products and ser- vices get to market.
To create strategic business advantages and get the products to market fast, leading companies are effectively using project management. They need the most advanced and modem project management software. Process and resource management software that offers the combined benefits of’ managing projects and building and using standard methodologies to help companies minimize project lifecycles, and deliver uncompromising pro- ject results [2] .
In the world of engineering and construction, a long time concern has been the concept of project scheduling by CPM. For many in the industry, CPM was seen as a system that somebody else should use. Of all modern engineering and construction tools, CPM was one of the very first to be computerized. CPM was initially aimed to improve construction labor productivity. Eventually other aspects such as, procurement and management de- cisions were included in the CPM planning networks and schedules. Today there are many CPM-based computer tools for scheduling and project management.
Where at one time most schedules showed only construction activities, current usage includes a full range of project phases and tasks, including producing, services, site analysis activities, schematic design services, design development services, and so on. PRIMAVERA offers to manage not only the project time, but also the costs and the use of resources. This is done by estimating activities in terms of their time, their resource requirements and their cost.
The initial concept of CPM was to plan activities using networks of logical relationships and then to create Gantt or bar charts to communicate the information. The current marketing approach, however, is to plan the work on screen using bar charts, connecting the bars for logic if needed, and then automatically converting the plan to a CPM/PERT network in a precedence diagram format [1] .
A widely used package, created for the engineering and construction industry, is Primavera software. PRIMAVERA is now written for Windows as GUI packages (Graphical User Interface). This means that the package depends on graphics and charts to illustrate the different tools and functions it provides. PRIMAVERA offers to organize, analyze, communicate and accelerate projects. The main planning graphic in PRIMAVERA, as in most other GUI packages is the Gantt charts.
PMSP although not specially written for the engineering and construction industry, includes: Microsoft Project and Computer Associates Super. Microsoft Project is seen by some as a lower cost, individual schedul- ing tool suitable as a feeder to more comprehensive PMSP for large projects, such as Open Plan and Primavera. Some lower price applications are Timeline Track from Primavera and Open Plan Desktop. Sure Track and TimeLine are simple, computerized bar chart drawing tools [1] .
According to Levine, PMSP can be classified according to three levels/types follows:
・ The Bare Bones Level: The software containing very basic functions operating within limitations.
・ The Mass-Market Level: This project management software attempts to provide the basic tools and features for a variety of project management activities.
・ The Advanced-Moderate Level: These highly sophisticated programs have a resultant increase in complexity and speed of operation [3] .
Although, this classification of PMSP was proposed by Levine in 1986, the available PMSP tools in the mar- ket today still fit neatly into this categorization. The Mass-Market level of PMSP includes packages such as CA Superproject, Micro planner, on Target, Project Scheduler, Timeline and MS-Project for Windows. These soft- ware packages range in price from US $200 to US $1000 and provide the general project management techniques of PERT/CPM, Gantt chart, resource leveling, WBS and cost tracking. The downfall of this range of PMSP is that they do not provide the ability to schedule and track multiple projects, and therefore are only use- ful for single project.
The advanced-moderate level of PMSP are highly sophisticated programs that provide a great number of functions ranging in price from approximately US $1500 to US $16,000, depending on the functionality and de- sired hardware platform. These packages include: Cresta PMS, Kemal PMS, Micrplanner X-pert, Primavera, X-pert for Windows, and the majority of the PMSP that require Unix as the hardware platform. These packages are recommended for multiprojects with several thousand activities or many subcontractors.
As software packages are generic, they apply to large segments of the market, ensuring sales and profits for the software vendor. However, the generalized nature of software packages guarantees that they are not designed to suit all of the functionality requirements of every organization.
The purpose of software packages is to provide a relatively cheap product that meets the basic needs of or- ganizations, attempting to solve a problem experienced by many. The other major advantages of software pack- ages are that they can be acquired, installed and operated within weeks, while the level of user support is high [4] .
In 1994, a survey was conducted with 1500 software professional in North America. The survey analyzed software product installations and purchase plans for the feature/function requirements. The objective of the re- search was to aid software vendors to better understand user requirements and the decision process buyers un- dergo when selecting software products. Respondent to the survey cited purchasing off-the-shelf applications as the most suitable and most popular choice. The survey has also found that product features and functions and ease of use have been rated respectively as the number one and two buying criteria for the past eleven years. However, in 1994 user support, that is service and training, overtook the number one and two positions.
Different methods are used by different people to select project management software. One approach is study- ing the experience of others, validating their experience and relating that to your situation. Another approach is to conduct trials on small problems, which will help to understand and use the software. Some project manage- ment experts prefer to apply the package or tool on real projects. This has one disadvantage that it is a lengthy process and this is only practical as a means of evaluating few products [5] .
In general, it is advisable to follow recommendations of national bodies, expert group (for example the Pro- ject Management institute) or consultants in the project management area. These groups are familiar with soft- ware packages and their potential uses for various activities and industry needs.
2. The Purpose of Software Evaluation
Today the use of information technology is growing; the number of critical computer systems is also growing. The quality of software in these systems is important because software faults may lead to serious consequences. Thus, it is important before starting the evaluation process to define, what is meant by the evaluation of project management software? It is the process of quantifying the different tools and features provided by a software package in a way that will help to compare it with different software packages.
Software products are extremely varied. They are written and produced to fulfill very diverse requirements. Their context for use can also be very varied, such as in the case of application software in a management in- formation system. The primary purpose of software product evaluation is to provide quantitative results con- cerning software product quality that is comprehensive and acceptable.
The evaluation process is described as a procedure that allows expression of evaluation requirements in terms of quality characteristics, as defined in ISO/IEC 9126. The evaluation takes into account various documents that can be considered as part of the software product. Design documentation, test or validation reports, and source code or user documentation. It is recommended by the ISO standards that the evaluator use a library of evaluation modules that define evaluation methods. These evaluation modules could be standardized, although no provision for that is proposed in the standard. The evaluation leads to the production by the evaluator, of an Evaluation report [6] .
There are different types of evaluation. The evaluation process can he classified by the party concerned with the evaluation, and the evaluation could be classified based on the stage at which the software is being evaluated [6] .
3. Objective & Methodology
To investigate the effectiveness of PMSP, the author has to collect an enormous amount of data for both soft- ware users and software vendors. The data collected can be applied using the scoring model developed by the author. The best method to achieve this objective is to use a survey/questionnaire method as this method will enable the collection of large amount of data from variety of software users in several locations and from dif- ferent industries.
Surveys are statistical research method used to collect data from a large group of participant. Surveys are very useful research instrument to collect data on phenomena that cannot be observed. Surveys could be of different types including: structured, semi-structured and unstructured questionnaire surveys. In a survey, researchers use a random sample to represent the entire population. Basha and Harter in 1980 define population as “any set of persons or objects that possesses at least one common characteristic” [7] .
The best method to achieve this objective was the use of questionnaire survey forms. Surveys are statistical research method used to collect data from a large group of participant. Surveys are very useful research instru- ment to collect data on phenomena that cannot be observed. Surveys could be of different types including: structured, semi-structured and unstructured questionnaire surveys. In a survey, researchers use a random sam- ple to represent the entire population.
Surveys may have advantages as a research method. 1) they are not expensive, 2) participants can respond quickly and can be widely spread geographically, 3) participants can respond after gathering information or tak- ing time to think about their responses, 4) it participants privacy while they respond, 5) allow respondents to read questions rather than hear them loud, 6) those who participate can do so according to what is most conven- ient for their schedule, 7) participants can look at the entire survey, giving context to questions and 8) partici- pants are also unaware of the feelings or wishes of the researcher [8] .
The objective of this research paper was to apply a scoring model developed previously by the author based on ISO/IEC 14598-1 software evaluation criteria to help evaluate two different software packages. The scoring model uses a score card which assigns scores for each software attribute in order to assess the effectiveness of the software package.
4. Data Collection
4.1. Overview
The data collected in this particular research, originates from questionnaires sent to the software user, repre- senting any individual or organization that is likely to use project management software to perform its activities; and the software vendor, being the project management software company that delivers this product to the user.
In order to test the validity of this scoring model, the author chooses to assess two different software packages by applying the model to it. The software packages chosen were: Microsoft Project and Primavera due to their common and wide application among engineering and construction firms.
4.2. Software User Survey Form
The software user questionnaire would be used to support the vendor questionnaire, by targeting organizations that use project management software to monitor and coordinate their projects. In this particular study, a soft- ware user survey form was distributed among 110 companies across Australia. The survey form was distributed in five cities: Sydney, Melbourne, Brisbane, Adelaide and Perth. Almost 35 companies responded to the survey which helped to gather data about the assessment of various software characteristic and attributes both from a technical perspective (so called operational characteristics) and system requirements (so called general informa- tion system characteristics).
The survey form consisted of the following sections:
・ Section A: which would include general information on the software used by the company, and the size of project management activities conducted by them? It would also contain company details. It was hoped that this section would reveal the popularity of project management software to those organizations and the de- gree of importance of the software itself.
・ Section B: which would help weighting the evaluation criteria? It will focus on obtaining a numerical as- sessment for the different attributes listed in the software evaluation criteria. This assessment represents the degree of importance of each attribute expressed as a percentage as seen by the software user. This will be referred to as the weight of the attribute. Each characteristic set itself will have a relative weight and the sub characteristics included in each criterion will have a different weight as well. The section will be organized in the form of tables that list the main characteristics used and the sub characteristic sets associated with it, as well as, the different attributes associated with each sub characteristic set in separate tables. Each table con- sists of two columns. Column one that lists the characteristics/attributes and column two contains the degree of importance given to the particular attribute in terms of the overall characteristics examined.
・ Section C: This would involve the attributes. This section was part of the questionnaire sent to software users who are using either MS-Project or Primavera in planning to help in scoring the attributes listed in Section B, and give their personal judgment of how the tested software achieves each attribute. The score will be assigned based on a point scale (0 - 3). It will reflect the performance of the software package in respect to each attribute. Where zero represents a poor performance and 3 represents an excellent performance.
Finally the user was allowed to comment on the contents of the questionnaire and the investigation itself.
To be assigned a score, the software package must achieve the condition stated for each performance category. Since this is a subjective assessment that might differ from one person to another, the four point’s scale as illu- strated in Table 1 was selected to minimize the personal error in each assessment. Once the scores are obtained for all the attributes, it will be installed into the scoring module. For each of these attributes listed in the model the following were calculated: a score that represents the assessment of the software user to how each software package covers this attribute, a relative weight (X) of each sub-characteristic set defined within the operational or general information system characteristics, and a weighted score was calculated by multiplying the score of each attribute with the relative weight of that sub-characteristic set. Finally a relative weighted score (Y) for each sub-characteristic set was calculated to be the sum of all the weighted scores of the attributes that belongs to it.
4.3. Software Vendor Survey Form
Two software vendors in the investigation were Primavera Systems Inc., and Microsoft Corporation, which produce the software Microsoft Project. Both vendors have an outstanding reputation in Australia and world- wide. Their products are widely used at both the individual and organizational level.
The data collected using this questionnaire was necessary to the evaluation process to be used in the scoring module. The structure of the questionnaire consists of three sections: Section A, the general information, Section B, the general system requirements and Section C, the life cycle cost information.
Section A covered general information including the software company details and the respondent’s name. This section meant to help estimate the size of the software company both in Australia and worldwide and to help establish a general background on the experience of the software company in the field of project manage- ment.
Section B will cover the data required in evaluating the general system characteristic, including information on technical support, compatibility, training, security and hardware requirements.
Section C presents the collected data necessary to work out all the different cost items associated with the software package. Such information will help in evaluating the third characteristic set, which is the cost of the software package. These cost items are listed in Table 2.
![]()
Table 1. Scoring criteria for each software attribute.
The cost is a significant part of the evaluation process, since a decision to purchase a software package will very much depend on the cost of the package as well as other considerations. It was for this reason that, the user was given the choice in the questionnaire form to rate the degree of importance of the cost from amongst the other criteria items. Two questionnaire forms were sent out one for Primavera Systems Inc. (Brisbane) and the other one to Microsoft Australia Corporation (Sydney). Results of the data collected in these questionnaires will be discussed later in this paper.
5. Data Analysis
Since software is considered the main topic of this investigation, it was extremely important to collect information re- garding both the software packages under investigation. Indeed this information was most useful in the evaluation process. Both companies: Microsoft Australia Corporation and Primavera Systems Inc. responded to the questionnaire.
As indicated in the second part of the survey forms, both software providers satisfy the documentation re- quirements identified in the evaluation criteria. They provide the user with user and technical manuals when purchasing the software. In addition to that, they provide auxiliary materials such as demo CDs, brochures, booklets, etc. This approach is important as the software user ranked the documentation of the software as the second most important item in the general system requirements of the software, with a relative weight of 20%. In the documentation characteristics, PRIMAVERA achieved superiority over MS-Project by scoring higher in terms of the area of support for user and technical manuals (2 and 2 for PRIMAVERA, 1.73 and 1.8 for MS- Project respectively). In contrast, MS-Project scored higher in terms of supporting auxiliary materials with the software (1.6 for PRIMAVERA, 1.7 for MS-Project respectively).
With a relative weight of 16% the compatibility of the software came third in terms of the general systems characteristic. Both MS-Project and PRIMAVERA satisfy this requirement. However, PRIMAVERA was found to be compatible with several project management systems. MS-Project was found to be better only in terms of compatibility with MS-Office applications, which was expected from a Microsoft product scoring 1.91. PRI- MAVERA only scored 1.20 on this attribute. In terms of compatibility with other project management software and information management systems, however PRIMAVERA achieved better results scoring 1.5 and 1.8 while MS-Project scored only 1.3 and 1.7 respectively. In terms of compatibility with programming languages and da- tabase management systems, MS-Project is compatible with C+, Visual Basic and C sharp. PRIMAVERA is al- so compatible with C+, as well as SAP and the Oracle database.
The technical support sub characteristics came forth in the degree of importance with a relative weight of 15%. This sub characteristic was achieved well in both software packages. Both software suppliers indicated that they provide technical services such as: implementation, consultation and training courses on the use and application of the software package. Additionally both vendors conduct seminars and workshops to introduce the new user to the last updated versions of their products. The attributes in the technical support were another success for PRIMAVERA over MS-Project. MS-Project was only better in the process of installing and operating the soft- ware, scoring 2.27, while PRIMAVERA scored only 1.83 on this attribute. Meanwhile, PRIMAVERA scored better in terms of support for technical services and training courses, (2 and 2 for PRIMAVERA, 1.38 and 1.9 for MS-Project respectively) on these attributes.
With a 40% relative weight, the user friendliness characteristics proved to be the most important item for the software user. In examining the different attributes of this group, MS-Project achieved better results than PRIMAVERA in terms of a shorter time period to learn the software package, clearer screen messages and di- alog boxes, and the undo redo option. PRIMAVERA was better than MS-Project only in terms of the organiza- tion of the menu structures and commands.
The security sub characteristics of the software came last in terms of the degree of importance for the soft- ware user with a relative weight of only 9%. This was because the user was more interested in characteristics like user friendliness and documentation. However PRIMAVERA achieved a higher success level than MS- Project in terms of both security on accessing project files and security on target plans.
6. The Scoring Model
In a previous research phase, the author developed a scoring model based on the ISO/IEC criteria illustrated in Figure 1 and data gathered from variety of software users in the engineering and construction industry. The scoring model included two type of characteristics to be evaluated in a software package: the operational char- acteristics which refers to the technical features and tools available within the package which affects the func- tionality of the software, and the general information system characteristics which refers to the non-technical software features that are essential to have not only in a project management software package but in any soft- ware product. Each level was further broken down into sub-characteristics that define the key aspects that should be present in a PMSP and finally each sub-characteristic was broken down into the smallest measurable units in the software (the so called attributes). Each attribute was given a weight (0% - 100%) to represent the degree of importance of this attribute in terms of the overall assessment criterion.
The first step in building the scoring module was to calculate the average score assigned by each software user for each attribute, the average score is called the “aggregated value” and is calculated using the below formula:
![]()
Figure 1. Evaluation process as identified by ISO/IEC: 14598-1.
(1)
where:
A: number of users assigning a score of ZERO for a specific attributes.
B: number of users assigning a score of ONE for a specific attributes.
C: number of users assigning a score of TWO for a specific attributes.
D: number of users assigning a score of THREE for a specific attributes.
After calculating the aggregated value for each attribute and for both software packages under investigation, the score is then installed into the scoring module.
The second step is to determine the weight assigned for each attribute which is a percentage assigned by each software user to represent the degree of importance of each software attribute in terms of the sub-characteristic set for which this attribute belongs. The average weight is then calculated by summing all the weights assigned divided by the number of software users responding to the questionnaire. This represents the average weight for each software attribute. This percentage is inserted in the scoring model.
The third step is to calculate the weighted score for each software attribute which is calculated according to the formula below:
(2)
The weighted score is then inserted into the scoring model against each software attribute.
The fourth step in building the scoring model was to calculate the relative weight (X) for each sub characte- ristic set which represents the degree of importance of each sub characteristics set in terms of overall software evaluation criteria. The relative weight is calculated as an average number resulted by adding all relative weight percentages assigned by software users and divide this by the number of software users. This number is then in- serted in the scoring model
The values of X as obtained from software users as follows:
X1 = 33 X6 = 40
X2 = 15 X7 = 20
X3 = 14 X8 = 16
X4 = 19 X9 = 15
X5 = 19 X10 = 9
The fifth step is to calculate the relative weighted score which represents the aggregate of all weighted scores for each sub characteristic set and is calculated using the formula below:
(3)
7. Evaluation Results: MS Project and Primavera Software
Now that we have built the structure of the scoring model which will be used to evaluate and compare the two different project management software package, we can start applying the above equations of the scoring model to the data obtained which will give the following values for Y (See the scoring module Table 3 for each soft- ware package)
・ For Microsoft Project:
Y1 = 1.99 Y2 = 1.75 Y3 =1.59
Y4 = 1.56 Y5 = 1.82 Y6 = 1.74
Y7 = 1.67 Y8 = 1.35 Y9 = 1.78 Y10 = 1.97
・ For Primavera
Y1 = 2.26 Y2 = 2.26 Y3 = 2.12
Y4 = 1.63 Y5 = 2.11 Y6 = 1.91
Y7 = 1.43 Y8 = 1.91 Y9 = 1.96 Y10 = 1.84
Now that we have the values of X, Y for each sub characteristic set, we can work out the values of A which represents the total score of the software package in terms of the operational characteristic as follows:
(4)
Applying this equation to each software package under investigation will result in:
A (for MS-Project) =178 points
A (for Primavera) = 209 points
Similarly the value of B representing the performance of the software package in terms of GIS characteristics can be calculated as follows:
(5)
Applying this equation to each software package under investigation will result in:
B (for MS-Project) =179 points
B (for Primavera) = 181 points
Using the values of (m, n), which represents the relative weight of the operational characteristics and the gen- eral system information characteristics of the software as assigned by software users to be 60% and 40% respec- tively. Applying this to the equation below will yield the value of C which represents the total weighted score for each software package
(6)
Therefore,
C for MS-Project = 178 × 0.60 + 179 × 0.40 = 178 points
C for Primavera = 209 × 0.60 + 181 × 0.40= 198 points
The scoring model discussed above is indicated in Table 3.
In order for the value of C to be a true measure of the performance of the software, it has to be compared to the worst and best outcome that could be achieved by the software tested using this module. Therefore, the minimum and maximum value of C should be calculated and will be called C.min and C.max respectively. This will give the user or the evaluator a range for the minimum and maximum level of quality of the software ac- cording to this scoring module. A better sense of the efficiency of the software package can therefore be achieved.
In calculating C.min and C.max the same weights (m, n) and the same relative weights (X values) are used. However, the scores will be totally different and hence, the values of Y will be different.
The best PMSP, according to this module will score the maximum value on all attributes, assuming the soft- ware satisfies all the attributes fully and achieves excellent performance. In this case, a score of three was as- signed to each attribute. It must he remembered that the worst PMSP will not score zero for all the attributes, because zero means that the attribute does not exist at all. A package that represents the minimum acceptable level of quality will score one point on all attributes.
To achieve the minimum level of quality in the software, the value of A and B should be 100 and to achieve the maximum level of quality, the value of A, B should be 300.
Using m = 60%, n = 40%, the value of C is calculated as follows:
(7)
(8)
According to the scoring module, the value of C for MS-Project is178 and the value of C for Primavera are 198. Comparing this with the value of C.min and C.max, it can be seen that both software packages fully satisfy the minimum quality level according to this scoring module. However, both are still far from reaching the high- est quality level able to be achieved by this scoring module. In comparing the value of C each package achieved to C.max will give the efficiency of the software in terms of the technical aspects examined.
(9)
Therefore,
Efficiency of MS-Project = 178/300 = 59%
Efficiency of PRIMAVERA = 198/300 = 66%
(a) ![]()
(b) ![]()
Table 3. The scoring model used in the evaluation process.
These results show that PRIMAVERA is better than MS-Project by 20, points or 7% in terms of efficiency. Moreover, by looking at the values of A and B for each package, it can be seen that PRIMAVERA achieved this superiority over MS-Project through the operational characteristics of the software (measured by the value of A). PRIMAVERA scored 209 points while MS-Project only scored 178 points. Thus the functions of the software and the technical features provided in PRIMAVERA are better than in MS-Project. For the general system cha- racteristics, measured by the value of B, both softwares achieved almost the same results with PRIMAVERA scoring 181 points and MS-Project scoring 179 points. This means that the general characteristics examined are almost the same for the two packages. However, comparing the different values of Y achieved by each software in these characteristics, it appears that MS-Project achieved slightly better results in compatibility. This is something to be expected from software produced by the biggest computer software corporation in the world Microsoft. However, PRIMAVERA provides better security tools for accessing project files and baseline plans.
So far this analysis has only examined the software in terms of the operational and general systems characte- ristics. As mentioned earlier, cost is a significant part of the evaluation criteria; therefore, a cost effectiveness analysis has to be performed.
The third part of the software supplier survey form was concerned with collecting information on all cost items associated with the software package. The following cost items were assessed:
・ Price of the software package
・ Installation cost
・ Maintenance and upgrading cost
・ Hardware equipment cost
・ Learning cost
Using the data obtained from the survey forms, the cost of the software was calculated as follows:
・ Cost for Primavera (Operation life of the software 2 years)
o Price of the software license = AUS S 8000/license
o Installation fee AUS $800/license
o Maintenance and upgrading cost 1600 S/year × 2 years:= AUS $3200
o Learning cost = 58 Si hour × 24 hour AUS $1400 (3 day course, 24 hours)
o Hardware equipment required to operate the software as indicated by the supplier arc: RAM 16 MB, CPU 486 at least, HDD 200 MB
o Hardware parts cost = AUS $85
Total Cost of Primavera Software = AUS $13485/one license
・ Cost for Microsoft Project (Operation life of the software 2 year)
o Price of the software license AUS $11001 license
o Maintenance and installation cost = 105 S/year × 2 years = AUS $210
o Upgrading cost = 600 $/year × 2 years = AUS $1200
o Learning cost = AUS $390 (2 day course, 16 hours)
o Hardware equipment required to operate the software as indicated by the supplier arc: RAM 32 MB, CPU Pentium 100 MHZ at least, HDD 1.0 GB
o Hardware parts cost = AUS $200
Total Cost of MS Project AUS $3100/one license
Finally to include the cost as part of the evaluation criteria, a cost effective ratio were calculated as follows:
[9] (p. 567) (10)
In general, this equation assumes the total cost to be as important as the operational and general information characteristics of the software. According to this equation 50% weight is assigned to the cost and 50% to the other criteria items. However, in this study the software user indicated in the survey form that the cost (on average) is only worth 24% as compared to other criteria items and not 50%. The operational and general system characteris- tics in the software were given 76%. Thus, a correction factor of 76/24 should be installed in this equation. Hav- ing said that, the cost effectiveness ratio calculated for MS-Project and Primavera Project Planner is as follows:
・ CER (MS-Project) = 178/3100 × (76/24) = 0.182
・ CER (PRIMAVERA) = 197/13485 × (76/24) = 0.046
After performing the cost effectiveness analysis, it can be seen that the results dramatically changed. Taking the cost into consideration, PRIMAVERA no longer is the preferred option. With a cost effectiveness ratio of 0.182, Microsoft project seems to be a better option to select, despite the success of PRIMAVERA in the other technical aspects of the software.
Again examining the detailed information on the cost of each software package, it can be seen that the cause for this huge difference in cost comes mainly from two items. Firstly, the cost of the software licenses itself. One license of PRIMAVERA costs AUS $8000 while MS-Project costs only AUS $1100/license. Secondly, the learning cost. At the time where Primavera provides the course of PRIMAVERA at a cost of AUS $1400/person, Microsoft can give a very reasonable price of AUS $390/person for the learning course in Microsoft project.
The final analysis of the scoring module recommends selecting Microsoft Project since it satisfies all the evaluation criteria items at a higher CER. However, if the cost was not a significant factor in the selection process, Primavera is recommended as it offers better tools and features from a technical point of view. Table 4 below summarizes the results of the evaluation process conducted on both software packages.
![]()
Table 4. Summary of the evaluation results.
8. Conclusions
On the evaluation of Microsoft Project, the application of an evaluation module on this software was successful. The evaluation results of Microsoft Project, according to the scoring module, indicate the following:
1) The software achieves an acceptable level of quality in terms of the operational and general system charac- teristics, The best performance of the software in the operational characteristics was found to be in the time management attributes, particularly in relation to setting up the project plan and identifying activities and rela- tionships. This was totally consistent with the user requirements in time management attributes, giving the soft- ware package a big advantage. The software also performed well in relation to control and monitoring, which was also important to the software user.
2) The worst results for Microsoft Project in the operational characteristics were in reporting attributes. Mi- crosoft Project scored under two points in all these attributes. This result is not totally unexpected, since a prac- tical examination of the reporting tools in the software will reveal that MS-Project has a very limited reporting capability. Indeed the reports available are very few, and there is a very limited ability for report customization and generation.
3) Microsoft Project achieved prominent success with the user friendliness characteristics. The user needs were a relatively short time in which to learn the basic functions of the software. In addition to that, the software has a very well organized menu structure and commands.
4) In terms of general systems characteristics, the weak point in the software was the security tool. The soft- ware performance was only average in providing security for project files and baseline plans.
5) The software user survey indicates that Microsoft Project remains the most popular and commonly used PMSP among the software packages in the market. Around 80% of the selected sample indicated that they use MS-Project in project planning and management.
6) Although MS-Project does not provide sophisticated tools and advanced control features like PRI- MAVERA, the results of the scoring module recommend the selection of this package. It satisfies the user’s re- quirements in terms of operational and general system characteristics combined with very low cost.
On the evaluation of Primavera Software, the evaluation results of Primavera according to the scoring module indicate the following:
1) With an overall efficiency of 66%, the software achieved an acceptable level of quality. Moreover, in terms of operational and general system characteristics, PRIMAVERA was found to be much better than Microsoft Project. In the attributes of time management, only setting up the project plan was better in Microsoft Project, while PRIMAVERA scored higher points in defining activities and relationships, network calculations and net- work graphics.
2) PRIMAVERA’s other advantage over MS-Project was in resource management. In all the attributes of this group, PRIMAVERA performed much better than MS-Project, especially in relation to resource allocation and leveling,
3) In the general system characteristics of the software, PRIMAVERA achieved acceptable results in all the attributes of this group. The best performance for PRIMAVERA was found in the technical support attributes, especially in terms of technical services and training courses on the software use and applications.
4) Only in terms of compatibility did PRIMAVERA score less than MS-Project This was because the latter being a Microsoft product, is more compatible with other Microsoft applications (Word, Excel, Access, etc.).
5) Despite the fact that PRIMAVERA is better than MS-Project in terms of both the operational and general system characteristics. It is not the most suitable PMSP to be selected according to the scoring module. The cost of the software package influences this outcome. While MS-Project is offered in a reasonable price of AUS $3100/license, PRIMAVERA is much more expensive (cost AUS SI3485/license). This places PRIMAVERA with a CER of 0.046 in a weak position against other PMSP available in the marketplace.
Generally speaking, this investigation revealed that the software user most of the time is not concerned with the most sophisticated PMSP, or the package that has more advanced tools and features. What is more important for the user is their need to produce simple time charts, simple resource and cost analysis and basic reports. However, it was found that large project management organizations are conducting professional project man- agement that require the use of sophisticated project management software packages, which offers a variety of tools and capabilities, and thus, the cost were irrelevant to them. On the other hand, a small organization is only concerned with the basic tools and capabilities of the software and thus, the cost of the software package is highly significant to them.
In addition to the technical aspects of the software package, it has to be user friendly and compatible with other Microsoft Office applications. If this was achieved, then the software is considered highly efficient. How- ever, this is only one part of the equation; the other part of the equation will be the cost, which is not less impor- tant than the other aspects. A reasonable price, in addition to the needs mentioned would make the chosen soft- ware the best alternative.
Finally, the results of the scoring module revealed that PRIMAVERA is ahead of MS-Project by 20 points in terms of operational and general system characteristics. However, a comparison between the two packages, in- corporating the cost, will make MS-Project a better alternative. The question remaining then is whether an addi- tional 20 points, or an increase of 7% in efficiency of the software, is equivalent to the huge difference in cost.