Findings from European Benchmarking Exercises on E-Learning : Value and Impact

Benchmarking is becoming a more and more commonly used method for quality assurance and enhancement in higher education today. The aim of this study was to explore the impact and value of benchmarking e-learning in universities. Two European benchmarking initiatives, E-xcellence+ through the European Association of Distance Teaching Universities [EADTU] and an e-learning benchmarking exercise through the European Centre for Strategic Management of Universities [ESMU], were studied using an exploratory multiple case study method and cross-case analysis of the findings. The findings indicated the value and impact of a variety of aspects of benchmarking e-learning on different levels, and stressed the need for a holistic and contextual approach as both benchmarking and e-learning are complex and comprehensive phenomena. This study also shows that benchmarking, in line with national and international quality boards/agencies, is a powerful tool for making improvements to teaching and learning in higher education.


Introduction
This paper focuses on research questions concerning the value and impact of benchmarking e-learning in higher education, which until now has remained a fairly unexplored area.This study uses as points for departure the experiences of two European benchmarking initiatives; E-xcellence+, which was carried out by the European Association of Distance Teaching Universities (EADTU) (Ossiannilsson & Landgren, in press;Ubachs, 2009;Williams & Brown, 2009) and the e-learning benchmarking exercise, carried out by the European Centre for Strategic Management of Universities (ESMU) (Comba et al., 2010;Ossiannilsson & Landgren, in press;Williams & Rotheram, 2010).Moriarty (2008) argued that although the literature on benchmarking is overwhelming, research and evidence on the value and impact of benchmarking is still missing.This research aims to contribute to the formation of a deeper understanding and knowledge of the value and impact of benchmarking e-learning in higher education.This research does not claim to focus on benchmarks as such, and neither has this study been conducted using a special method for analysing impact.
Major changes are taking place in European higher education today, and quality in education and research is the key to supporting development and innovation.In this context, enhancing the performance of universities and modernising university management must be on the agenda for all university leaders and decision-makers in Europe (Bates, 2010).A clear understanding and transparency of the modes of operations and processes with the aim of continuous improvement is needed.Universities are facing new challenges in the twenty-first century, as they are required to be competitive not just in terms of their educational, social, managerial and technological aspects, but they are also called to work globally as drivers for innova-tion and to contribute to sustainable development.In this environment, in parallel with institutional, regional, national and international demands and challenges, improving the performance and quality of universities has become more important than ever.At the same time, respect for individual students and accountability for the use of funding has to be taken into account, in addition to contributing to economic growth (European Commission, 2009;van Vught, 2008avan Vught, , 2008b)).Enhanced quality and increased transparency are strong driving forces behind competition in education and research.Ossiannilsson and Landgren (in press) showed in their study on three international benchmarking projects that the identification of success factors for e-learning and its implementation in the educational arena is of the utmost significance.
Benchmarking has become a more commonly used method for quality assurance and enhancement in higher education, as it deals with identifying gaps and making changes, but also with improvement and successful implementation of new procedures and schemes (Bacsich, 2009a(Bacsich, , 2009b;;Ossiannilsson, 2010Ossiannilsson, , 2011)).In a recent study, Ossiannilsson (in press) advocated the use of benchmarking as a method and a powerful tool for making improvements to teaching and learning (concerning elearning/blended learning) in the twenty-first century, and to support improved governance and management in higher education.The study showed that further research has to be done according to holistic perspectives in order to answer questions on the value and impact of benchmarking e-learning in higher education.These themes include: why is benchmarking used, what is benchmarking e-learning?When is benchmarking used?etc.Thus, this study is based on multiple case studies, together with cross-case analyses, in order to contribute to the existing knowledge and understanding of the value and impact of benchmarking e-learning in higher education.
In the following section, the concepts of benchmarking and e-learning will be further elaborated in order to set the scene.After this, the projects will be described briefly, and this will be followed by a summary of the findings from the multiple case studies and the cross-case analysis, with the focus on value and impact.Finally, some reflections and conclusions will be presented.

The Concept of Benchmarking
The concept of benchmarking has its origins in the management and business context.Nevertheless, it started in the private sector, and was introduced by the Xerox Corporation in the ways in which they conducted their successful development.Looking what others were doing, and especially what their competitors were doing, led Xerox to make changes internally in order to improve their quality and processes and to enable the company to gradually regain its market position.Their original definition of benchmarking was: "...a process for improving performance by constantly identifying, understanding and adapting best practices and processes followed inside and outside the company and implementing the results.The main emphasis of benchmarking is on improving a given business operation or a process by exploiting 'best practices', not on 'best performanc'.Simply put, benchmarking means comparing one's organization [sic] or a part of it with that of the other companies." 1amp one of the most frequently cited scholars with regard to benchmarking (cited in Bacsich, 2009a;Hämäläinen, Jessen, Kaartinen-Koutaniemi, & Kristoffersen, 2003;Johnson & Seborg, 2007;Ossiannilsson, in press;ReVica, 2009;van Vught, 2008a), explored the benchmarking process using five continuous steps: determining what to benchmark, forming a benchmarking team, identifying benchmarking partners, collecting and analysing benchmarking information and finally taking action.The steps are illustrated in Figure 1.
Benchmarking has developed into an essential tool for organizations, and is regarded as a vital component of good managment practice.It is internationally respected, not just in busisses, organisations and management, where the concept came from, but also now in education and higher education (Moriarty, 2008).Moriarty and Smallman, (2009: p. 484) stated that "the 'locus' of benchmarking lies between the current and desirable states of affairs and contributes to the transformation processes that realise these improvements".Furthermore, Moriarty (2008) discussed the fact that although the literature on benchmarking is overwhelming, research and evidence on the value and impact of benchmarking is still missing.The criticisms of benchmarking are, according to Moriarty (2008), based mainly on lack of information, difficulties with implementation and a lack of theory.He stressed that benchmarking may require another definition, and that benchmarking is intended to be a means towards the end of achieving a more desirable organisational state of affairs.Benchmarking may identify the changes which are necessary to achieve that end.The concept of change seems to be inherent in benchmarking.Benchmarking is, however, not just about change, but about improvement, or as Harrington (1995) put it, "all improvements is [sic] change, but not all change is improvement" (cited in Moriarty, 2008: p. 29).Mo-Figure 1. Illustration inspired by the benchmarking process described by Camp, 1989.riarty continued by stating that benchmarking is not just about making changes, as it is more about identification and successful implementation.Therefore, he suggested a provisional definition: "Benchmarking is an exemplar-driven teleological process operating within an organization [sic] with the objective of intentionally changing an existing state of affairs into a superior state of affairs."(Moriarty, 2008: p. 30).
Benchmarking is a rather new phenomenon in higher education, especially with regard to e-learning (Bacsich, 2009a;Ossiannilsson, in press;Ubachs, 2009;van Vught, 2008avan Vught, , 2008b)).Quality assurance, quality indicators, benchmarks and critical success factors for e-learning have neither been taken seriously nor incorporated into regular quality assurance procedures in higher education.These concepts have not been conceptualised, and are even sometimes taken as being synonymous.The concept of quality in e-learning has been discussed, considered and managed in a very disconnected manner, and it has not been embedded in learning and quality contexts (National Agency of Higher Education in Sweden (NAHE), 2008;Soinila & Stalter, 2010).Ossiannilsson also showed in earlier studies (2010, in press, 2011) that there is a lack of experience of the value and impact of benchmarking in higher education.
The ESMU has worked with benchmarking in several projects in different areas within higher education.The definition of benchmarking used by ESMU is: "Benchmarking is an internal organizational [sic] process which aims to improve the organization's performance by learning about possible improvements of its primary and/or support processes by looking at these processes in other, better-performing organizations" (van Vught et al., 2008: p. 16).
As shown in the definitions above, benchmarking is very much a process designed to enhance quality, to identify gaps and to bring about the implementation of changes.Benchmarking initiatives are often conducted as self-evaluations, including systematic data and gathering information from predefined benchmarks, as well as formulating roadmaps.The goal of benchmarking is to formulate, together with other colleagues, strengths and challenges for the purpose of improvement (Ossiannilsson, in press;van Vught et al., 2008b).The benefits of benchmarking were expressed by the ESMU in 10 statements: "self-assess institution, better understand the process, measure and compare, discover new ideas, obtain data to support decision-making, set targets for improvement, strengthen institutional identity, enhance reputation, respond to national performance indicators and benchmarks [and] set new standards for the sector" (van Vught et al., 2008b).
Participating in a benchmarking process can potentially lead to change in the area being investigated.In addition, an increased awareness, both individual and collective, of the organisation itself occurs as a result of participation, which can be considered as a direct and substantial value (Ossiannilsson, in press, 2011).This awareness can often lead to reflections.Reflections are crucial and can even be a method for organisational change (Højrup, 2004;Ossiannilsson & Landgren, in press).

The Concept of E-Learning
Information, Communication and Technology (ICT) has become embedded in the social and economic sectors, and ought to be similarly embedded in the education and training systems.However, new and more efficient ways of operating, supporting pedagogical and organisational innovation need to be found.In the twenty-first century and with the new millennium learners existing in a digital world, the e-phenomenon has to be embedded in all learning and educational activities in order to push the boundaries in daily life, in a global sustainable environment (Bates, 2010;Bonk, 2010;Conole, 2010;Ehlers & Pawlowski, 2006).Hence, several scholars have argued that there is no longer a need for definitions, as e-learning has implications on a vast number of fields (Ehlers & Schneckenberg, 2010;Johnson, Smith, Willis, & Haywood, 2011;O'Reilly & Batelle, 2009).However, research has shown that respect has to be given to critical success factors in educational environments in order to succeed in the field of e-learning and to benefit learners, teachers and management (Ossiannilsson & Landgren, in press).McLoughlin and Lee (2008) explored success factors under the headings of personalisation, participation and productivity.Bonk (2009) went a step further, and expressed this concept as ubiquitous learning (u-learning), with the focus in the 'you perspective' on personalisation and the learner's rights and responsibilities.The EADTU emphasises within their E-excellence Associates label four success factors regarding e-learning, namely accessibility, interactivity, flexibility and personalisation (Ubachs, 2009).Connectivism is also considered as another concept which is essential to success i.e. that knowledge is distributed across networks of connections, and learning therefore consists of the ability to construct and traverse these networks (Siemens, 2005).The concepts and success factors related to e-learning in the twenty-first century will surely change the current learning climate, and may have an impact on how benchmarking e-learning in higher education will be conducted in the future and the kinds of quality-related issues which matter (Ossiannilsson & Landgren, in press).
Management is fundamental regarding the integration and implementation of ICT (Ubachs, 2008;Sangra, 2008).Sangra (2008) stated that the integration of technology, organisation and pedagogy is crucial for success, as all of the elements are needed to increase productivity and processes.New challenges for universities in the twenty-first century include bringing together all of the aspects of e-learning in a holistic framework, and perceiving these concepts in context (Ehlers & Schneckenberg, 2010;NAHE, 2008;Soinila & Stalter, 2010).Bates (2009) argued for the need for experimentation, innovation and vision where there are challenges, in order to bring together three competing factors: increasing access; increasing quality or improving outcomes, and reducing costs.

The Projects
Two European benchmarking initiatives regarding e-learning were recently implemented.The first one was the E-xcellence+ project, carried out during 2008 through the EADTU (Ubachs, 2009).The second was an e-learning benchmarking exercise, carried out during 2009 by the ESMU in collaboration with the EADTU (Comba et al., 2010;Williams & Rotheram, 2010).The focus of this article is on these two projects.Neither the research, nor this paper, will primarily focus on single benchmarks, indicators, critical success factors or methodologies as such, but rather on the value and impact of benchmarking on e-learning.Both initiatives aimed to identify good practices with regard to e-learning by learning from other participants.

The E-Xcellence+ Project
The EADTU (Ubachs, 2009) initiated the E-xcellence project in 2004 as part of the e-learning programme on behalf of their members, who had indicated that specific e-learning criteria were missing from the current quality assurance systems in place in Europe.The E-xcellence instrument was developed in co-operation with The European University Association (EUA), United Nations Educational, Scientific and Cultural Organization (UNESCO) and The European Association for Quality Assurance in Higher Education (ENQA), in order to complement the existing quality assurance systems in higher education, rather than to interfere with them.The project is more explored and investigated in more detail in research papers by Ossiannilsson (2010Ossiannilsson ( , 2011) ) and by Ossiannilsson and Landgren (in press).The quality benchmarking assessment instrument which was developed covered pedagogical, organisational and technical frameworks, with special attention being paid to accessibility, flexibility and interactivity.After the piloting phase, and under the E-xcellence Associates label, personalisation was also highlighted.The instrument was based on a manual covering 33 benchmarks on e-learning, across three main areas: management (strategic management), products (curriculum design, course design and course delivery) and support (student and staff support), with indicators relating to benchmarks, guidance for improvement and references how to achieve excellent levels of performance.In addition, assessors' notes were included to provide a more detailed description of the issues and approaches, as well as the tools, comprising the online instrument, the quick scan and the full assessment.E-xcellence+ was piloting during 2008 at local seminars (Ossiannilsson, 2011), and three universities carried out the full assessment, together with site visits.Several universities carried out the quick scan.Universities who conducted the full assessment on the level of excellence and underwent site visits, worked out roadmaps and committed themselves to carrying out the benchmarking process for e-learning in higher education every other year awarded the E-xcellence Associates label.

The E-Learning Benchmarking Exercise
The e-learning benchmarking exercise, devised by the ESMU in co-operation with the EADTU (Comba et al., 2010;Williams & Rotheram, 2010), was carried out by nine European universities during 2009-2010.This initiative combined the ESMU's collaborative benchmarking approach through a comparison of the good practices of different universities with the EADTU's more individualistic approach, as described above.The exercise started with the self-evaluated quick scan from the E-xcellence+ scheme (Ubachs, 2009).Furthermore, two workshops with international experts were carried out, and roadmaps for improvement were designed by the partners.In order to find some kind of common point of departure within this partnership, e-learning was considered to be. 2

Material and Methods
In order trying to explore a multifaceted phenomenon in depth, this study used an exploratory multiple case study strategy, as described by Yin (2003Yin ( , 2009)).A mixed-methods approach was applied, utilising a combination of qualitative data sources and integrated methods for data analysis (Yin, 2003(Yin, , 2009;;Creswell & Clarke, 2007).The data were mainly processed using cross-case analysis, with embedded and multiple units of analyses.

Respondents
The cases for the current study were selected from both projects: E-xcellence+ (three respondents) and the e-learning benchmarking exercise (four respondents).One of the respondents from the latter project took part in both projects (Table 1).The respondents from the E-xcellence+ project were chosen because they had been the first in Europe to be awarded the E-xcellence Associates label.All nine partners from the elearning benchmarking exercise were invited to take part in the study, and five of them agreed (however, one agreed too late to be included in this paper).In the following work, the respondents are not called by their real names, but instead are called Alpha, Beta, Gamma, Delta, Epsilon and Zeta.

Data Collection and Procedure
A case study protocol was worked out for the data collection procedure, which was then sent to the respondents in advance.The procedures were carried out so as to increase the reliability of findings and to maintain the chain of evidence, in line with the work of Yin (2003Yin ( , 2009)).
Existing survey data.Basic information about the respondents and their universities was found out in advance, and more information was gathered throughout the projects and their completion.The experts' reports from the benchmarking exercises were used as additional data (Comba et al., 2010;Williams & Brown, 2009;Williams & Rotheram, 2010).
Additional data for this study were collected through in-Table 1.

Respondents E-xcellence+ e-learning benchmarking exercise
depth qualitative interviews or more as narratives recorded through Adobe Connect.One interview was carried out at the respondents' office (in this case, two respondents were both involved in the dialogue throughout the interview, which is also the way in which they work and teach) and recorded with a manual audio recorder.The interviews were typed out in verbatim.Following the case study protocol, open-ended questions on the following main themes were used for the interviews in order to capture the respondents own experiences:  Why did your university joined the benchmarking project? What was your opinion of the process after its completion? In your opinion, where there any drawbacks within the project process? Do you have any additional thoughts regarding the involvement in the benchmarking project?

Data Analysis and Process
The existing survey data served to illustrate the case studies.Subsequently, during the analysis process, the data sources were analysed and integrated in order to contextualise the findings (Creswell & Clarke, 2007) and to elucidate the issues regarding the value and impact of benchmarking on e-learning in higher education.Each case was first analysed on an individual basis, followed by cross-case analysis (Miles & Huberman, 1994;Yin, 2003Yin, , 2009)).The qualitative data from the cases were processed in the following way.In order to obtain a general and holistic sense of the whole situation and to become familiar with the information in the cases, the recordings and the transcriptions of the in-depth interviews were repeatedly listened to and read through case by case.Listening to the material repeatedly reminded is of its meaning and content, but also gave a sense of the feelings and emotions contained in the narratives.The essential statements which had been made i.e. regarding the significant issues (natural units) were distinguished (Yin, 2003(Yin, , 2009)).The words were counted and a "wordle"3 was worked out (Figure 2).The text, case by case, was further condensed into units of meaning in relation to the respondents' expressions which served to highlight certain issues.Using this method ensured that the findings stayed close to the original data.The findings were aggregated or clustered according to key issues, reflecting the value and impact of taking part in 2 "…covering a wide set of applications and pedagogical processes supported by ICT learning, such as web-based learning, computer-based learning, virtual   benchmarking exercises.The analyses produced a summarised description of each case.The descriptions were intended to reflect the respondents' narratives and core experiences as closely as possible.The findings begin with an overview of the six cases, and then the findings will be presented case by case.Finally, a cross-case analysis, exploring similarities, differences and variations, will be presented at the end of the findings.

Overview of the Cases
The six cases and all of the respondents were from research intense universities and so-called traditional universities which employ a blended approach.Three of them were in a process of transformation, in that they were merging universities and faculties, which was one of the reasons for participating in the benchmarking exercises.They were also in the process of formulating e-learning strategies.Three of the respondents were from universities in which e-learning was not separated from other forms of learning e.g. in policies and plans; however, they strongly emphasised new technologies and e-resources throughout the university.The universities had well-known international reputations for e-learning and distance education.
In the report from the e-learning benchmarking exercise, the experts noted that participation in benchmark exercises always involves reflections on the part of the institutions involved on the lessons learnt and on new approaches and methods that the experience can be expected to bring.They suggested "...integration of benchmarking into strategic planning and to conduct benchmarking exercises as a regular practice to support ongoing organisational evaluation and retaining a competitive edge.Such a change agenda depends crucially on strong leadership to set clear directions and ensure their implementation.Highly-performing higher education institutions use a variety of tools, including benchmarking, to better understand their operations and progress towards increased performance" (Williams & Rotheram, 2010: pp. 29-30).The experts also emphasised the important lesson that, in reality, the most important work in the benchmarking process actually starts when the benchmarking procedures are over and the implementation phase begins i.e. when changes are made with regard to human capital and implemented within the organisation.
In the report by Williams and Brown (2009), they pointed out, in addition to their findings from the site visit, some problems with international benchmarking.Some of the problems to be aware of are, for example, the fact that benchmarks often relate to more than one point and thus require a multi-faceted response and a team-based approach.An important issue which they noted concerns the interpretation of languages and linguistics, as well as cultural interpretations.In this case (E-xcel-lence+), some benchmarks were also lacking e.g.library resources and student involvement, which were reported back to the EADTU.Williams and Brown (2009) also expressed that university campus courses would gain a great deal if they had the same transparency as the e-learning courses which were involved in the benchmarking process.They emphasised the importance of commitment at the management level in question when starting a benchmarking exercise, as benchmarking is about the implementation of results and (eventually) about change.

The Six Cases
The findings from the cases are summarized and presented below case by case.Each case starts with setting the scene followed by their experiences expressed in their narratives.

Alpha
Setting the scene.Alpha expressed two reasons for joining the project (the e-learning benchmarking exercise).One of the reasons was the desire to be part of one of the leading universities in the country within the e-learning arena i.e. to measure performance and to measure value for the money invested.Some ranking aspects were also considered.Another reason was to learn from others and to network for the purposes of further collaboration.
Being a multi-campus university creates challenges when it comes to staff and student support and the organisation of competence development schemes for both administrative and teaching staff in the field of Internet-based communication, collaboration and learning.At the same time, the multi-campus situation provides a natural driving force for the implementation of online technology that can help to save time and money on transportation between campuses.
Experiences.Alpha expressed and referred to the ten summarised values which were described above by van Vught et al. (2008b).However, Alpha expressed that one of the most important values was possibly decision-making (i.e. during and after a benchmarking exercise, when an institution receives basic data and documentation).Other core values were perceived as an increase in awareness and understanding at the individual, collective and organizational levels.This is a sustainable value, as it involves the internal involvement of stakeholders within the organization.Alpha also expressed that they had obtained both explicit documentation and also tacit knowledge and understanding which, due to the benchmarking exercise, became explicit.In addition, Alpha articulated that when one undertakes a benchmarking exercise with benchmarks to relate to, and holds discussions and interviews with stakeholders in an organization, a collective picture of the situation is gained, which had a fairly large impact.Relationships were strengthened and team members became closer to each other.According to this respondent, benchmarking is very much a question of involvement.This has resulted in spin-off effects in other projects within the organization.
Alpha also expressed that one of the lessons learnt was that, during the benchmarking itself, a great deal of attention was devoted to gathering data, collaborative discussions, writing reports and formulating action reports with a lot of valuable help received from partners and experts.Conversely, less attention and support was gained during the implementation phase.Choosing stakeholders is another important issue, especially with regard to organizational learning, when the management needs to be involved.Alpha stressed the importance of commitment at the managerial level.This is probably the most important step in terms of organizational learning and development.Alpha felt that they came out relatively well in the benchmarking exercise, both in terms of their self-evaluation and also compared with the others.Accordingly, they learnt that they needed to look even more at their practices, and that improvement and consolidations could be made.Additional lessons were to consider a more constructive approach to learning in relation to e-learning, and that learning needs to be more deeply embedded in educational processes, i.e. to focus on how learning can be enriched by technology, rather than how technology can enrich learning.The fact that a new project had been created within the organization with the aim of implementing some of the results of the benchmarking exercise was an explicit result.The values of the exercise, for Alpha, lay in extended networking with partners in the project, which had led to spin-off effects such as new projects and collaborations.
Alpha emphasised the importance of benchmarking as a method for quality assurance, as it involves self-evaluation and learning through and with others.Nevertheless, it requires a large amount of resources.Alpha put forward the idea that, in this context, it is probably of value to consider the content and the number of benchmarks.The drawbacks were that there were too many extensive benchmarks.Another issue was the linguistic sense, (the benchmarks were in English), but also in the translated form, there were different cultural interpretations among the partnership, but also within the organization.

Beta
Setting the scene.One of the main reasons why Beta joined the project (the e-learning benchmarking exercise) was the desire to be able to compare their e-learning efforts with those of other universities.Other reasons to participate were a need to gain new inspiration and insights, and also to learn from others in a spirit of collaboration.Beta is also a multi-campus university, and so the same arguments apply as in the case of Alpha.
Experiences.Beta expressed a whole range of benefits for their university.On a personal level, Beta learnt a lot about their organization.An unexpected advantage was that they became aware of some new contacts, for strategic and practical work, even within the organization.They also appreciated that they gained a good overview of what they were actually doing within the field of e-learning, as well as their strengths and weaknesses, what could be improved and what targets to set for the years ahead.Beta expressed this as follows: "I think it surprised me how many different benefits on so many different levels in the organization we actually gained.It was valuable to focus on the entire organization and the future of e-learning in the university."At a managerial level, the project report was seen as important, as the university came out with good results.This also showed that the resources allocated to the area of e-learning at the university were meaningful.It focused their attention on quality and the need to link e-learning to quality management efforts and to secure the widespread use of e-learning.Within Beta's quality organization and quality committees, they ensured the involvement of stakeholders and the dissemination of the project.With this level of organization and commitment, it was easier to work with the lessons learnt.Though, this process was not perfect from the very beginning, but they caught up in many ways during the process.From the start, Beta was under pressure to come out as one of the best universities in the partnership.
Conversely, having gone into the process and learnt more and more about the organization, Beta felt that "the lessons you learn and the knowledge you gain in the process is perhaps the most valuable part of the benchmarking process".Of course, they were happy to point out their strengths, but also to record that there were some points which needed improvement, "so you do not just rest on your laurels".
Beta also gave voice to the fact that the benchmarking process and results motivated people to developments, either by continuing with current developments or through innovation.Beta articulated that there were some limitations to the ways in which the benchmarks were formulated, in that they were pragmatic and depended on a certain way of looking at elearning.Thus, Beta would recommend more open-ended questions and also more time to work through and revise the benchmarks within the partnership.It should be more like a joint effort and a collaborative approach.It was also suggested that time frames could be added for the further development of the process, i.e. time allocation for follow up processes.
In addition, Beta emphasised that the successful areas within e-learning are mainly based on backup from top management, with clear and obvious visions and strategies throughout the whole organization.The development of teachers' competences within the fields of learning and blended learning are also of the highest importance, together with pedagogical and technical support.Well-implemented e-learning platforms and tools are necessary.Knowledge sharing and the sharing of experiences both nationally and internationally are furthermore of the utmost significance.Finally, Beta expressed that funding should be provided for pilot projects or the most innovative projects within a certain area.

Gamma
Setting the scene.Gamma expressed three main reasons for participating in the project.First, they expressed that they were in the middle of a major merging process, and that they were also reducing the number of their faculties.Second, they expressed that there was a wide variety of e-learning standards and procedures at the university.Getting involved in the benchmarking exercise was therefore one obvious way in which to gain an overview of what was going on and a way to commence internal co-operation.Finally, their participation was a way for them to generate a roadmap and to get some new ideas, thanks to this international co-operation.They had no expectations regarding how well their participation would go or whether or not they would succeed.Gamma is also a multicampus university, and so the same arguments apply as were mentioned for Alpha and Beta.
Experiences.Gamma articulated that, in principle, benchmarking had a great deal of value and a significant impact, as suggested by van Vught et al. (2008).The value for Gamma was not so much in learning with and from collaborators, as in coming to know one's own organisation, especially as they were in the process of an extensive merger (so they did not have any working practice to relate to).Therefore, the main impact upon Gamma was related to values and lesson learnt regarding how they formulated themselves.Gamma expressed that "benchmarking is a great tool to point out flaws, to explore and spot missing areas and to raise attention.It is also a good opportunity to work with best practice and it is also a method to create inspiration".Gamma is in a phase of reorganization, and due to this they had many ideas relating to the processes of innovation.Gamma suggested that benchmarking ought to be mandatory for universities, either on an annual basis or every other year.Benchmarking should take place in different areas in order to enhance quality and with the aim of being in the frontline.In this case, benchmarking is a great method for educational movement; enhancing quality and working processes in order to relate to best practice etc. in a more structured and more professional manner.Universities need to have benchmarks and indicators in order to speak the same language.In addition, benchmarking allows universities to extend their networks and to collaborate on setting up new projects or bilateral agreements (spin-off effects).
Gamma expressed some pitfalls of benchmarking, which of can be turned into opportunities as and when they became visible.Some pitfalls are language and cultural differences, but also the question of how to set benchmarks e.g. in this case, they were rather unfashionable, so Gamma had to go above and beyond in order to connect to their area and context.In order to improve, it is essential to consider cultures, such as the university's own culture but also international perspectives.The disadvantages also include that benchmarking is time-consuming and demands focus.However, Gamma expressed that if one spends time and focuses on the work in a benchmarking exercise, one will get more out of it and therefore gains more from it: "What comes in goes out".
Gamma also stated that they now focus more on learning and how students interact with information technology (IT) and the social media in their context.This raises questions about the contextualization of learning and education.Therefore, in the event that a benchmark exercise should be set up again, they will consider these phenomena to a greater extent.

Delta
Setting the scene.Delta expressed that as they were selected to take part in the benchmarking project (E-xcellence+), they naturally appreciated the opportunity and wanted to join in.They were also curious with regard to obtaining an awareness of how far they had reached, even though they had established an international reputation many years ago, and had many international students enrolled.Second, they saw the advantages of the process, such as obtaining an internal awareness and knowledge about their own organization.The third driving force was the fact that there was an opportunity to attain official international recognition; this, of course, was a motivating factor.Another motivating factor was the future situation of the university, as they from 2011 have to charge tuition fees for students outside Europe.Competition will thus be more explicit, and there will be an even stronger drive to ensure that students get what they pay for.Being internationally distinguished is, in this context, very significant.
Experiences.Delta was the first university in Europe to be awarded the E-xcellence Associates label by EADTU as a concrete recognition of the high quality of their e-learning programme.As a result of this, the level of their national and in-ternational collaboration, as well as student recruitment, was increased.The real impact of this will first be noted in a few years when the first students (who are paying tuition fees) enroll and they are asked why they chose this programme instead of others.In addition, students who are already enrolled are impressed and appreciate being enrolled in a programme which has been labelled as excellent.They have also articulated a positive response to the educational culture and level of professionalism they have experienced.Delta also emphasised that they had received an increased level of recognition for their work at different levels within the university, which has resulted in spin-off effects.
Delta stated that, in terms of their positive experiences, they have had the opportunity to undertake the benchmarking and internal processes, which increased their understanding and knowledge.To have the chance to reflect and to share within the centre, but also with colleagues within the university, was very valuable, and made their work more effective.In addition, it was valuable to perform a self-evaluation, as well as the documentation and the experts' report.This was also a way for Delta to gain an awareness of the areas in which they fall short.In addition, they had opportunities to review and improve their routines.
Thus far, they have not been involved with or taken part in the Excellence Associates club, but they will in the future and can see the advantages of such a network.The roadmap that they worked out has served as an inspiration and as a driving force.It is not a formal action plan, but a working plan and an inspirational document.They expressed that they have either worked through some of the actions or at least considered them.It is an honest document, referring to what they ought to do, considering time restraints and financial opportunities.They articulated that, during the process, they benefited from support and commitment from the management team, and they also stated that, without this support, it would be more or less impossible to carry out changes or improvements in relation to the benchmarking exercise.Other issues were that the exercise was so time-consuming due to the need for data gathering, internal discussions and analyses that support and commitment were needed.They expressed that they had experienced continuous support, even now the real work has started with the implementation and transition phase.
Delta emphasised the importance of involvement and teamwork during the process.The benchmarking exercise was as such also important due to the need to "attract attention, gather and spread information and engage staff and students (three core concepts".Delta articulated that the benchmarks were comprehensive, but maybe too comprehensive.On the other hand, this contributed to discussions, interpretations and the development of knowledge within their own educational culture and context.If Delta were to undertake the exercise (or a similar exercise), again, they stated that it would be valuable to estimate time requirements, and probably also to have a more explicit overview of the process.The drawbacks of the exercise, when one is sensitive to them, can be turned into opportunities.Another issue is whether the university could or should market and raise awareness of the benchmarking process to an even greater extent.

Epsilon
Setting the scene.Epsilon indicated three incentives for their participation (in E-xcellence+ and the e-learning benchmarking exercise).First, they expressed that they wanted to obtain an awareness of what was state-of-the-art in the area.Second, they wanted to learn from others and to obtain an awareness of their progress in the area in an international context.Finally, they expressed that participating was a way to obtain insights and knowledge with an eye to the future.They also expressed that their participation was a way of gaining both internal and external processes.They stated that benchmarking is an optimal method for raising awareness and understanding of what is going on, both internally and externally.Both advantages and disadvantages become explicit, but primarily and most importantly, potential areas for success and where quality can be enhanced become visible within the common understanding in an organization.
Experiences.Epsilon expressed having found the exercise to be valuable both internally and externally, and that lessons can be learnt in and through both dimensions.For Epsilon, the projects had been extremely valuable.Benchmarking is a way of watching what is happening in the surrounding world.Regarding Epsilon's participation, it can be stated that they found it to be valuable on several levels; within departments, on the university management level, and on national and international levels.One example which was articulated several times in the in-depth interview was the fact that a new programme is going to be set up in collaboration with several other universities in the country.With this programme, there is a need for high quality work to be carried out with new technologies, not just for the purposes of content and inter-university collaboration, but also to create a model and to become part of an IT society in the twenty-first century.Even within other areas of the university, schemes are being implemented as result of the benchmarking projects in combination with the current discourse in this area.
Epsilon articulated that they found it interesting how many outstanding things had happened throughout their faculties and departments; it seems likely that these things have happened due to the fact that Epsilon is a decentralised university.They therefore expressed that freedom, enthusiasm and development on demand are crucial for a reflective and growing organization.The key is not always efficiency, as sometimes quality enhancement is more important.Both bottom-up strategies and top-down strategies require flexible methods.Benchmarking helps universities to choose a direction.At a research intensive university, where the teachers and programmes are at a variety of levels in terms of IT proficiency, from early beginners to those who excel, there are a variety of demands and requirements; however, there needs to be something for everyone, even at a central organizational level.The most important thing at a large university is to develop maturity and "teaching environments to learning environments to web environments and also that it [e-learning, new technologies, social media etc.] is embedded in [the] curriculum".This journey has to take time.An awareness of maturity is of great importance, as if an organization is immature, then no development will happen.There needs to be maturity, at both individuals and organizational levels.Each faculty/department has their own structures for dealing with demands and cultures.This is good.In addition, one has to make choices, and one has to be familiar with elearning and technology development.When maturity has been reached, co-ordination can begin.
The fact that two of the programmes at Epsilon were awarded the E-xcellence Associates label indicates that they have been recognised both internally and externally.For the programmes that were awarded the label and already have large groups of international students from outside Europe, this award is very valuable and will have a significant impact as they are facing a new situation regarding tuition fees for students from outside European Union.Of course, this is due to other programmes as well.The E-xcellence Associates label supports the entire university in its strategic work towards achieving consolidation, internationalization and distinction, as universities are striving to be world class.Epsilon stated that they valued all of the documentation which had been a part of the benchmarking projects, which they will now use for other purposes.One example is that when the National Agency for Higher Education conducted a survey of distance education in the country (Sweden), then the benchmarking reports supported the documentation required for this survey to a large extent.
Epsilon stressed the fact that e-learning needs to be userfriendly, integrated and embedded into courses and educational arenas.Epsilon stressed further the importance of freedom in a large organization, as creativity grows and things happen when there is freedom.

Zeta
Setting the scene.The main reason why Zeta chose to participate in the project (E-xcellence+) was to ensure the quality of its education and courses and to provide evidence that these were of the highest possible quality.The main reasons for this are due to student rights and student satisfaction.Other reasons were of both an internal and external character.The internal reasons were a desire to raise awareness of advantages and disadvantages concerning range, support and implementation of the e-learning arena at Zeta, and in addition student satisfaction, and through the process find gaps for possible improvements.
The external reasons were mainly a desire for international co-operation and networking, but also an attempt to learn from and through others.As benchmarking is a form of self-evaluation, it is more honest and the motivation to make changes is thus higher, as any criticism tends to be more constructive.There were also other reasons, such as the fact that each member of the staff including the students need to feel appreciated for their work.It is also important to note who the actors are who initiated and took part in the benchmarking e.g. in this case, the benchmarking was initiated by the EADTU, the National Agency for Higher Education and the university itself.The chance to attain the E-xcellence Associates label was also an incentive to join the project.
Experiences.Zeta expressed that through the benchmarking project, advantages and disadvantages concerning range, support and implementation of e-learning became visible and ideas for changes were considered.The fact that Zeta earned the certification and the label was, of course, very encouraging, and the label demonstrated that the university had been recognised internally as well as in international collaboration and networking.In addition, more students are being recruited.This is, of course, a triumph, especially when the university was cooperating with countries that tend to distrust distance education.However, Zeta expressed that they might be able to market distance education to an even greater extent, but they already have more than enough students (i.e. more than they get paid for).Naturally, this was an occasion for self-evaluation and this influenced all staff and their work.Of course, the label as such is not as important as the knowledge and understanding of the work which the university is doing, as well as an awareness of their shortcomings.Nobody is happy with just the award itself.Zeta, therefore, articulated a very humble attitude towards its recognition.However, two of the leading staff at Zeta awarded the university a prize for excellence in teaching and pedagogy in the following semester.
Zeta stated that the roadmap served as inspiration and not as a dogmatic document.Some of the actions have already been taken and some are being considered, due to limited time and resources.They expressed that they have been supported by the university, although perhaps not always at the closest administrative level i.e. the faculty level, probably due to a degree of mistrust, as they are too far ahead and too innovative in their course development and delivery.Maybe this had been the fact anyway, if they had been innovative and therefore succeeded in other areas, they would have experienced mistrust there as well.
Concerning the benchmarks, Zeta expressed that they were particularly extensive and that they worked in teams in order to get the staff and students involved.They used micro and mini meetings to go through everything and to discuss different aspects of the process.Zeta gives voice to the idea that one of the most important issues is flexibility at all levels and dimensions in any attempt to satisfy students.The ultimate goal is to satisfy students, and they frequently demand a high degree of flexibility.Satisfied students are important quality indicators.Flexible and boundless education with regard to pathways, time (even starting time), free resources, flexible materials/resources (texts, audio, video) etc. is of paramount importance and forms the foundation of the programme.Student evaluations have also shown that this flexibility is crucial.
One explicit impact was an inspection of the learning platform and the overall university administrative system in order to make communication more flexible.Zeta stated that when e-learning programmes begin, the most important aspect seems to be the learning dimension.This is natural, but when courses are scaled up, it is crucial that administrative systems communicate well.
Thus far, Zeta has limited experience of the E-xcellence Associates club, due to limited time and human resources.Obviously, such initiatives are of the highest importance.Zeta's final comment was that universities which are striving for the highest quality in an international field should certainly consider benchmarking.

Cross-Case Findings
Overall, the findings indicated many similarities between the universities, but also differences, or rather different ways of expressing experiences across the cases.The cases illustrated how different aspects interact to a varying extent and indicated the value and impact of the international benchmarking exercises which were conducted.In the section below, some of the key areas are summarised.

Internal Processes and Involvement
The benchmarking allowed the teams to maintain their focus.Involvement and shared responsibilities in the work with benchmarking enabled not just the benchmarking itself, but also contributed to commitment and appreciation among the co-workers within the institution.The fact that advantages and disadvantages in the field of e-learning became explicit supplied the motivation to make the changes which led to the impact of the scheme.In addition, the fact that the exercise was mainly a self-evaluation motivated the institutions to examine even negative issues, which often could be turned to challenges.
In addition, the solid documentation which was collected, as well as the knowledge and institutional awareness regarding elearning was important and led to a significant impact.The institutions gained a lot from the solid work with its transparency, which could even be used and valued in other contexts.The awareness of the infrastructural support for e-learning, which was one of the results of the benchmarking exercise, was made explicit and this led to closer collaboration between the infrastructural units in question, as well as further collaboration with other faculties and departments, for example within pedagogical areas and e-resources.Additional benefits were clearly expressed regarding teamwork, creating a dialogue, policy making, quality assurance and transparency within the organization.

Quality Enhancement
Benchmarking is an advanced method, but the cases have indicated that it is quite easy to use with explicit opportunities for self-reflection.Even though certain steps have to be taken, one can set the structure within the partnership as either a smaller or a larger exercise; one can also define the benchmarks, partners and the time schedule.Nearly all of the cases reported that enough time and support was allocated during the project period.However, they emphasised that they misallocated time and support for the reflection and implementation phases.This process needs to be followed by reflections on the lessons learnt, which can provide incentives for further development in the institution.This may involve changes in structure, organisation and resource allocation, which may require strong leadership at all levels.Furthermore, the importance of integrating benchmarking processes as a natural part of strategic quality assurance work was emphasised.The value of continuously following up earlier benchmarking exercises as well as committing to new projects was highlighted.All of the respondents agreed on the value of benchmarking as a method for quality assurance.

Management and Commitment
The cases showed, to a significant extent, the importance of full support and commitment from all levels of the management team during the whole benchmarking process.Such support is necessary in order to maintain the focus on the project, to allow staff and students to be involved, to work in an interdisciplinary manner and award status for the ongoing work and dissemination during the process, but most of all with regard to data gathering and reports i.e. the implementation phase, if changes have to be made.In these cases, several departments were involved in full.They were therefore able to focus on and discuss common areas and processes, creating togetherness, trust, commitment and involvement.This will certainly contribute in turn to enriching the future employment situation and potential areas for development.

Collaboration and Networking
When taking part in benchmarking exercises, universities have the chance to meet other institutions that are facing the same challenges as themselves.There are opportunities to approach one another, to learn from best practices and to take part in joint discussions on how to handle challenges.This can evolve into continued benchmarking on a smaller scale, where two or more universities from the group collaborate and formulate new benchmarks.Such new benchmarks can help the institution to improve important areas and can play a major role in continuing quality assurance efforts.On a more practical level, the knowledge exchange between benchmarking participants can lead to mutual inspiration and can help the individual universities to understand and handle topical issues within the field of elearning.
In the benchmarking exercise, it became apparent that each university possessed best practices within certain areas that were of importance for all of the universities.These areas included pedagogy, technology, strategy etc.In the benchmarking exercise itself, there was unfortunately no time to study the best practices of the other universities in depth.
However, the exercise itself provided a good overview of the strengths of each individual university.Mutual trust and understanding has been gained, and future activities such as joint applications and agreements for further co-operation will naturally be established.A network has come into being, in which participants can use each other as experts when knowledge of a certain topic is needed.

Cultural Issues
As this was an international benchmarking initiative, English was the chosen language.All of the respondents raised a problem with this, as none of them had English as native language.It was not just the language and linguistics as such, but also cultural aspects of the language and how it is used in different contexts.The benchmarking was mainly carried out in teams, and even within the teams there were differences in how certain concepts were understood.Another issue was the somewhat old-fashioned concepts regarding e-learning.Nearly all of the cases work within a blended mode context.In addition, nearly all of the universities in question are decentralised, and thus there was not only one way of acting and performing.

Attitudes
Through the in-depth interviews, all of the cases voiced their enthusiasm and positive feelings regarding the exercise itself, but also regarding the methods.It was, for example, noted that during a comparatively short time, many lessons had been learnt, both about the institution themselves, but also about wider perspectives including other universities and the surrounding international context.Furthermore, it was articulated that they had also gained a deep understanding and knowledge in the area of e-learning through the benchmarks and indicators, areas which to some extend seemed to be tacit knowledge.In addition, they expressed their enthusiasm and interest in continuing to work, or as they put it, they had the energy to tackle future challenges.All of the respondents expressed that it would be useful to carry out the exercise again, but maybe on a smaller scale with time to define their own benchmarks in a more useful way.

Discussion
This study explored the experiences of respondents who had taken part in an international benchmarking project.The integrated case study findings provided critical information on the value and impact of benchmarking in higher education.Most importantly, this study showed the complexity of the value and impact of taking part in such exercises.The complexity of benchmarking e-learning in higher education is based on the fact that values can refer to either immediate or more long-term values.Furthermore, these values can apply to different aspects and different levels, and to both individuals and organizations.Moreover, some effects take years to evaluate.
One contribution of this study to our present knowledge and understanding of benchmarking as a method seems to be a recommendation that time, resources and maybe even the use of experts and/or co-partners is allocated for the implementation phase, as this is an important step in the benchmarking process.All of the respondents in the study clearly emphasised the lack of support they had received afterwards.Another contribution is the apparent importance of commitment from managerial levels.In the event that structural changes or other changes have to be made, resources will probably need to be allocated or re-allocated, and due to this it is extremely important to have commitment from the leadership.This works both ways; if the institution produced as "good result", it is essential to communicate and market such benefits, internally as well as externally.It is never easy to speak up for oneself, and so support for marketing is often needed.
We are facing a paradigm shift within the twenty-first century, and e-learning as a phenomenon was considered among the respondents to have a very broad meaning.Therefore, the learning dimensions were emphasised.Changes in technology have to be considered in education, even through a social-contextual approach.This study shows, therefore, that benchmarks must be chosen with care and that linguistic, cultural and contextual issues have to be taken into account.Even if this study did not aim to focus on benchmarks as such, critical issues within the area were articulated, in accordance with the success factors described by the EADTU (Ubachs, 2008) and Ossiannilsson and Landgren (in press).The need for flexibility in all of the concepts and dimensions was clearly highlighted.Even the structure of the partnership is crucial, as are similarities and/ or differences within the partnership.How to choose benchmarks that provide direction is one of the aims within the benchmarking process.Trust, involvement, collaborative learning, honesty and courage are of the utmost importance in benchmarking processes for fruitful and sustainable work and collaboration.As mentioned above, there can be pitfalls, but awareness and thoughtfulness can turn them into challenges.
Turning to the theoretical frame of reference, this study showed, as the discourse on benchmarking also shows, that benchmarking is about change, and thus lessons have to be learnt from benchmarking processes (Moriarty, 2008;Moriarty & Smallman 2009;Ossiannilsson, 2010Ossiannilsson, , in press, 2011;;Ossiannilsson & Landgren, in press).Once again, as Moriarty & Smallman (2009: p. 484) stated, "… 'the locus' of benchmarking lies between the current and desirable states of affairs and contributes to the transformation process that realise[s] these improvements".In this study, the findings on the value and impact of benchmarking as described by the respondents were close to and in accordance with the ten good reasons to conduct benchmarking as described by van Vught (2008b).Some more critical issues were also touched upon, such as the significance of the first and last phases of benchmarking; the identification and implementation phases.Furthermore, the value and impact of tacit knowledge (Elliott & Stemler et al., 2011) were explicitly expressed.However, they were not only expressed; the possibility of sharing this tacit knowledge, as well as knowledge and experiences, was articulated as a possible impact.The case studies indicated the importance of support and management when making changes, as Sangra (2008) also argued.
Contributions to the transformation processes that realize improvements require commitment, but also reflection.Through the cross-case analysis, the respondents experiences in this study indicated that a process of philosophical interpretation is required, and that there is a need to modify the theoretical assumptions concerning benchmarking.Høyrup (2004) argues, for example, that there is a need for a learning organization to reflect, but he distinguished between reflection and critical reflection.He emphasised critical reflection in particular as one of the core processes of organizational learning.He stated, moreover, that critical reflection was necessary on both the individual and organizational levels.Critical reflection includes social, cultural and political aspects and concerns the 'why' (meaning) rather than questions about how to act or perform.The meaning of reflection and the importance of reflective practices for change was also clearly expressed in the study.
Another dimension of critical reflection can be seen as Deleuze's concept of "becoming" (cited in Giger, 2010).Becoming share the common meaning of becoming which even is true in this context.Thus, the process of becoming is about changes, just like benchmarking is about changes.Becoming, as a more philosophical concept, has something to do with becoming us as individuals.The rhizome concept, also put forward by Deleuze (cited in Giger, 2010), was described in brief as connectivity and multiplicity.Both concepts by Deleuze are new dimensions which are valuable for benchmarking and organizational change.
Reflections on this study also touch on limitations of the research approach.Of course it would have been valuable to have had all participating universities involved in the study.Probably the findings had been somewhat more valid.On the other hand the findings from the cases involved are seen as representative.Within the six cases variations, but also similarities were found.

Conclusion
Benchmarking is a powerful strategic tool used to assist decision-makers to improve the quality and effectiveness of organisational processes.This study showed that the quality of e-learning has to be considered and must be embedded in suitable contexts.This study explicitly showed the value and impact of benchmarking e-learning in higher education.In the case studies, the respondents expressed that the benchmarking process had a significant impacts i.e. the power to make changes.Lessons from this study have to be taken into consideration.Lessons on how benchmarking is conducted, on objectives, on how it is organized and set up, how to choose partners and how to choose benchmarks must be acknowledged.In addition, how to generate commitment and allocate time and resources during the entire benchmarking process, including the implementation phase, has been emphasised in certain cases.Some characteristics which may not be mentioned as often in a positive light are the intentions, i.e. the meaning of the bench-marking process as such.The involvement of individuals within the organization creates an impact on both individual and organizational levels, not least through critical reflection.The concepts of "rhizome" and "becoming" used in this study have leant other and new dimensions to benchmarking.Thus, this study is not only relevant to management, but to philosophy as well.
The study shows that benchmarking, in line with national and international quality boards/agencies, is a powerful tool for making improvements to teaching and learning in higher education.
classrooms and digital collaboration with an added value of increased accessibility, flexibility and interactivity."(personal communication, ESMU workshop, 25 May 2009.

Figure 2 .
Figure 2.An example of a Wordle made from the interview texts.