On the Ontogeny of Regional Tech Transfer Systems: The Case of Brussels ()
1. Introduction
Technology transfer (TT) is a rapidly growing field of research, accompanied by a growing set of empirics (Cunningham, Menter, & Young, 2017). Research in this domain has focused on the analysis of various themes, sectors and or actors and has done so in varying institutional and regional settings (Cunningham, Menter, & Young, 2017). Macro level analyses of TT systems, such as the one that is presented in this paper, do not often focus on analyzing the triple helix as presented by Leydesdorff and Etzkowitz (1996). Most of the research focuses on the emergence and functioning of different subcomponents of technology transfer (Clarysse, Wright, Bruneel, & Mahajan, 2014; Rothschild, 1990; Moore, 1993; Alvedalen & Boschma, 2017). Next to this, the current body of work studies critical success factors, impacts generated, and the benefits gained from geographically scoped TT (Cunningham, Menter, & Young, 2017). Although the components are well described and often intricately conceptualized (Good, Knockaert, Soppe, & Wright, 2019), we know little about how TT systems emerge and evolve. Knowing this, however, stands to impact the creation of future technology transfer ecosystems and allows for participants in contemporary technology transfer ecosystems to assess their own performance, the emergence of idiosyncratic procedures and processes and furthermore offers a benchmark that can be used for further analyses. This paper therefore presents the evolutions that guided the formation of the Brussels TT system (Jacobides, Cennamo, & Gawer, 2018).
We employ a qualitative method to capture the distinctive technology transfer system, its building blocks and support mechanisms given the research approaches’ fitness towards studying technology transfer (Cunningham, Menter, & Young, 2017). We present the results from 21 semi-structured interviews with regional TT experts to gauge how the technology transfer system evolved over time. We complement these interviews with an in-depth case study on the evolution of technology transfer in Brussels, based on publicly available data and secondary sources. As such, we analyze the emergence and evolution of this system on the following relevant system levels: actors, governance/organization, functions, and power.
The paper is structured as follows, first we introduce the key concept employed in our research, i.e., technology transfer systems. Then we address our research method including the rationale behind our case selection and selection of respondents after which we present both our descriptive as well as qualitative results, before coming to our conclusion.
2. On Technology Transfer Systems
Empirical support for the basic conceptual components of the field of technology transfer research is mounting. Yet, several gaps in the literature remain (Ankrah & AL-Tabbaa, 2015). Cunningham et al. (2017) also point towards a need for more longitudinal approaches to the investigation of the dynamics that underpin the transfer of technology (Cunningham, Menter, & Young, 2017). Furthermore, the authors highlight the need for investigations into technology transfer that are more holistic in their conception, thereby addressing the shortcomings of the current state of literature on technology transfer which is typified as atomistic (Good, Knockaert, Soppe, & Wright, 2019). Good et al. (2019) state that: “Our first finding is that papers discussing the TT ecosystem as a whole are scarce and those that do mention it typically take a regional or national perspective rather than an organizational or management perspective” (Good, Knockaert, Soppe, & Wright, 2019). As such, holistic and longitudinal approaches towards the study of technology transfer systems (ecosystems being one sort of systems) remain scarce and, especially, the application of the systems metaphor as a managerial frame of analysis is often absent at the macro level. What is more, research on technology transfer from a systems perspective seems preoccupied with the study of sub-elements of the system, rather than the system as a whole (Good, Knockaert, Soppe, & Wright, 2019).
In this research, we therefore focus on a particular geographical region and provide a long-term overview of the events that gave rise to and altered the evolution of the TT system through a managerial frame of reference (Bagur & Guissinger, 1987; Tornatzky, 2001; Reagans & McEvily, 2003). We therefore do not focus on the institutional and or multi-level governance frame of reference as this has already been developed in the work of amongst others (Hooghe et al., 2001; Christopherson & Clark, 2007).
3. Research Method
We employ a qualitative research method to engage in the factual inquiry into the emergence of an idiosyncratic regionally developed technology transfer system. We conduct 21 semi-structured interviews, complemented by an in- depth case study, in order to establish technology transfer and its genesis in the Brussels region. Next to the descriptive statistics used to establish the case, we opt for a qualitative strategy for two reasons. First of all, this method allows us to analyze coordination and concept related issues in an in-depth manner (Eisenhardt, 1989; Yin, 2003; Gibbert, Ruigrok, & Wicki, 2008; Pratt, 2009). Secondly, our choice to utilize a qualitative research approach stems from its tolerance towards the study of socially occurring phenomena. These phenomena based on human interactions, include the human perspective, in other words, this approach is sensitive to the fact that people attribute meaning to their environment. Hence, when studying the meaning that is attributed to concepts such as a stakeholder, or power, then, looking through the eyes of the people being studied is understood to be beneficial (Peters & Peters, 1998; Bryman, 2004). We address how university-industry technology transfer policy and practice evolved in the Brussels region by looking at the system-wide evolutions on a number of levels:
· Organization and functions of the TT system (Adner & Kapoor, 2010; Walrave et al., 2018)
· Stakeholders and their roles in the TT system (Hayter, 2016; Good et al., 2019; Dedehayir et al., 2018)
· The power balance and interdependencies in the TT system (Adner, 2006; Autio & Thomas, 2014; Jacobides et al., 2018)
4. Case Study Analysis
This paper approaches the history of technology transfer in Brussels from the case-study angle as we collect information about specific events and activities that took place within a geographically scoped region (Hair, Wolfinbarger Celsi, Money, Samouel, & Page, 2011). The reason for the application of this method is that obtaining a complete picture of the entire situation requires examining a real-life example. In doing so, the unearthing of interactions and developments becomes more workable and allows for a sound implementation of longitudinal research based on archival information and retrospective interviewing (Yin, 2003). The approach furthermore has shown its merits as it was applied to geographically scoped regions in the past by authors such as Golob (2006), and Hu et al. (2005), but also to broader regions such as done by Gulbrandsen and Nedrum, 2007; Hu, Lin, & Chang, 2005; Golob, 2006; Gulbrandsen & Nerdrum, 2007.
5. Data Collection
The case description, focusing on the evolution of the technology transfer within the region, employs a methodology that is based on descriptive statistics used to visualize some of the output indicators for successful technology transfer. The secondary data sources themselves include governmental statistics, business data, patent data, and internal university reporting. For the analysis of the secondary data, all relevant legislation implemented after 1991 until the present day was analyzed. This amounted to 49 documents ranging from 1991 until 2018 and consisting of both legislation (Federal and regional), policy preparation and policy analysis documents. The patent data included in the research addresses all patents registered by the Vrije Universiteit Brussel (VUB) and Université Libre de Bruxelles (ULB) since their inception and was downloaded from the European Patent Office’s official website (https://www.epo.org/). The total number of patents included in the overview is 289 patents for the VUB and 343 for the ULB. The internal university documents that we analyzed consist of the yearly reports produced by the technology transfer offices of both universities going back to their initial creation and in so far as they were still available. We complement this with an in-depth analysis of the yearly financial reports registered by all spin-off companies of both universities. In total 73 yearly financial reports were analyzed for the spin-off companies and 34 internal university documents were assessed.
For the second part of the research, we used a semi-structured questionnaire to capture the assessments made by key ecosystem participants retrospectively. Therefore, we conducted face-to-face interviews with technology transfer participants, functioning in universities, funding agencies, companies and government. We interviewed 21 people during a 6-month period in 2019. The number of respondents is limited due to the characteristic of the regional setting. All respondents were in charge of handling technology transfer matters, and or worked on the transfer of technology amongst ecosystem stakeholders.
Table 1 provides an overview of the respondents including a classification of their stakeholder type, their title as well as the years of experience that they have with technology transfer in the Brussels region. The interviewees were selected through recommendations made by the Brussels Institute for Research and Innovation Innoviris1. The ecosystem manager active at Innoviris provided the
![]()
Table 1. List of respondents and years of experience.
researchers with a shortlist of 36 people and organizations to be approached for an interview. Ultimately 21 people were found and willing to participate in the research.
6. Coding Method
Based on the conceptual positioning of our research and the constructs of the system that we retrospectively wish to explore, we decided to define an exhaustive list of questions to be included in the semi-structured questionnaire that we employed to conduct our interviews (Bryman, 2003). Appendix 1 provides an overview of the questions that were included and provides further information as to the reason for which they were included and also informs the reader on the scientific basis from which these questions were distilled. For each question, key constructs for discovery were defined and these elements are placed in a 3-phase timeframe: past, present & future (Fereday & Muir-Cochrane, 2006). Once the interviews were conducted, all information was centrally stored, and responses were analyzed on a question by question basis first. This analysis entailed the capturing of core ideas represented in the responses of the respondents as to enable us to summarize, consolidate and condense the information. Based on this analysis, additional pairing of responses allowed us to improve the richness of the obtained data.
7. Case Selection
We have selected Brussels or officially the Brussels-Capital Region (BCR), as the subject of our study. The BCR is a Belgian region comprising 19 municipalities, including the City of Brussels, which is the capital of Belgium. The Brussels-Capital Region is located in the central region of the country and is a part of both the French Community of Belgium and the Flemish Community but is separate from the territories of Flanders (in which it forms an enclave) and Wallonia. The BCR is furthermore an autonomous region within the country of Belgium. As such, it has a Parliament and a regional Government which were created as the result of the Belgian State Reform of 1988-89. The Parliament of
![]()
Table 2. Innovation and science competencies of the Brussels region.
Source: https://be.brussels/about-the-region/the-regional-competences, last consulted July 2019.
the Brussels-Capital Region exercises its legislative power by means of ordinances. The executive power lies with the Government of the BCR. The Ministry of the Brussels-Capital Region is the most important instrument of the Brussels government for the execution of its policy. The science and innovation policies are part of the regional government’s control in so far as it is applied to its regional powers. This means that scientific policy can be made when it relates to one of the topics listed above (CIRB, 2020).
Given the relatively recent devolution of power, this makes the BCR the ideal candidate to study the origins and evolution of technology transfer systems. This is due to the fact that the system can be seen to evolve from this point of devolution forwards and the fact that eyewitnesses who were present at the creation of this system are still available for interviews due to the recent history. As such it is easier to recount the creation and evolution of the system and with more detail and accuracy. In addition, much of the written material is also still available, allowing more depth to be brought to the analysis. Finally, given the regional scope of the region, the smaller size of the region, this also means that the number of stakeholders involved is limited and that the complexity of the interactions is restricted. The latter is helpful in reconstructing the evolution as it reduces complexity both in terms of the number of actors involved as well as the number of interactions that were had.
What is furthermore noteworthy about Brussels is that it serves as the center of administration for Belgium and Europe. The region contributes to one fifth of Belgium’s GDP, and its 550,000 jobs account for 17.7% of Belgium’s employment. Its GDP per capita is nearly double that of Belgium as a whole, and it had the highest GDP per capita of any NUTS 1 region in the EU, at ~$80,000 in 2016. With approximately 50,000 business and the presence of most Belgian centers of government and administration, the region contains a vested business community, as well as a policy community (BISA—Brussels Institute of Statistics & Analysis, 2015).
8. Results
8.1. Establishing Technology Transfer in the BCR
A 2017 report by the Université Libre de Bruxelles (ULB, 2017) indicates that the Brussels region counts over 50 universities, university colleges and other institutes for higher education. The key anchor tenants for the Brussels region are the Vrije Universiteit Brussel and the Université Libre de Bruxelles. This is due to their size, combined they account for approximately 40% - 60% of the student population in Brussels and their total budgets, combined exceed ?00 m. per year. In terms of the technology that these anchor tenants transfer, Figure 1 & Figure 2 provide an illustration of the historical evolution of the technology transfer output as performed by the two universities under scrutiny, expressed as the number of patents published yearly and cumulatively over the total period for which patent data is available.
![]()
Source: Author, based on information gathered via https://worldwide.espacenet.com. Last searched November 2018.
Figure 1. Evolution of the registered patents for the VUB (by publication date).
![]()
Source: Author, based on information gathered via https://worldwide.espacenet.com. Last searched November 2018.
Figure 2. Evolution of the registered patents for the ULB (by publication date).
As such the latter indicates that the region can make use of and rely on a well-established knowledge community that actively engages in patenting and can therefore safely be assumed to engage in the protection and exploitation of its intellectual property. What is more, as expressed in Figure 3 below, spin-off companies created by the VUB and ULB also generate value added. The yearly total of the generated added value by all spin-offs reached a total of ?63,469,214.00 for the year 2016. The latter being substantially higher than the year before where the total value added generated by ULB and VUB spin-offs amounted to a total of ?04,020,800.00 and a substantial increase since 2008 where the total value-added generated yearly represented ?9,731,126.00, in turn
![]()
Source: Author, based on facts and figures retrieved via https://statbel.fgov.be/nl. Last searched November 2018.
Figure 3. Historical evolution of the total added value generated by ULB & VUB spin-offs.
representing a compound annual growth rate of close to 30%. The total value added that was generated by university spin-offs over the whole period amounts to ?14,635,610.00.
However, when we dig deeper in the data on the total generated added value by both universities’ spin-offs, we notice one large outlier (see Figure 4). This is the Ablynx spin-off which solely amounts for over ?50 m. in total cumulative added value generated over the whole period or almost 65% of the total added value generated over the whole period for all spin-offs under scrutiny.
Figure 5 below addresses the total cumulative added value per spin off from a geographical distribution perspective. It shows the VUB spin-offs (Green) and the ULB spin-offs (Orange), their location and size (representation of the total cumulative added value) as located on a map of Belgium. The map shows that most of the added value that has been generated by spin-offs of the VUB and ULB has ultimately benefited regions outside of Brussels. Only 9.7% of the cumulative total added value is generated within the Brussels region. Once again, the outlier Ablynx jumps out at the reader, as it represents a cumulative total added value that is much higher than those of the peers in this cohort.
Nonetheless, these figures indicate that there is noticeable growth both in terms of number of spin-offs being founded and the value added that they represent, as well as a steadily rising increase in the number of patents being made by the two main anchor tenants. The later allows us to assume that technology transfer is taking place in the region, and that this is being done at scale. It is however striking that, given Brussels’ regional autonomy and focus on support for innovation and R & D, that the added value that is generated by its spin-offs escapes the region wherein it was created. What is furthermore characteristic of the Brussels case is that even within its regional autonomy, policy makers are restricted in the extent to which they can support scientific research,
![]()
Source: Author, based on facts and figures retrieved via https://statbel.fgov.be/nl. Last searched November 2018.
Figure 4. Comparison of total generated added value per spin-off/university.
as this support has to relate to the exhaustive list of competencies that is mentioned in Table 2. Given these conditions and the current state of technology transfer in this region, what remains unclear is how this transfer of technology came about and what its key characteristics are. The latter will be addressed in the following paragraphs.
8.2. Historical Organization and Function of the Technology Transfer System in the BCR
The organization of technology transfer in the Brussels region, as a part of Belgium, is guided by policies that reside at different policy levels. Historically, Belgium was a federal constitutional monarchy with a parliamentary system of governance. Today, still a constitutional monarchy, Belgium is divided into three highly autonomous regions and three communities. The geographically bound regions are the Dutch-speaking region of Flanders in the north, the mainly French-speaking Wallonia region in the south and the bilingual (French and Dutch) Brussels-Capital Region.
![]()
Source: Author, based on facts and figures retrieved via https://statbel.fgov.be/nl. Last searched November 2018.
Figure 5. Geographical distribution of the total added value generated by ULB & VUB spin off companies.
Belgium’s linguistic diversity and related political conflicts are reflected in its political history and complex system of governance (made up of six different governments) but also in its science and innovation policy. After the Belgian independence of 1830, it took until 1920 before the Belgian University Foundation was created with the mission to support higher education and scientific research. By 1928 The National Fund for Scientific Research (NFSR) was created, which mainly focused on fundamental research at that time and which in its conception already established a clear link between industry actors and academia. The latter is reflected in the fact that its board was composed of university representatives, bankers and industrialists. By the 1960’s, the National Council on Science Policy was created and by 1965 the initial reorganization of the organizations promoting scientific research within Belgium started as a result of the linguistic diversity that was mentioned above. Between 1970 and 1993 the country evolved into a federal structure. This was done through five state reforms (in 1970, 1980, 1988-89, 1993 and 2001).
As a result of these reforms, the power to make decisions no longer belonged exclusively to the federal government and the federal parliament. In terms of innovation and science policy, what remained at the national level is provided by the Belgian Federal Science Policy Office (BELSPO), which to this day prepares various research and science policy activities resorting under the competence of the Belgian federal government, including the supervision for ten Federal scientific institutions. The National Fund for Scientific Research however, as a result of the 1970 constitutional reform, started to reflect the linguistic duality of Belgium as its scientific committees, board members and subsidies started to become driven by and attributed in a split that reflected the divide between the language communities starting from 1969 and onwards. As a result of the 1988 state reform, the NFSR no longer fell under the authority of the national Minister of Science Policy, de facto generating a situation where the regional communities each managed their own budget for research and higher education. From this moment onwards, separate chambers were created within the NFSR as to reflect the dual lingual modality of the country. By 1992, the fund is split into two autonomous institutions, leaving the BELSPO entity (addressed higher) and an umbrella board of national directors at the national level, yet creating a separated board and scientific body structure for the communities, through which they are able to enact and define their own science policy. During this period, innovation and collaboration remained covered under Belgian Company Law, regulating for instance the characteristics of drafting and signing commercial contracts on distribution, licensing, franchising, and partnering.
At the regional level, as a consequence of the 1970 and 1988 constitutional reforms, regional management of research and science needed to start taking shape. It is in this instance that the Brussels Capital region starts developing its own science and educational policies, institutions and administration. The region was able to inherit some of the institutions that had already been active in the past, e.g. The Regional Development Company for the Brussels-Capital district (GOMB) had been created in 1974 and from the 1980’s onwards was transformed to accommodate the devolution of competencies to the regional level. Others like the Regional Investment Company (GIMB), created in 1984, worked/ works as a government holding company with mixed capital that aims to provide incentives for the creation and development of companies (in particular SMEs) in the Brussels-Capital Region. In establishing these regional competencies, and in order to support innovation at the business side, in 1991 The Technopole ASBL/VZW was created in order to support Brussels based companies to innovate. By the end of the 1990s, the Brussels-Capital Region had equipped itself with institutions, with a strategic and legal framework, and instruments for pursuing a STI policy at the level of the research and economic potential of the Region. Exemplary of this effort to establish the competencies at the heart of Brussels are the 2001 legislation on the regional development plan for Brussels2 and the establishment of the Science Policy Council of the Brussels-Capital Region in the year 2000 (this is the principal advisory instrument of the regional government in the development of its STI policy); the proclamation in 2001 of the decision to encourage and finance scientific research and technological innovation in the region, which sets the regulatory framework for regional aid in this regard; and the establishment in 2003 of the IWOIB-IRSIB (Institute for the Encouragement of Scientific Research and Innovation in Brussels, now ‘Innoviris.Brussels’), which provides the main structure for the policy implementation of the Region. By 2003, the support for Brussels based companies is consolidated in the merger of Technopole and SDRB InfoPoint/ECOBRU which then becomes ABE-BAD-BEA, the Brussels Enterprise Agency, which later became “Impulse.Brussels” and which is now called “Hub.Brussels”.
From 2006 onwards, policy makers in Brussels started presenting their visions for the future of Brussels in terms of innovation and science. The first regional innovation plans were included in the overall regional policy declarations with a first approved and signed policy taking effect in 2006 (Regional Innovation Plan (GIP) 2006). After a revision in 2012, the updated regional innovation plan foresaw several commitments and objectives for the 2014-20193 period. As such the Brussels regional government committed itself to an expenditure for research and innovation up to a level of 3% of the Gross Domestic Product. Key objectives in this updated innovation plan include the use of smart specialization to drive the development of the economy and employment, the fostering of a favorable environment for innovative companies, the increased attractiveness of Brussels as the European hub for knowledge, the increased participation of Brussels based organization in European projects and finally, the strengthening of the governance of innovation.
From what the respondents have added during our interviews we have gathered that the organization of technology transfer in Brussels runs parallel to the state reform and the devolution of the science and education related responsibilities that became part of the responsibilities of the region. When asked about the past organization of technology transfer in Brussels, the most experienced respondents were able to give feedback and noted that in the era before the 1980s, technology transfer was not organized in Brussels. ‘In the large research groups there were nearly no contacts with the industry in the beginning. Contact with the industry was not done, patents were not sought after or at least academics did not share that information. There was no framework for IP, so the inventors tried to get a patent without the knowledge of the university. Getting a patent was not easy, so the easiest solution was making a contract with the industry… The mentality of the industry was that the university was subsidized, so the industry did not want to pay for the work that universities performed for them. There however became an issue with the valorization, as the industry claimed that the IP had to move to the industry’ (Respondent 9, 13/02/2019).
From the 1980s onwards an awareness grew at the university side. Increased cooperation with the industry was needed, yet this development also endured a lot of resistance from within the university. ‘I became dean in 1985, I wanted an employee at the secretary for the contacts with the industry. At this stage the contacts were at an embryonic stage. The support left and the university reorganized hence the emphasis on industry contracts disappeared’ (Respondent 9, 13/02/2019). Many in the academic staff did not see technology transfer or industry-university collaboration as being an integral part of the mission of the university and as such preferred not to take part. ‘In the beginning it was clearly not the core business of the university. So, the industry was the driver. Central government of the university was not in favor of these kind of activities. But what we now see is that it has become one of the three objectives of the university researcher. Where it was demand driven in the beginning it now is opportunity driven and the push comes more from the university’ (Respondent 16, 11/03/2019). Very few academics did at this stage engage in industry-university collaboration, and as one of the respondents, states: ‘this led to a very unhealthy and unbalanced situation where there was a lack of awareness as to the value of what was being transferred. And also, there was a lack of understanding of the legal liabilities. So, the first step was to establish good contracts’ (Respondent 16, 11/03/2019).
The key instigators behind this embryonic type of university-industry cooperation therefore were the large industrial corporations that were already active in the region. In an effort to gain access to the knowledge that was being generated within universities, these large corporations started to set up direct collaborations with academics. This created a situation where the awareness within the university grew that other academics could also benefit from this type of cooperation. There was however a need for incentives to perform similar activities, whilst simultaneously making sure that the objectives and expectations related to this cooperation, became aligned and offered a fair distribution of the value generated through the collaboration.
As the awareness for the possibility of these types of collaborations grew and as the responsibilities for the support of innovative actions started to devolve to the level of the Brussels Capital region, during the 1990s the foundations were arranged for the centralized organization of technology transfer by government. ‘We were already busy with the TT activities before it was picked up in Brussels. In Brussels there was an emphasis on the guidance of companies towards TT but the role of universities in this system was not clearly recognized. The start of considering universities in this process started around 1995, when the Brussels Government under Minister Grijp started with the financing of TTOs in Brussels. At that point 1 person at university was financed within university. This was the interface, this fit within the broader Technopole initiative’ (Respondent 16, 11/03/2019). The latter, according to some of the respondents, is also related to developments that were ongoing in the regions that surrounded Brussels. Initiatives such as the Third Industrial Revolution (DIRV) (1982) in Flanders, the creation of strategic research institutes such as IMEC (1984), VITO (1991), VIB (1996) and the installment of the Flemish agency for Science and Innovation (IWT) during the 1990s are illustrative of the speed and intensity with which Flanders was aiming to explore and exploit its new found responsibilities. ‘It was DIRV (Third Industrial Revolution Flanders) under Gaston Geens and initiatives such as Flanders Technology with André Leysen or the instigation of VIB, IMEC, and others that technology transfer came on the agenda of the university (VUB) and it was at that moment the university jumped on that boat’ (Respondent 10, 19/02/2019).
For Brussels, the era at the start of the 21st century meant consolidation, professionalization and further organization of the administrations that were tasked with the structuring, monitoring, funding & controlling of technology transfer between universities and industry in Brussels. Exemplary for this period are the creation of regional legislation on the organization and financing of scientific innovation. The former is addressed in the 2003 Ordonnance establishing the Institute for the Promotion of Scientific Research and the Innovation of Brussels (IWOIB-IRSIB) and the 2009 Ordinance to promote research, development and innovation. As a result of these ordonnances, policy support and grants for innovation are provided to universities and higher education institutions as laid out in the Regional Innovation Plan (GIP).
Given the gradual centralization and organization of the support structure for industry-university collaboration during the 1990’s and at the start of the 21st century, respondents were asked whether or not the technology transfer became more efficient over the years in Brussels.
Most respondents stated that they don’t know or can’t tell if the transfer of technology in Brussels became more efficient. Knowledge community representatives tended to be more positive; the business side representatives were mainly not concerned with the issue or had insufficient information and the policy community representatives felt that a lot can still be done to improve the efficiency of the technology transfer. The policy community felt that one of the main reasons as to why TT did not become more efficient was the lack of hard KPIs to measure whether or not projects are launched easier, maintained easier and lead to outcomes in a more efficient manner. Part of this also has to do with the fact that, from the policy community perspective, no progress monitoring was done in the past. As such, it is hard to benchmark the performance of current projects to similar past projects. ‘We have more collaborative projects today than in the past and we spend more money on these projects, but at this moment I don’t have any idea of whether the transfer is very efficient. The projects run for 3 or 4 years and I don’t know if there is a precise ex-post evaluation of these projects. As such it is simply too early for the moment to make the analysis’ (Respondent 18, 28/03/2019).
Another reason why the transfer of technology in the region is not perceived to be more efficient now than in the past, is the fact that over the year’s normalization occurred, meaning that initial targets for technology transfer and the stimulation of innovation were reached, raising the bar and thereby allowing participants to expect more from current interactions. ‘(Did TT become more efficient over the years?) No, I don’t think so, because we expect more from this dynamic. It has become more complex, what was initially intended is now done more efficiently, but we expect more, so this is not yet efficient’ (Respondent 6, 5/02/2019).
Overall respondents agreed that it has become easier to set up collaborations between universities and industry actors, but this is not the same as a guaranteed successful outcome. Especially knowledge community respondents pointed towards the difficulties they endure in measuring the uptake of technology by the companies they work with. One clear assumption that was stated was that it is safe to assume that companies would not invest in collaborative projects with universities if there was no upside to the time and resources invested. Next to this, these interactions have also become more complex due to specialization and therefore ask more from the participants involved. As such this lowers the operational efficiency. ‘TT has become more structured. The question is whether the uptake of technology is more efficient, the number of interactions and collaborations has increased a lot, but the uptake cannot be measured, and we can therefore only assume it is. Companies would not invest in recurrent collaborations if there was no value.’ (Respondent 16, 11/03/2019).
Next to the efficiency, respondents were also asked to address the scale at which technology transfer interactions were had and whether or not the organization of technology transfer in Brussels was able to facilitate these interactions. Overall, the respondents felt that the scale had increased over the years, yet that the collaborations with the industry are still often with the larger companies that have a clear focus on R&D. ‘The scale has grown due to Innoviris’ (Respondent 3, 04/02/2019); ‘The number of collaborations is increasing, and we feel that our coordination efforts are the driving factor behind the increased collaborations. The key is that we can focus more on communication and match making’ (Respondent 14, 7/03/2019). The respondents furthermore indicate that the growth in the number of collaborative R&D projects is due to more funding, more communication, better coordination, more incentives and a change in the mindset and culture within the knowledge community which allowed more collaborative projects to arise. ‘The number of interactions is growing. I assume that the coordination and organization have an impact on this growth, but we do not know for sure. There is also a change of mentality at the researcher side, they are becoming more open to collaborating with the industry, this has as a consequence off course that the number of collaborations is increasing’ (Respondent 11, 20/02/2019).
8.3. Power and Stakeholder Interactions
What stands out from the interviews is that the interactions at all levels between companies and universities, but also between universities and the policy makers have transformed over the years. What used to be very informal contacts, within a small set of actors, has now changed into a more formal, more complex setting. Where the initial interactions between academics and companies were found to be bilateral, the respondents state that through the years they have perceived a move towards more multi-actor relations, which adds complexity to the interactions. ‘In terms of (university) support you need to professionalize so that people have the correct information when creating a company or new venture. You have to take into account a lot more information and more aspects. This is also related to the learning curve, as we now have more experience and hence can foresee more trouble and on the other hand the partners now have more requirements and we now have to match that. In the past the collaboration was more based on inter-personal contacts, now it is becoming more institutionalized and hence the collaboration also sits at the institution level. This makes TT even more complex and also with open innovation this adds more difficulty in relation to the valorization path’ (Respondent 11, 20/02/2019).
In terms of power, the respondents indicate that power was located at the side of the larger companies in the beginning. Unstructured approaches both at universities and at the policy side allowed larger companies to get disproportional returns as some of the value of what was generated at university was provided to the companies with too much ease. Hence, within universities a need arose to structure the interactions with industry as to guarantee value extraction for the universities.
As time passed, the power shifted to the policy community. This community gained its power through the devolution of the responsibilities to the regional level and as such was able to set policy priorities for the region through the funding that was provided to the universities for their collaborative efforts. Funding and scale therefore allowed the funding agency and the associated governmental services to claim a larger stake in these interactions. This shift did however not come about without power clashes and conflict at the individual level. The power shift was furthermore influenced by the appointment of new general managers for the governmental agencies, new ministers for the involved ministerial departments and by the rebranding of governmental services multiple times. ‘The organization of TT in Brussels (BXL) is still struggling to clearly establish an organization and this causes a mix of informal and formal methods. The influence of the current director at Innoviris was also important in this case… In the beginning TT expanded fast, now it is slowing down a bit. The driving force behind it was ‘impulse.brussels’ in the beginning, driven by certain individuals. There were some clashes with other participants in the network, due to the concentration of power. Atrium was also an important player which has now been rebranded to 1819’ (Respondent 3, 04/02/2019).
What is worth noting is that when respondents were questioned about the origin or the ontogeny of technology transfer in Brussels, the respondents that represent the knowledge community felt that overall TT had an emergent character and that it only became policy driven in a later stage. The policy community respondents on the other hand felt that TT was deliberate & university driven as universities were clearly engaged with the intent to attract funding and promote the research and political agendas of the actors that are part of the knowledge community. The latter shows a clear difference in the perception of the ontogenesis of the system.
8.4. Current Organization, Functioning & Stakeholders
Looking at the current state of technology transfer in Brussels, technology transfer is mainly driven by a set of objectives that is shared across the different communities involved in technology transfer. The interactions within the system are mainly focused on increasing technological development, allowing funding for R&D projects, fostering innovation and increasing deal flow. Next to these, some of the respondents also indicate that there is an increased focus on generating societal impact and on stimulating entrepreneurship both for citizens in general, as well as for students and academics of the Brussels based universities. Overall, these objectives are rather generic and fit with the more commonly retrieved motivations behind the implementation of industrial policies and regional innovation systems. The same is true for the stakeholders that are involved in the technology transfer interactions. Table 3 lists the stakeholders that were identified by the respondents. Here again the usual suspects are retrieved, although what stands out is that civil society is not explicitly mentioned, even if it is implicitly captured in many of the calls that are put forwards by the Brussels funding agency for science and research.
What is particular for technology transfer in Brussels is that the system of interactions is driven bottom-up and top-down simultaneously and that the mode of operations is perceived differently depending on the background of the respondents. For the knowledge community, there is a sense of self-similarity when addressing the opinions of people that are working for the university technology transfer offices (TTOs). These respondents reported mimetic behavior in terms of processes and operations and as such felt a drive towards increased similarity between TTO operations, as instructed by the regional funding agency. Also striking is that the business community representatives tended to have no opinion on this matter or felt insufficiently informed as to address whether or not the mode of operations had become more self-similar across TT service-providers. The policy community representatives finally, focused much more on the
![]()
Table 3. Stakeholders identified in the Brussels’ technology transfer system.
Source: Author
differences that still exist in the ways in which different TTOs address similar themes and or problems. They furthermore pointed towards differences in the way the different TTOs are organized and the effect this has on their performance. As such, these latter respondents were far more focused on differences than on similarities. ‘The VUB is clearly different due to the separation of the R&D department and the TTO. That is a choice. In practice the other organizations function more or less the same way’ (Respondent 5, 05/02/2019); ‘There are differences in the way the universities work. The aggressiveness to go to the industrial tissues is different among the TTOs. We have the impression that the universities have different strategies to approach the industry… 80% of the jobs they do is however the same. The 20% that is different relates mostly to aggressivity of approaching the industry’ (Respondent 14, 7/03/2019).
Coordination within the technology transfer system takes place through several mechanisms. Table 4 below offers an overview of the coordination mechanisms that were identified and used by the respondents.
The calls put forward by the funding agency, the top-down mechanisms, are often temporary thematic calls. According to some of the technology transfer officers, this approach may lead to lost opportunities. As the calls are thematic and only temporarily available, this generates difficulties in matching the competencies needed to answer the calls. What is more, some respondents question whether innovation and the promotion of science and research are always the main drivers behind the thematic of the calls. ‘The Innoviris calls are the main coordination mechanism, ‘hub.brussels’ has different clusters and we cooperate much with them…’ (Respondent 7, 06/02/2019); ‘The calls for projects by Innoviris work quite well, the negative aspect is that the calls are limited in time, run
![]()
Table 4. Overview of the technology transfer coordination mechanism.
Source: Author.
for a period, and if one does not answer within this period, the opportunity is lost’ (Respondent 3, 04/02/2019).
The thematical restrictiveness that is felt by some of the respondents is also due to the fact that the restricted regional autonomy of Brussels limits the scope of R&D themes that can be addressed. Thematically, the calls put forward by the regional government and through its funding agency can only relate to elements that fall under the responsibility of the region. Due to these institutional boundaries, academics and companies are not allowed to mix and or integrate multiple regional sources of funding, which in practice would be possible and could lead to greater returns. ‘There should be more collaboration between the regions, as a Flemish university we are confronted with a lack of capacity to finance larger projects or to integrate money from other regions. We do not have research parks here this is also a limitation... VC money for startups is there but not for scaling’ (Respondent 16, 11/03/2019).
What is furthermore particular for Brussels is the fact that technology transfer in Brussels occurs in a relatively small region. This creates a geographically limited situation which generates spatial or geographical proximity for the actors involved. The respondents clearly indicate that proximity is a key facilitator for collaboration in the region, since proximity allows for easy access to the different communities and facilitates dialogue amongst the system participants. By being close to each other there is also more room for informal contacts which also increases the level of flexibility and adaptiveness of the system as whole. ‘What are the key strongpoints for TT in Brussels? The presence of so many research organizations in such a small territory. Being a region that attracts European institutions is also a competitive advantage and sufficient budget to finance all good projects that we receive’ (Respondent 1, 10/10/2018).
When asked about limiting factors for the technology transfer interactions, whilst the proximity and the size of the region are seen as key facilitators to the transfer of technology in Brussels, respondents also evaluate this as one of the key bottlenecks in this particular system. ‘The Brussels funding agency requires the valorization to happen in Brussels. But this restrains the interest of the company that has to drive the priorities. There is a strange geographical limitation that delimits the playing field and the companies still have to go their own way (both spin off and existing ones)’ (Respondent 10, 19/02/2019). Given the relatively small size of the region and the composition of the region both demographically as well as in terms of industrial composition, this means that not all types of industrial actors are present in the region. This limits the scope of technological applications that can or need to be developed within the region even further. ‘One of the difficulties is that the region is relatively small, you have little hard industry, there is no producing industry. So, if you want to setup this transfer or setup projects only with Brussels companies, then this is not ideal, because your needed type of actors is not fully active in Brussels.’ (Respondent 12, 20/02/2019). Furthermore, as was stated before, the region and its institutional context also generate a technology transfer system in which scientific valorization is mainly if not solely Brussels focused. ‘The political framework should facilitate more flexibility for the design of support programs’ (Respondent 15, 7/03/2019). Not only does this hamper the development of inter-regional cooperation, it also leads to the fact that only some types of research are funded, which conflicts with the universal nature of knowledge development within a university setting.
What is key towards explaining the bottlenecks in the system is the fact that too many actors rely on incomplete information and yet have to or choose to take part in the technology transfer system. Respondents identified incomplete information as the key explanation. They state that there is insufficient education, professionalization and training in certain research fields and professions which creates difficulties for new entrants and or experienced yet not fully informed participants in TT. One example that was given was that in some of the TTOs, there is no professional in-house training for TT officers. ‘Too much need for contractual and legal aspects, not all TTOs provide enough resources to cater for that need. The number of legal advisors is too limited at our end’ (Respondent 4, 5/02/2019); ‘The other bottlenecks are not related to Brussels, but to Belgium. Hiring new people is too expensive’ (Respondent 17, 12/03/2019). Another element that leads to frustration due to incomplete information are the IP discussions between partners. Moreover, illustrative of a situation of incomplete information is the fact that the business community, due to its strategic objectives and the competitive nature of the market, is unable to provide sufficient transparency in terms of its industrial needs. As such, it is not always clear what the regional industry expects from the knowledge community, nor is it clear for the business community what it is that they can expect from the knowledge community. The respondents also highlight that there is a mismatch in incentives for the knowledge and business communities to communicate their true mutual value.
8.5. Directions for the Future Organization of Technology Transfer in Brussels
In relation to the future organization of technology transfer in Brussels, respondents indicated that the system can scale further in the future, but it is and will remain dependent on the human resources that make up the backbone of the system. Scale growth will therefore have to come in parallel to attracting the necessary human resources in the future. What is more, public administrations will not scale forever, so there are natural limits to the scale. Brussels as a region is limited in scale by its size and location and as such should not be governed in an administrative system that outgrows its geographical scope. Finally, in relation to the organization of technology transfer, respondents warn that scale growth is not automatically translated into more ROI for society and more spin-offs at university, hence there is also a cost-benefit trade-off at play which needs to be taken into account. ‘Knowledge transfer will become a challenge, mixing economic valorization with societal valorization this will also be a challenge. Which are the right KPIS and how to measure this. Very vague. Multilateral elements will also become more relevant; hence it makes the management of these deals more challenging’ (Respondent 11, 20/02/2019).
In terms of the coordination improvements, the respondents primarily focused on what the other communities could do to improve the coordination. Policy and business community representatives, for instance, stated that the knowledge community could do more to exploit the potential that resides at university, whilst also focusing on a common governance structure and management method for the different TTOs. In essence, they were pointing towards standardization efforts that would reduce transaction costs for the other communities when pursuing or entering into collaborations with universities. The knowledge community was furthermore asked to lower the time needed to find connectors at university that are able to navigate the knowledge community in order to expediate the creation of fruitful interactions with the other communities. ‘An initial step forwards would be the simplification of the ecosystem in order to reduce the fragmentation in the system and lower the time needed to find the correct connector or entry point. If we cannot reduce the number of dots, then we need to make the links between the dots more visible and stronger.’ (Respondent 1, 10/10/2018).
The policy community on the other hand, was urged by the two other communities to reduce administrative burdens when applying for research or project funding and to increase the transparency in the motivations for thematical calls for projects (mainly requested by the knowledge community). The business community was urged to improve transparency in terms of their critical business needs in order to inform the other actors involved in technology transfer in the region. Overall, for the coordination aspect it became clear that the respondents want to ensure less fragmentation in the system and want to improve the ability to detect business potential and business needs in the region.
In terms of the outputs to be generated by the system in the future, the respondents focused on the fact that the incubators present in the region needed to better integrated and that the convenience with which knowledge is transferred to society would need to increase. Another clearly articulated suggestion for the future was to provide more follow-on money for starting companies and scaling companies as this was perceived as to be lacking at the moment. Overall, the knowledge community was asked to focus more on the service to society and to provide more incentives to academics and students to pursue entrepreneurial activities. The policy community was asked to provide more autonomy to TTOs in terms of the financing of small projects and in order to provide more support to SMEs. Next to this, several respondents stated that funding in the future would need to be much more oriented towards failure, meaning that the policy community is too risk-averse in the projects and the types of research it funds today (respondents 8, 9, 10, 15). In so doing, respondents stated that more support for high-risk high-reward projects should be provided, potentially generating greater returns on investment for society. The business community finally was urged to be more open about their business needs, as to allow the other stakeholders to better serve their needs. ‘We need a better means of detection of the potential in the region in terms of companies that we have no access to today.’ (Respondent 11, 20/02/2019).
In terms of future challenges, in general respondents stated that data sharing, trust, global competition and the speed of change would remain challenging in the future. Additionally, training and attracting skilled human resources will also remain challenging. The knowledge community representatives saw challenges in the fact that not enough big issues were being tackled. Due to the funding structure and the institutional limitations, the knowledge community representatives felt as if there is a strong focus on small issues and small matters. They furthermore stated that the position of the academic professor and the career advancement of academics would become quite puzzling in the future as both the role and the value of fundamental research is being questioned in society, whilst universities are being asked to also focus on informing society through direct knowledge transfer. ‘Trust remains a problem. The lack of companies in certain sectors, but that is specifically for Brussels. This is more general. Another challenge is the cultural shift that has to be made within the universities, to also build career advancement on the basis of collaboration with companies.’ (Respondent 19, 28/03/2019). Additional difficulties for the future of technology transfer in Brussels reside in obtaining standardized IP solutions, managing multilateral relationships and in guaranteeing the independence of the universities. For the other communities, the key issues for the future revolve around conflicts and competition between regions and improving the matching of minds and needs.
In terms of solutions, the respondents added that in the future there will be a need for more cooperation-based business models, more openness and transparency instead of IP protection, the use of more adaptive and flexible TTO models that both focus on new KPIs as well as incorporate the social sciences in order to overcome barriers between technical and non-technical people. ‘We need more integration and new business models based on cooperation to tackle the complexity of our sector and or to tackle the scarcity of labor in the sector’ (Respondent 12, 20/02/2019); ‘If TT stays at university, this puts the academic logic first. Private initiatives could provide more incentives to people; hence we could think about moving to a hybrid structure’ (Respondent 9. 13/02/2019).
9. Implications
Table 5 summarizes our findings and compares different time periodes for the Brussels Technology Transfer system. Specifically, for this case, the research shows that a considerable part of the added value generated by technology transfer interactions (in casu spin-offs) in the region, spills-over into the surrounding regions. What is more, the thematically and geographically limited approach towards the launching of calls for funding, aims to focus the research
![]()
Table 5. Overview of the evolution of the Brussels’ technology transfer system
Source: Author.
efforts on Brussels, yet it is clear that the outcomes of research projects in many instances will have a domain of application that far exceeds the Brussels region. The latter generates a situation where policy makers and or knowledge community representatives need to evaluate the value and necessity for upholding regional and thematical limitations in terms of R&D subsidies given that the finality of the technology transfer cannot be contained within the region.
Additionally, the lack of clear benchmarks and KPIs, present both in the past and in the current situation, create a situation where the performance of the system and its actors is hard to establish. Installing a rigid monitoring system and or introducing competitive pressures in terms of innovation and research funding may be extreme over compensations to this situation, yet the complete lack of verifiable efficiency and effectiveness improvements seems striking to say the least.
10. Conclusion
This paper attempts to fill a gap in the literature that addresses the longitudinal study of technology transfer systems. The empirical implications allow us to broaden the research field through comparative research methods. The results of the analysis show that the actors involved, the formalism in the interactions within the system, the organization of these interaction, and the coordination of these interactions, changing over time in relation to changes that take place within one or more of the communities involved in technology transfer in this regional setting, and as the result of competitive forces that come from outside of the region.
The conceptual implications related to the usage of an ontogenesis frame of reference are multiple. Firstly, the creation of this particular technology transfer system is largely directly assignable to both the devolution of competencies to the regional level and due to regional competitive pressures arising as a result of this devolution of power. What is more, the time needed for these competencies to become institutionalized also offers a time frame within which the system’s stakeholders adapt and respond to the changes taking place in the policy community. Secondly, power within the system shifts from one triple helix community to the other over time. The same is true for the functions that are fulfilled by the technology transfer system, as they transform in relation to the increasing level of maturity, professionalization and institutionalization in and of the system. From this conceptualization, an image arises that marks a technology transfer system as a complex, multi-layered system anchored in a geographical delimited space, existing of a number of constructs, located within the triple helix, generating interaction between its actors through which relationships are purposefully oriented towards several objectives, of which the primary are: the strengthening of companies, the reduction of uncertainty, the generation of innovation and the fostering of learning (Scaringella & Radziwon, 2018). Conceptualized as such, the latter provides ample broadness and richness in order to encompass the notion of a technology transfer ecosystem and study of its ontogeny, which can be pursued in future research. What is furthermore worth considering for future research is whether the notion of sustainability and durability, i.e. the independence of a technology transfer system form external sources of funding, is also part of this definition as suggested by some of our respondents. Empirically, the study finds that most of the added value that is generated by university spin-offs in the region, flows out of the region rather quickly, within several years after the creation of the spin-offs. Furthermore, for this case, the study shows that insufficient benchmarking in the past and in the present creates a situation where the performance of the technology transfer system is hard to measure or to verify, which leads to further implications related to the justification of public expenditure towards these activities and the management thereof.
11. Discussion & Future Research
The technology transfer ecosystem
The boundary space within the triple helix, as defined by Champenois & Etzkowitz (2018) and conceptually refined by Scaringella & Radziwon (2018) is made up of three layers. In this conceptual model, the regional element takes center stage and consists of an inner territorial layer, surrounded by a territorial ecosystem layer, which in itself is surrounded by an ecosystem layer (Scaringella & Radziwon, 2018). For the case of Brussels, our results indicate that the ecosystem layer, the most outer layer, which focuses on entrepreneurial or business activities that can or may exceed the territorial boundaries of the two inner layers is clearly functional and exhibits traits such as co-creation, coopetition and interdependence to some extent. As was stated by the respondents, companies are unlikely to invest in R&D projects with universities if there is no value to be gained from these interactions. Future research would have to assess if excessive rent-seeking behavior and or commercial dominance over regional resources, as described by Christopherson and Clark (2007) is the case in this region and in accordance with the theory put forwards by these authors (Christopherson & Clark, 2007). What is however clear from our findings is that the innovations themselves, the companies spun out of universities and the added value as well as associated employment do largely escape the region.
The second layer, the intermediary layer is characterized by a sense of belonging, the simultaneous existence of collaboration and competition and innovative outcomes. This second layer is clearly also present for the Brussels TT system, even if the actors were evaluated as mainly being focused on their own self-interest. Illustrative of the fact that they felt that there is a sense of community was the respondents’ response in relation to parasitism within the system and the notion that such behavior would be acted upon. In this secondary layer of the model stakeholders interact on the regional level, whilst at the same time engaging in knowledge dynamics that reduce uncertainty. Uncertainty clearly was an element that needs to be studied further as both the business community as well as the knowledge community indicated not knowing what they could expect from each other (Scaringella & Radziwon, 2018). As such the synergy and complementarity in this particular regional system, stands to be improved if the actors involved are able to increase the levels of mutual trust. Trust as an element of study was not explicitly included in our study and therefore needs to be integrated in future efforts that attempt to study the ontogenesis of technology transfer systems. It is furthermore our understanding that it is at this second layer of the ecosystem that the creation of hybrid organization occurs (Champenois & Etzkowitz, 2018). Examples of such hybrid organizations in our results were found in the TTOs found in both universities and in the emergence of regionally active funding and support agencies. The most inner layer finally, is what fits most closely with the notion of the knowledge community as it is created and established by universities and research centers and focused on developing locally anchored social capital (Scaringella & Radziwon, 2018). For our study, it is clear that these entities exist and interact in this particular system and that they work towards the agglomeration of knowledge spillovers, illustrative of which are the patents and spin-offs described above.
As such, this boundary space and conceptual definition of technology transfer as an ecosystem, based on the limited empirical evidence presented her, does seem to hold water. Future research should therefore address this issue from both the theoretical perspective and the empirical perspective.
On the importance of ontogeny
From what is retrieved through the analysis presented in this paper it is clear that, at least for this case, technology transfer systems undergo change and as such form ideal settings for the study of dynamic organizational capabilities and organizational metamorphosis. What is interesting in the composition of the set of respondents that participated in this study is that whilst on average these respondents had 11 years of experience with technology transfer in this setting, most of them, when asked about the ontogeny and or origin of the system, were unable to answer or had no idea how technology transfer had come about in Brussels. Performing longitudinal analysis about the origin or genesis of technology transfer therefore, at least from an instrumental point of view then seems redundant, given that system participants are apparently able to participate without information on the origins of the system and or do not seem to need this information to perform managerial or other tasks. Normatively however, given the importance of trust as a key facilitator for industry-university collaboration and the potential impact of knowledge sharing as a future challenge for technology transfer as reported by our respondents, our aim with this paper has been descriptive as our contribution is holistic in nature and aims to improve our understanding at the systems level. From this perspective the insights gained from the analysis of the system’s evolution do hold value as the findings can be compared with other cases and or may allow others to replicate and or evaluate the findings against future findings. What is more, as the past influences the present and the future, by looking back and by analyzing the past, our aim is to allow the system participants to better understand the cultural differences that exist within the system and in turn reduce their reliance on incomplete information, which they themselves listed as one of the key hurdles in facilitating technology transfer.
Appendix 1. Overview of the Questions Included in the Semi-Structured Questionnaire
NOTES
1More information on Innoviris can be retrieved through the following URL: https://innoviris.brussels.
2The legal text that is referred to can be found here: http://www.ejustice.just.fgov.be/cgi_loi/change_lg.pl?language=en&la=N&cn=2001092034&table_name=wet, last consultation July 2019.
3The regional innovation plan can be consulted via: https://innovadm.irisnet.be/nl/ooi-beleid/gewestelijk-innovatieplan/pri-2016-nl-revision-2me-lecture-web.pdf, Last consulted July 2019.