Unequal Distribution of Innovation Efforts for Neglected Tropical Diseases: The Role of Funding Evaluation Criteria ()
1. Introduction
Neglected tropical diseases (NTDs) constitute a group of diseases that primarily affect resource poor populations and have traditionally been neglected [1] . The severity of the problem was first recognized in 1990, when research showed that only 10% of health research funding was allocated to health issues in developing countries, despite the fact that these issues account for 90% of the global burden of disease [2] . The disparity became known as the 10/90 gap and led to international efforts, including the formation of the Health Forum and the formulation of a Millennium Development Goal dedicated to “combat HIV/AIDS, malaria and other diseases”. In practice, however, the majority of efforts were primarily focused on the big three diseases: HIV/AIDs, malaria and tuberculosis. In 2005, the term NTDs was introduced to refer to the “other diseases” that were still being neglected despite a clear unmet need [3] .
Efforts to decrease the burden of NTDs take a variety of forms such as public-private partnerships, drug donation programs [4] , R&D incentive programs [5] , and the London Declaration. Funding for research and development (R&D) on neglected tropical diseases (NTDs) has increased with 13%, in the last ten years [6] . This trend can be attributed to many strategies that were introduced to stimulate investments in NTDs, by lowering the risks and increasing financial benefits [5] . The growth in total amount of funding is also the result of increased public pressure on pharmaceutical companies leading to the development of access to medicines initiatives [7] .
As a result of these efforts, some diseases have seen a significant decrease in the burden of disease and others are heading towards elimination [8] . The incidence of other diseases, however, continues to grow as is most evident from a 610% increase in dengue cases and a 170% increase in leishmaniasis cases between 1990 and 2013 [9] . These numbers raise the question whether resource allocation, which is known to be unequally distributed amongst the individual NTDs [6] [10] [11] , should be improved. It could be argued that resource allocation should be prioritized towards diseases with the highest burden of disease, or diseases with the highest increase in the burden of disease. In both cases, policy decisions should be informed by insights in the current distribution of efforts compared to the burden of disease. Moreover, funding allocation mechanisms should take such prioritization into account.
These changes, however, cannot be made without sufficient insight in the problem at hand. While disparities in funding are described in literature [11] [12] [13] [14] and some studies compare specific efforts to the burden of disease [15] [16] [17] , to date there is no comprehensive overview of the distribution of innovation efforts between NTDs and the relative neglect of the individual NTDs. Furthermore, while existing literature includes studies on: correlation between funding and firm or disease characteristics [18] [19] [20] , funding organizations’ processes for research allocation [21] and priority setting [22] , and frameworks for improving resource allocation [23] [24] ; there is no in-depth analysis on the mechanisms through which current allocation processes lead to the observed disparities. Most funding allocation is dependent on the assessment of research proposals and therein applied evaluation criteria play a key and yet understudied role.
Therefore, the aim of this research is to investigate distribution of innovation efforts between NTDs and gain in-depth understanding of the mechanisms of evaluation and assessment of research proposals. To this end, this study includes a systematic literature review on innovation efforts for NTDs, followed by an in-depth analysis on the role of evaluation criteria based on semi-structured open-ended interviews with funding organizations.
2. Methods
2.1. Distribution of Innovation Efforts
A systematic literature search was conducted to collect data on health R&D and innovation efforts for NTDs. Due to the absence of a uniform definition of neglected tropical diseases (NTDs) [25] , this study does not adhere to a predefined list of NTDs. Instead, we include all NTDs mentioned in the articles selected for analysis. The search syntax used “neglected tropical diseases” in combination with the following well known indicators of efforts: funding, publications, clinical trials, publications, patents, and investigational products [26] . PubMed and Scopus were used as these databases have a high coverage of biomedical and health sciences [27] and grey literature reporting on unique data was added. The search was conducted on Jan 13, 2019 and limited to the period since Jan 1, 2000 given the increase in initiatives to promote health R&D in NTDs since this date (Appendix 1). As the corona-pandemic made huge changes to the funding landscape, the search was not updated after 2020, in order to show a clear picture of funding allocation in the absence of pandemic circumstances. However, this exclusion introduces a limitation, potentially skewing the findings toward a pre-pandemic understanding of funding allocation.
2.2. Data
For the selection of relevant articles, the PRISMA guidelines for systematic reviews were applied [28] . A total of 527 unique records were retrieved, of which 490 were excluded after screening the title and abstract on relevance. Of the remaining records, 2 full text articles were not accessible due to a pay wall, after which the eligibility of the remaining 35 full text articles was reviewed. Non research articles (e.g. news articles, editorials) were excluded as these do not contain original data. Articles with a limited scope (e.g. focus on a geographical area, type of drug, or specific funder) are also excluded. Assessment of the eligibility led to the exclusion of 23 articles, 4 other were relevant but based on overlapping data (Appendix 2). Relevant articles reported on at least one indicator of innovation effort for more than one NTD since 2000. A total of 8 articles were included in the analysis because these quantify the innovation efforts and present these per disease. One article was based on a database that included more up-to-date data so it was consulted instead [6] . Information regarding the background and focus of these articles is presented in Table 1.
2.3. Analysis
All data on innovation efforts from 2000 onwards retrieved from papers (Table 1), were inserted in a spreadsheet and harmonized. The data was combined with data on DALYs to calculate the R&D effort relative to the burden of disease in terms of DALYs, which were retrieved from the GBD Results tool that is available from the IHME [37] . We acknowledge the limitation of relying on DALYs as a measure of disease burden and linking it to funding priorities, in that it may simplify some of the complexity and intricacies of health economics. The database does not cover all diseases, either because the data is unavailable or because they are grouped under one heading or the disease is underreported. Diseases for which no DALYs are available from the GBD Results Tool are excluded from this study because estimates of the burden of disease largely depend on the methodology and complementing data from other sources would hamper comparison between diseases. Efforts per DALY were calculated over the specific period covered by the selected articles. To calculate relative efforts compared to other diseases the resulting numbers were plotted on a 0 - 100 scale.
![]()
Table 1. Background of selected articles.
2.4. Funding Evaluation Criteria
A funding organization’s evaluation criteria are the primary determinants through which that organization allocates funding. Some make the criteria they use publicly available, while others do not publish their evaluation criteria. Moreover, evaluation criteria can be interpreted and applied differently by different organizations and noticeable differences between the content of documents and data from interviews could point to an implementation gap. Therefore it is important to gain understanding of the meaning of these concepts and the interpretation and application of these criteria in practice [38] . To this end, the study used a qualitative approach.
2.5. Participant Selection
Interviewees were purposively selected. Grant managers and other representatives of private, public, philanthropic and intermediary organizations were identified via an internet search and selected via preliminary phone calls. Based on these phone calls and preliminary invitation e-mails, some respondents provided the name of a colleague that would be more suitable to answer our questions. Participants were invited via an e-mail, which included a brief introduction of the research, the purpose of the study and the interview design. When participants did not respond to the invitation e-mail, several attempts were made to contact participants by e-mail and telephone. When participants agreed to participate, they received the interview questions beforehand to allow them to prepare and enable the interview to go in-depth with follow-up questions.
2.6. Guide and Concepts
For the purpose of data collection, an open-ended interview guide was used as it provides direction while allowing for flexibility to adapt to the answers given by the interviewees. Important concepts that influence the funding decision were derived from literature and used as sensitizing concepts during the interviews. Analysis of literature revealed eight major categories of criteria that influence the likelihood that a research proposal will receive funding either in a positive or negative way. Characteristics of these categories provide insight in the likelihood that funding organizations will fund research proposals related to NTDs: value of the product to society, revenue generation model, resources of the applicant available for the research, quality and costs of the proposed research and competition [12] [39] - [47] . Such concepts are not conclusive nor precisely defined and serve as a starting point for the interviews [48] [49] . The applicability and meaning of these concepts to the respective interviewee was tested and defined [49] , while remaining open to concepts that may emerge from the interview [48] . This way, sensitizing concepts allowed for the identification of other concepts not previously described in literature.
2.7. Data and Analysis
Thirteen in-depth interviews were conducted with representatives of public (4), private (2), philanthropic (4) and intermediary (3) funding organizations. The interviews started with an open question about the evaluation criteria that are used by the funding organization. Follow-up questions were used to further discuss the definition of evaluation criteria and an understanding of how they were applied in practice by the organization. Criteria that are used in the agenda setting phase or to check the eligibility or approval phase of a research evaluation process were considered out of scope. Interviews were conducted by two researchers via Skype between 2015 and 2018.
Interviews were transcribed verbatim directly after the interviews were conducted and tape records were deleted. Researchers’ notes were transformed into an initial coding sheet directly after the interviews that were used for the initial, open coding directly after the interviews. The coding sheet was adapted and completed during the open coding process. After this initial analysis, one page summaries of the interviews were written and sent to participants for a member check. No changes in interpretation were required. Subsequent axial coding leads to the identification of the overarching evaluation criteria that are used by all different types of organizations. Next, selective coding of the evaluation criteria and the application thereof led to the identification of common themes that are specific for NTDs.
3. Results
3.1. Distribution of Innovation Efforts
The selected articles (Table 1) use a wide variety of definitions for NTDs, covering from 14 up to 45 diseases. The articles rely on definitions used by WHO, PLoS NTD journal, and G-FINDER, although the interpretation and categorization of diseases still differ per article. The definitions only slightly overlap, which results in a list of 66 (sub)diseases. Data on the burden of disease was available for 27 diseases, which are the NTDs that are included in the analysis (Table 2).
![]()
Table 2. The number of diseases considered an NTD varies widely per article. Only 9 out of 66 diseases were covered in all articles. Diseases that are considered an NTD in at least one of the articles and for which DALYs are included in the analyses.
3.2. Indicator Comparison
Figure 1 shows an overview of absolute data on innovation indicators. The diseases for which data in all five indicators is available, show that there is a weak link between the different indicators. Leishmaniasis and human African trypanosomiasis, for example, receive similar amounts of funding, but have very different outcomes in terms of publications, patents and clinical trials. The same goes for lymphatic filariasis, leprosy and onchocerciasis.
3.3. Absolute versus Relative Innovation Efforts
Table 3 shows the data on R&D efforts that were retrieved from the selected articles per R&D indicator. Data that is shown for compounds and clinical trials are integrated from two and three articles, respectively (Appendix 1). The results show a large difference between the absolute innovation efforts and innovation efforts per DALY. The difference is largest for publications and patents on dracunculiasis, or Guinea Worm Disease, which is covered in only 48 publications and 63 patents. These absolute numbers are lower than for any other disease, but highest in terms of innovation efforts per DALY (Table 3, bold). Another substantial difference is seen in the absolute funding and funding per DALY allocated to HIV/AIDS, which receives the highest amount of funding by far: more than twice the disease ranking second, and more than 1000 times the disease ranking last. When corrected for the burden of disease, however, HIV/AIDS ranks only 11th out of 19.
3.4. Innovation Efforts Relative to Other Neglected Tropical Diseases
Figure 2 shows a notably large difference in the extent to which different diseases are neglected. The difference between the most and least neglected diseases
![]()
Figure 1. There is poor correlation between funding and other indicators of innovation effort.
![]()
Table 3. Efforts per DALY provide a highly different perspective on efforts than absolute numbers. Numbers that are highlighted indicate the lowest and highest efforts per indicator.
![]()
Figure 2. The extent to which a disease is neglected varies greatly between NTDs. The difference in innovation efforts per DALY is large, please note the logarithmic scale on the y-axis, while the trend between indicators is fairly stable. The number 100 is allocated to the disease with the highest amount of R&D, for exact data see Table 3.
is at least a factor 370, which is the case for the number of publications per DALY (left), and a factor 9300 for the number of compounds per DALY (right).
The lines display a generally constant pattern across indicators. In other words, a disease that is relatively more neglected than other NTDs in terms of one innovation indicator, is likely to also be more neglected in terms of other indicators. An exemption to this is dracunculiasis, which scores highest for publications and patents per DALY, whereas no clinical trials have been reported for this disease. Other exceptions are trachoma and Ebola, which breaks the pattern with a relatively high score on patents per DALY and compounds per DALY, respectively.
Diseases that score high on all five indicators, and are thus least neglected, are leprosy and American trypanosomiasis. Tuberculosis, malaria and Ebola also score high, but were only covered in two articles. Diseases that score low on all five indicators are lymphatic filariasis and onchocerciasis. Roundworm (ascariasis) and syphilis are both covered in only one article, where they are the most neglected diseases.
3.5. Funding Evaluation Criteria
Funding organizations vary in terms of the definition used to demarcate NTDs from other health causes, the scope of NTDs covered, the research phase funded, the types of applicants that can apply for funding, the distribution mechanism, and the application of evaluation criteria. Respondents’ organizations with a broad scope, accept NTD proposals in specific calls or in general calls competing with proposals addressing other public health issues. Conversely, there are some organizations that are dedicated to address one specific disease. The type of research funded ranges from basic research to clinical development. Consequently, also the applicants eligible for funding varied from academia, tax-exempt organizations, product development partnerships, small- and medium-sized enterprises and pharmaceutical companies. The mechanisms of distribution can be divided in calls for proposals (requests for proposals), open calls (letters of inquiry, unsolicited proposals) and invitations to organizations.
According to the interviewees, all funding organizations assess research proposals based on criteria that relate to the potential impact of the research project (impact), the probability that the impact will be achieved (probability of success) and the fit between the envisioned impact and the goals of the funding organization (strategic fit). The definition and application of evaluation criteria differs between organizations from applying a fixed weight per criterium, sometimes in combination with a threshold per criterium, or in some cases different criteria can outweigh each other. Most organizations emphasise it is a case-by-case evaluation in which all criteria must be in place. Some funders take a portfolio perspective in which they either construct or fund portfolios consisting of different projects that can outweigh each other on different criteria. In general, however, philanthropic organizations emphasise impact, public organizations focus on probability of success and private organizations prioritize strategic fit.
Funding organizations primarily mention factors that positively influence funding decisions, or are even considered a requirement. Some factors, however, are also considered to negatively correlate with funding allocation. Compared to non-NTD funding, criteria that are specifically mentioned to not play a role in funding decisions are: target group, economic impact and competition. Table 4 shows the main evaluation criteria and the direction of their influence.
![]()
Table 4. Evaluation criteria used for the assessment of NTD research proposals by type of funding organization. Number of interviews in which the (sub)evaluation criteria are mentioned and the direction of influence. Direction: − = negative; o = neutral, or context-dependent; + = positive; req = requirement.
3.6. Impact
The impact can concern impact on society (societal impact), science (scientific impact) or economy (economic impact). All respondents emphasised that the merit of NTD R&D funding is societal impact and any research proposal must strive to address major public health problems. The weight that is given to societal impact versus other types of impact differs per funding organization. Compared to other types of organizations, philanthropic organizations put more weight on societal impact, public organizations on scientific impact and private organizations on economic impact.
3.7. Societal Impact
Societal impact is considered the most important criterion in the evaluation of proposals in NTDs. All respondents mention that projects should address true unmet medical needs and contribute to the product pipeline with products that are complementary to existing products. Unmet medical needs were considered true when there is currently no medical product accessible to the most affected populations and the development of a new medical product is considered to be able to lower the burden of disease.
The most common measure for societal impact mentioned in the interviews was the impact of a product on public health in terms of the number of DALYs prevented. Respondents indicated projects would score high on societal impact if they 1) concern a disease with a current high burden of disease; 2) can be largely solved by new medical products (unmet medical needs) and; 3) aim to develop a product with characteristics that fulfil these unmet medical needs (competitive advantage over existing products).
To fulfil an unmet need, a product may be highly innovative compared to the standard of care, in which case a research proposal must describe technical characteristics of the product (quotes 1 - 2).
Another way to fulfil an unmet need is for a product to be more accessible to the most affected populations compared to existing products, in which case research proposals must describe aspects such as scalability (e.g. easy to produce), affordability (e.g. low costs-of-goods) and acceptability (e.g. fit with local needs). Moreover, accessibility is deemed higher for projects that are expected to lead to results in the short- to medium-long term (short time-to-market) and that can convincingly show that they will be continuously produced in the future (sustainability). The accessibility criterion is given high priority and includes for example the positive assessment, or even requirement, of involving partners that can ensure adoption rates (quote 3).
![]()
There is a wide variety of other definitions of societal impact used by funders. Alternative measures for societal impact in the context of public health include: contribution to the elimination of a disease, health of a specific population, and quality of life. Besides public health, some funding organizations look for health equity or the impact on local communities through empowerment and capacity building. In contrast, organizations that fund basic research emphasise that societal impact is important but also difficult to assess in this stage (quote 4).
3.8. Economic Impact
The majority of respondents mention that economic impact is not an important aspect in the funding decision of an NTD research project. This is most apparent from the fact that potential profitability is not a requirement or positive factor, and at times even considered a negative factor in the funding decision for public and philanthropic organizations.
Nevertheless, interviews revealed several pathways through which economic impact does influence funding decisions in the field of NTDs. One public funding organization evaluates the macroeconomic impact of a project in terms of employment and economic competitiveness in its own geographical area. Private funding organizations evaluate the economic impact on their own organization. Although private funding organizations are not looking for profitability, they do assess the ability to recoup investments or prevent extensive losses to justify the investment (quotes 5 - 6).
3.9. Scientific Impact
Scientific impact relates to the extent to which outcomes of research advance the field by addressing specific critical barriers. Another aspect that is assessed in relation to scientific impact is the dissemination of research findings via (open access) publications and (socially responsible) licensing. Projects with a large contribution to science often concern basic research or the development of technically innovative products. Therefore, this criterion is mainly important to public funding organizations. The criterion of “open access” seems to have more weight in NTD R&D compared to other research areas, due to the nature of projects.
3.10. Probability of Success
Following impact, funding decisions depend on the probability that a project will succeed in achieving direct deliverables as well as long term objectives that these deliverables contribute to after the project is finished. This includes the likelihood that the project’s results, when promising, will be further developed and taken up all the way to market introduction. This is especially prioritised by public funders from HICs as they fund early stage research. The probability of success is assessed based on the technical, organizational, financial and regulatory feasibility.
3.11. Technical Feasibility
Technical feasibility refers to technical characteristics of the research project that influence its feasibility and has a direct relation with the strength of the science underlying the project. A research project is considered technically feasible when there is sufficient “scientific evidence” that the proposed solution will provide a solution to the problem at hand. The technical feasibility and probability of success is evaluated higher when there is more supporting scientific evidence (quote 7).
Furthermore, technical feasibility also strongly depends on the “scientific excellence” of the research design. Even with ample evidence, results can be achieved only when the research is well-designed and suitable for answering the research question (quote 8).
3.12. Organizational Feasibility
Organizational feasibility refers to the extent to which the research design can be properly executed by the research organization(s). This includes the access to patients, research facilities and intellectual property rights that are needed to carry out the research, as well as the quantity and quality of human resources. Individuals involved in the project should be capable, a criterion that is assessed mostly based on the “track-record” of researchers. In addition, funding organizations also assess the quality of collaboration. A good track-record or previous partnership will positively influence the probability of success (see quotes 9 - 10).
3.13. Financial Feasibility
Financial feasibility relates to the extent to which the financial resources fit with the financial needs to perform the research activities. Respondents highlight that assessment of the financial feasibility concerns the current project and extends to the likelihood of funding subsequent R&D phases. It is in this light that funding organizations positively evaluate the involvement of pharmaceutical companies (quote 11), which can again lead to the exclusion of new research organizations (quote 12).
3.14. Regulatory Feasibility
Regulatory feasibility concerns the likelihood that the results will be accepted and supported by regulatory authorities. A research proposal scores must at least be compliant with international standards and may score higher on regulatory feasibility when regulatory agencies are involved in an early research phase.
3.15. Strategic Fit
Strategic fit is defined as the extent to which a research project aligns with the strategy of the funding organization. For a positive assessment of strategic fit, funders consider that projects should contribute to the overall organizational goals, concern research that is not extensively funded by other organizations and should fit with in-house assets. The latter criterion is applied mostly by private organizations with significant weight and specifically for projects concerning neglected tropical diseases. None of the funding organizations explicitly formulated strategic fit as evaluation criterion, but interviews reveal that it is an important aspect in NTD funding. Strategic fit is mostly applied as a criterion (as a requirement) in the agenda setting phase of funding opportunities, but the criterion is also used to evaluate and compare competing research proposals.
3.16. Contribution to Goals
In funding decisions, funding organizations assess contribution to the goals of the organization and/or the specific call for proposals. In the field of NTDs, these goals are largely influenced by international goals. Interviewees mention the Sustainable Development Goals, WHO list of priorities and the London Declaration of health as important guidelines funding allocation. The importance of contributing to these goals is emphasised explicitly and repeatedly by private funding organizations, which highlights the importance considering optimization of investments (quote 13).
3.17. Fit with In-House Assets
For private funding organizations, fit with in-house assets is an important criterion for funding in the field of NTDs. To receive funding, R&D projects should match with the expertise of employees, with compound libraries or research facilities available at the funding organization. In interviews, the rationale for this fit as a criterion is to provide focus and enhance chances of success (quote 14).
3.18. Fit with Funding Gap
All respondents emphasise that the main reason to invest in NTDs is the existence of a funding gap. The funding gap relates to the lack of funding by industry, due to market failure, but also to the absence of other funding organizations. By focusing on subjects that are not yet covered by others, funding organizations aim to make a substantial contribution to a specific subject (quotes 15 - 16). However, a funding gap that is considered too big to cover by the budgets of the funding organizations can influence the funding decision in a negative way (quote 15).
3.19. Interrelatedness
Interviews show that evaluation criteria do not stand on their own but are interrelated. Most of the criteria reinforce each other, but there are also criteria that contradict, introducing trade-offs in funding decisions. Though these trade-offs are not unique to the field of NTDs, they have an outcome that seem characteristic for this field.
3.20. Innovation Trade-Off
Respondents describe the difficult trade-off between incremental innovations and disruptive innovations. Research concerning highly innovative products that are very different from the standard of care have a potentially high social impact and a low probability of success (quotes 1 - 2). For incremental products, this is the other way around (quote 7). Interviews also reveal that there are two criteria that compete with highly innovative products and thus increase the funding chances of incremental innovations. These are 1) accessibility to achieve societal impact, and 2) the strategic fit of private funding organizations. Incremental innovation more often becomes available in the short-term and for an affordable price, scoring higher on societal impact. Strategic fit also steers towards incremental product innovations rather than to research addressing real barriers and finding innovative solutions on the long-term.
This innovation trade-off is handled differently by different organizations. Some are happy to fund technically challenging projects, in line with the merits of their organization. Others solve the issue by constructing a balanced portfolio to allow more risky projects. Most, however, emphasise the value of incremental innovations to the field of NTDs. Thereby, the definition of innovativeness appears to be altered for the context of NTDs (quotes 17 - 18).
3.21. Reinforcement
For public and philanthropic funding organizations, the involvement of pharmaceutical industry is considered to secure further product development (financial feasibility) and a sustainable supply chain (societal impact). Therefor this involvement is generally assessed positively or even considered a requirement for funding (quotes 18 - 19). As such, the evaluation criteria that are applied by private funding organizations, such as the likelihood to recoup investments via dual market opportunities, indirectly influence to the funding decisions of other funders (Figure 3).
4. Conclusions and Discussion
This is the first study to provide a comprehensive overview of innovation efforts and resource allocation in NTDs, revealing a large difference in the extent to
![]()
Figure 3. The criteria used by pharmaceutical companies directly and indirectly (dashed line) influence the funding decisions of other funding organizations. Green = positive; red = negative.
which the individual NTDs are neglected in terms of innovation efforts per DALY. The study further shows that while societal impact is the main rationale for investing in NTD R&D, it is not the sole determinant of funding, nor the most decisive. In-depth interviews reveal the interrelatedness of different evaluation criteria that may even have a perverse effect on developing products with high societal impact. As such, there is a gap between societal impact as the rationale for NTD-funding and significant challenges related to the application of societal impact as a funding criterion in practice. Furthermore, the application of evaluation criteria leads to a primary focus on short-term impact. Finally, the study shows that the pharmaceutical industry plays a more important role in NTD research funding than expected. Current practice thus reinforces the relative neglect of the most neglected diseases, and a long-term strategy is needed to resolve this.
4.1. Innovation Efforts and Burden of Disease
The literature review demonstrates an unequal distribution of research efforts for NTDs. The NTDs that are relatively most neglected are up to 10,000 times more neglected than other NTDs (Figure 2). Some diseases receive 24-fold more funding than other diseases, despite similar needs for investments. The “big three” diseases HIV/AIDS, tuberculosis and malaria receive the highest and increasing share of 71% of all funding; whereas a scarce and declining share of 1% of funding is allocated to bacterial pneumonia and meningitis. This is not commensurate to the burden of disease in terms of deaths or DALYs [37] . Moreover, the results show that the relative innovation efforts towards NTDs are almost unrelated to the burden of disease. The top three diseases that pose the highest burden on society also score highest in terms of absolute efforts (Figure 1; Table 3), but HIV/AIDs is found to be amongst the most neglected disease in terms of funding per DALY while tuberculosis and malaria remain with relatively high efforts per DALY (Figure 2; Table 3). These findings confirm that allocation of resources is not based on the burden of disease alone [21] . Further comparison shows that relative efforts are also unrelated to a change in DALYs. The burden of the relatively neglected diseases lymphatic filariasis and onchocerciasis, for example, reduced by 30% since 1990 [9] , while the burden of leprosy continues to rise [50] . Leprosy also receives lots of effort in absolute terms (Figure 2), showing that efforts in one disease are more successful in creating societal impact than efforts in other diseases.
An explanation for the difference in efforts between NTDs can be sought in the extent to which diseases are covered in the definition of an NTD. Since definitions are generally formulated with the intention to set the global health agenda, it could be argued that exclusion of a disease in the definition of NTD would automatically lead to neglect. Another explanation can be sought in arguments that are used to distinguish NTDs from other diseases, namely the (limited) market size as an important incentive for innovation. The market size is influenced by both quantitative and qualitative market characteristics, such as the prevalence of a disease in high income settings, the disease trajectory and the existence of medicines [51] ; which affect the relative neglect of NTDs.
4.2. Challenges with Evaluation of Societal Impact
Societal impact plays an important, yet unexpected role in allocating R&D funding for NTDs. The importance of this evaluation criterion is evident from the fact that achieving societal impact is the main reason for all types of funding organizations to invest in NTDs. A tangible contribution to public health challenges is rather a prerequisite and funders are unlikely to fund research for pure scientific curiosity or economic reasons (e.g. quotes 11 and 18). Although societal impact is a prerequisite for funders, it is important to note that its role as a primary funding rationale has not yet been substantiated by quantitative analysis. In general the same criteria are applied, but the difference between NTD research and other fields of (health) research, lies in the weight and operationalization of the societal impact criterion.
The weight differs between funding organizations as well, largely dependent on their mandate. Philanthropic organizations are dedicated to achieving societal impact in a certain area [36] , enabling them to prioritize this criterion. Public organizations have the prime responsibility to protect the health of their own citizens and strengthen their national economies. For a long time, investments in basic research were seen as a means for competitive economic markets and therefore, research was [52] and is primarily assessed solely on scientific merit (see quote 8). Private organizations on the other hand, must act in accordance with the wishes of their shareholders. It is therefore no surprise that these companies mention the prevention of economic losses (quotes 5, 6) and optimization of investments (quote 13) as important evaluation criteria. Through these criteria, the business case for investing in NTD research is optimized, which is an effective way to justify NTD funding as part of a CSR strategy to investors [43] . Thus, the results show that the type of organization is the main indicator of the importance of societal impact in the funding decision.
An important explanation for challenges in the evaluation of societal impact is that this criterion is relatively new as it was introduced in the distribution of public funding only a decade ago [52] . Efforts to develop measures for the ex ante assessment of societal impact are still ongoing [53] as it has proven difficult to develop measures that are reliable, meaningful and quantifiable [52] . Moreover, reviewers often are researchers with field-related expertise themselves who lack expertise and experience to evaluate societal impact [54] . Challenges related to the ex ante assessment of social impact hamper the prioritization of this criterion in the funding decision.
The impact of the challenges regarding the societal impact evaluation criterion can be seen in the amount of funding that is allocated to different diseases by the different types of funding organizations [6] . As shown, funding patterns of philanthropic organizations most closely resemble the burden of disease (Figure 4). Compared to private and public organizations, philanthropic organizations allocate less money to the big three diseases and a large share of their budgets is
![]()
Figure 4. Funding patterns of major funding organizations in 2019. Philanthropic funders allocate funding to a more diverse and equal way, while private and public funders spend more than three-quarter of their budgets to the big three diseases. HIC = high-income countries; MNC = multinational pharmaceutical companies; ML = multilateral organizations. Data retrieved from: G-FINDER 2020 [33] .
allocated to diseases with the highest burden of disease, namely diarrhoeal diseases and bacterial pneumonia/meningitis [55] . In this light, it is unfortunate that growth in funding budgets in the last decade was largely due to increased budgets from private, multinational pharmaceutical companies [33] . Considering the funding criteria and weight of funding criteria, it is expected to be attributed towards the least neglected diseases, reinforcing the unequal distribution.
4.3. Focus on Short-Term Impact
This study highlights that evaluation criteria that are used in the field of NTDs steer funding decisions towards incremental innovation with high probability of success and short-term results. The demand for this can be justified by the urgent medical needs that characterise the field. From interviews, it becomes clear that funders now generally accept that the field of NTDs benefits most from incremental innovations (quotes 17 - 18). Whereas previous research shows that R&D outcomes, like patents and clinical trials, reflect the focus on incremental innovations such as repurposing existing drugs [56] [57] , we now show that this is at least partly due to funding decisions: research has a higher chance of acquiring funding when it concerns low-risk, incremental innovations.
Funding organizations also de-risk NTD R&D in other manners, for example by focussing on renowned institutes and researchers with long-track records (quotes 9 - 10). This introduces the Matthew effect in science, meaning that research topics and researchers that received funding in the past are more likely to receive funding in the future [58] . In practice, it means that the relatively most neglected diseases will remain neglected, and a long-term strategy is needed to increase funding for these diseases.
This study is the first to describe how the trade-off between short-term and long-term societal impact affects the field of NTDs. In the past, incremental innovations indeed led to significant advancements for some diseases, for example, those listed on the WHO Essential Medicines List [57] . Moreover, in this already underfunded research area, the impact is significant. As exemplified by studies diving in-depth into unmet needs and innovation barriers [51] [59] both radical and incremental R&D are needed to advance medical countermeasures against neglected diseases.
4.4. Key Role of the Pharmaceutical Industry
This study clearly shows the significant role of pharmaceutical companies in the funding landscape of NTDs, both directly and indirectly. While market failure is mentioned as the most important reason to invest in NTD R&D in the first place, interviews reveal that the involvement of pharmaceutical industry in a research project can enhance funding chances via different routes. The evaluation criteria that are used by pharmaceutical industry (e.g. fit with in-house assets, fit with London Declaration and possibility to recoup investments), thereby indirectly influence funding decisions from other funding organizations in a positive way.
This provides an explanation for empirical data showing that the top 10 diseases for public funding organizations from high-income countries and multilateral organizations and private multinational pharmaceutical companies are highly similar (Figure 4). Amongst the funding criteria employed by pharmaceutical companies is the likelihood to recoup investments, through dual market opportunities, which confirms the suggestion from Corporate Social Responsibility literature [46] . That market size plays a significant role in guiding philanthropic activities. Importantly, it implies that without deliberate changes to this rationale, diseases that are currently relatively most neglected, will likely remain so.
4.5. Implications
The unequal distribution of efforts and relative neglect of NTDs has important implications for policy, since current R&D incentive mechanisms often treat NTDs as a single group of diseases for which increasing the return on investment is the most important strategy to stimulate innovation [5] . Although grouping these otherwise neglected diseases has had clear and positive effects on the total amount of efforts dedicated to NTDs, the literature review shows that it may also lead to the ignorance of disease specific barriers to innovation. These unique barriers should be taken into account to maximize the impact of current and novel programs on the control of NTDs [51] . Since some diseases are profoundly more neglected than other diseases, we advocate that NTD related programs should be redesigned. Importantly, the findings provide practical indications for stakeholders on how to improve their efforts in line with their mandate.
The review also highlights the need for a global definition of NTDs. Current literature is based on multiple and highly diverse definitions of NTDs, that reflect the discrepancy in definitions used by the WHO [60] , PLoS NTD [61] , and Policy Cures Research [6] . Moreover, the variety of definitions leads to major data gaps that not only affect the current study, but also the whole body of literature (Table 2) and programs dedicated to NTDs. The results of this study can aid the discussion on what constitutes an NTD, by serving as a benchmark for determining the neglect of other diseases compared to NTDs. Based on such future research, a threshold should be determined to decide what constitutes an NTD. A globally accepted definition of NTDs and disease specific incentive mechanisms should be adopted to subsequently align global efforts and enhance their impact.
Analysis of the interview data identifies two manners to increase the societal impact of R&D funding. On the long term, further development of impact criteria could increase the confidence and weight that is given to the impact assessment. On the short time, funding organizations could benefit from embedding the impact assessment in an earlier stage of the funding distribution process. One way to do so is working with (highly) targeted funding schemes, in terms of issued calls or solicited calls for prioritized health needs. To maximize impact, the health care needs follow from a process of identification, articulation and prioritization of health R&D needs. This way, funding can steer research towards specific demands for health research. More research is needed on their efficacy and impact on e.g. scientific quality. There is no evidence that such focus would deteriorate other, scientific requirements. Nevertheless, these funding schemes are not used often [14] while they may be able to correct for skewness caused by evaluation criteria shown in this study.
As discussed, current practice reinforces the problem that the relatively most neglected diseases will remain neglected. One way to go about this is to build upon extended commitment by the pharmaceutical industry through international goals. The results of this study show that internationally agreed priorities, and primally those listed in the London Declaration, are effective in fostering commitment from the pharmaceutical industry (quote 13). Literature shows that these priorities have also lead to real declines in the burden of NTDs [50] . The uptake of the most neglected diseases in the formulation of new goals may thus also be effective in shifting the attention of pharmaceutical industry to other disease indications. Renewed international priorities will most likely improve but not entirely solve disparities in funding, as it will not alter and rather establish other criteria that are used by pharmaceutical companies such as fit with in-house assets and the possibility to recoup investments. Therefore, alternatives ways for funding allocation should be explored.
The results of this study pinpoint the need for exploring the de-linkage of pharmaceutical industry from R&D on urgent societal problems, including but not limited to NTDs. Nowadays, the public sector seems to focus on incentivizing private companies to fund late stage development through push and pull factors. There is a growing body of evidence, including the current study, showing that these strategies are insufficient to overcome urgent societal problems and pointing to the need of increased involvement of the public sector in late stage development [62] and even vaccine production [63] . We encourage policy makers to build upon these findings in the transition policies that are expected to follow post-COVID.
Acknowledgements
The authors would like to thank Marloes Nooitgedagt, Shaquille Taylor and Adrienn Lencses for organising and conducting the interviews and Jelle Feddema for his help in conceptualising the results.
Funding
This study was financially supported by the Veni SGW program (project number VI.Veni.201S.044) of the Dutch Research Council (NWO).
List of Abbreviations
CSR = corporate social responsibility
HIC = high income countries
MNC = multinational pharmaceutical companies
NTD = neglected tropical disease
R&D = research and development
SME = small and medium pharmaceutical and biotechnology companies
Appendix 1. Search Syntax. Date Data Search: 13 Jan. 2019
Appendix 2. Flow Chart Based on the PRISMA Guidelines