Institutional Factors That Impact the Success of Big Data Science Projects ()
1. Introduction
The success of Big Data Science projects is critical to the growth and development of organizations in the modern data-driven economy [1]-[4]. However, many Big Data Science projects fail to achieve their intended goals, often due to challenges such as unclear objectives, poor project management, and the complexity of data science workflows [2] [3]. Big Data Science projects aim to extract actionable insights from vast datasets, improve decision-making processes, and drive innovation. Despite their potential, research indicates that many Big Data Science projects fail to deliver value, resulting in wasted resources and missed opportunities [5] [6]. According to Darwish [7], approximately 85% of big data science projects fail to progress beyond the experimental stage. VentureBeat reported that 87% of data science projects never finished production. This high failure rate indicates a significant gap between the promise of data science and its practical implementation [8]. Several factors contribute to the failure of big data projects, including technical challenges, business-related issues, and organizational obstacles. These failures underscore the crucial need to comprehend the obstacles that impede the success of Big Data Science projects and to devise strategies to mitigate these risks.
Despite technical expertise and sophisticated models, data science projects can still falter, underscoring the field’s diverse nature and the need for both technical and business knowledge for successful project delivery. Effective communication and collaboration among project stakeholders are crucial for the success of Big Data Science projects. Stakeholders must have a shared understanding of project goals, clear communication channels, and mechanisms to address conflicts and concerns promptly. Additionally, organizational culture plays a significant role in determining project outcomes. A culture that fosters innovation, collaboration, and adaptability can enhance the success of Big Data Science projects by encouraging open communication, continuous learning, and employee empowerment. Resource allocation is another critical factor in the success of Big Data Science projects. Adequate resources, including skilled personnel, computational infrastructure, and funding, are essential to achieving project objectives. Effective resource management throughout the project lifecycle can significantly impact the efficiency and effectiveness of Big Data Science initiatives. Therefore, it is essential to consider the business domain when evaluating the success of data science projects [9].
This study is necessary because it addresses a critical gap in understanding how external factors, including cultural differences, technological change, and resource allocation, affect the success of Big Data Science projects. While the technical aspects of these projects, including algorithm performance and data engineering, have been extensively studied [10], there is limited empirical research on the broader organizational and external factors that influence project outcomes [11]. This study aims to provide actionable insights to enhance project management practices, resource allocation, and stakeholder collaboration by investigating these external factors. Understanding these dynamics is crucial for organizations to optimize the value of their data-driven initiatives and to mitigate the high failure rates associated with Big Data Science projects.
1.1. Statement of the Problem
Despite the increasing adoption of Big Data Science in organizations, the success rate of these projects remains alarmingly low. According to Darwish [7], 85% of data science projects fail to deliver value due to unclear objectives, poor project management, and a lack of stakeholder alignment. The complexity of data science workflows and the rapid pace of technological change exacerbate these challenges, increasing the risk of project failure [12].
The problem addressed in this quantitative study is the lack of empirical evidence regarding cultural differences, technological change, and resource allocation that influence the success of Big Data Science projects. While researchers have extensively studied technical aspects such as algorithm performance and data quality, there is limited research on the non-technical factors that contribute to project outcomes. The lack of research on the non-technical aspects of data science projects has been identified as a significant obstacle to success in this field ([7] [13]-[15]). The persistently high failure rate of Big Data Science projects underscores the need for a deeper understanding of the factors that contribute to project success. Although previous studies have focused on technical aspects, researchers have conducted limited investigations into the roles of cultural differences, technological change, and resource allocation in determining project outcomes. This study seeks to address this gap by investigating the impact of these external factors on the success of Big Data Science projects.
1.2. Purpose of the Study
This quantitative study aims to examine the factors that influence the success of Big Data Science projects, focusing on cultural differences, technological changes, and resource allocation. This research aims to provide actionable insights for data science practitioners and organizations seeking to improve their project management practices by identifying the key drivers of project success. Herath and Chong [16] stated that the ability to adhere to the original budget, timeline, and established goals determines a project’s success. Internal and external elements play a role in the success or failure of a project. Project managers recognize that managing cost and schedule parameters alone is insufficient to prevent failures; they must also consider other factors [12].
The findings of this study contribute to the growing body of knowledge on data science project management by providing empirical evidence on how these external factors collectively influence project outcomes. This research also aims to enhance project success rates by identifying strategies to address cultural challenges, adapt to technological advancements, and optimize resource allocation. By addressing these factors, the study aims to bridge the gap between technical and organizational considerations, ultimately enhancing the success of Big Data Science projects in dynamic and resource-intensive environments.
1.3. Research Questions and Hypotheses
The study aims to answer the following research questions:
RQ1. How do technological changes influence the success of big data science projects?
H10. There is no relationship between project success (PS) and technological changes (TC).
H1a. There is a relationship between project success (PS) and technological changes (TC).
RQ2. How do cultural differences influence the success of big data science projects?
H20. There is no relationship between project success (PS) and cultural differences (CD).
H2a. There is a relationship between project success (PS) and cultural differences (CD).
RQ3. How does resource allocation influence the success of big data science projects?
H30. There is no relationship between project success (PS) and resource allocation (RA).
H3a. There is a relationship between project success (PS) and resource allocation (RA).
2. Literature Review
The success of Big Data projects depends on a complex interplay of organizational, technological, and human factors within research institutions and data-driven enterprises [17]. Researchers have indicated that big data projects continue to fail at an alarming rate, despite significant investments in technology infrastructure and human resources ([15] [18] [19]). Understanding these failure rates requires examining the institutional dynamics and cultural barriers that may prevent the successful implementation of big data initiatives across organizations. In a recent study, Sajid et al. [20] found that the success of Big Data Science projects depends on various factors, including organizational culture, leadership support, technical infrastructure, data quality standards, team expertise, and resource allocation strategies. However, there is limited empirical research examining the relationships between external institutional factors and project outcomes in big data science initiatives across various organizational contexts ([13] [21]).
2.1. Evolution of Big Data Science
The evolution of Big Data Science has transformed from a simple data collection and analysis process to a complex, multidisciplinary field that integrates advanced analytics, machine learning, and distributed computing systems across organizations [7]. This rapid evolution has led to the emergence of specialized roles, tools, and methodologies to address the increasingly complex data challenges faced by modern organizations. The term “Big Data” emerged in the early 2000s to describe datasets that exceed the capacity of traditional data processing tools and methods [22]. As the field of Big Data Science has grown, it has become increasingly important to understand the factors contributing to the success of Big Data projects within institutions. Since then, the field has grown to encompass advanced machine learning algorithms, cloud computing platforms, and real-time analytics ([2] [23]). The introduction of frameworks such as Hadoop and Spark has revolutionized data processing, enabling organizations to analyze large datasets efficiently [24]. Today, Big Data Science is a multidisciplinary field that integrates computer science, statistics, and domain expertise to extract actionable insights from complex datasets [25]. A recent study by Mazumder [26] highlighted that significant organizations, including healthcare and financial institutions, have adopted Big Data Science to drive innovation, improve decision-making, and secure a competitive advantage.
2.2. Importance of Big Data Projects in Modern Institutions
Big Data projects play a crucial role in modern institutions by enabling data-driven decision-making, enhancing operational efficiency, and fostering innovation [27]. Abdikhakimov [2] noted that these projects have transformed organizational processes by offering advanced analytics capabilities, predictive modeling, and automated decision-support systems across various sectors of the industry. For instance, healthcare organizations use Big Data analytics to predict patient outcomes and optimize resource allocation [26]. In manufacturing, Big Data and predictive analytics enhance cost and operational performance [28]. Similarly, academic institutions can use Big Data to improve research, teaching, and administrative processes. Financial institutions also harness Big Data to detect fraud and assess credit risk, while other institutions analyze and predict climate change using Big Data [29]. A study by Darwish [7] suggests that the potential economic impact of Big Data could reach $3 trillion by 2025, as organizations utilize advanced analytics to enhance decision-making capabilities and operational efficiency.
Organizations that effectively leverage Big Data are significantly more likely to achieve competitive advantages and drive innovation across their operations compared to traditional organizations [30]. These initiatives have also become vital for market research and customer behavior analysis, with organizations investing heavily in data analytics infrastructure, DevOps (development and operations), and MLOps (machine learning operations) to generate actionable business intelligence [31]. However, the success of Big Data initiatives often depends on various organizational and institutional factors that can either facilitate or obstruct their implementation [20].
2.3. Common Challenges in Big Data Science Projects
Big Data Science projects encounter numerous challenges that can impede their success. A significant challenge is the complexity of integrating diverse datasets from multiple sources, which often requires substantial preprocessing and necessitates significant data cleaning [32]. Other challenges encompass data quality issues, security concerns, limitations in technical infrastructure, and the demand for specialized talent with advanced analytical skills [33]. Additionally, the shortage of skilled data scientists restricts organizations’ capacity to execute projects effectively. Furthermore, Jonathan and Raharjo [13] indicated that insufficient governance and management processes can hinder the success of Big Data Science initiatives. Organizational culture and resistance to change can also present significant barriers [18]. Moreover, cultural differences within data science teams can create communication barriers and misaligned objectives [34]. Finally, the rapid pace of technological change necessitates continuous learning and adaptation, which can strain organizational resources ([31] [35]).
2.4. Volume and Variety of Data
One of the main challenges in Big Data projects is managing the vast volume and variety of data. As organizations collect large amounts of information, the sheer volume becomes challenging to manage. For example, the Centre for Environmental Data Analysis (CEDA) has developed strategies to manage extensive data volumes by organizing them into file sets, which facilitate planning and tracking of dataset storage [36]. The rapid growth in both the volume and variety of data generated from diverse sources, such as social media, sensors, and transaction systems, presents significant challenges for organizations in effectively managing and analyzing this data [22].
2.5. Technological Integration
The integration of diverse data sources, technologies, and applications poses a significant challenge for Big Data Science projects [37]. These projects often involve heterogeneous data formats, legacy systems, and incompatible technologies, necessitating considerable effort to harmonize and integrate them effectively. This complexity is further exacerbated by the rapid pace of technological innovation and the proliferation of open-source tools, creating an ever-evolving landscape that teams must navigate [38]. As organizations adopt new technologies, they frequently face issues such as incompatible data pipelines or tools that do not align with existing workflows, which compound the difficulty of ensuring seamless integration. Furthermore, integrating technologies that support iterative workflows and agile development processes is essential throughout the lifecycle of Big Data Science projects, from data collection to deployment. According to Oursatyev [39], projects face risks of inefficiencies, redundancy, or failure to achieve their objectives without a holistic approach to technological integration.
2.6. Data Management and FAIR Principles
In scientific research, effective management of large datasets is essential. The necessity for data to be Findable, Accessible, Interoperable, and Reusable (FAIR), as outlined by Hey [40], introduces its own set of challenges. Ensuring that data is Findable requires the use of persistent identifiers and rich metadata, while Accessibility necessitates open protocols and authentication mechanisms. Data Interoperability relies on formal, accessible, shared, and broadly applicable languages for knowledge representation, and Reusability demands detailed provenance, usage licenses, and compliance with community standards. Addressing these FAIR principles is crucial for maximizing the value and impact of big scientific data [41].
2.7. Interdisciplinary Collaboration
Successful Big Data Science projects often require collaboration among individuals with varied backgrounds, including domain experts, data scientists, and information technology specialists [14]. Nevertheless, coordinating these interdisciplinary teams and aligning their goals can pose challenges. The complexity of Big Data projects often demands cooperation across multiple disciplines. Prakash [27] emphasized the importance of interdisciplinary research programs in tackling the challenges inherent in data-intensive studies, noting that nurturing such collaboration can be difficult due to the differing methodologies and terminologies across fields.
2.8. Factors Impacting Big Data Science Project Success
The success of big data science projects is crucial for harnessing the immense potential of vast datasets to drive innovation, uncover hidden insights, and propel organizations toward their strategic goals. Factors such as data volume, velocity, and variety pose significant challenges to the effective management and analysis of big data [27]. Additionally, the need for data accessibility and integration, as well as addressing regulatory concerns regarding data ownership and privacy, further complicates the landscape [42]. Existing literature highlights several key internal factors that can impact the success of Big Data Science projects. Chang [43] identified management support, commitment to significant data initiatives, and government support and policy as influential factors in organizations’ adoption of big data. However, studies have comprehensively examined the numerous external factors that can contribute to or hinder the success of Big Data Science projects [13]. Beyond the technical hurdles, Big Data Science endeavors must navigate and examine external factors that may affect project outcomes, including cultural differences, technological change, and resource allocation [32]. Overall, the successful implementation of big data science projects requires a holistic approach that addresses both internal and external factors.
2.8.1. Cultural Differences
Cultural diversity presents both opportunities and challenges in Big Data Science projects. Teams composed of members from diverse cultural backgrounds can foster innovation and offer unique perspectives in problem-solving. However, cultural differences may lead to communication challenges and differing expectations, particularly in multinational or diverse teams ([44] [45]). Effectively managing these cultural differences is crucial for enhancing collaboration and ensuring project success. Attribution Theory illustrates how cultural backgrounds can influence team members’ perceptions of project outcomes, thereby shaping their behaviors and decisions [46]. The research underscores the importance of inclusive leadership, cross-cultural training, and clear communication in overcoming these challenges ([14] [47]-[49]). This section examines the role of cultural diversity in Big Data Science projects, the challenges it presents, and practical strategies for managing these differences.
2.8.2. The Role of Cultural Diversity in Big Data Science Teams
Cultural diversity within Big Data Science teams can be both an asset and a challenge. Diverse teams bring unique perspectives, problem-solving approaches, and innovative ideas, which can enhance the quality of project outcomes [50]. However, cultural differences can also result in communication barriers, conflicting priorities, and misaligned expectations if not managed effectively [44]. For instance, Panda [14] found that cultural misalignment was a significant factor in the failure of several multinational Big Data projects. In another study, Hsu [51] underscored the importance of researchers being cognizant of and respectful toward differing perspectives on the value of Big Data approaches, as trust is crucial for fostering successful collaborations.
Furthermore, cultural diversity in Big Data Science teams encompasses varying perceptions and attitudes toward data privacy and ethics. According to Schoentgen and Wilkinson [52], researchers from different cultural backgrounds may hold divergent views on the appropriate use and protection of sensitive data. This variation can pose challenges in establishing consistent data governance policies and procedures across the team. Differences in work styles, decision-making processes, and attitudes toward risk often exacerbate these challenges. Yao [53] noted that leaders must foster an inclusive environment that values diverse viewpoints and provides mechanisms for effective collaboration. According to the Data and Analytics Leadership Executive Survey conducted by New Vantage Partners, 79.8% of chief data officers (CDOs) reported cultural issues as their primary challenge in establishing a data-driven organization [54]. Success in Big Data Science projects often depends on team members’ ability to bridge cultural divides and work cohesively toward common goals [45].
2.8.3. Managing Cultural Differences in Big Data Science Projects
Effective management of cultural differences promotes collaboration and ensures project success [47]. A study by Burrows [55] found that clear communication, a shared understanding of objectives, and trust-building were critical success factors in multinational information systems projects. Strategies such as cross-cultural training, inclusive leadership, and the use of collaborative tools can help mitigate the adverse effects of cultural diversity [56].
Grander [57] asserts that leaders overseeing Big Data Science projects must consider the cultural context surrounding data collection and interpretation. They highlight that the multicultural nature of Big Data can introduce biases that, if not adequately addressed, may distort the validity of analytical results. Research by Wang and Goh [58] emphasizes the importance of open communication. Additionally, attribution theory provides a valuable framework for understanding how cultural backgrounds influence team members’ perceptions of project outcomes, thereby enabling managers to proactively address potential conflicts ([46] [59]).
3. Methodology
This study employs a quantitative methodology with a correlational research design to investigate whether institutional factors, including cultural differences, technological changes, and resource allocation, significantly influence the success of Big Data Science projects. A correlational approach determines whether changes in these independent variables are statistically associated with changes in project success outcomes ([60] [61]). The study employs convenience sampling to recruit participants and gathers data through an online survey targeting data scientists and Big Data Science project managers. The researchers will analyze the data using statistical techniques to evaluate the relationships among the variables of interest.
3.1. Data Collection
The study relies on primary data, collected through an online survey administered via SurveyMonkey. Participants completed structured, closed-ended questions using validated instruments. The target population comprises data scientists and Big Data project managers with experience in Big Data Science projects. The data were self-reported, with no reliance on institutional archives or secondary online sources. The researchers recruited participants through professional networks, such as LinkedIn, particularly in the Big Data Science open groups, to ensure relevance and accessibility. Each participant had at least one year of experience in data science project environments, thereby enhancing the contextual accuracy of the data.
3.2. Scope
The survey targeted professionals in the United States, with particular focus on those in New York State. The expected age range for participants was 25 to 60 years, corresponding to mid- to senior-level professionals. To ensure the reliability of the statistical analysis, the study recruited at least 100 participants. A power analysis conducted using G*Power 3.1 software determined that for a multiple linear regression with three predictors and an anticipated medium effect size (f2 = 0.15), a sample of 77 participants is required to achieve a statistical power of 0.80 at α = 0.05 ([62] [63]). The targeted sample size exceeds this threshold, allowing for incomplete responses and improving generalizability.
This design enables the study to 1) assess the strength and direction of relationships between independent and dependent variables, 2) identify which institutional factors most significantly predict project success, and 3) generate actionable insights for practitioners. The use of a non-experimental, correlational methodology is justified by its empirical alignment with recent research in data science, project management, and organizational behavior. Mixed-methods or qualitative approaches were deemed less suitable given the study’s focus on statistically measurable associations across a larger population ([15] [61]).
3.3. Research Questions and Hypotheses
The purpose of this study is to investigate the relationship between institutional factors, including cultural differences, technological advancements, and resource allocation, and the success of Big Data Science projects. In alignment with the problem statement and purpose outlined in Chapter 1, the following research questions and hypotheses guide this quantitative, correlational investigation:
Research Question 1 (RQ1). How do technological changes influence the success of big data science projects?
H10. There is no relationship between project success (PS) and technological changes (TC).
H1a. There is a relationship between project success (PS) and technological changes (TC).
Research Question 2 (RQ2). How do cultural differences influence the success of big data science projects?
H20. There is no relationship between project success (PS) and cultural differences (CD).
H2a. There is a relationship between project success (PS) and cultural differences (CD).
Research Question 3 (RQ3). How does resource allocation influence the success of big data science projects?
H30. There is no relationship between project success (PS) and resource allocation (RA).
H3a. There is a relationship between project success (PS) and resource allocation (RA).
3.4. Population and Sample
3.4.1. Population
This study examines professionals who have worked on Big Data Science projects in the technology sector, including data scientists and data analysts. Empirical research suggests that these professionals play a crucial role in designing, executing, and evaluating data science initiatives, positioning them well to assess the institutional factors that influence project success [64]. Big Data Science projects are inherently interdisciplinary, requiring the integration of technical, managerial, and strategic perspectives. Prior studies have found that data science professionals routinely engage with issues such as cultural dynamics, technological adaptation, and resource management, which are critical determinants of project success ([14] [21]). Moreover, [15] reported that data scientists and analysts are among the most influential roles in cross-functional data science teams, frequently interacting with business leaders, developers, and stakeholders to align technical goals with organizational outcomes. These roles are also most affected by institutional barriers, including poor resource allocation, cultural misalignment, and a lack of technological readiness [42]. For these reasons, targeting this population ensures that the study captures informed, experience-based insights into how institutional factors affect the outcomes of Big Data Science projects.
3.4.2. Sample Size and Power Analysis
For this study, the researchers included 100 participants. To determine the minimum sample size needed to detect a medium effect size (f2 = 0.15) with 80% statistical power and a 5% significance level (α), the researchers used G*Power 3.1 for a multiple linear regression model with three predictors: cultural differences, technological changes, and resource allocation [62]. Following guidelines from Murayama [63] and Arkes [65], the minimum required sample size is 77 participants. However, the researchers slightly oversampled to account for any incomplete or invalid responses.
3.4.3. Sampling Method
For this study, the researchers used a non-probability, convenience sampling approach to recruit participants. This method is suitable for exploratory, cross-sectional surveys and allows for efficient access to a relevant population without requiring randomization ([66] [67]). The researchers recruited participants through professional social media platforms, such as LinkedIn. Recruitment materials outlined the study’s purpose and eligibility criteria and provided a link to an anonymous online survey on SurveyMonkey. Participants participated voluntarily and anonymously, and the researchers did not collect any data until the Institutional Review Board (IRB) approved the study. Following IRB guidelines, participants received an informed consent form explaining the study’s scope, data protection measures, and their right to withdraw at any time without penalty.
3.5. Instrumentation
For this study, the researcher used a structured survey with multiple validated scales to measure key variables, including project success, cultural differences, technological changes, and resource allocation. The researcher distributed the complete survey via SurveyMonkey.com®, a secure online platform that supports anonymous participation and efficient data collection. This platform enables researchers to design content and analyze results objectively [68]. The researcher compiled all instruments into a single questionnaire and included it in the appendix. If any instruments were not publicly accessible, he obtained permission to use them.
3.5.1. Project Success
To measure project success, the researcher used a modified version of the multidimensional project success scale developed by Shenhar [69] as the dependent variable. This scale encompasses key performance indicators, including schedule adherence, budget compliance, technical performance, and stakeholder satisfaction. Respondents rated items on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). This scale is commonly used in Data Science and project management research and is well-suited to Big Data Science projects ([70] [71]).
3.5.2. Cultural Differences
To assess cultural differences, the researcher utilized the Cultural Intelligence Scale (CQS), developed by Ang [9]. The CQS evaluates four main components of cultural intelligence in cross-cultural interactions: metacognitive, cognitive, motivational, and behavioral. This 20-item tool was included in the SurveyMonkey questionnaire and scored on a 5-point Likert scale. Previous research has validated the instrument’s reliability and validity in international work environments ([72] [73]).
3.5.3. Technological Changes
To evaluate technological changes, the researcher used a revised version of the Technology Readiness Index 2.0 (TRI 2.0), developed by Parasuraman and Colby [74]. This 16-item scale measures individual attitudes toward technology adoption across four dimensions: Optimism, Innovativeness, Discomfort, and Insecurity. The researcher administered the scale through SurveyMonkey.com®, using a 5-point Likert response format. Günaltay [75] noted that TRI 2.0 demonstrates strong psychometric performance across different populations. The researcher selected this scale because it aligns with the research questions, demonstrates validity and reliability, and is suitable for an online survey format.
3.5.4. Resource Allocation
The researcher evaluated resource allocation using the Information Systems Resource Allocation Survey (ISRAS), a survey-based instrument grounded in established information systems and project management frameworks ([76] [77]). The ISRAS assesses the availability, equitable distribution, and alignment of key resources, including human capital, technology infrastructure, budget support, and executive sponsorship, with project objectives. Designed to be customizable for specific project needs, the ISRAS is well-suited to Big Data Science initiatives that often require flexible, scalable resource deployment across various parts of an organization [78]. The researcher administered the ISRAS via SurveyMonkey.com®, enabling participants to complete the survey anonymously and electronically. The ISRAS consists of approximately 15 items, each rated on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). Research indicates that the ISRAS is reliable and valid in IT Data Science project environments. Its adaptability has also been confirmed through studies exploring the impact of resources on agile, hybrid, and analytics-driven projects ([79] [80]).
3.6. Variable/Construct
3.6.1. Project Success
Project success is the dependent variable in this study. The researcher measured it using a modified version of the Project Success Scale, developed by Shenhar [69]. This scale has been widely used in research on IT and data science projects. It assesses four dimensions of success: meeting time and budget goals, achieving technical performance, stakeholder satisfaction, and long-term business impact. Participants responded to 12 questions on a 5-point Likert scale, from 1 (strongly disagree) to 5 (strongly agree). Prior studies documented the scale’s reliability, Ali [6]. Furthermore, Mangla [71] demonstrated the construct validity of this model in evaluating large-scale public sector IT projects. Its multidimensional nature makes it well-suited to assessing the outcomes of Big Data Science projects. The instrument is included in the appendix, and the researcher administered it via SurveyMonkey.com®.
3.6.2. Cultural Differences
Using the Cultural Intelligence Scale (CQS) developed by Ang [9], the researcher evaluated cultural differences. The CQS measures how effectively individuals can operate in diverse cultural environments. It includes 20 items assessing four dimensions of cultural intelligence: metacognitive, cognitive, motivational, and behavioral. Each item is rated on a 5-point scale, with higher scores indicating greater cultural intelligence. The CQS is widely used in cross-cultural research and has proven to be reliable [72]. Recent studies support the validity of this tool, demonstrating its ability to predict leadership effectiveness and cross-border team performance ([81] [82]). This tool is suitable for this study because Big Data Science projects often involve global teams, where cultural awareness plays a crucial role in collaboration and decision-making [38]. The researcher administered the instrument via SurveyMonkey.com® and included it in the appendix.
3.6.3. Technological Changes
The researcher assessed technological change using a modified version of the Technology Readiness Index 2.0 (TRI 2.0), developed by Parasuraman and Colby [74]. The TRI 2.0 includes 16 items across four subscales: optimism, innovativeness, discomfort, and insecurity. Participants rated their agreement with each item on a 5-point Likert scale. Higher scores in optimism and innovativeness, along with lower scores in discomfort and insecurity, indicate greater readiness for technological adoption. The TRI 2.0 has been commonly used in research on organizational technology adoption and digital transformation [75]. Parasuraman and Colby [74] validated the scale by demonstrating its significant correlation with actual adoption behavior and perceived ease of use. Since the organization’s technological landscape influences Big Data Science projects, TRI 2.0 offers a reliable and valid method for assessing participants’ views on the pace and impact of technological change [83]. The researcher administered this tool via SurveyMonkey.com®, and it is included in the appendix.
3.6.4. Resource Allocation
To evaluate resource allocation, the researcher used the Information Systems Resource Allocation Survey (ISRAS), an adapted instrument based on the DeLone and McLean [77] model. The ISRAS has 15 items that measure access to key organizational resources, including staff, infrastructure, funding, and executive sponsorship. Participants rated each item on a 5-point Likert scale, and the average scores reflect their perceptions of adequacy and fairness in resource distribution. Researchers have adapted this tool to suit Big Data Science environments, where projects often require support from multiple departments and have changing resource requirements [78]. Previous studies have shown the tool’s reliability [79]. The ISRAS draws on the Technology-Organization-Environment (TOE) framework and the IS success model, which strengthen its construct validity. This instrument is well-suited to addressing RQ3, which examines whether institutional support mechanisms, such as resource planning, influence project success. The researcher distributed the survey via SurveyMonkey.com®, and it is included in the appendix.
4. Results
The purpose of this quantitative, correlational study was to investigate the relationship between institutional factors, specifically cultural differences, technological changes, and resource allocation, and the success of Big Data Science projects. Despite the critical role of data science in the modern economy, high failure rates persist due to complex organizational and external barriers ([12] [15]). While researchers have extensively studied the technical aspects of Big Data Science, such as algorithm development and data engineering, limited empirical evidence exists regarding how institutional factors collectively influence project outcomes [13]. This study addressed this gap by examining three specific research questions: 1) How do technological changes influence the success of Big Data Science projects? 2) How do cultural differences influence the success of Big Data Science projects? Moreover, 3) How does resource allocation influence the success of Big Data Science projects?
4.1. Data Preparation and Screening
After collecting the data, the researchers systematically prepared it to guarantee its integrity and appropriateness for analysis. Once the survey was closed, the researchers downloaded the raw data from SurveyMonkey in SPSS format and imported it into IBM SPSS Statistics Version 29.0. The initial dataset included 103 responses.
4.1.1. Data Cleaning and Case Exclusion
The researchers examined all cases to verify completeness and data integrity. Incomplete survey responses led the researchers to exclude one participant, resulting in a final sample of 102. This number exceeds the minimum of 77 participants needed, as determined by an a priori power analysis. In no other cases were missing data or response issues present that required exclusion.
4.1.2. Variable Coding and Scoring
The researchers numerically coded all demographic variables for analysis; refer to Appendix C for the complete codebook. Likert-scale items ranged from 1 (Strongly Disagree) to 5 (Strongly Agree). The researchers calculated composite scores for each scale by averaging the responses: Cultural Differences (4 items), Technology Readiness and Change (4 items), Resource Allocation (3 items), and Project Success (3 items).
4.1.3. Data Screening and Assumptions Testing
The researchers examined the data for outliers, normality, linearity, homoscedasticity, and multicollinearity. Boxplots and z-scores revealed no extreme outliers (z > ±3.29). Skewness values ranged from −0.65 to −0.22, and kurtosis values ranged from −0.48 to 0.82, indicating acceptable normality [84]. Visual inspection of Q-Q plots and scatterplots confirmed that the assumptions of normality and linearity were satisfied. Variance Inflation Factor (VIF) values ranged from 1.12 to 1.58, well below the threshold of 10, indicating no multicollinearity concerns. Residual plots showed no evidence of heteroscedasticity. All assumptions for multiple linear regression were adequately satisfied. Consequently, the data analysis continued as outlined in Chapter 3 without any need to alter the research design or statistical methods.
4.2. Population and Sample
Following the pilot study, the researchers conducted the primary data collection phase using an online survey distributed through LinkedIn Big Data Science open groups. A total of 103 individuals initiated the survey; however, the researchers excluded one participant from the final analysis because the participant did not complete the questionnaire. This resulted in a final sample of N = 102 participants, which exceeded the minimum required sample size of 77 participants determined a priori via a power analysis using G*Power 3.1 (α = 0.05, power = 0.80, medium effect size, f2 = 0.15) for multiple linear regression with three predictors.
The achieved sample size of 102 participants provides adequate statistical power to detect medium effect sizes and reduces the likelihood of Type II errors, thereby supporting the validity and reliability of the statistical inferences made in this study. All participants met the inclusion criteria: they were at least 18 years old, had at least 1 year of professional experience working on Big Data Science projects in team-based or cross-functional environments, were fluent in English, and provided informed consent.
4.2.1. Sample Characteristics and Demographics
The demographic profile of the study participants offers key insights into the characteristics and backgrounds of Big Data Science professionals. This section analyzes the sample across four dimensions: age, gender, experience with Big Data Science projects, and organizational size. Understanding these demographics is crucial for assessing the sample’s representativeness and the applicability of the findings to the broader population.
The researcher collected demographic data through self-report survey questions on SurveyMonkey at the start of the survey. The researcher asked participants to indicate their age range, gender identity, years of experience with the Big Data Science project, and organization size. The researcher selected these variables for their relevance to the research questions and their potential influence on perceptions of cultural differences, technological changes, resource allocation, and project success. Previous studies show that factors such as career stage, gender diversity, experience, and organizational context can significantly impact data science project outcomes ([15] [21]). Table 1 provides a comprehensive summary of the demographic characteristics of the 102 participants, including frequencies and percentages for each category.
Table 1. Demographic characteristics of the participants.
|
|
N |
% |
Age range |
22 - 31 years |
33 |
32.40 |
32 - 41 years |
35 |
34.30 |
42 - 51 years |
24 |
23.50 |
52 - 61 years |
7 |
6.90 |
62+ years |
2 |
2.00 |
Missing/No Response |
1 |
0.90 |
Gender |
Male |
59 |
57.80 |
Female |
39 |
38.20 |
Prefer not to answer |
2 |
2.00 |
Missing/No Response |
2 |
2.00 |
Years of experience working on Big Data Science projects |
1 - 5 years |
73 |
71.60 |
6 - 10 years |
16 |
15.70 |
11 - 15 years |
6 |
5.90 |
16 - 20 years |
5 |
4.90 |
21+ years |
2 |
2.00 |
Approximate size of your organization (all employees) |
1 - 50 |
26 |
25.50 |
51 - 500 |
28 |
27.50 |
501 - 5000 |
24 |
23.50 |
5001+ |
24 |
23.50 |
Not all participants responded to every demographic question. One participant (1.0%) did not provide age data, and two participants (2.0%) did not provide gender data. These missing responses are minimal and do not affect the study’s validity, as all 102 participants completed the primary instruments assessing cultural differences, technological change, resource allocation, and project success.
1) Age Distribution
The sample’s age distribution indicates a predominantly early- to mid-career group engaged in Big Data Science projects, as depicted in Figure 1. The largest group consisted of individuals aged 32 - 41 years (n = 35, 34.3%), followed by those aged 22 - 31 years (n = 33, 32.4%). Together, these two groups accounted for approximately 66.7% of the total sample, underscoring the field’s youthful, evolving nature. Participants aged 42 - 51 comprised 23.5% of the sample (n = 24), reflecting mid-career professionals with broader organizational and project experience. Fewer individuals were in older groups: 6.9% aged 52 - 61 (n = 7) and 2.0% aged 62+ (n = 2). One participant (0.9%) did not provide age information. This aligns with industry trends, as Big Data Science is a recent, rapidly evolving field that mainly attracts younger professionals trained in modern data analytics, machine learning, and computational techniques.
Figure 1. Age distribution of big data science professionals.
2) Gender Composition
The sample was primarily male, with 57.8% (n = 59) identifying as male and 38.2% (n = 39) as female, as shown in Figure 2. A small fraction (2.0%, n = 2) chose not to share their gender. Additionally, two participants (2.0%) did not provide gender information. This gender breakdown reflects broader trends in the technology and Data Science fields, which have historically been male-dominated, although initiatives to enhance diversity and inclusion in STEM are ongoing [38].
3) Professional Experience with Big Data Science Projects
The sample displayed varied professional experience with Big Data Science projects, with most participants still in the early stages of their careers in this field, as shown in Figure 3. The largest group, comprising 71.6% of the sample (n = 73), reported having 1 - 5 years of experience in Big Data Science. This dominance of early-career professionals emphasizes the relatively recent emergence of Big Data Science as a specialized field and the rapid growth of data-driven roles in organizations over the past decade.
Figure 2. Gender distribution of big data science professionals.
Participants with 6 - 10 years of experience made up 15.7% of the sample (n = 16), while those with 11 - 15 years accounted for 5.9% (n = 6). A smaller group reported 16 - 20 years (4.9%, n = 5), and only 2.0% (n = 2) had 21 or more years of experience in Big Data Science projects. Although most participants were early-career professionals, all met the minimum requirement of at least one year of team-based Big Data Science project experience. This ensured they had sufficient familiarity with factors such as cultural differences, technological changes, and resource allocation, enabling them to offer informed perspectives on project success outcomes.
Figure 3. Distribution of professional experience with big data science projects.
4) Organizational Size and Context
Participants represented a diverse range of organizational sizes, reflecting the widespread adoption of Big Data Science initiatives across small, medium, and large enterprises, as shown in Figure 4. The largest group of participants worked in medium-sized organizations with 51 - 500 employees (27.5%, n = 28). About a quarter of the sample worked in small organizations with 1 - 50 employees (25.5%, n = 26), indicating that even smaller firms are investing in Big Data Science capabilities to stay competitive and data-driven.
The sample also featured a significant number of large enterprises. Participants from organizations with 501 - 5000 employees and those with over 5000 employees each accounted for 23.5% (n = 24) of the sample. This balanced distribution across various organization sizes enhances the external validity and relevance of the findings, as it represents the experiences of data science professionals from diverse institutional settings, resource levels, organizational structures, and project scales.
Figure 4. Distribution of participants across organizational sizes.
4.2.2. Comparison with a Priori Sample Size
As discussed in Chapter 3, a priori power analysis conducted with G*Power 3.1 indicated that at least 77 participants were needed to detect a medium effect size (f2 = 0.15) with 80% power at α = 0.05 for multiple linear regression with three predictors [62] [63]. The final sample included 102 participants, surpassing the minimum by 32.5%, thereby increasing statistical power and reducing the risk of a Type II error. Since the actual sample size exceeded the initially calculated minimum, the researcher did not need to change the research design or statistical methods, nor acknowledge additional limitations. The larger sample size enhances the accuracy of parameter estimates and broadens the relevance of the results to the broader population of Big Data Science professionals.
4.3. Descriptive Statistics
Before conducting hypothesis tests, the researcher calculated descriptive statistics for the four main scale variables: Cultural Differences, Technology Readiness and Change, Resource Allocation, and Project Success. These statistics offer an initial overview of the response central tendency, variability, and range, as summarized in Table 2.
Table 2. Statistical measurement of the study parameters.
Scale |
Mean |
Median |
SD |
Minimum |
Cultural Differences |
4.0539 |
4 |
0.58406 |
1.75 |
Technology Readiness and Change |
3.8775 |
4 |
0.63397 |
1 |
Resource Allocation |
3.6569 |
4 |
0.80182 |
1 |
Project Success |
3.9085 |
4 |
0.59915 |
2 |
4.3.1. Measures of Central Tendency
The mean scores for the four scales ranged from 3.66 (Resource Allocation) to 4.05 (Cultural Differences), indicating generally positive ratings across all constructs, as shown in Figure 5. The median score for each was 4.0, suggesting most participants agreed on cultural awareness, technological readiness, resource adequacy, and project success. Cultural Differences scored highest (M = 4.05, SD = 0.58), indicating that participants viewed their Big Data Science teams as highly culturally aware and adaptable, reflecting efforts to foster inclusive environments.
Figure 5. Mean scores for the four study variables.
Technology Readiness and Change had a mean of 3.88 (SD = 0.63), indicating moderate to high perceptions of technological preparedness; however, responses varied, highlighting some challenges in adapting to rapid technological changes. Resource Allocation scored lowest (M = 3.66, SD = 0.80), with variability suggesting uneven resource distribution across organizations, underscoring the need for strategic planning. Project Success averaged 3.91 (SD = 0.60), reflecting positive perceptions of recent project outcomes regarding schedule, stakeholder satisfaction, and goal achievement.
4.3.2. Variability and Range
Standard deviations ranged from 0.58 (Cultural Differences) to 0.80 (Resource Allocation), indicating moderate variability in participant responses (see Figure Descriptive Statistics). The higher standard deviation for Resource Allocation suggests that perceptions of resource adequacy vary more widely across organizations and projects than perceptions of cultural awareness and technological readiness.
The minimum and maximum values for each scale ranged from 1 to 5, aligning with the 5-point Likert scale used in the survey. Notably, the Cultural Differences scale had a minimum value of 1.75, which is higher than that of the other constructs. This indicates that even participants with the lowest scores demonstrated some degree of cultural awareness in their project environments.
4.4. Reliability Analysis
To evaluate the internal consistency and reliability of the multi-item scales in this study, the researcher calculated Cronbach’s alpha coefficients for four constructs: Cultural Factors, Technology Readiness and Change, Resource Allocation, and Project Success. Cronbach’s alpha is a standard measure of scale reliability, with acceptable values being 0.70 or higher for social science research [85].
Table 3 shows the results of the reliability analysis. All four scales demonstrated acceptable to high internal consistency, suggesting that the measurement tools effectively captured the intended constructs in the study sample. Resource Allocation had the highest reliability (α = 0.830), indicating strong internal consistency among the three items measuring perceived adequacy of skilled staff, technical infrastructure, and project budgets. Technology Readiness and Change (α = 0.763) and Project Success (α = 0.754) both showed good reliability, with coefficient values comfortably above the acceptable threshold. Cultural Factors (α = 0.719) also exhibited acceptable reliability, confirming that the four scales consistently measured a single underlying construct.
These reliability coefficients confirm the study’s internal validity and provide confidence that the scales were suitable for future hypothesis testing and regression analyses.
4.5. Data Collection
After obtaining IRB approval and completing the pilot study, the researcher collected the data via SurveyMonkey, distributed through LinkedIn’s Big Data Science professionals’ groups. Data collection took place over six weeks, during which 103 participants began the survey. Following data screening and cleaning, the researcher removed one incomplete response, yielding a final sample of N = 102 participants. This number exceeds the minimum required 77 participants based on a prior power analysis, ensuring sufficient statistical power (0.80) to identify medium effect sizes (f2 = 0.15) at α = 0.05 for multiple linear regression with three predictors [86].
Table 3. Statistical measurement of the study parameters.
Scale |
Cronbach’s Alpha |
N of Items |
Cultural Factors |
|
|
Technology Readiness and Change |
|
|
Resource Allocation |
|
|
Project Success |
|
|
4.6. Results of Research Question One
Research Question 1 (RQ1). How do technological changes influence the success of Big Data Science projects?
Null Hypothesis (H10). There is no relationship between project success (PS) and technological changes (TC).
Alternative Hypothesis (H1a). There is a relationship between project success (PS) and technological changes (TC).
To address Research Question 1, the researcher conducted a Pearson correlation analysis, followed by a linear regression, to examine the relationship between Technological Changes (independent variable) and Project Success (dependent variable). The researcher analyzed data from all 102 participants.
Pearson’s correlation coefficient indicated a statistically significant positive relationship between Technology Readiness and Change and Project Success. As shown in Table 4, there is a significant positive correlation (r = 0.392, p < 0.001), suggesting a moderate positive relationship between technological change and project success.
Table 4. Correlation analysis of project success and technological change.
|
r |
p |
N |
Project Success * Technology Readiness and Change |
0.392** |
<0.001 |
102 |
**Correlation is significant at the 0.01 level (2-tailed).
This relationship is also evident in the scatter plot of Technology Readiness and Project Success shown in Figure 6, where higher levels of Technology Readiness are generally associated with higher Project Success scores. Consequently, the researcher rejected the null hypothesis and concluded that technological changes have a significant effect on the success of Big Data Science projects.
Figure 6. Scatter plot of technology readiness and change vs. project success.
The researcher also tested this hypothesis using regression analysis in SPSS. The results in Table 5 show that technological changes significantly impact project success. The regression model is statistically significant (F (1, 100) = 18.111, p < 0.001) and explains 15.3% of the variance in project success (R2 = 0.153). This confirms that technology readiness and change have a strong positive impact on the success of Big Data science projects.
Table 5. Regression analysis of project success and technological change.
|
B |
SE |
β |
t |
p |
Constant |
2.474 |
0.342 |
|
7.241 |
<0.001 |
Technology Readiness and Change |
0.370 |
0.087 |
0.392 |
4.256 |
<0.001 |
F (1, 100) = 18.111, p < 0.001, R2 = 0.153.
Based on these findings, the researcher rejected the null hypothesis (H10). The data provide sufficient evidence to support the alternative hypothesis (H1a), indicating a statistically significant positive relationship between technological change and success in Big Data Science projects. The results imply that greater technological change is associated with higher perceptions of project success among Big Data Science professionals.
4.7. Results of Research Question Two
Research Question 2 (RQ2). How do cultural differences influence the success of Big Data Science projects?
Null Hypothesis (H20). There is no relationship between project success (PS) and cultural differences (CD).
Alternative Hypothesis (H2a). There is a relationship between project success (PS) and cultural differences (CD).
To answer Research Question 2, the researcher analyzed the relationship between Cultural Differences (independent variable) and Project Success (dependent variable) using Pearson’s correlation and linear regression. The study utilized data from all 102 participants in the final sample.
Pearson’s correlation coefficient revealed a statistically significant positive link between Cultural Differences and Project Success. As detailed in Table 6, this correlation is significant (r = 0.281, p = 0.004), indicating a small-to-moderate positive relationship.
Table 6. Correlation analysis of project success and cultural differences.
|
r |
p |
N |
Project Success * Cultural Differences |
0.281** |
0.004 |
102 |
**Correlation is significant at the 0.01 level (2-tailed).
The scatter plot of cultural differences and project success in Figure 7 also demonstrates this pattern, indicating that higher ratings of cultural differences generally correspond to higher levels of project success. Therefore, the analysis rejects the null hypothesis and concludes that cultural differences significantly impact the success of Big Data Science projects.
Figure 7. Scatter plot of cultural differences vs. project success.
The researcher examined the hypothesis that cultural differences influence project success by conducting regression analysis with the Statistical Package for the Social Sciences (SPSS). The results in Table 7 show that cultural differences significantly predict project success (B = 0.288, t = 2.924, p = 0.004). The regression model is also statistically significant (F (1, 100) = 8.551, p = 0.004) and explains 7.9% of the variance in project success (R2 = 0.079). This also confirms the correlation analysis results, which show that cultural differences have a moderate positive impact on the success of Big Data science projects.
Table 7. Regression analysis of project success and cultural differences.
|
B |
SE |
β |
t |
p |
Constant |
2.741 |
0.403 |
|
|
<0.001 |
Cultural Differences |
0.288 |
0.098 |
0.281 |
2.924 |
0.004 |
F (1, 100) = 8.551, p = 0.004, R2 = 0.079.
The researcher rejected the null hypothesis (H20), and the evidence supports the alternative hypothesis (H2a), confirming a statistically significant positive link between cultural differences and project success. These results suggest that higher cultural intelligence and better management of cultural diversity within Big Data Science teams are associated with greater project success.
4.8. Results of Research Question Three
Research Question 3 (RQ3). How does resource allocation influence the success of Big Data Science projects?
Null Hypothesis (H30). There is no relationship between project success (PS) and resource allocation (RA).
Alternative Hypothesis (H3a). There is a relationship between project success (PS) and resource allocation (RA).
To address Research Question 3, the researcher analyzed the relationship between Resource Allocation (independent variable) and Project Success (dependent variable) using correlation and regression analyses. All 102 participants provided complete data for both variables.
The Pearson correlation coefficient indicated a statistically significant positive relationship between Resource Allocation and Project Success. The data in Table 8 show that the correlation between project success and resource allocation is significant (r = 0.476, p < 0.001), reflecting a moderately positive relationship.
Table 8. Correlation analysis of project success and resource allocation.
|
r |
p |
N |
Project Success * Resource Allocation |
0.476** |
<0.001 |
|
**Correlation is significant at the 0.01 level (2-tailed).
The scatter plot in Figure 8, which shows resource allocation versus project success, further supports this, as the researcher observes that higher resource allocation tends to be associated with greater project success. Therefore, the researcher rejects the null hypothesis and concludes that resource allocation is a key factor in the success of Big Data Science projects.
Figure 8. Scatter plot of resource allocation vs. project success.
The researcher assessed the third hypothesis using a regression analysis in SPSS, which suggests that resource allocation affects project success. The results in Table 9 show that resource allocation significantly predicts project success (B = 0.355, t = 5.406, p < 0.001). The regression model is significant (F (1, 100) = 29.226, p < 0.001) and explains 22.6% of the variance in project success (R2 = 0.226). This confirms the correlation analysis results, showing that resource allocation has a significant positive impact on the success of Big Data Science projects.
Table 9. Regression analysis of project success and resource allocation.
|
B |
SE |
β |
t |
p |
Constant |
|
|
|
|
<0.001 |
Resource Allocation |
|
|
|
|
<0.001 |
F (1, 100) = 29.226, p = 0.001, R2 = 0.226.
Based on the statistical evidence, the researcher rejected the null hypothesis (H30). The data support the alternative hypothesis (H3a), showing a statistically significant positive relationship between resource allocation and project success. These results suggest that effective allocation of human, financial, and technological resources is associated with better outcomes in Big Data Science projects.
4.9. Limitations of the Study
The researcher acknowledges several limitations affecting the interpretation of the findings. First, data were self-reported via online surveys, which may introduce response biases, such as social desirability and recall bias. Although anonymity aimed to encourage honesty, the researcher could not independently verify participants’ perceptions of resource allocation, cultural dynamics, or project outcomes. Future research should incorporate objective performance metrics, archival data, or multi-source assessments to enhance validity through triangulation.
The sample primarily consisted of early- to mid-career professionals (ages 22 - 41, 66.7%) with limited experience in Big Data Science projects; 71.6% had 1 - 5 years of experience. This demographic might limit the relevance of the results for more seasoned practitioners, senior project managers, or data science leaders, who could have different perspectives on institutional factors and success. Future studies should involve participants with a broader range of experience levels to examine potential differences across career stages or organizational roles.
The cross-sectional design of this study captured data at a single point in time, which limits the ability to observe changes in project dynamics, technological evolution, or resource shifts over the course of project lifecycles. This temporal limitation precludes the establishment of causal relationships or the identification of trends in how institutional factors influence project success over time. Future research should consider longitudinal or repeated-measures designs to examine these dynamic relationships.
Another limitation is that using convenience sampling from LinkedIn Big Data Science groups may introduce selection bias, as active participants may differ from others in terms of technological readiness and organizational context. Future studies should consider random or targeted sampling across industries, organization types, or regions to enhance representativeness and external validity.
5. Conclusion
Whereas Chapter Four presented the quantitative results of this study, Chapter Five shifts focus to interpreting these findings, exploring their meaning, and discussing their implications for theory, practice, and future research. Big Data Science projects are essential for the growth and competitiveness of organizations in the modern data-driven economy, yet they continue to fail at alarmingly high rates. Research indicates that approximately 85% of these projects fail to deliver value due to unclear objectives, poor project management, and inadequate alignment between technical and business goals ([12] [18] [87]). These failures result in significant financial losses, missed opportunities, and diminished stakeholder confidence [7]. Although extensive research has focused on the technical aspects of Big Data Science projects, such as algorithm development and data engineering, there is limited empirical evidence on how external institutional factors influence project outcomes ([13] [88]). Chapter 4 presented quantitative findings from the correlational analysis examining these institutional factors; this chapter interprets those empirical results within the context of Attribution Theory, the CRISP-DM framework, and existing literature.
5.1. Findings
Quantitative analysis of data from 102 Big Data Science professionals revealed that all three institutional factors were positively and significantly associated with project success. The findings show that technological readiness and adaptability to change, cultural intelligence and diversity management, and strategic resource allocation significantly predict project outcomes in Big Data Science initiatives. These results empirically support the theoretical proposition that external institutional factors critically influence project success, extending beyond the technical capabilities and algorithmic sophistication traditionally emphasized in data science research. Table 10 summarizes the hypothesis-testing results, indicating whether each null hypothesis was rejected or the alternative hypothesis was accepted, based on the statistical analyses presented in Chapter 4. The researchers rejected all three null hypotheses, confirming statistically significant relationships between each institutional factor (technological changes, cultural differences, and resource allocation) and Big Data Science project success.
Table 10. Results hypotheses.
Hypotheses |
Results |
H01: There is no relationship between project success (PS) and technological changes (TC). |
Rejected |
H02: There is no relationship between project success (PS) and cultural differences (CD). |
Rejected |
H03: There is no relationship between project success (PS) and resource allocation (RA). |
Rejected |
HA1: There is a relationship between project success (PS) and technological changes (TC). |
Accepted |
HA2: There is a relationship between project success (PS) and cultural differences (CD). |
Accepted |
HA3: There is a relationship between project success (PS) and resource allocation (RA). |
Accepted |
Note: The table summarizes the acceptance or rejection of the null and alternative hypotheses based on the statistical analyses conducted.
While all three institutional factors demonstrated significant positive relationships with project success, the strength of these relationships varied considerably. Resource allocation exhibited the most substantial effect (β = 0.476, r = 0.475, p < 0.001), explaining 22.6% of the variance in project success, which indicates that adequately skilled staff, technical infrastructure, and project budgets constitute the most influential institutional factor examined in this study. Technological readiness and change showed the second-strongest relationship (β = 0.392, r = 0.391, p < 0.001), accounting for 15.3% of the variance in project success, suggesting that organizations’ capacity to keep pace with evolving data science tools, frameworks, and technologies significantly affects outcomes. Cultural differences demonstrated a significant but comparatively minor effect (β = 0.281, r = 0.281, p = 0.004), explaining 7.9% of the variance in project success. However, this is the weakest of the three factors; it remains statistically and practically significant, particularly for organizations with globally distributed or culturally diverse teams. These findings confirm that institutional and organizational dimensions are critical determinants of Big Data Science project success, with resource-related factors exerting the most significant influence on outcomes.
The findings support and expand upon the existing literature on project management factors in IT and data analytics contexts ([15] [21]). Moreover, this study is among the first to provide quantitative evidence linking cultural differences, technological change, and resource allocation to the success of Big Data Science projects within a unified empirical framework.
5.2. Research Questions
5.2.1. Discussion: Research Question One
Research Question One: How do technological changes influence the success of big data science projects?
The analysis revealed a statistically significant positive relationship between technological changes and project success (r = 0.392, p < 0.001), with technological readiness and adaptability accounting for 15.3% of the variance in project outcomes. The regression model demonstrated that technology readiness significantly predicts project success (B = 0.365, t = 4.247, p < 0.001; F (1, 100) = 18.037, p < 0.001). These findings support rejecting the null hypothesis (H01) and confirm the alternative hypothesis (HA1) that technological change is positively associated with project success in Big Data Science environments.
These findings align with the Attribution Theory, which suggests that team members’ perceptions of technological capabilities and challenges shape their behavior and collaboration [89]. When teams credit success to their technological readiness and adaptability, they are more likely to adopt innovative tools, experiment with new methodologies, and remain flexible amid changing technical needs. These findings also align with the CRISP-DM framework, which emphasizes continuous technological adaptation across the iterative phases of data science projects. The iterative nature of data science, as highlighted by the CRISP-DM framework, requires the ongoing integration of new tools throughout the project. Organizations with high technological readiness are better equipped to manage these cycles [90].
The findings align with recent empirical research highlighting the importance of technological adaptability in Data Science. Gureyev and Mazov [25] emphasized that practitioners must stay current with emerging trends, including machine learning frameworks, cloud computing platforms, and data visualization tools, to maintain project relevance and competitiveness. Pandurang [32] noted that the rapid adoption of artificial intelligence and cloud computing necessitates continuous organizational adaptation. A recent study by Bollineni [91] on DevOps (development and operations) practices in data science and MLOps (machine learning operations) highlights the importance of technological updates and flexibility.
Technology, adaptability, and the growing use of Big Data are crucial factors in the success of data science projects [31]. This psychological dimension of technological readiness has received limited attention in the Data Science literature, but emerges as consequential in this study. The findings imply that organizations should consider cultural attitudes toward technology adoption when managing technological change in data science. Rather than viewing technological change solely as a technical issue requiring infrastructure investment, the results highlight the importance of fostering a positive culture around technology. Organizations should establish formal governance mechanisms, such as cross-functional technology committees and structured adoption frameworks, to systematically evaluate, pilot, and deploy emerging tools while supporting a culture of innovation and experimentation. This involves offering training and ongoing learning opportunities [12].
5.2.2. Discussion: Research Question Two
Research Question Two: How do cultural differences influence the success of big data science projects?
The analysis identified a significant positive association between cultural differences and project success (r = 0.281, p = 0.004), with cultural intelligence accounting for 7.9% of the variance in project outcomes. The regression analysis indicated that cultural differences are a significant predictor of project success (B = 0.288, t = 2.924, p = 0.004; F (1, 100) = 8.551, p = 0.004). These results support rejecting the null hypothesis (H02) and affirm the alternative hypothesis (H12) that a positive relationship exists between cultural differences and project success.
These findings support Attribution Theory by illustrating how team members interpret cultural diversity [89]. From a cultural dynamics perspective, Attribution Theory explains how teams’ causal interpretations of cultural differences influence interpersonal responses, trust, and patterns of collaboration. When teams attribute project problems to cultural misunderstandings rather than individual skills, they shift the focus toward improving cross-cultural communication and fostering inclusive teamwork. Effective diversity management can turn cultural differences into a strategic asset, supporting the “value-in-diversity” idea that diverse teams enhance outcomes through creativity and multiple perspectives ([14] [15]). Research on data science projects confirms this, indicating that unmanaged cultural differences can cause communication failures and misaligned goals ([34] [92]). For example, when team members from different cultural backgrounds hold divergent expectations about decision-making authority or communication urgency, misalignment can manifest as delayed approvals, unaddressed conflicts, or siloed work that fragments the project effort if not explicitly managed through cross-cultural dialogue. Oguine [46] also highlights the strong effect of cultural diversity on team dynamics, communication, and decision-making, aligning with the CRISP-DM framework, which emphasizes stakeholder alignment and iterative collaboration while recognizing diverse communication and decision-making styles [93].
Organizations should view cultural diversity not merely as a demographic characteristic but as a strategic capability that requires active development. This involves implementing structured onboarding for diverse teams, establishing clear communication guidelines, providing language assistance as needed, and creating spaces for cultural exchange and shared learning.
5.2.3. Discussion: Research Question Three
Research Question Three: How does resource allocation influence the success of big data science projects?
The analysis revealed a statistically significant positive relationship between resource allocation and project success (r = 0.476, p < 0.001), with resource allocation explaining 22.6% of the variance in project outcomes, the strongest association among the three institutional factors examined. The regression model showed that resource allocation significantly predicts project success (B = 0.355, t = 5.406, p < 0.001; F (1, 100) = 29.226, p < 0.001). These findings support rejecting the null hypothesis (H03) and confirm the alternative hypothesis (HA3) that resource allocation is positively associated with project success.
The moderate-to-strong correlation (r = 0.476) indicates a threshold effect: minimum resource levels are essential for project viability; below this threshold, projects cannot succeed regardless of other strengths, whereas above it, the strategic distribution and timing of resources become as critical as the overall quantity of resources allocated. This substantial effect size indicates that the strategic allocation of human, financial, and technological resources is a crucial determinant of outcomes in Big Data Science projects. This pattern of threshold effects is consistent with findings in the IT project management literature, which similarly shows that adequate resource provision is a necessary but not sufficient condition for success, and that effective resource governance is the differentiating factor among well-resourced projects.
This finding supports the existing literature, which identifies resource allocation as an ongoing challenge in data science projects. Research by Bateni [76], Pratama [94], and Yang [49] highlights the need to strategically manage financial, human, and technological resources to achieve success. The study demonstrates that resource allocation accounts for more than 20% of the variation in project success. Consistent with Attribution Theory, teams with sufficient resources tend to focus on technical issues rather than on constraints [89]. Aligned with the CRISP-DM iterative methodology, achieving optimal results depends on strategically allocating resources across phases and adjusting them as project needs evolve. Dynamic resource management, which involves ongoing monitoring and reallocation, is more effective than fixed resource commitments. Furthermore, as Ambore [95] highlighted, the success of resource allocation becomes equally important as the number of resources used.
Organizations should adopt structured resource allocation processes aligned with the data science project phases. This involves establishing resource governance systems, conducting ongoing resource-needs assessments, introducing flexibility to support iterative development, and making resource allocation decisions that account for the entire project lifecycle rather than focusing solely on modeling and algorithm development.
5.3. Limitations of the Study
This study offers valuable insights into institutional factors that influence the success of Big Data Science projects, but it must also acknowledge several limitations. The researchers organize these limitations into methodological, sampling, and measurement categories. Collectively, they affect the interpretation, generalizability, and scope of the findings and highlight opportunities for future research. These limitations reflect design-informed constraints that shape how the findings should be interpreted rather than deficiencies in the study’s rigor.
5.4. Methodological Limitations
This study employed a cross-sectional research design, capturing institutional factors and project success at a single point in time. Although appropriate for identifying relationships among variables, this design provides a static view of inherently dynamic processes. Big Data Science projects evolve across multiple phases, during which technological readiness, resource allocation, and organizational culture may change substantially. As a result, the study cannot capture how these institutional factors interact over time or establish causal relationships. Accordingly, the findings should be interpreted as reflecting relationships observed at a specific point in time rather than as evidence of developmental or causal processes. Future research using longitudinal designs that follow projects from initiation through deployment would provide deeper insight into temporal dynamics and cause-and-effect relationships.
5.5. Measurement Limitations
This study relied exclusively on self-reported survey data, which introduces several measurement-related concerns. Social desirability bias may have influenced participants to report more favorable assessments of technological readiness, cultural intelligence, or resource adequacy than objective measures would reveal. Additionally, recall bias and retrospective memory bias may have affected participants’ ability to accurately report past project outcomes and resource conditions [15].
Although the study provided anonymity to encourage honest responses, the researchers could not independently verify the accuracy of the self-reported data. Moreover, key constructs, particularly project success, are inherently subjective and may be interpreted differently across organizational contexts, roles, and industries. These variations in interpretation may have influenced participants’ responses and introduced measurement inconsistency. Consequently, researchers should interpret these findings as subjective assessments influenced by organizational roles and contexts.
5.6. Control Variables: Justification and Recommendations
The current study collected relevant demographic data—including years of experience with Big Data Science projects (71.6% with 1 - 5 years), organizational size (ranging from 1 - 50 to 5000+ employees), age distribution (66.7% aged 22 - 41), and gender composition (57.8% male, 38.2% female)—but did not include these as control variables in the regression analyses. The exclusion of control variables can be justified on theoretical grounds: Attribution Theory and the CRISP-DM framework suggest that institutional factors (cultural differences, technological changes, resource allocation) operate at the organizational and project-team level and should influence project success regardless of individual practitioner characteristics, making these relationships theoretically robust across demographic subgroups. However, to strengthen the validity of findings and rule out alternative explanations, a hierarchical multiple regression approach is recommended, where: 1) Block 1 enters theoretically motivated controls—years of Big Data Science experience (to control for expertise effects), organizational size (as a proxy for resource availability and project complexity), and practitioner role if differentiated (e.g., data scientist vs. project manager, since managerial roles may perceive success differently)—and 2) Block 2 adds the three institutional factors (cultural differences, technological changes, resource allocation) to assess their incremental contribution beyond demographics. This approach would reveal whether the observed relationships (r = 0.281 to 0.476) represent genuine institutional effects or are confounded by experience levels (e.g., more experienced practitioners work in better-resourced organizations) or organizational context (e.g., larger organizations have both more resources and better technology adoption), with ΔR2 values indicating the unique variance explained by institutional factors after accounting for individual and organizational characteristics, thereby providing more conservative and interpretable effect size estimates.
5.7. Common-Method Bias Mitigation and Assessment
To mitigate common-method bias inherent in the single-survey, self-report design, several procedural remedies were implemented: 1) participant anonymity was ensured to reduce social desirability bias and encourage honest responses, 2) validated, psychometrically sound instruments with established reliability (Cronbach’s α ranging from 0.719 to 0.830) were used to enhance measurement quality, 3) predictor and criterion variables were measured using different scale formats and item structures (CQS with 20 items, TRI 2.0 with 16 items, ISRAS with 15 items, and Project Success Scale with 12 items) to reduce method overlap, and 4) clear instructions emphasized that there were no right or wrong answers to minimize evaluation apprehension. Statistically, Harman’s single-factor test should be conducted by entering all items from the four constructs into an exploratory factor analysis to determine if a single factor accounts for the majority of variance (>50% threshold); if a single factor does not emerge or accounts for less than 50% of total variance, this suggests common-method bias is not a pervasive concern. While these procedural and statistical controls reduce the likelihood of inflated correlations due to common-method variance, the self-report nature means that observed associations (r = 0.281 to 0.476) may still be partially attributable to shared method variance rather than purely substantive relationships, suggesting that the reported effect sizes should be interpreted as upper-bound estimates and that triangulation with objective performance metrics (e.g., actual budget/schedule variance, deployment success rates) in future research would strengthen causal inferences about how institutional factors influence Big Data Science project success.
5.8. Operationalization of Project Success
Project success in this study is operationalized using Shenhar’s [69] multidimensional scale, which assesses four key dimensions: meeting time and budget goals (delivery performance), achieving technical performance standards, stakeholder satisfaction, and long-term business impact. This operationalization captures both immediate project execution metrics (schedule/budget adherence) and broader strategic outcomes (business value and stakeholder perceptions), providing a comprehensive assessment of Big Data Science project effectiveness.
5.9. Justification for Self-Reported Measures
Self-reported success measures are appropriate for this research because 1) the study’s focus on institutional factors (cultural differences, technological changes, resource allocation) requires capturing practitioners’ lived experiences and perceptions of how these factors influence outcomes in their specific organizational contexts, and 2) Big Data Science professionals with direct project involvement are uniquely positioned to assess multidimensional success criteria that span technical, organizational, and stakeholder domains, which may not be fully captured by objective metrics alone. While self-reporting introduces potential biases (social desirability, recall bias), the use of validated instruments, anonymous data collection, and the correlational nature of the research questions make practitioner perceptions a valid and contextually rich data source for examining relationships between institutional factors and perceived project success.
5.10. Recommendations for Future Research
Future research should address the limitations of this study’s methodology and deepen the understanding of institutional factors through complementary approaches. Researchers using longitudinal designs can establish causal links by tracking Big Data Science projects from start to finish, including post-deployment phases. This approach can reveal how institutional factors evolve over the project lifecycle and identify key moments when they most significantly affect outcomes. Employing methods such as repeated measures, event history analysis, or survival analysis would overcome the current study’s correlational limitations and provide more robust causal insights [96].
Future studies should supplement self-reported survey data with objective performance indicators and organizational factor metrics. These success indicators, such as schedule and budget variances, model performance metrics, deployment success rates, and business impact data, can be sourced from project records. Institutional factors like diversity metrics, technology adoption rates, actual resource allocations, and communication patterns derived from organizational systems also provide valuable insights. Integrating both subjective and objective measures enables researchers to assess convergent validity and distinguish perceived from actual performance gaps [97].
Future research should employ mixed-methods designs that integrate quantitative measurement with qualitative exploration to provide richer insights into how institutional factors influence the success of Big Data Science projects [98]. While this study’s quantitative approach established the strength and significance of relationships among cultural differences, technological changes, and resource allocation, qualitative methods can illuminate the mechanisms, contextual nuances, and practitioner experiences underlying these statistical associations. Sequential explanatory designs, beginning with survey data and following with in-depth interviews or focus groups, could clarify how teams interpret and respond to resource constraints, why specific cultural configurations enhance collaboration, and how technological readiness manifests in daily project activities. Mixed-methods inquiry would also address measurement limitations inherent in self-reported surveys by triangulating perceptions with observational data, project documentation, and stakeholder interviews [99].
5.11. Current Analysis Approach
The study conducted three separate simple linear regressions (one predictor at a time) rather than a single multiple regression model with all three predictors entered simultaneously. Each research question was analyzed independently: technological changes (R2 = 0.153), cultural differences (R2 = 0.079), and resource allocation (R2 = 0.226) were each regressed on project success in separate models, which explains the unique variance for each predictor but does not account for shared variance or the unique contribution of each predictor when controlling for the others.
5.12. Recommendation for Alignment
To properly assess the unique contributions of institutional factors and address potential multicollinearity or shared variance among predictors (given that VIF values of 1.12 - 1.58 were reported), the study should conduct and report a single multiple regression model including all three predictors simultaneously. The full model should report: 1) overall model fit statistics (R2, adjusted R2, overall F-test with df and p-value), 2) standardized beta coefficients (β) for each predictor showing unique contributions while controlling for others, 3) 95% confidence intervals for each beta coefficient, 4) individual t-tests and p-values for each predictor’s unique effect, and 5) a comparison between R2 from individual models (0.153 + 0.079 + 0.226 = 0.458 total if independent) versus the multiple regression R2 (likely lower due to shared variance), which would clarify whether these institutional factors independently predict project success or if their effects overlap substantially.
5.13. Conclusions
Big Data Science projects represent critical strategic investments for organizations competing in data-driven economies, yet failure rates exceeding 85% underscore persistent challenges in delivering value from these initiatives. This quantitative, correlational study addressed a critical gap in the literature by empirically examining how institutional factors, cultural differences, technological changes, and resource allocation influence project success. Grounded in Attribution Theory and the CRISP-DM framework, the research provides evidence that these external factors significantly predict project outcomes beyond technical capabilities and algorithmic sophistication.
Analysis of data from 102 Big Data Science professionals showed significant positive correlations between all three institutional factors and project success. These results indicate a need for substantial changes in data science practices. Organizations should establish structured resource governance frameworks that strategically allocate personnel, budgets, and infrastructure throughout project lifecycles, rather than concentrating resources solely in the modeling phases. Technology readiness programs must address infrastructure, training, and psychological aspects of change, fostering optimism and innovation while mitigating discomfort and insecurity. Developing cultural intelligence should involve metacognitive, cognitive, motivational, and behavioral dimensions, supported by cross-cultural training, inclusive communication protocols, and psychological safety initiatives.