Assessment of Migrants’, Refugees’ and Asylum Seekers’ Hard Skills: Cultural Adaptation and Psychometric Properties of the NADINE Hard Skill Tests ()
1. Introduction
Since 2015, Europe experienced a major migrant/refugee crisis characterised by a dramatic increase of migrants and refugees crossing its borders seeking asylum (Clayton, 2015). The vast majority of the newcomers were fleeing war conflicts in Syria, Afghanistan, Libya or other areas that posed a major threat for their life and integrity (Cosgrave, Hargrave, Foresti, & Massa, 2016). Although the sense of emergency for the refugee crisis that was strongly felt in 2015-2019 has dissipated (Kushnir, Kilkey, & Strumia, 2020; Rankin, 2019), more than 2.5 million people have entered EU (Cosgrave et al., 2016) and they need to be integrated the European labour market. Furthermore, the EU is still recovering from the aftermaths of the 2008 financial crisis with some countries having more than 15% unemployment rate (Greece and Spain) (Statista, 2021). Moreover, the EU has to anticipate the impact of the United Kingdom’s withdrawal from the Union and more importantly the consequences of the current pandemic in the labour market. The International Monetary Fund has characterised the pandemic period as the “Great Lockdown” highlighting that its economic impact is expected to be analogous to the Great Depression and far graver than the 2008 financial crisis (Gopinath, 2020). This interaction between the refugee crisis and the other crises has put migrants, asylum seekers and refugees among the social groups that are most impacted by the pandemic and suffer disproportionately from its economic and psychosocial consequences (UN, 2020).
Work integration of migrants and refugees is essential for achieving social justice and it is recognised as a priority of the European Commission (EC, 2016, 2020) and the International Labour Organization (ILO, 2016). The European Commission has stated that assisting work integration of refugees is paramount for Europe’s social cohesion and highlighted the need for accurate skill assessment to ensure the best use of their skills and talents (EC, 2020). Furthermore, it is a prerequisite for achieving the objectives of the Global Compact on Refugees (UN, 2018) and the 2030 Agenda for Sustainable Development (UN, 2015). We should note that migrants/refugees face many additional difficulties in entering the labour market due to limited access to socioeconomic resources (Udayar, Fedrigo, Durante, Clot-Siegrist, & Masdonati, 2020). Being forced to arrive in a new country without prior knowledge of the language and the culture makes their transition very difficult and the need for work integration may become a major source of stress. Some of the most usual obstacles are the loss of previously held professional credentials, and professional status, and change from professional to service and manual labour (Desiderio, 2016). Career counselling can play a major role in assisting work integration, but requires a good understanding of this population’s unique and complex characteristics, and practical tools designed to address their needs.
The development of career assessment tools that are designed specifically for this population is crucial in order to provide career counselling services to vulnerable migrants, asylum seekers and refugees. The NADINE project, a three-year Horizon 2020 innovation project, aims at supporting migrant integration across Europe, and addresses the aforementioned issue by creating a series of tests designed to assess hard skills for migrants (the NADINE hard skills tests), refugees and asylum seekers in the framework of the broader scope to facilitate their labor market integration (see also Tountopoulou, Karaminas, Drosos, & Vlachaki, 2020). Employment and access to resources that facilitate work acquisition are considered as a prerequisite for smooth social inclusion of migrants and refugees in their new societies.
2. The NADINE Hard Skills Tests
2.1. Description and Theoretical Background
The NADINE Hard Skill Tests aim at assessing a vast variety of different skills in order to provide an accurate profile of each migrant/refugee/asylum seeker. Hard skills are strictly job-specific, closely connected with knowledge, easily observed, measured and trained. They constitute the core occupational requirements of a job (CEDEFOP, 2020) and mainly represent cognitive abilities. The “hard skills” constructs’ operationalization was the result of thorough literature review and focus groups with employers and migrants/refugees/asylum seekers. Various European and national frameworks on skills were used, such as ESCO skills classification, OECD competency frameworks, definitions and descriptions used at the Survey of Adult Skills (OECD PIAAC), the OECD Learning Framework 2030 (OECD, 2005, 2014, 2018a, 2018b), ILO competency standards (ILO, 2008, 2015, 2018), the EU Key Competencies Framework (2006, 2018), the EU Skills Profile Tool for Third Country Nationals, as well as international surveys and studies on skills needed to enable migrants labour market integration in the host countries (e.g. ILO, 2017, The Adecco Group, 2017a, 2017b).
Additionally, the development of the NADINE Hard Skill Tests was largely based on the Cattell-Horn-Carroll Theory of Cognitive Abilities. The Cattell-Horn-Carroll (CHC) theory is one of the most comprehensive and empirically supported theories regarding the structure of cognitive abilities to date. It represents the integrated works of Raymond Cattell, John Horn, and John Carroll (Alfonso, Flanagan, & Radwan, 2005; Horn & Blankson, 2005; McGrew, 2005; Schneider & McGrew, 2012). Raymond Cattell made a distinction between “fluid” and “crystallized” intelligence. He defined fluid intelligence as the ability to perceive relationships independent of previous specific practice or instruction concerning those relationships. Fluid intelligence involves being able to think and reason abstractly and solve problems. On the other hand, crystallized intelligence involves knowledge that comes from prior learning and past experiences and therefore is based upon facts, skills, and information learned in school or from past experience (Cattell, 1957, 1971). Horn expanded Catell’s theory into an eight-factor model which included: Fluid intelligence (Gf), Quantitative knowledge (Gq), Crystallised intelligence (Gc), Reading/Writing (Grw), Visual processing (Gv), Short-term memory (Gsm), Long-term retrieval (Glr), Processing speed (Gs), Decision speed (CDs) and Auditory processing (Ga) (Horn, 1991; Horn and Blankson, 2005). Finally, Carroll (1997) differentiated factors or abilities into three strata that varied according to the “relative variety and diversity of variables” included at each level. The various G abilities are the most prominent and recognized abilities of the model. They are classified as broad or stratum II abilities and include abilities such as Gf and Gc, the two original factors. Broad abilities represent “basic constitutional and long standing characteristics of individuals that can govern or influence a great variety of behaviors in a given domain”, while narrow abilities “represent greater specializations of abilities, often in quite specific ways that reflect the effects of experience and learning, or the adoption of particular strategies of performance”. The broadest or most general level of ability in the Gf-Gc model is represented by stratum III, located at the apex of Carroll’s (1993, 2005) hierarchy. An integrated Cattell-Horn and Carroll model was proposed by McGrew (1997), while several revisions of the model were made (McGrew & Flanagan, 1998; Flanagan et al., 2000), with the most recent the one being proposed by Schneider and McGrew (2012).
NADINE Tests’ development methodology follows the relevant and internationally used scientific statistical and analytical psychometric procedures for psychological testing (Nunnally, 1978, American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999, Society for Industrial and Organizational Psychology, 2003), so as to minimize measurement errors and guarantee the quality, suitability, and usefulness of the tool to be developed. Such standardized psychometric tools are widely used to objectively measure individuals’ mental abilities and skills. A standardized test requires all test takers to answer the same questions in the same way, and is scored in a “standard” or consistent manner, which makes it possible to compare the relative performance of individuals. One main consideration was the tests’ length. In the first stages we included a large number of questions having in mind that a significant number was likely to be rejected as inappropriate in the next stages. The NADINE tests’ first version included more than 10 to 48 items per test with the aim to reduce items for use in the final version after the piloting.
Items were developed taking into consideration the following: 1) thorough review of relevant literature to ensure content adequacy, 2) make sure items address a single issue, and 3) make sure statements are culturally appropriate. For each skill to be measured, a working group of experts was composed, consisted of 2 - 3 experts in the field, such as mathematicians, ICT experts, physicians, engineers, neurologists, so as to propose the appropriate items for each relevant field. Moreover, the group included 2 psychologists, 1 sociologist and an expert in psychometrics. The test development team specified an exhaustive list of potential indicators of each scale and the representative sample of indicators was selected from this list. Then, the team proceeded to the development of items that supposedly measure the selected constructs. Each member of the working team proposed several items measuring the construct under evaluation. The next step was to evaluate each of the proposed items in terms of representativeness and content redundancy. The process resulted in a shorter, but comprehensive enough pool of items, following the rule of including 2 or 3 times larger number of items (Devellis, 2017), with the aim to reduce those finally used after the item analysis/factor analysis. Dichotomous and Multiple choice questions are used for different skills evaluation. Dichotomous questions are used for the assessment of Spatial ability and of Accuracy, where the options/given answers are “Different - Same”. Multiple choice questions are used for all the other hard skills, with 4 or 5 options/given answers, so as to distinguish successfully those who are less capable in a field and reduce the chance of randomly choosing the right answer (25% or 20%).
Taking into consideration the aforementioned criteria the NADINE team developed 10 tests to evaluate the specific hard skills. Moreover, scores are provided for two combined scales as well. The tests are presented in Table 1.
In Figure 1 and Figure 2 are presented two examples of test items.
The present study has a two-folded aim: 1) to investigate whether the different language versions of the NADINE hard skill tests are linguistically and culturally equivalent and 2) to assess their psychometric properties. The procedures that we followed to investigate the aforementioned are described below.
![]()
Table 1. The NADINE hard skills tests.
2.2. Items’ Review by Expert Groups
Before the translation and adaptation of the tests the items had to undergo a process of review in order to assess the content validity. It is widely acknowledged that the most common and arguably useful form of review is the use of expert groups (DeMaio & Landreth, 2004; Presser & Blair, 1994; Willis, Schechter, & Whitaker, 2000). The expert groups that were assigned for the review of the NADINE skill assessment tools consisted of different experts, experienced in the field of the construct under measurement, such as experts in ICT, physics, mathematics, language learning, as well as a sociologist, a cognitive psychologist, a neurologist and an organizational psychologist. In any case, 4 to 5 persons, who were experts in each subject and 1 expert in test design participated in each expert focus group. The major concern of “subject matter experts” was to provide input about the initial pool of items regarding their relevance to the content domain, accuracy, & simplicity, and to evaluate potential item bias. The expert in test design was mainly concerned with assessing whether items conform to commonly accepted rules of good item design.
The items’ review procedure was as follows: each expert panel member was given a list of the items along with the content dimension they belong. They were asked to evaluate each item on a 4-point scale based on relevance, clarity, and simplicity as 1 (not relevant), 2 (somewhat relevant), 3 (relevant), or 4 (very relevant). Only items that scored a 3 or 4 were considered relevant and therefore used to calculate the actual Content Validity Index (CVI) (Polit & Beck, 2006). The formula of content validity ratio is I-CVR = Ne/N, in which the Ne is the number of panelists indicating “essential” and N is the total number of panelists, while for calculating S-CVI the sum of I-CVIs is divided by the total number of items. The higher score indicates further agreement of panel members on the necessity of an item in an instrument. The results from the experts’ panels in scale-level CVIs ranged from 0.95 to 0.99, and item-level CVIs ranged from 0.80 to 1.0, higher than 0.79 which is considered the cut-off point for needing revision. According to the review results, no items were deleted from the developed tools, while only minor revisions were suggested regarding the clarity or wording of the items, which were incorporated into the tools.
3. Linguistic and Cultural Adaptation
3.1. Translation and Adaptation of the Tests
The first phase of linguistic and cultural adaptation involved the translation of NADINE assessment tools from English to the target languages: Farsi, Arabic, French and Spanish. The second phase refers to the item review by target population, while the third phase refers to psychometric evaluation of the different language versions of the tests to ensure cultural equivalence.
First, in order to translate the NADINE questionnaires from English to Farsi, Arabic, French, Greek and Spanish, the working team combined the back-translation design (Hambleton, 2005; Sireci et al., 2006) and the Committee Approach as well (Harkness, Pennell, & Schoua-Glusberg, 2004), in order to benefit from both methods advantages. Particularly, the English version of the test was first translated into the target languages: Arabic, French, Farsi, and Spanish by two translators per language, working individually. The translators were fully proficient in both languages of interest and familiar with the cultures associated with the respective languages of each. Such characteristics enable the translation process to consider the nuances of the language for which the instrument is intended, which ensures a greater cultural fit of the adaptation process group (Hambleton, 2005). Translators were given the instruction not to pursue the literal translation of items because that often results in incomprehensible statements or rather limited target language fluency. Therefore, they tried to work towards a balanced treatment of linguistic, cultural, and contextual information, so as the people to whom the tools are targeting to be able to understand them in the same way (Tanzer, 2005; Kankaraš & Moors, 2010). To successfully adapt a test in another language, the translators have to ensure the use of concepts, words and expressions that are culturally equivalent in the second language. After the two translators finished the translations of items, a combined translation came out in a way that best represents the views of both of versions. The result of the synthesis of the translations was then given to two different translators, in order to perform the back translation of the items into English following the same instructions. The original English version of the hard-skills assessment tool was then compared to the produced English versions after the back translation in terms of meaning of each item. No revision was needed.
3.2. Cultural Relevance and Review by Migrants, Asylum Seekers and Refugees’ Groups
This next stage of the linguistic and cultural adaptation process aimed to verify whether the items, the response scale and the instructions are comprehensible and cultural relevant for the target population. Thus, this procedure aimed to investigate whether the instructions are clear, whether the terms found in the items are appropriate, whether the expressions correspond to those used by the group, and other aspects. For the process of the tests review by the target population, three focus groups were formed with 9, 9, 6 and 7 migrants, asylum seekers and refugees living in Greece. The first group with Farsi speakers, was comprised of 3 men and 6 women coming from Syria, Lebanon and Sudan, while the group of Arabic speakers included 5 men and 2 women from Iran and Afghanistan. The French speaking group consisted of 5 men and 4 women from Congo and Ivory Coast, while for the Spanish speakers 3 were women and 3 men from Venezuela and Mexico. All participants in the focus groups were selected in cooperation with NGOs in Greece working with migrant groups. The responsible persons in the NGOs were informing their “clients” on the process and the scope of it and those who expressed their interest in participating in the focus groups. The process of review was assisted each time by a culture mediator speaking Arabic, Farsi, French of Spanish, so as to facilitate the exchange of ideas and feedback on the tests items.
To obtain rich information about comprehension and cultural relevance of the translated items semi-structured cognitive interviewing was conducted (Oremus et al., 2005). The group discussion focused on clarifying the meaning and relevance of the items’ content to the target culture. The semi-structured cognitive interviewing technique was applied for all items comprising the assessment tools. The set of questions guiding the group discussion elicit: 1) comprehension of the item content (i.e. What does this item mean to you and people of your community?), and 2) cultural relevance of the item content (i.e. Do you and people in your community believe and/or experience what this item reflects? Is the content of the item offensive to people of your community? What words in your language can be used to express the same idea or indicator as in this item?). Additionally, for each item the participants were asked to rate in a 10-point scale the comprehensiveness (comprehension index) and the cultural relevance (cultural relevance index). When a given item was not clear to the participants, the facilitators were encouraged to provide synonyms that best exemplify the vocabulary of the target group.
The review process aimed at ensuring that all items are culturally relevant and clear and at identifying items that require modification to enhance their relevance. Items that are deemed irrelevant or unclear would be omitted. The rule for accepting, modifying or omitting an items was the following: the items that are rated as clear to understand (i.e., comprehension index > 5) and culturally relevant to the target culture (i.e., cultural relevance index ≥ 78%; Polit & Beck, 2006) are maintained in the tool, item modification is necessary if the item was rated as not easy to understand (i.e., comprehension index < 5) but relevant (i.e., cultural relevance index ≥ 78%) and exclusion of items are foreseen if they are rated as not easy to understand (i.e., comprehension index < 5) and not relevant to the target culture (i.e., cultural relevance index ≤ 78%) (McGorry, 2000). The review resulted in just minor changes in the wording of the instructions of the tests and of some items in numeracy and mechanical skills.
3.3. Preliminary Assessment of Psychometric Properties
To assess the psychometric properties of the items we proceeded in calculating the difficulty level of each item and its contribution to the scale’s internal reliability coefficient. The sample consisted of 120 asylum seekers, vulnerable migrants and refugees currently residing in refugee structures in Greece: 41 participants were given the Farsi version of the tests, 41 participants the Arabic version and 36 participants the French version. With respect to gender participants 66 (55%) were male and 54 (45%) female. Regarding age 31.7% was 17 - 25, 40% was 26 - 40 and 28.3% was 41+ years old. Moreover, 30.8% had attended only primary education, 36.7% had finished secondary education and 28.3% had a bachelor degree. The participants were selected again in cooperation with NGOs in Greece working with migrant groups. The NGOs having in mind that we need representativeness of the different groups and variables (refugees, vulnerable migrants, asylum seekers speaking the three different languages) chose among their users, those who could provide us with the appropriate characteristics for the process.
We used the Farsi, Arabic and French versions of the NADINE Hard Skill Tests. Dichotomous and Multiple choice questions are used for different skills evaluation. Dichotomous questions are used for the assessment of Spatial ability and of Accuracy, where the options/given answers are “Different-Same”. Multiple choice questions are used for all the other hard skills, with 4 or 5 options/given answers. An impromptu questionnaire consisted of 5 items was used to assess demographic information. The scale was administrated in the refugee camps by the researchers or by camp staff who has been trained by the researchers. Participation was voluntary and this was made clear to participants who gave their written consent. They were asked to contribute in a study about developing valid and reliable career assessment instruments by filling out the research tests. They were instructed to answer all questions that they could.
We calculated item frequencies and descriptive measures to assess whether the items have adequate difficulty level to distinguish between individuals with high levels of the construct being measured from those with lower levels. Moreover, each item’s contribution in internal reliability coefficient’s of the corresponding scale. Thirty-nine items seemed to have extremely low or high difficulty level or reduced the reliability coefficients and were eliminated. The remaining items had difficulty level between 0.20 and 0.80 (with few exceptions), while all reliability coefficients were above 0.60; and the vast majority above 0.70. The study resulted in revising the NADINE Hard Skill Tests, and the items of each test were put in order according to their difficulty level beginning from the easiest and resulting to the most difficult.
4. Ensuring Cultural Equivalence
A major concern when translating and adapting a test in other languages is cultural equivalence, which is necessary for scores obtained by different language versions of the test to provide comparable scores. Van de Vijver and Leung (1997) refer to structural equivalence, wherein correlations between variables are identical in the different groups. In other words, structural equivalence shows that the same construct is measured across cultures despite using different versions of the same test. Therefore, a significant stage in adapting a new test is performing statistical analysis to determine construct equivalence showing that scores of the adapted versions measure the same construct. Therefore, we proceeded with the check of cross-cultural equivalency.
Subjects
The sample consisted of 427 participants (141 vulnerable migrants, 149 asylum seekers and 137 refugees) currently residing in Greece and Spain: 117 participants were given the Farsi version of the tests, 112 participants the Arabic version, 115 participants the French version and 83 the Spanish version. With respect to gender participants 217 were (50.8%) male and 210 (49.2%) female. Regarding age 29% was 17 - 25, 41% was 26 - 40 and 30% was 41+ years old. Moreover, 30.2% had attended only primary education, 36.8% had finished secondary education and 33% had a bachelor degree. The engagement of participants was made through NGOs working with migrants in Greece and Spain, who were responsible for the organization of the testing administration.
Method & Materials
We used the Farsi, Arabic, French and Spanish versions of the NADINE Hard Skills Tests. The number of items per test is presented in Table 2.
The scales were administrated in the NGOs’ premises by the researchers and trained facilitators. Participation was voluntary and this was made clear to participants who gave their written consent. As mentioned before dichotomous questions were used for the assessment of Spatial ability and of Accuracy, where the options/given answers are “Different-Same”, and multiple choice questions were used for all the other hard skills, with 4 or 5 options/given answers.
Results
The equivalence of the structure in different cultures was checked by means of calculations of Tucker’s phi, which is the congruence index of two sets of factor solutions (van de Vijver & Leung, 1997). Exploratory factor analyses were conducted to categorize the scale’s items into larger homogeneous groups, in order to better describe the internal structure of the participants’ responses. In Table 3 we present the Factor analyses results for the ICT Skill Test for the three language groups, as an example of the procedure we followed. Principal Component Analysis was used as extraction method with a fixed number of factors. In this example, the final solution (having as the cut point loading the 0.25) revealed the one dimension we expected for all groups. For these analyses we had the following results: Farsi version: ΚΜΟ = 0.88, χ2 for the Bartlett’s test of sphericity = 455.23 (df = 66, p < 0.001); Arabic version: ΚΜΟ = 0.81, χ2 for the Bartlett’s test of sphericity = 384.64 (df = 66, p < 0.001); French version: ΚΜΟ = 0.87, χ2 for the Bartlett’s test of sphericity = 531.11 (df = 66, p < 0.001) and Spanish version: ΚΜΟ = 0.82, χ2 for the Bartlett’s test of sphericity = 301.28 (df = 66, p < 0.001). All tucker’s phi coefficients were above the cut-point of 0.90, which indicate cultural equivalency of the four versions of the tests.
![]()
Table 2. Number of items per hard skills’ test.
![]()
Table 3. Factor analyses (PCA) for the 12 items of the “NADINE” ICT Skills Test according to the language used.
Based on the factor analyses, the item loadings and the tucker’s phi coefficients we can support structural equivalence among the different language versions of the NADINE Hard Skill Tests. In some tests the tucker’s phi was below 0.90 but above 0.80 which indicate similarity. Twenty-four items in total showed poor psychometric properties and were omitted to ensure better results (numeracy test: 2 items, accuracy test: 12 items, mechanical test: 2 items, phonetic spelling: 1 item, spatial ability test: 7 items).
5. Discussion
The present paper aimed at presenting both the NADINE Hard Skill Tests for refugees, migrants and asylum seekers, and how we ensured the linguistic and cultural adaptation for all different language versions of the Tests. We investigated the adaptation procedures, and the structural equivalence of the ten tests. Prior to translating and checking the tests on a sample of each of the target groups, the generated items were subjected to a process of review from expert groups, so as to ensure the content validity of the tests. Multiple methods were used during the translation/adaptation process such as back-translation and the committee approach. Moreover, three focus groups were formed with Farsi-speaking, French-speaking and Arabic-speaking migrants, asylum seekers and refugees. Each scale’s items were further reviewed by the three focus groups. This process aimed at assessing the tests’ content and at examining whether the items, the response scale and the instructions are comprehensible and culturally relevant for the target population. The review of the tests resulted in minor changes, while participants agreed that the items were culturally relevant and clear. Thus, the different languages’ versions of the NADINE Tests are comprehensible and culturally relevant for the different categories of migrants, asylum seekers and refugees (Arabic, Farsi, Spanish and French speaking).
The psychometric properties of the tests were further investigated in two subsequent studies with 120 and 427 participants respectively. Our findings supported both the existence of one dimension in each test and the cross-cultural equivalency of all Tests’ versions. We examined the test items’ difficulty level, the reliability and the structure of each test and cross-cultural equivalency. The last study showed that 39 items had poor psychometric properties and were therefore eliminated, resulting in a shorter and more coherent test. Structural equivalence among different language versions of the tests was examined by calculating Tucker’s congruence coefficient for each one of the Tests on a sample of 427 subjects. There was strong evidence for cross-cultural equivalency, while all reliability coefficients were above 0.70. The shorter tests are easier and less time-consuming for the target population, while there is no information loss.
Therefore, the different language versions of the NADINE Hard Skill tests are linguistically and culturally equivalent, and constitute valid and reliable instruments to measure migrants’, refugees’ and asylum seekers’ skills.
Work integration is a key ingredient for social inclusion of migrants and refugees. A comprehensive career guidance system is essential to assist them not only in accessing the labour market, but more importantly in acquiring self-knowledge and constructing their careers in their new societies. Understanding their skills and their career interests is essential in order for them to design their future careers and make long-term career plans, considering the initial work placement as the first step in their new career. Without effective career guidance services, public policy measures to integrate migrants at individual or societal level are likely to fail. The NADINE tests are user-friendly tools for employability and career guidance that provide an accurate assessment of migrants’ and refugees’ skills. This allows guidance professionals to offer a more personalised and contextualized service to migrants based on real needs and the opportunities available to them. The NADINE hard skill tests can have multiple functions: 1) they are a useful tool for career practitioners to assess their clients’ skills; 2) they can facilitate the selection of skillful employees by employers and organizations and 3) contribute in designing trainings for upskilling that correspond to the needs of migrants/refugees/asylum seekers. It can be used as a complimentary tool to other EU tools, such as the Labour-Int (http://www.labour-int.eu/) that has developed tools to assess the qualifications and the skills that asylum-seekers’ and refugees’ have acquired via their job experience.
Acknowledgements
This article is part of the NADINE project, which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 822601.