The NADINE Soft Skills Tests for Migrants, Refugees and Asylum Seekers: Cultural Adaptation and Preliminary Testing of Their Psychometric Properties ()
1. Introduction
In recent years, Europe faces a major migrant/refugee crisis. Although migration is not a new phenomenon, 2015 saw a rise in the number of refugees and migrants coming to Europe seeking asylum. The wars in Syria, Libya, and other areas have forced hundreds of thousands to seek a better future in the countries of the European Union. Migrants and refugees will have to be integrated into the European labour market, and career counselling practitioners will have to play a major role in this. Migrants/refugees may come to Europe with the hope of a better future for their families and themselves, but they usually have limited access to economic and occupational resources. They arrive in a new place with a different culture, without knowing the language; and they have to build lives, establish careers, and find ways to adapt. This is a difficult transition during which vocational obstacles become a major source of stress. Usual problems often include unemployment, underemployment, loss of previously held professional credentials and professional status, change from professional to service and manual labour, extended work hours, and lengthy periods of time away from families. Career counselling with migrants requires understanding of this population’s unique and complex circumstances, and a set of practical tools specifically tailored to address issues of immigrant career development.
The questions that naturally arise are related to the tools and the amount of knowledge that we have in order to provide career counselling services to migrants and refugees. We live in times of major socio-economic changes that have put in question the most dominant career counselling theories, not only for migrants but for mainstream population as well, and have opened the way to new theories which will address the new challenges.
The NADINE project addresses the aforementioned challenges. The project consortium comprises 13 partners from France, Spain, the UK, Belgium, Italy, Luxembourg and Greece. Smooth access to employment is considered to be a key ingredient in the successful integration of migrants and refugees in their new societies. Furthermore, concrete assessment of their skills is a prerequisite for providing successful career counselling and guidance services. The present paper aims at introducing the NADINE Soft Skills Tests, which were created via the NADINE project, and at assessing the linguistic and cultural adaptation and equivalency of the French, Farsi, Arabic and English versions of the Tests.
2. The NADINE Soft Skills Tests
The NADINE Soft Skills Tests’ scope is to facilitate the production of a concrete and realistic profile of each migrant’s/refugee’s/asylum seeker’s specific soft skills. These tests can have multiple functions: 1) career practitioners will be able to provide better services to migrants/refugees/asylum seekers, 2) employers will be facilitated in selecting appropriate employees according to their needs, and 3) educational institutions will be facilitated in designing training programs that correspond to the needs of migrants/refugees/asylum seekers. Soft skills, which are non-job specific, are traversing jobs and sectors, closely connected with intangible personal attitudes and competences as well as social competences (CEDEFOP’s Skills Panorama Glossary, 2020). Regarding the construct definitions of the soft skills, the constructs’ operationalization was based on extensive literature review and focus groups with employers and migrant groups. European and national frameworks on skills were used, such as ESCO skills classification, OECD competency frameworks, definitions and descriptions used at the Survey of Adult Skills (OECD PIAAC), the OECD Learning Framework 2030 (OECD, 2005, 2014, 2018), ILO competency standards (ILO, 2008, 2015), the European Commission (2018), the EU Skills Profile Tool for Third Country Nationals, as well as international surveys and studies on skills needed to enable migrants labour market integration in the host countries (e.g. ILO, 2017; The Adecco Group, 2017a, 2017b etc.).
NADINE Soft Skills Tests’ development methodology follows the relevant and internationally used scientific statistical and analytical psychometric procedures for psychological testing (Nunnally, 1978; American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999; Society for Industrial and Organizational Psychology, 2003), so as to minimize measurement errors and guarantee the quality, suitability, and usefulness of the tool to be developed. The development of the tests consists of several stages that will be implemented in 3 phases. The present study aims at presenting the results of the initial phase, including the linguistic and cultural adaptation process, and the preliminary investigation of the tests’ psychometric properties.
Test length is an important consideration for any study, and especially critical for large-scale assessments which need to be as efficient in the use of testing time as possible. The number of items and therefore the size of each NADINE psychometric test was determined based on the need to include as many questions as possible, given that a number of them are likely to be rejected as unsuitable in the stages of pilot-testing that will follow. Furthermore, in the case of personality traits (soft skills), short self-report scales represent the most efficient form of assessment and are also easiest to administer. Shorter scales reduce the burden on respondents and thus potentially improve quality of data. However, shorter scales may have worse measurement properties than longer scales, as having fewer questions might decrease their reliability and validity (Kankaraš, 2017). Therefore, NADINE attempts to have good psychometric properties with as few items as possible. The NADINE tests’ first version included more than 10 items per main scale and 6 per sub-scale, aiming at reducing the number of items used in the final version after pilot-testing.
Items were developed taking into consideration the following: 1) thoroughly reviewing all relevant literature to ensure content adequacy, 2) making sure that all items address a single issue, and 3) making certain that all statements are simple and familiar to target respondents and are culturally appropriate. For each skill to be measured, a working group of experts was formed, consisting of 2 psychologists, 1 sociologist, and an expert in psychometrics. Each member of the working team proposed several items measuring the construct under evaluation. The next step was to evaluate each of the proposed items in terms of representativeness and content redundancy. The process resulted in a shorter but comprehensive enough pool of items. Items used in psychological research tests are closed-ended thus better allowing the generated data to be analyzed (Coolican, 2014; Furr, 2011). A 5 pointer Likert Scale Multiple Choice Questions type was used, being the most appropriate in personality trait evaluation. The rating scale is from 1 - 5, where 1 = totally disagree and 5 = totally agree. The item development guidelines for Likert type items proposed by Irwing & Hughes (2018) were followed.
Taking into consideration the aforementioned criteria, the NADINE team developed 4 scales to evaluate the 19 specific soft skills:
· Interpersonal skills: Social interaction, Team working, Intercultural competence, Coping with Authority, Conscientiousness, Extraversion (item examples: “I believe it is always better to cooperate with others rather than to compete with them”, “I can easily resolve arguments with others”, “I find it enjoyable to team up with others and help them” etc.).
· Organizational skills: Work efficiency, Time management, Independent Work, Organization (item examples: “I dedicate a substantial amount of time to checking a project, before I consider it completed”, “I leave things for the last minute”, “When carrying out a task, I always try to stick to the deadlines” etc.).
· Personal skills: Adaptability, Reliability, Willingness to learn, Stress tolerance (item examples: “I tend to feel uneasy when things change around me”, “I can manage to control the way I react, even when I am emotionally tense”, “I do not usually lose my patience” etc.).
· Entrepreneurial skills: Creativity, Managerial skills, Initiative, Risk tolerance, Leadership (Item examples: “I tend to come up with new ideas and proposals”, “I am in constant search for new incentives”, “You can’t win if you don’t risk” etc.).
3. Linguistic and Cultural Adaptation
Prior to translating and pre-testing on a sample of migrant groups, the generated items were subjected to a review process, so as to evaluate the content validity of the tests. It is a widely accepted fact that the most common and arguably useful form of review is the use of expert groups (DeMaio & Landreth, 2004; Presser & Blair, 1994; Willis, Schechter, & Whitaker, 2000). The expert groups that were assigned to review the NADINE skill assessment tools consisted of various experts, such as psychologists (cognitive, organizational, etc.), psychometricists, as well as a sociologist and a neurologist. The test design expert was mainly concerned with assessing whether items conform to commonly accepted rules of good item design.
The item review procedure was as follows: each expert panel member was given a list of the items along with the content dimension to which they belonged. They were then asked to evaluate each item on relevance, clarity, and simplicity on a 4-point scale [ex. 1 (not relevant), 2 (somewhat relevant), 3 (relevant), or 4 (very relevant)]. Only items that scored a 3 or 4 on relevance were considered suitable and therefore used to calculate the actual Content Validity Index (CVI) (Polit & Beck, 2006). The formula for content validity ratio is I-CVR = Ne/N, where Ne is the number of panellists indicating “essential” and N is the total number of panellists, while for calculating S-CVI the sum of I-CVIs is divided by the total number of items. The higher the score, the more panel members agree on the inclusion of an item in an instrument. Expert panel results concerning scale-level CVIs ranged from 0.95 to 0.99, while item-level CVIs ranged from 0.80 to 1.0, both higher than 0.79 which is considered the cut-off point for revision. According to the review results, no items were deleted from the developed tools, while only minor revisions were suggested regarding the clarity or wording of the items, which were incorporated into the tools.
The first phase of linguistic and cultural adaptation involved the translation of NADINE assessment tools from English to the target languages: Farsi, Arabic & French. The second phase refers to preliminary statistical and psychometric evaluation of the translated versions of the NADINE soft skill assessment tools in order to verify the linguistic and the conceptual equivalence between the different versions of the test (English and Farsi, Arabic, French), while the third phase refers to the item review by the target population.
First, in order to translate the NADINE tests from English to Farsi, Arabic and French, the working team combined the back-translation design (Hambleton, 2005; Sireci et al., 2006) and the Committee Approach as well (Harkness, Pennell, & Schoua-Glusberg, 2004), in order to benefit from the advantages that both methods have to offer. Particularly, the English version of the test was first translated into the target languages: Arabic, French, and Farsi by two translators per language, working individually. The translators were fully proficient in both of the languages concerned and familiar with the cultural aspects associated with these languages. Such characteristics enable the translation process to consider the nuances of the language for which the instrument is intended, which ensures a greater cultural fit of the adaptation process group (Hambleton, 2005). Translators were instructed to avoid the literal translation of items because that often results in incomprehensible statements or rather limited target language fluency. Therefore, they tried to work keeping a balance among linguistic, cultural, and contextual information, so that the people whom the tools are targeting will interpret them in the same way (Tanzer, 2005; Kankaraš & Moors, 2010). Following the two separate translations, a final combined version came out which best represented both of the original translations. This final version was then given to two different translators in order to perform the back translation of the items into English, following the same instructions. The original English version of the soft-skills assessment tool was then compared to the produced English back translation versions in terms of meaning of each item. In the French version, there were no differences on the meaning of any item, while in the case of Arabic 4 items needed rephrasing and in the Farsi version 5 items also needed revision.
3.1. Study 1
The next step for the language-cultural adaptation of the soft skills tests was the preliminary statistical evaluation of the translated versions. In order to statistically compare the translated versions of the NADINE soft-skills tests with their respective original versions, on linguistic, conceptual equivalence and cultural adaptation terms, a number of bilingual subjects were given both the source and target versions of the test. Half of them first completed the English and then the translated versions, while the other half were given the tests in the opposite order, in order to control for order effect. The interval between the two administrations was approximately 4 weeks. The general criterion, in order to ensure equivalence, was that the difference in scores between the two versions should not be statistically significant.
Subjects
In the case of the French version, 30 individuals agreed to participate in the study as subjects filling both versions of the NADINE soft-skills tests. Of the participants who spoke both French and English, 13 (43%) were male and 17 (57%) were female, while 18 (60%) were of medium and 12 (40%) of high educational level. In the case of persons speaking both English and Farsi or Arabic, the number of persons engaged in the process was lower due to difficulties encountered in finding the required number of bilingual speakers. Specifically, 15 persons speaking both English and Arabic and 12 persons speaking both English and Farsi took part in the process. The first group consisted of 9 (60%) men and 6 (40%) women, where 8 (53%) were of medium and 7 (47%) of high educational level. The English-Farsi speaking group consisted of 7 (58%) men and 5 (42%) women, 62% of whom were of medium and 48% of high educational level.
Method & Materials
The English, French, Farsi and Arabic versions of the NADINE Soft Skills Tests utilize a 5-point Multiple Choice Likert Scale (1 = totally disagree to 5 = totally agree) and consist of 4 scales: Interpersonal skills, Organizational skills, Personal skills and Entrepreneurial skills.
Results
The t-test was used to check for statistical significance in the difference between subject responses to individual items on the two versions of each test. In order to ensure conceptual equivalence between the two languages, we hypothesized that no statistically significant difference should be found. According to the analysis, there were very few items that revealed significant difference in the different versions. All the differences (either in terms of the means or in terms of the standard deviations) between the two versions of the tests, in all language versions, were very small, a result which allows us to argue (at least to a first descriptive level) that there is a relative uniformity in the subjects’ responses in both languages, for all test versions.
Significant differences were found only in a few cases between the English-Farsi and English-Arabic versions. Eight items of the Arabic version and 9 items of the Farsi version were subsequently revised.
3.2. Study 2
This next stage of the linguistic and cultural adaptation process aimed to verify whether the items, the response scale and the instructions are clear, comprehensible and culturally relevant to the target population. Culturally relevant means that the items are appropriate and the expressions used correspond to those used by the group.
Subjects
For the process of the tests review by the target population, three focus groups were formed with 9, 9 and 7 migrants, asylum seekers and refugees living in Greece. The first group with comprised 3 men and 6 women, all Farsi speakers, from Syria, Lebanon and Sudan, while the group of Arabic speakers included 5 men and 2 women from Iran and Afghanistan. The French speaking group consisted of 5 men and 4 women from Congo and Ivory Coast. A culture mediator speaking Arabic, French or Farsi, was present during the review process so as to facilitate the exchange of ideas and feedback on the test items.
Method & Materials
To obtain further qualitative information on clarity and cultural relevance of the translated items, semi-structured cognitive interviews were conducted (Oremus et al., 2005). The group discussion focused on clarifying the meaning and relevance of the items’ content to the target culture. The purpose was to identify: 1) items that are acceptable to the target culture; 2) items that require modification to enhance their relevance to the target culture. The semi-structured cognitive interviewing technique was applied for all items comprising the assessment tools. The set of questions guiding the group discussion elicit: 1) comprehension of the item content (i.e. What does this item mean to you and people of your community?), and 2) cultural relevance of the item content (i.e. Do you and people in your community believe and/or experience what this item reflects? Is the content of the item offensive to people of your community? What words in your language can be used to express the same idea or indicator as in this item?). Additionally, for each item, the participants were asked to rate in a 10-point scale the comprehensibility (comprehension index) and the cultural relevance (cultural relevance index). When a given item was not clear to the participants, the facilitators were encouraged to provide synonyms that best exemplify the vocabulary of the target group.
The review process was oriented toward making final decisions about accepting items deemed culturally relevant and clear, modifying items to enhance their relevance, omitting culturally irrelevant and unclear items. The rule for accepting, modifying or omitting an item was the following: the items that are rated as clear to understand (i.e., comprehension index > 5) and culturally relevant to the target culture (i.e., cultural relevance index ≥ 78%; Polit & Beck, 2006) are maintained in the tool, item modification is necessary if the item was rated as not easy to understand (i.e., comprehension index < 5) but relevant (i.e., cultural relevance index ≥ 78%) and exclusion of items are foreseen if they are rated as not easy to understand (i.e., comprehension index < 5) and not relevant to the target culture (i.e., cultural relevance index ≤ 78%) (McGorry, 2000).
Results
The review resulted in a few minor changes in the instructions. Additionally, in the Farsi version, one item was revised.
3.3. Study 3
To assess the psychometric properties of the items we proceeded in calculating the discriminative power of each item and its contribution to the scale’s and subscale’s internal reliability coefficient. Moreover, we calculated Tucker’s congruence coefficient for each Soft Skill Tests’ subscales to ensure cross-cultural equivalency.
Subjects
The sample consisted of 120 asylum seekers and refugees currently residing in refugee structures in Greece: 41 participants were given the Farsi version of the tests, 41 participants the Arabic version and 36 participants the French version. With respect to gender participants 66 were (55%) male and 54 (45%) female. Regarding age 31.7% was 17 - 25, 40% was 26 - 40 and 28.3% was 41+ years old. Moreover, 30.8% had attended only primary education, 36.7% had finished secondary education and 28.3% had a bachelor degree. None of them was employed at the time of the study.
Method & Materials
We used the Farsi, Arabic and French versions of the NADINE Soft Skills Tests, which consist of 131 items in total. A 5-point Likert Scale Multiple Choice Questions was used (1 = totally disagree and 5 = totally agree). The scale was designed to assess four distinct soft skills, namely: Interpersonal skills, Organizational skills, Personal skills and Entrepreneurial skills. A simple 5 item questionnaire was used to collect demographic information.
The scale was administrated in the NGOs’ premises by the researchers and trained facilitators. Participation was voluntary and this was made clear to participants who gave their written consent. They were asked to contribute in a study about developing valid and reliable career assessment instruments by taking the NADINE Skills Tests. They were instructed to answer all questions by clicking on the response of their choice on a 5-point Likert scale for each item. No name was asked for during testing in order to assure anonymity.
Results
Descriptive measures pertaining to response frequencies per item were calculated and used to assess each item’s discriminative power (Price, 2017). Moreover, each item’s contribution to the corresponding subscale’s internal reliability coefficient was determined. Fifty items were subsequently eliminated. The remaining items had discriminative power between 0.20 and 0.80, while all reliability coefficients were above 0.70; the vast majority being above 0.80.
Although the sample size was rather small, we decided to proceed with checking for cross-cultural equivalency. The equivalence of content interpretation by different cultures was checked by means of Tucker’s phi which is the congruence index of two sets of factor solutions (van de Vijver & Leung, 1997). Exploratory factor analyses were conducted to categorize each scale’s items into larger homogeneous clusters, in order to better describe the internal structure of the subjects’ responses.
In Table 1, we present the Factor analyses results for the Entrepreneurial Skill Scale for the three language groups, as an example of the procedure we followed. Principal Component Analysis with Varimax rotation was used as the extraction method.
In this example, the final solution (with a cut-off point loading of 0.45) revealed the five (5) expected factors for all groups. Results were as follows: Farsi version: ΚΜΟ = 0.67, χ2 for the Bartlett’s test of sphericity = 969.17 (df = 171, p < 0.001); Arabic version: ΚΜΟ = 0.74, χ2 for the Bartlett’s test of sphericity = 865.32 (df = 171, p < 0.001); French version: ΚΜΟ = 0.59, χ2 for the Bartlett’s test of sphericity = 543.95 (df = 171, p < 0.001).
All tucker’s phi coefficients were above the 0.90 cut-off point, with the exception of the Farsi-French and the Arabic-French comparison of the first component (Initiative). Nonetheless, Tucker’s phi was above 0.80 which shows similarity. Moreover, despite the modest sample, we had high loadings that reveal the same items comprising subscales in all 3 language groups. Therefore, we decided to keep the items and proceed to future studies with larger samples.
Based on the content of item loadings, the five components (considered to be subscales of the original scales) that we expected were derived for all groups. These subscales can be labelled as follows. Item examples are offered for each one.
![]()
Table 1. Factor analyses (PCA) for the 19 items of the “NADINE” Entrepreneurial Skills Scale according to the language used.
*Reverse scored.
· Subscale 1, labelled Initiative, comprises 5 items that represent the subject’s willingness to take initiatives to achieve his/her goals.
· Subscale 2, labelled Creativity consists of 4 items that assess the subject’s tendency to generate ideas, alternatives, or possibilities that may be useful in solving problems.
· Subscale 3, labelled Risk tolerance, comprises 4 items that assess the subject’s willingness to take risks in order to achieve his/her goals.
· Subscale 4, labelled Leadership Skills, comprises 3 items that assess the ability to communicate effectively with other team members, to address their needs and to inspire them to work for the common cause.
· Subscale 5, labelled Managerial Skills, consists of 3 items that assess the subject’s willingness to succeed and to manage other people.
4. Discussion
It is an accepted fact that career counsellors should consider a range of factors, besides psychometric quality, when selecting an assessment tool for use. One of the main factors is the target-population. The current refugee crisis has contributed to the complexity of career assessment due to the fact that the target-population is characterized by vast diversity both in terms of culture and language. As standardized assessments are frequently used for provision of effective intervention, it is important that assessment instruments are psychometrically solid and culturally suitable. The NADINE Soft Skills Tests have been developed to address the current needs for culturally appropriate, reliable and valid tools for refugees, migrants and asylum seekers. These tools have been designed to produce assessments which are appropriate to the needs of the target-population, career counsellors and potential employers. The NADINE Soft Skills Tests are fundamentally based on the idea of matching careers to talents, skills and personality, initially derived from the Trait and Factor Theory of Occupational Choice developed by Frank Parsons. The theory assumes that it is essential for career counselling to assist people to gain a clear and accurate work related self-profile, by measuring their skills so that they can be matched to best fitted occupations, in order to achieve best performance and productivity (Parsons, 1909). However, the NADINE tests do not aim at an oversimplified matching of skills and occupations, but at gaining a better self-understanding to assist people in crafting their own careers. The matching approach, the Big Five Factor Model, and John L. Holland’s theory of Career Choice, have been the conceptual frameworks mainly employed in the design of the NADINE psychometric instruments for skills assessment.
The present paper examined the results of three different studies regarding the cultural and linguistic adaptation of a series of Soft-Skills tests that were developed via the NADINE project. The NADINE Soft Skills Tests for refugees, migrants and asylum seekers were developed in English and adapted in three language versions: French, Arabic and Farsi. We investigated the adaptation procedures, and the items’ and scales’ structural equivalence. The findings confirmed the cross-cultural equivalency of all 3 of the language versions.
Prior to translating and checking the tests on a sample of each of the target groups, the generated items were subjected to a process of review from expert groups, so as to ensure the content validity of the tests. To translate the tests, the working team combined the back-translation design and the Committee Approach to benefit from advantages offered by both methods. This resulted in the revision of two items. From then on, the first study evaluated the linguistic, conceptual equivalence and cultural adaptation of the translated versions of NADINE Soft Skills Tests with their original version using bilingual subjects. Subjects were administered both tests versions with an interval of 4 weeks between the administrations. Differences between the two language versions of the tests were very small or non-existent. Therefore, we may argue that there is a relative uniformity in the subjects’ responses in both languages and that the items are conceptually equivalent. These findings suggest that the different languages’ versions of NADINE Tests are conceptually equivalent and reliable.
The second study aimed at verifying whether the items, the response scale and the instructions are comprehensible and culturally relevant for the target population. Three focus groups were formed with Farsi-speaking, French-speaking and Arabic-speaking migrants, asylum seekers and refugees. The review of the tests resulted in minor changes, while participants agreed that the items were culturally relevant and clear. Thus, the different languages’ versions of the NADINE Tests are comprehensible and culturally relevant for the different categories of migrants, asylum seekers and refugees (Arabic, Farsi and French speaking).
Finally, the third study aimed at examining the test items’ discriminative power, checking the reliability of each subscale and ensuring cross-cultural equivalency by calculating Tucker’s congruence coefficient for each one of the Soft Skills Tests subscales on a sample of 120 subjects. Fifty items showed poor discriminative power or reduced the reliability coefficients and were therefore eliminated, resulting in a shorter and more coherent instrument. The remaining items had good discriminative power, while all reliability coefficients were above 0.70; the vast majority being above 0.80. Furthermore, there was strong evidence for cross-cultural equivalency. The third study resulted in shorter scales with all items having good discriminative power and contributing to the scale’s reliability. The shorter tests are easier and less time-consuming for the target population, while there is no information loss.
Therefore, these studies suggest that the different language versions of the NADINE Soft Skill tests are linguistically and culturally equivalent. Moreover they constitute valid and reliable instruments to measure interpersonal skills, organizational skills, personal skills, and entrepreneurial skills. The tests are a useful asset for career practitioners to assess the level of soft skills of migrants, refugees and asylum seekers and design the most appropriate interventions for their work integration. Furthermore, potential employers can make use of the tests to assess the skills of candidates for a job.
Nonetheless, our studies have limitations. Sample sizes were rather modest as they had to be split among three different versions of the tests. Further research is required to ensure the reliability and validity of the tests with larger samples. The next phase of the NADINE pilot-testing will include standardization on a significantly larger sample of refugees, asylum seekers and migrants.