Going Global: 39 Language Versions of the BFI-2-XS

Assessing the Big Five in the Survey of Adult Skills (PIAAC)

The Programme for the International Assessment of Adult Competencies (PIAAC) is an established international comparative large-scale study program initiated and run by the Organisation for Economic Co-operation and Development (OECD). Its main product, the Survey of Adult Skills, assesses key cognitive skills such as literacy and numeracy in the adult population, allowing for analyses of differences in the level of skills within and between countries as well as the predictors and outcomes of skills.

Complementing its in-depth assessment of adults’ (cognitive) competencies, in its current second cycle (2022/2023), PIAAC has, for the first time, included social-emotional skills as one of several background variables in its extensive background questionnaire. For this purpose, it was decided to employ a measure assessing the Big Five dimensions of personality, which are often used as proxies of social-emotional skills (Lechner, Anger, & Rammstedt, 2019). To investigate which Big Five instrument best fit its needs, the OECD convened an expert panel and conducted comprehensive national and international pilot studies in which the full BFI-2 was assessed. Results of these pilots suggested that the variance explained in numerous outcomes of interest to PIAAC was highly comparable for the 60-item BFI-2, the 30-item BFI-S, and the 15-item BFI-XS (Rammstedt, Lechner, & Danner, 2024). Because of time constraints in the administration of the background questionnaire, the OECD initially decided to assess only the two dimensions of Conscientiousness and Openness with six items each from the BFI-2-S in the PIAAC Field Trial. Nonetheless, twenty-four countries opted to administer the full 30-items BFI-2-S in the Field Trial. For the Main Study, the decision was taken to cover all dimensions, but to use the shorter BFI-2-XS instrument to save time. National project teams of the participating countries were given the option to administer the additional items from the BFI-2-S if they wished. Out of the 28 countries that opted to participate in the Big Five assessment in the Main Study, 12 decided to test the full 30-item BFI-2-S in the Main Study. In total, the instruments were translated into 23 languages and adapted to 29 countries, resulting in 39 language versions.

Aims of the Present Study

The aims of the present study are to make these language versions of the BFI-2-S and BFI-2-XS available to a wider research audience and to present the general state-of-the-art translation and adaptation procedure followed in PIAAC. In the Results section, we will describe the major decisions taken and adaptations made in the multi-step development of the final BFI-2-XS fielded in the Main Study of PIAAC. In doing so, we will adopt a national or, where necessary, a language-by-country perspective. Details of the adaptation steps by country and item are documented in an online appendix on the Open Science Framework (OSF, see Roemer et al., 2024). Finally, and most importantly, we also share the resulting final translated BFI-2-XS versions (and BFI-2-S versions if available) to enable their use in the broader scientific community. Although OECD guidelines regarding data confidentiality preclude the publication of empirical results from the PIAAC Field Trial, widely sharing these language versions of the BFI-2-S and BFI-2-XS, as well as the procedures used to develop and validate them, will facilitate both within-country and international research on personality traits and social-emotional skills in a broad range of languages and cultures around the world.

Countries and Language Regions

The social-emotional skills module, which comprised the BFI-2-XS, was an international option of the PIAAC 2023 background questionnaire. This means that countries could decide whether to include it or not. Of the 31 participating countries, only three—Korea, Japan and the United States—opted not to do so. In Korea, however, the BFI-2-S instrument was administered during the Field Trial and subsequent changes were implemented, resulting in a validated language version (even though it was not fielded in the Main Study). The Korean adaptation of the BFI-2-S thus fully followed the process described below and is therefore included in the present documentation.

The 29 countries for which the social-emotional skills module for PIAAC 2023 was developed are listed in Table 1. In eight of these countries, the module was administered in multiple languages (e.g., Spain, where regional languages such as Basque, Catalan and Galician are used in addition to Spanish). For the different language groups within these countries, separate instruments were prepared. Some countries (e.g., Germany, Austria, and Switzerland) share an official language. However, as the spoken language differs slightly between these countries, different language versions were prepared, albeit from a common starting point. This resulted in a total of 39 different language versions of the social-emotional skills module in PIAAC 2023.

Source Instrument

The source instrument for the social-emotional skills module in PIAAC 2023 was the Anglo-American original version of the BFI-2-XS (Soto & John, 2017b), which consists of 15 short-phrase items—three per Big Five domain. These items are answered on a 5-point rating scale ranging from disagree strongly (1) to agree strongly (5); the neutral category is labeled “neutral; no opinion” (3).

To facilitate translation and comparability, all 15 items of the BFI-2-XS were adapted slightly to form full sentences rather than phrases (see Appendix A in the Supplementary Materials, Roemer et al., 2024). The formulation of the response categories was also adapted slightly to fit the format typically applied in PIAAC. The adapted labels were "strongly disagree," "disagree," "neither agree nor disagree," "agree," and "strongly agree."

Translation Procedure

State-of-the-art translation procedures were applied for all instruments assessed in PIAAC. The specific procedure applied for the translation of the social-emotional skills module deviated from that applied for the other measures administered in PIAAC, in that the Big Five module was translated centrally by the PIAAC Consortium rather than by teams of the individual participating countries. The intention was to take account of the sensitivity of the Big Five items to even slight meaning shifts and to ensure a monitored double translation and reconciliation design for these items. In preparation for the translations, the PIAAC Consortium (together with Christopher J. Soto, the first author of the BFI-2-XS) compiled a list of existing translations of the BFI-2-XS. These existing (and often already validated) versions (see Table 2) were used as a source for the national adaptations of the BFI-2-XS to be used in PIAAC. Further, a list of one or more content experts for the Big Five in each of the 29 countries participating in the Big Five assessment was compiled, many of whom were authors of existing BFI-2 translations or at least similar personality inventories. These experts were later contacted to serve as expert reviewers for any new translations into the target languages.

Translation Notes

To ensure item comparability across language versions, and to prevent mistranslations, translation and adaptation notes (also known as item-specific guidelines or translation annotations) were provided for the BFI-2-S. These notes describe what specific words or phrases mean in measurement terms so that translators can transfer this meaning correctly without having to adhere too closely to the wording and structure of the source instrument. For the present purpose, the translation notes for the BFI-2 provided by Soto and John (personal communication, November 14, 2018) were used and expanded by cApStAn. For instance, the item “I am compassionate, have a soft heart” was accompanied by the note: “‘Has a soft heart’ means ‘is caring and compassionate’.” The adapted translation notes for the BFI-2-S items are provided in Appendix B (see Roemer et al., 2024).

Initial Translated BFI-2-S Versions

In a first step, it was checked whether a translation of the BFI-2-S into the target language already existed (for a full list of translations used see Table 2). This could be a version used in a previous OECD pilot study (see Rammstedt, Roemer, & Lechner, 2024) or a version from an independent translation project (e.g., for Germany, Rammstedt et al., 2020). These translated versions were then used as the basis for the corresponding national adaptations. The respective translators, who were selected, trained, and supervised by cApStAn on behalf of the PIAAC Consortium, reviewed these preexisting translations and suggested edits if needed—for example, if the existing item translation did not conform to the translation notes. Revisions to the adaptations made to the BFI-2-S source version used for PIAAC were also carried out at this stage.

If no translation of the BFI-2-S into the target language previously existed, the PIAAC Consortium produced translations of the items following a double translation and reconciliation approach (Lyberg et al., 2021), a slightly modified version of the TRAPD² procedure (Harkness, 2003). Specifically, two independent translations into the target language were produced by professional translators with extensive experience in translating surveys and psychological assessments. These two versions were then reconciled into one translation by a senior questionnaire translator who merged them by (a) selecting the best components from each version; (b) selecting one version over the other; or, very rarely, (c) proposing a new version in case neither of the provided versions was deemed satisfactory. Problematic issues were discussed and resolved in a subsequent meeting between the initial translators and the reconciler. For all corresponding countries, scholars from the above-mentioned list of domain experts for the Big Five were then contacted. Whenever possible, the resulting translations were reviewed by these experts.

In the case of countries sharing an official language (e.g., German for Germany, Austria, and Switzerland), one language version (either preexisting or newly translated) served as the starting point for all countries and subsequently underwent further country-specific adaptation).

Regardless of the translation approach (use of an existing translation version or translation from scratch), national teams were asked to review the translations provided by the PIAAC Consortium and request changes if problems were identified. These requests were reviewed by independent verifiers (linguists trained to identify potential equivalence issues in translated or adapted questionnaires) commissioned by cApStAn, and in some cases also by further members of the Consortium (e.g., domain experts). This step was called verification. Based on this feedback, the initial translated versions were finalized and assessed in the PIAAC Field Trial.

All steps in the process from translation to finalization of the instrument were rigorously documented, including the different translation versions at each step and additional comments by translators, verifiers, country teams, and domain experts if translation challenges or problems arose.

Final National Versions of the BFI-2-XS

Data from the PIAAC Field Trial were analyzed centrally by the PIAAC Consortium.³ For the BFI-2-XS, the analyses focused on descriptive statistics and distributions of responses as well as dimensionality, scale reliability, and validity of the five domains. Besides overall inspection of predictive validity for several criteria, confirmatory factor analysis was used to inspect within each country the model fit for each dimension and for a joint 5-dimensional measurement model. In addition, multi-group confirmatory factor analysis was applied to inspect the cross-national functioning of the measurement of the five domains.

Country teams received a common international report, as well as country-specific national reports. Based on an alignment procedure within multi-group confirmatory factor analysis, countries could identify any flagged severe national deviations from the international results. Based on these findings, country teams were asked to review their respective item translations for potential translation biases.

Country teams could then request revisions to their national BFI-2-S/BFI-2-XS versions from the PIAAC Consortium. Requests had to be limited to major issues, such as mistranslations that caused errors in the Field Trial responses, preferential changes or minor (inconsistency) requests were not approved. The final BFI-2-XS/BFI-2-S versions administered in the PIAAC Main Study were thus either the initial versions used in the Field Trial or the slightly modified versions incorporating country team feedback based on the Field Trial results. The general translation process is summarized in Figure 1.

Click to enlarge

Figure 1

Translation and Adaptation Process for the BFI-2-S/BFI-2-XS Followed in PIAAC

Development of the Initial BFI-2-S/BFI-2-XS Language Versions

For 15 of the 24 languages in which the PIAAC social-emotional skills module was administered, BFI-2-S/BFI-2-XS versions already existed, some which had already been validated (see Table 2). These existing language versions were consulted and adapted for use in PIAAC. For all these country versions, the adaptations applied to the source version (see above) were implemented. In addition, the PIAAC Consortium sometimes also changed item wording. These adaptations were often minor, for example, adding the feminine gender (as done in Czech) or correcting typing errors. However, in some cases, for example in the Spanish and Slovakian adaptations, items were reformulated to better represent the construct in the target language, align with translation notes, or capture common language use.

For nine languages (or 12 language versions), there were no preexisting BFI-2-S adaptations. For these, the PIAAC Consortium conducted translations and, where possible, had them reviewed by Big Five experts in the respective countries.

Review of the Initial BFI-2-S/BFI-2-XS Language Versions (Before the Field Trial)

All language versions were thoroughly reviewed by the respective national teams in the verification phase prior to the Field Trial, and changes were requested when needed. These requests were then discussed within the PIAAC Consortium and implemented where appropriate.

For the purpose of this study, we coded the requests for changes—and thus the errors in the existing translations identified by the national centers—according to a customized MQM (Multidimensional Quality Metrics) error typology, focusing only on relevant top error categories. MQM is a widely used framework for systematic translation quality assessment (https://themqm.org/). We differentiated the following error types: (a) accuracy errors (i.e., meaning-related errors); (b) accuracy errors as flagged in the Field Trial; (c) linguistic convention errors related to the linguistic correctness of the text (spelling, grammar, punctuation, etc.); (d) errors of style that reflect an inappropriate language use (awkward or unidiomatic text, etc.); (e) errors related to survey-specific terminology that reflect inappropriate wording of response scales; (f) gender errors (i.e., missing or inconsistent gendering); and (g) other (e.g., changes to the English source wording or to the wording used in another country). In Appendix D, we provide further details on our coding approach.

In general, change requests were typically approved, with the exception of three requests that were considered preferential and therefore not implemented. Some countries also requested that the labeling of the response scale options be modified. These cases are documented in Appendix C. In the following, we will therefore concentrate on requests for changes to item formulations, which were also approved and implemented.

The number of requested and implemented changes before the Field Trial classified into the various error types are shown in the left-hand panel of Figure 2. As can be seen in the left panel of Figure 2, across all country versions, 49% (n = 37) of the requested and implemented changes to item formulations during the verification phase prior to the Field Trial related to adaptations of item gendering. Twenty-four percent (n = 18 requests) of the changes were due to linguistic conventions, and 21% (n = 16 requests) were due to accuracy of the item formulations. As an example of the latter, the French team requested that the preexisting French BFI-2 adaptation of the original item “I can be cold and uncaring” should be changed. It was originally translated as “Je suis parfois dédaigneux/euse, méprisant(e)” [I am sometimes disdainful and contemptuous]. The national center argued that “dédaigneux/euse" and "méprisant(e)" were stronger terms in French than "cold" and "uncaring" in the source item. This would have compromised the comparability of the translation with the other language versions and could have resulted in comparatively lower item means. The French team therefore requested that the item should be reformulated to "Je suis parfois indifférent(e), insensible" [I am sometimes indifferent, insensitive]. Averaged across all country versions, 10% of the BFI-2-S items were changed before the Field Trial, with changes per country version ranging between 0% and 80% of items.

Click to enlarge

Figure 2

Revisions to the Translations of the BFI-2-XS Before the Field Trial and the Main Study by Multidimensional Quality Metrics (MQM) Category

Testing the BFI-2-S/BFI-2-XS Language Versions in the Field Trial

The finalized BFI-2-S versions (or in some cases only two of the five trait domains) were then administered in a large-scale Field Trial in each country (see Footnote 1) with a total sample size of nearly 30,000 respondents. The resulting data were analyzed centrally by the PIAAC Consortium as described above. Based on these analyses, countries received detailed feedback on the psychometric performance of the national BFI-2-S adaptations. The psychometric evaluation comprised inter alia descriptive statistics on both the scale and the item level, scale reliability, exploratory and confirmatory factor analyses, and measurement invariance analyses across countries. Through these analyses, divergences from the international results were flagged, and country teams were asked to inspect these cases especially.

Review of the BFI-2-S/BFI-2-XS Language Versions Tested in the Field Trial

Based on this psychometric feedback, country teams examined the item formulations, looked for specific causes of item misinterpretations, and suggested reformulations. The option to request changes was availed of for 13 of the 39 language versions. As can be seen in the right panel of Figure 2, nearly half of the changes requested and later implemented were due to specific items being flagged in the Field Trial (46% or n = 15 requests).⁴ For example, in the Finnish-language version for Finland, the item “I tend to be quiet” was flagged. The national center reviewed the Finnish translation and concluded that the formulation used in the Field Trial—“Olen yleensä hiljainen” [I am usually quiet]—was stronger than the source formulation. They thus requested that the item wording be changed to “Olen usein hiljainen” [I am often quiet].

Other requests for changes were also due mostly to issues of accuracy in the item formulation—however, without the item issue being explicitly referred to in the documentation as having been flagged in the Field Trial (27% or n = 9 requests). Further, in a few cases, item translations were adapted to harmonize them with other versions of the same language or to correct typing errors. On average, 6% of the BFI-2-XS items per country version were changed at this point (with a range of 0% to 53%), resulting in the final BFI-2-XS versions administered in the PIAAC Main Study.