The majority of factor analysis models assume that the response of an individual to a test item is influenced by only one dimension. This is such that cross-loadings are usually deemed as item ambiguity and lack of objectivity of the factors. Conversely, multidimensional within-item models comprise items belonging to more than one dimension at a time, requiring a more complex loading structure able to model the relationship between the latent traits under investigation. These models allow for cross-loading items and do not treat the dimensions as separate scales, providing a more robust insight about convergent and discriminant validity. As opposed to between-item multidimensional models, within-item multidimensional models make use of procedures which take full advantage of the information in the data as opposed to relying on limited information. These models are therefore called “full information” models (Bock, Gibbons, & Muraki, 1988) since they are based on individual’s patterns of response rather than on the correlational structure of the multivariate latent response distribution (Wirth & Edwards, 2007).
In modern psychometrics, Item Response Theory (IRT) is deployed for the calibration of items belonging to individual scales so that each dimension is regarded as unidimensional. According to IRT models, a person’s response to a test item depend solely on the item characteristics and the person’s single parameter, the latent trait, also called q. The extension of IRT to within-item models is called Multidimensional Item Response Theory (MIRT). As with IRT models, MIRT also considers that the probability of an individual with a given q is a function describing the relationship between the item parameters and the individual’s latent trait. However, as more than one latent trait can be accounted for by a single item, different q-coordinates are expected to represent the person’s characteristics. In this case, cross-loadings have a real meaning since it informs the position of an individual in relation to the latent traits assessed. Similar to the z-scores scale, the theta-coordinates typically range from -3 to +3.
Taking the items of the Multidimensional Turnover Reasons Scale as an example, under the MIRT approach, if one responds that they strongly agree with the statement “I would move to another organization if it offered me greater flexibility”,this decision is meant to be made not only based on what another organization has to offer (external aspects), but on the level of flexibility currently perceived by the worker regarding his current organization (internal aspects). All of the items were therefore designed to reflect both external and internal factors so that the reasons why one wants to leave the organization could be better determined. Consequently, a respondent with scores q1 = 1.25 and q2 = 0.5 in the dimensions external and internal, respectively, could be identified as someone whose external elements seem more enticing than the current characteristics of her organization while deciding whether to leave or stay in an organization.
To illustrate how MIRT can contribute to the science of organisational behaviour, we will show the psychometric properties of the Multidimensional Turnover Reasons Scale, first instrument designed by us in order to concurrently provide information about internal and external forces driving intentions to quit an organisation.
As MTRS is a Likert-type scale, the Multidimensional Graded Response Model (MGRM, Muraki & Carlson, 1993) was chosen for this investigation. MGRM is a generalization of traditional polytomous IRT models and takes into account both item difficulty and discrimination. In a nutshell, MGRM assumes that the selection of a response category requires a number of steps and reaching step k requires acceptance of step k – 1. For instance, since the answer choices to MTRS items range from 1 (totally disagree) to 6 (totally agree), the probability of one endorsing higher categories increases monotonically with an increase in any of the dimensions of turnover reasons, as represented by the elements of the q-vector.
That said, the probability of accomplishing k or more steps is modeled by a two-parameter normal ogive model, with the person parameter determined by a linear combination of the elements in the q-vector weighted by discrimination parameters (Reckase, 2009). The normal ogive form of the MGRM can be seen in Equation 1:
Equation 1. A normal ogive form for multidimensional models.
where k is the step/score on the item, ai is a vector of discrimination parameters, qj is a vector of individual scores per dimension, and dik is the difficulty parameter, i.e. how ease a person will endorse the kth step of the item. The interpretation of the difficulty parameter (dik) is not as straightforward as for unidimensional IRT dichotomous models, since d is in slope/intercept form, representing aq – ab. Also, there are as many difficulty parameters as the number of steps or thresholds between the category responses, making it more complicated to compare them against the individual’s latent traits.
The studies carried out for the development and validation of standardized instruments to measure turnover reasons have applied factor analytic techniques, with a broad application of between-items multidimensional models. Nevertheless, considering that most sets of test items are more likely to measure several latent traits instead of a single one, such models are not as informative as within-item multidimensional models. Moreover, although MIRT models share some similarities with factor analysis, item parameters, such as discrimination and difficulty level, are considered as nuisances expected to be removed when performing factor analysis (Reckase, 2009). Built on the advantages of deploying a within-item multidimensional polytomous compensatory model, this study innovates by showing not only how such a model can be applied to investigate the construct validity of a turnover reasons scale, but also brings significant contributions for the investigation of psychological constructs in organizations.
Results
The results of the Full Information Factor Analysis for the two-dimensional model showed that 28 out 30 items measure turnover reasons adequately (table below). There are no items with null slopes in both dimensions. As can be seen in the table below, twelve items loaded primarily onto the external dimension, followed by thirteen items whose factor loadings are higher for the internal dimension. Different from the factor analysis carried out in the context of CTT, where an item is considered ambiguous when a difference between its factor loadings is less than 0.1 (Berkenbosch et al, 2013), the MIRT approach will regard this event as an important information to be accounted for. As such, three items had high factor loadings in both dimensions and can be considered as appropriate for construct measurement. Item 2 was removed since its factor loadings were below 0.3 in both dimensions. Also, item 7 was ultimately removed due to low discriminations in both dimensions as well because its MDISC is below 0.8.
Except for items 10, 13, 15, 17 and 18, all items from the External dimension have combinations where both dimensions contribute positively toward a high probability of leaving an organization. When internal elements are the main forces for leaving, the internal factors weigh more in favor of turnover reasons and external elements collaborate with the motives to remain.
Regarding internal consistency, the Cronbach’s alpha for External dimension was 0.83 and for the Internal dimensions, 0.89. After the calibration of item parameters, factor scores were computed for each individual. Taking a random participant as an example, if she decided to leave her organization, internal aspects would likely play a major role since she scored 2.16 in the Internal dimension. On the other hand, as she scored -1.82 in the External dimension, if she eventually decides to leave, the elements and characteristics of other organizations would likely influence her decision the least.
Discussion
When classical factor analytic techniques are deployed, contributions of secondary factors are typically disregarded and much of the information deemed as significant for explaining the relationship between different elements that contribute for the decision-making process is lost. Nevertheless, the Multidimensional Item Response Theory approach provides additional information with respect to both the individual, whose scores can be computed for each dimension, and the items, as their parameters are estimated by testing models of how different factors interact to better explain psychological constructs.
The application of MIRT models to the field of organizational behavior reflects the complexity and multi-determination of the human behavior so that even the test items which have been designed to be unidimensional measure a complex of latent traits rather than a single trait (Reckase, Ackerman & Carlson, 1988). For this investigation, a compensatory MIRT model was deployed for the validation of the Multidimensional Turnover Reasons Scale (MTRS) and therefore the estimation of individual scores. The two-dimensional factor structure was confirmed, being the interactions between external elements of other organizations and internal elements of the current organization the major feature leading the individual’s turnover reasons.
The findings regarding the MTRS factor structure show that the External aspects are those driving the maximum probability of leaving the organization, though most of the items are influenced by internal aspects alike, as expected. Even though the items had been developed to be multidimensional, most of them had low factor loadings in the second factor, yet significant when discrimination parameters are considered.
When items are investigated under a compensatory MIRT perspective, it is common that one of the dimensions accounts for a higher variance. Moreover, the higher the discrimination parameters across the different dimensions, the higher the information an item will provide with regard to the trait measured. Notwithstanding, items with discriminations equal or close to zero in one of the dimensions are not desirable when multidimensional items are under investigation and should be ultimately considered as unidimensional. Having an instrument with unidimensional items, yet thought to be multidimensional, does not impinge on the construct validity, but it does not also contribute to identifying the reasons why one wants to leave the organization nor brings further information about specific individual scores for the multiple dimensions. Furthermore, when multidimensional discrimination (MDISC) is taken into consideration, some unidimensional items, such as Item 9, have a total MDISC greater than for some multidimensional items, such as Item 5, since the slopes in both dimensions are not high. While these items still provide information about the trait measured, this is a partial and therefore limited information since only one dimension has actually been measured.
Practical implications
Given the purpose of the measure, the inferences from the analytical method, and the universally relevant nature of turnover in organizations, there are substantial practical implications from our approach and findings. First, this is not merely a tool for scientific study, but one that (at a minimum) HR professionals may utilize in providing senior management with precise insights about both the volume and nature of potential staffing challenges. While some organizations do offer to tailor incentive packages, many retention efforts focus on a corporate mindset, as opposed to bespoke plans targeting those issues specifically identified. Generalized approaches offer some benefit but may otherwise miss out on opportunities for retention by lacking that precision. Having this information also allows managers to assess cost-effectiveness of retention strategies, as well as relevant timelines. For example, if it becomes clear that a common reason for potential employee departure is based on the lack of skills training, management may assess the return on investment for providing such opportunities and compare against the cost of losing those employees. They may also assess better the timeline at which this concern may arise based on how long employees are in the organization, plus monitor changes over time and employment variations.
Conclusions
This study intended to widen the understanding of the construct turnover reasons by devising an international scale based on the Multidimensional Item Response Theory perspective and finally provide the field with a new standardization technique. Despite MIRT models have received little attention from organizational researchers, their potential to explain complex phenomena in organizational settings is enormous. The Multidimensional Turnover Reasons Scale can help companies to work beyond their turnover rates, mainly on the analyses of their talented employees with a stronger reason to leave the organization, and then create new measures aimed at worker retention. Moreover, the validation and standardization of MTRS may provide insights into the field of organizational behavior and further contribute to the development of this topic and its measurement. When established, this may then be useful in identifying potential levers for improving retention by tailoring to the needs of employees, benefitting both organizations and the individuals who make them function.
The complete version of this paper has been submitted to the European Journal of Psychological Assessment and it is under assessment.
People Analytics and Psychometrics Lab 