Detection of differential item functioning in imbalanced groups. Are INVALSI tests fair among pupils from different academic schools?

Differential Item Functioning (DIF) and bias measurement are often used as synonyms in standardized tests fairness evaluation between individuals belonging to different groups. Recently, Zumbo et al. (2016, 2017) have provided a redefinition of DIF/bias term and proposed a new methodology for DIF/bias detection analysis. The new definition of bias requires attributional reasoning; therefore, there is a need to find a way to control for possible confounding factors. Only by balancing groups with respect to covariates, it is possible to attribute DIF to group membership. Propensity score matching techniques allow to carry out groups balancing and bias is detected if item is flagged as DIF, after balancing groups. The conditional logistic regression is proposed for DIF detection analysis after matching because it allows to consider the data structure generated by matching. The aim of this work is twofold. Firstly, we assess the efficacy and performance of the new methodology in imbalanced groups, comparing its performance to performance of traditional DIF detection methods (Mantel-Haenszel statistic, logistic regression and Lord's χ2). Our research, through a simulation study, shows that the new methodology outperforms traditional DIF detection methods in imbalanced groups in situations of large samples and DIF items presence. Nevertheless, the new methodology suffers to I error inflation for large samples and simulation results suggest that the use of an effect size measure (ΔR2) reduces significantly this issue. Secondly, the proposal methodology is applied to data coming from the large-scale standardized test administered by the National Evaluation Institute for the School System (INVALSI) to evaluate pupils' Italian language and mathematics competencies. The idea is to detect possible DIF items among pupils from different academic tracks. The results reveal that very few items are flagged as DIF, indicating the fairness of INVALSI tests.