Differential functioning of PISA science items in Brazil and Japan
DOI:
https://doi.org/10.18222/eae.v28i68.4584Keywords:
Differential Item Functioning – DIFF, PISA, Scientific Literacy, Sciences.Abstract
The Programme for International Student Assessment (PISA) is an international comparative assessment program applied to samples of 15-year-old students. Together with other countries, Brazil and Japan have participated in this program since its first edition in 2000. DIF (Differential Item Functioning) analysis was used to identify factors that could explain performance differences in scientific literacy between Brazilian and Japanese students, in the 2006 edition. Based on the test results, this analysis showed patterns of differentiated effects. To identify the items that showed differential functioning between Brazil and Japan, we used the Bayesian integrated model. In addition to confirming this occurrence, this model may also explain the DIF. DIF was found in all covariates selected. However, differential functioning did not always favor either of the two countries. There are competencies that discriminate more against the Brazilian students and areas of application of the items that sometimes were easier for Brazilian and sometimes for Japanese students.Downloads
References
AGUIAR, Glauco. Estudo comparativo entre Brasil e Portugal, sobre diferenças nas ênfases curriculares de Matemática, a partir da análise do Funcionamento Diferencial do Item (DIF) do PISA 2003.
246f. Tese (Doutorado em Educação) – Pontifícia Universidade Católica do Rio de Janeiro, Rio de Janeiro, 2008.
ANASTASI, Anne. Psychological testing. New York: MacMillan. 1988.
ANDRIOLA, Wagner. Descrição dos principais métodos para detectar o funcionamento diferencial dos itens (DIF). Psicologia: Refl e Crítica, Rio Grande do Sul, v. 14, n. 3, p. 643-652, 2001. DOI: https://doi.org/10.1590/S0102-79722001000300020
BARROSO, Marta; FRANCO, Creso. Avaliações educacionais: o PISA e o ensino de ciências. In: ENCONTRO DE PESQUISA EM ENSINO DE FÍSICA, 11.,2008, Curitiba. Anais... Curitiba, 2008. Disponível em: <http://www.if.ufrj.br/~marta/artigosetal/2008-epef11-PISA.pdf>. Acesso em: 27 jul. 2014.
BERBEROGLU, Giray. Differential item functioning (DIF) analysis of computation, word problem and geometry questions across gender and SES groups. Studies in Educational Evaluation, Great Britain, v. 21, n. 4, p. 439-456, 1995. DOI: https://doi.org/10.1016/0191-491X(95)00025-P
BRASIL. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. Resultados nacionais – PISA 2006: Programa Internacional de Avaliação de Alunos (PISA). Brasília, DF: Inep, 2008.
CLAUSER, Brian; MAZOR, Kathleen. Using statistical procedures to identify differential item functioning test items. Educational Measurement: Issues and Practice, Philadelphia, v. 17, n. 1, p. 31-44, 1998. DOI: https://doi.org/10.1111/j.1745-3992.1998.tb00619.x
COLE, Nancy. History and development of DIF. In: HOLLAND, Paul W.; WAINER, Howard (Ed.). Differential item functioning. Hilldsdale, NJ: Lawrence Erlbaum, 1993.
FERRER, Alejandro Tiana. Que variáveis explicam os melhores resultados nos estudos internacionais? In: AZEVEDO, Joaquim. Avaliação dos resultados escolares. Porto: ASA, 2003.
GAMERMAN, Dani; LOPES, Hedibert. Markov Chain Monte Carlo: Stochastic simulation for Bayesian inference. New York: Chapman & Hall / CRC, 2006. DOI: https://doi.org/10.1201/9781482296426
GAMERMAN, Dani; SOARES, Tufi GONÇALVES, Flávio. Bayesian analysis in item response theory applied to a large-scale educational assessment. In: O’HAGAN, Anthony; WEST, Mike. The Oxford handbook of applied Bayesian analysis. New York: Oxford University, 2010. p. 624-652.
GIERL, Mark; BISANZ, Jeffrey; BISANZ, Gay; BOUGHTON, Keith. Identifying content and cognitive skills that produce gender differences in mathematics: a demonstration of the DIF analysis framework. Journal of Educational Measurement, Philadelphia, v. 40, n. 4, p. 281-306, 2003. DOI: https://doi.org/10.1111/j.1745-3984.2003.tb01148.x
HAMBLETON, Ronald; SWAMINATHAN, H.; ROGERS, Jane. Fundamentals of Item Response Theory. Newburry Parks: Sage, 1991.
HOLLAND, Paul; THAYER, Dorothy. Differential item performance and the Mantel–Haenszel procedure. In: HOLLAND, Paul W.; WAINER, Howard (Ed.). Differential item functioning. Hilldsdale, NJ: Lawrence Erlbaum, 1988. p. 129-145.
LINN, Robert; DRASGOW, Fritz. Implications of the golden rule settiemernt for test construction. In: HOLLAND, Paul W.; WAINER, Howard (Ed.). Differential item functioning. Hilldsdale, NJ: Lawrence Erlbaum, 1993.
O’NEIL, Kathleen; McPEEK, Miles. Item and test characteristics that are associated with differential item functioning. In: HOLLAND, Paul W.; WAINER, Howard (Ed.). Differential item functioning. Hilldsdale, NJ: Lawrence Erlbaum Associates, 1993. p. 255-276.
PASQUALI, Luiz. Psicometria: teoria dos testes psicológicos. Brasília, DF: Prática, 2000.
SCHMITT, Alicia P.; BLEISTEIN, Carole A. Factors affecting differential item functioning for black examinees on scholastic aptitude test analogy items (ETS RR-87-23). Princeton, NJ: Educational Testing Service, 1987. DOI: https://doi.org/10.1002/j.2330-8516.1987.tb00227.x
SCHMITT, Alicia; HOLLAND, Paul; DORANS, Neil. Evaluating hypotheses about differential item functioning. In: HOLLAND, Paul W.; WAINER, Howard (Ed.). Differential Item Functioning. Hilldsdale, NJ: Lawrence Erlbaum, 1993. p. 281-316.
SOARES, Tufi Utilização da Teoria de Resposta ao Item na produção de indicadores sócio-econômicos. Pesquisa Operacional, Rio de Janeiro, v. 25, n. 1, p. 83-112, jan./abr. 2005. DOI: https://doi.org/10.1590/S0101-74382005000100006
SOARES, Tufi GENOVEZ, Silene; GALVÃO, Ailton. Análise do Comportamento Diferencial dos Itens de Geografi estudo da 4a série avaliada no Proeb/Simave, 2001. Estudos em Avaliação Educacional, São Paulo, v. 16, n. 32, p. 81-110, jul./dez. 2005. DOI: https://doi.org/10.18222/eae163220052137
SOARES, Tufi GONÇALVES, Flávio; GAMERMAN, Dani. Na integrated Bayesian model for DIF analysis. Journal of Educational ad Behavioral Statistics, Washington, v. 34, n. 3, p. 348-377, Sep. 2009. DOI: https://doi.org/10.3102/1076998609332752
SWAMINATHAN, Hariharam; ROGERS, Jane. Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, Philadelphia, v. 27, p. 361-370, 1990. DOI: https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
THISSEN, David. IRTLRDIF v.2.0.b: software for the computation of the statistics involved in item response theory Likelihood-Ratio Tests for differential item functioning. 2001.
THISSEN, David; STEINBERG, Lynne; WAINER, Howard. Detection of differential item functioning using the parameters of item response models. In: HOLLAND, Paul W.; WAINER, Howard (Ed.). Differential Item Functioning. Hilldsdale, NJ: Lawrence Erlbaum, 1993. p. 67-114.
ZIMOWSKI, Michele F.; MURAKI, Eiji; MISLEVY, Robert J.; BOCK, R. Darrell. BILOG-MG: Multiple Group IRT Analysis and test maintenance for binary items. [Computer software]. Chicago: Scientifi Software International, 1996.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2017 Andriele Ferreira Muri, Tufi Machado Soares, Alicia Bonamino
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
a. Authors retain the copyright and grant the journal the right to first publication.
b. All works are licensed under the Creative Commons Attribution License, which allows the sharing of the paper with acknowledgment of authorship.
Until 2024, Estudos em Avaliação Educacional adopted the Creative Commons Attribution-NonCommercial (CC BY-NC) license for its publications. For texts published from 2025 onwards, the journal will adopt the Creative Commons Attribution (CC BY) license, in line with the principles of Open Science.
