International Large-Scale Assessment Data: Issues in Secondary Analysis and Reporting
Source: Educational Researcher, 39: 142-151. (March 2010).
(Reviewed by the Portal Team)
The technical complexities and sheer size of international large-scale assessment (LSA) databases often cause hesitation on the part of the applied researcher interested in analyzing them. Further, inappropriate choice or application of statistical methods is a common problem in applied research using these databases.
This paper serves as a primer for researchers on the issues and methods necessary for obtaining unbiased results from LSA data.
The authors outline the issues surrounding the analysis and reporting of LSA data, with a particular focus on three prominent international surveys.
The authors conclude with a set of general steps that researchers should consider when planning analyses involving the studies discussed in this article.
1. With a clear research question in mind, carefully choose an appropriate sampling weight. Weights should generally be applied to every analysis. This is important for both point estimates and standard error estimates.
2. When proficiency estimates are of interest, plausible values should be used in conjunction with survey software for general analyses, including calculating means and standard errors and single-level linear and logistic regression analyses.
3. Resampling variance estimation techniques, available for standard analyses supported by survey software, should be used whenever possible to ensure unbiased estimates of sampling variability.
4. When conducting analyses that involve teacher-level data, always analyze and report teacher variables as attributes of the student.
5. Finally, consider that TIMSS, PIRLS, and PISA are cross-sectional and observational studies. Because of this design, it is generally not appropriate to make causal inferences using these data.