Item Response Theory, though it has become a widely recognized tool in language testing research, is still not used frequently in practical language assessment projects. This book intends to provide a theoretical overview as well as to give practical guidance concerning the application of IRT in item bank building in a language testing context by presenting a particular project in a higher education setting.
This book explores the appropriateness of Item Response Theory (IRT) in language testing. It investigates the dimensionality of the reading tests of the Cambridge First Certificate of English Test (FCE) and the Test of English as a Foreign Language (TOEFL), and the relative fit of 1, 2, 3 parameter IRT models in which the Rasch model is closely examined. Finding that the Rasch model fails to provide an adequate fit for the data, the study recommends that its predominant use in language testing be re-evaluated. Moreover, the 2 and 3 parameter models fit the data much better than the Rasch model. Finally, it shows that moderate departures from unidimensionality do not necessarily lead to an unacceptable model fit, nor does the use of IRT in test development guarantee that the unidimensionality assumption will be satisfied.
Even though the application of IRT to language testing has recently attracted much attention, no model-data fit research has been conducted to explore the appropriateness of IRT modeling in language testing. The tenability of the strong assumption of unidimensionality has not been studied systematically, and little is known concerning the effects of departure from unidimensionality on the estimation of parameters and on model fit. Furthermore, no study has examined the adequacy of the Rasch model which has been predominant in language testing. The present study investigated the dimensionality of the reading and vocabulary sections of two widely-used English as a foreign language proficiency tests, the University of Cambridge First Certificate of English (FCE) and the Test of English as a Foreign Language (TOEFL). It also compared the relative model fit of three IRT models: 1, 2, and 3 parameter model. Dimensionality of the tests was investigated using Stout's method, factor analyses, and Bejar's method. Secondly, employing fit statistics, invariance check, and the residual analyses, the current study investigated the adequacy of the Rasch model, and the effects of multidimensionality on parameter estimation and model fit. The results of this study suggest the following: (1) Even the TOEFL reading subtest, developed using the three-parameter IRT model, was multidimensional. This appears to be due to underlying factors associated with the reading passages. (2) The FCE reading and vocabulary subtest, based on the traditional British examination system, was found to be essentially unidimensional. (3) Bejar's approach to checking dimensionality appears to be inadequate in that the results differ across the 1, 2, and 3 parameter models. (4) The finding that the Rasch model clearly fails to provide an adequate fit for these data suggests that the prevailing use of the Rasch model in language testing needs to be re-evaluated. (5) The 3 parameter model fit the data only marginally better than did the 2 parameter model. This suggests that for language tests, the discrimination parameter is more significant than is the guessing parameter. (6) A moderate departure from unidimensionality does not appear to invalidate IRT modeling with the data. This finding suggests the possibility of more justified implementation of IRT modeling in language testing.
In the decade of the 1970s, item response theory became the dominant topic for study by measurement specialists. But, the genesis of item response theory (IRT) can be traced back to the mid-thirties and early forties. In fact, the term "Item Characteristic Curve," which is one of the main IRT concepts, can be attributed to Ledyard Tucker in 1946. Despite these early research efforts, interest in item response theory lay dormant until the late 1960s and took a backseat to the emerging development of strong true score theory. While true score theory developed rapidly and drew the attention of leading psychometricians, the problems and weaknesses inherent in its formulation began to raise concerns. Such problems as the lack of invariance of item parameters across examinee groups, and the inadequacy of classical test procedures to detect item bias or to provide a sound basis for measurement in "tailored testing," gave rise to a resurgence of interest in item response theory. Impetus for the development of item response theory as we now know it was provided by Frederic M. Lord through his pioneering works (Lord, 1952; 1953a, 1953b). The progress in the fifties was painstakingly slow due to the mathematical complexity of the topic and the nonexistence of computer programs.
Using familiar concepts from classical measurement methods and basic statistics, Hambleton and colleagues introduce the basics of item response theory (IRT) and explain the application of IRT methods to problems in test construction, identification of potentially biased test items, test equating, and computerized-adaptive testing. The book also includes a thorough discussion of alternative procedures for estimating IRT parameters, such as maximum likelihood estimation, marginal maximum likelihood estimation, and Bayesian estimation in such a way that the reader does not need any knowledge of calculus to follow these explanations. Including step-by-step numerical examples throughout, the book concludes with an exploration of new directions in IRT research and development.
A complete discussion of fundamental and advanced topics in Item Response Theory written by pioneers in the field In Item Response Theory, accomplished psychometricians Darrell Bock and Robert Gibbons deliver a comprehensive and up-to-date exploration of the theoretical foundations and applications of Item Response Theory (IRT). Covering both unidimensional and multidimensional IRT, as well as related adaptive test administration of previously calibrated item banks, the book addresses the growing need for understanding of this topic as the use of IRT spreads to other fields. The first book on the topic that offers a complete and unified treatment of its subject, Item Response Theory prepares researchers and students to understand and apply IRT and multidimensional IRT to fields like education, mental health and marketing. Accessible to first year-graduate students with a foundation in the behavioral or social sciences, basic statistics, and generalized linear models, the book walks readers through everything from the logic of IRT to cutting edge applications of the technique. Readers will also benefit from the inclusion of: • A thorough introduction to the foundations of Item Response Theory, including its logic and origins, model-based measurement, psychological scaling, and classical test theory • An exploration of selected mathematical and statistical results, including points, point sets, and set operations, probability, sampling, and joint, conditional, and marginal probability • Discussions of unidimensional and multidimensional IRT models, including item parameter estimation with binary and polytomous data • Analysis of dimensionality, differential item functioning, and multiple group IRT Perfect for graduate students and researchers studying and working with psychometrics in psychology, quantitative psychology, educational measurement, marketing, and statistics, Item Response Theory will also benefit researchers interested in patient reported outcomes in health research.
This volume includes selected papers from the 28th Language Testing Research Colloquium, held at the University of Melbourne (Australia) in July 2006. The papers selected for this volume share a common theme - that of 'performance'. Not only do they focus on performance assessments of second or foreign language speaking and writing, but they also focus on the performance of the participants - the candidate (or candidates) and the raters - and the construction of that performance through the tasks and the assessment criteria.
Item response theory has become an essential component in the toolkit of every researcher in the behavioral sciences. It provides a powerful means to study individual responses to a variety of stimuli, and the methodology has been extended and developed to cover many different models of interaction. This volume presents a wide-ranging handbook to item response theory - and its applications to educational and psychological testing. It will serve as both an introduction to the subject and also as a comprehensive reference volume for practitioners and researchers. It is organized into six major sections: the nominal categories model, models for response time or multiple attempts on items, models for multiple abilities or cognitive components, nonparametric models, models for nonmonotone items, and models with special assumptions. Each chapter in the book has been written by an expert of that particular topic, and the chapters have been carefully edited to ensure that a uniform style of notation and presentation is used throughout. As a result, all researchers whose work uses item response theory will find this an indispensable companion to their work and it will be the subject's reference volume for many years to come.
This volume guides its reader through the basics of Item Response Theory, with an emphasis on what and how to include relevant information in the methods and results sections of professional papers. The author offers examples of good and bad write-ups.
