Read and Download eBook Full
Author: Diane Kelly
Publisher: Now Publishers Inc
Published: 2009
Total Pages: 246
ISBN-13: 1601982240
DOWNLOAD EBOOKProvides an overview and instruction on the evaluation of interactive information retrieval systems with users.
Author: Christopher D. Manning
Publisher: Cambridge University Press
Published: 2008-07-07
Total Pages:
ISBN-13: 1139472100
DOWNLOAD EBOOKClass-tested and coherent, this textbook teaches classical and web information retrieval, including web search and the related areas of text classification and text clustering from basic concepts. It gives an up-to-date treatment of all aspects of the design and implementation of systems for gathering, indexing, and searching documents; methods for evaluating systems; and an introduction to the use of machine learning methods on text collections. All the important ideas are explained using examples and figures, making it perfect for introductory courses in information retrieval for advanced undergraduates and graduate students in computer science. Based on feedback from extensive classroom experience, the book has been carefully structured in order to make teaching more natural and effective. Slides and additional exercises (with solutions for lecturers) are also available through the book's supporting website to help course instructors prepare their lectures.
Author: Donna Harman
Publisher: Springer Nature
Published: 2022-05-31
Total Pages: 107
ISBN-13: 3031022769
DOWNLOAD EBOOKEvaluation has always played a major role in information retrieval, with the early pioneers such as Cyril Cleverdon and Gerard Salton laying the foundations for most of the evaluation methodologies in use today. The retrieval community has been extremely fortunate to have such a well-grounded evaluation paradigm during a period when most of the human language technologies were just developing. This lecture has the goal of explaining where these evaluation methodologies came from and how they have continued to adapt to the vastly changed environment in the search engine world today. The lecture starts with a discussion of the early evaluation of information retrieval systems, starting with the Cranfield testing in the early 1960s, continuing with the Lancaster "user" study for MEDLARS, and presenting the various test collection investigations by the SMART project and by groups in Britain. The emphasis in this chapter is on the how and the why of the various methodologies developed. The second chapter covers the more recent "batch" evaluations, examining the methodologies used in the various open evaluation campaigns such as TREC, NTCIR (emphasis on Asian languages), CLEF (emphasis on European languages), INEX (emphasis on semi-structured data), etc. Here again the focus is on the how and why, and in particular on the evolving of the older evaluation methodologies to handle new information access techniques. This includes how the test collection techniques were modified and how the metrics were changed to better reflect operational environments. The final chapters look at evaluation issues in user studies -- the interactive part of information retrieval, including a look at the search log studies mainly done by the commercial search engines. Here the goal is to show, via case studies, how the high-level issues of experimental design affect the final evaluations. Table of Contents: Introduction and Early History / "Batch" Evaluation Since 1992 / Interactive Evaluation / Conclusion
Author: Jiqun Liu
Publisher: Springer Nature
Published: 2022-05-31
Total Pages: 75
ISBN-13: 3031023196
DOWNLOAD EBOOKSince user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies. To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and "collaborations" among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies.
Author: Jiqun Liu
Publisher: Springer Nature
Published: 2023-02-17
Total Pages: 220
ISBN-13: 3031232291
DOWNLOAD EBOOKThis book brings together the insights from three different areas, Information Seeking and Retrieval, Cognitive Psychology, and Behavioral Economics, and shows how this new interdisciplinary approach can advance our knowledge about users interacting with diverse search systems, especially their seemingly irrational decisions and anomalies that could not be predicted by most normative models. The first part “Foundation” of this book introduces the general notions and fundamentals of this new approach, as well as the main concepts, terminology and theories. The second part “Beyond Rational Agents” describes the systematic biases and cognitive limits confirmed by behavioral experiments of varying types and explains in detail how they contradict the assumptions and predictions of formal models in information retrieval (IR). The third part “Toward A Behavioral Economics Approach” first synthesizes the findings from existing preliminary research on bounded rationality and behavioral economics modeling in information seeking, retrieval, and recommender system communities. Then, it discusses the implications, open questions and methodological challenges of applying the behavioral economics framework to different sub-areas of IR research and practices, such as modeling users and search sessions, developing unbiased learning to rank and adaptive recommendations algorithms, implementing bias-aware intelligent task support, as well as extending the conceptualization and evaluation on IR fairness, accountability, transparency and ethics (FATE) with the knowledge regarding both human biases and algorithmic biases. This book introduces a behavioral economics framework to IR scientists seeking a new perspective on both fundamental and new emerging problems of IR as well as the development and evaluation of bias-aware intelligent information systems. It is especially intended for researchers working on IR and human-information interaction who want to learn about the potential offered by behavioral economics in their own research areas.
Author: Katja Hofmann
Publisher:
Published: 2016-06-07
Total Pages: 134
ISBN-13: 9781680831634
DOWNLOAD EBOOKProvides a comprehensive overview of the topic. It shows how online evaluation is used for controlled experiments, segmenting them into experiment designs that allow absolute or relative quality assessments. It also includes an extensive discussion of recent work on data re-use, and experiment estimation based on historical data.
Author: Mark Sanderson
Publisher: Now Publishers Inc
Published: 2010-06-03
Total Pages: 143
ISBN-13: 1601983603
DOWNLOAD EBOOKUse of test collections and evaluation measures to assess the effectiveness of information retrieval systems has its origins in work dating back to the early 1950s. Across the nearly 60 years since that work started, use of test collections is a de facto standard of evaluation. This monograph surveys the research conducted and explains the methods and measures devised for evaluation of retrieval systems, including a detailed look at the use of statistical significance testing in retrieval experimentation. This monograph reviews more recent examinations of the validity of the test collection approach and evaluation measures as well as outlining trends in current research exploiting query logs and live labs. At its core, the modern-day test collection is little different from the structures that the pioneering researchers in the 1950s and 1960s conceived of. This tutorial and review shows that despite its age, this long-standing evaluation method is still a highly valued tool for retrieval research.
Author: Maristella Agosti
Publisher: Springer
Published: 2013-01-24
Total Pages: 184
ISBN-13: 3642364152
DOWNLOAD EBOOKThe research domains information retrieval and information visualization have always been independent from each other. However, they have the potential to be mutually beneficial. With this in mind, a writer school was organized in Zinal, Switzerland, in January 2012, within the context of the EU-funded research project PROMISE (Participative Research Laboratory for Multimedia and Multilingual Information Systems Evaluation). PROMISE aims at advancing the experimental evaluation of complex multimedia and multilingual information systems in order to support individuals, commercial entities, and communities who design, develop, employ, and improve such complex systems. The overall goal of PROMISE is to deliver a unified environment collecting data, knowledge, tools, and methodologies, and to help the user community involved in experimental evaluation. This book constitutes the outcome of the PROMISE Winter School 2012 and contains 11 invited lectures from the research domains information retrieval and information visualization. A large variety of subjects are covered, including hot topics such as crowdsourcing and social media.
Author: Matthias Hagen
Publisher: Springer Nature
Published: 2022-04-05
Total Pages: 734
ISBN-13: 3030997367
DOWNLOAD EBOOKThis two-volume set LNCS 13185 and 13186 constitutes the refereed proceedings of the 44th European Conference on IR Research, ECIR 2022, held in April 2022, due to the COVID-19 pandemic. The 35 full papers presented together with 11 reproducibility papers, 13 CLEF lab descriptions papers, 12 doctoral consortium papers, 5 workshop abstracts, and 4 tutorials abstracts were carefully reviewed and selected from 395 submissions.
Author: Marti A. Hearst
Publisher: Cambridge University Press
Published: 2009-09-21
Total Pages: 13
ISBN-13: 1139642812
DOWNLOAD EBOOKThe truly world-wide reach of the Web has brought with it a new realisation of the enormous importance of usability and user interface design. In the last ten years, much has become understood about what works in search interfaces from a usability perspective, and what does not. Researchers and practitioners have developed a wide range of innovative interface ideas, but only the most broadly acceptable make their way into major web search engines. This book summarizes these developments, presenting the state of the art of search interface design, both in academic research and in deployment in commercial systems. Many books describe the algorithms behind search engines and information retrieval systems, but the unique focus of this book is specifically on the user interface. It will be welcomed by industry professionals who design systems that use search interfaces as well as graduate students and academic researchers who investigate information systems.
