Common use of Research Challenges Clause in Contracts

Research Challenges. Challenges are faced on each of the areas that the proposed research covers. The proposed research touches on the collection over which the search engine operates, the user’s interaction with the search system, the user’s cognitive processes, and the evaluation of the changes to the user’s knowledge state or performance on tasks. At the level of the collection, we are concerned with the mix of information that is available. For large scale collections, such as the web, it is very difficult to understand the amount of material on a given topic, and thus is it hard to know what the existing biases are in the collection. For example, we might be interested in measuring the accuracy of decisions that users make after using a search engine. Collections of interest will contain a mix of correct and incorrect information, but the scale of the collection will make it difficult to understand the amount of correct and incorrect information in the collection apriori to the user’s search session. The field of IR is still in its infancy with respect to understanding user interaction and user cognitive processes. For us to be able to design systems that lead users to their desired knowledge state or decision, we will need to better understand how their cognitive processes affect their inter- action with the system and how the stream of information that they consume changes their mental state. A challenge here will be a lack of expertise in cognitive science and psychology (how people learn, how people make decisions, biases). Progress in this area will likely require collaboration outside of IR and require input from and engagement of other communities, including: cognitive science, human-computer interaction, psychology, behavioural economics, and application/domain specific communities (e.g., intelligence community, clinical community). The envisioned systems may require radical changes to aspects of user interfaces. Uptake of new UI solutions, however, is often difficult and poses extra onus on users, thus creating a high barrier to entry for the proposed new systems. Finally, evaluation ranges from the simple to the complex. We are interested both in sim- ple measures such as decision accuracy, and complex measures such as increases in curiosity. Evaluation is envisioned to embrace larger aspects of the user-system interaction than just the information seeking phase, e.g., evaluation of decisions users take given the information systems provided. Given that almost all evaluation will be with respect to changes in the user, evaluation will be as costly in time and effort as all user studies and human research is. Evaluation may also be hindered by difficulties in evaluating aspects such as learning, or the unavailability of a normative reference to evaluate decisions. Indeed, there are many circumstances in which the “right decision” or the “right knowledge” depends on personal circumstances, or cultural/societal frameworks.