Obstacles and Risks. To enable this research we need broad collaborations between IR researchers and communities outside IR. Finding effective ways of collaborating and finding a shared language requires con- siderable effort and investment that may not be properly “rewarded” by funding bodies and evaluation committees. An important risk concerns the diversity of perspectives on the definition of core concepts such as fairness, ethics, explanation or bias across scientific and engineering disciplines, gov- ernments or regulating bodies. Having more transparent IR systems could make systems more vulnerable for adversaries as knowledge about the internals of systems need to be shared through explanations. A potential obstacle is initial resistance from system developers and engineers, who might have to change their workflows in order for systems to be more transparent. Another possible obstacle is the tension between transparency and fairness, and an enterprise’s commercial goals. An inadvertent risk is introducing a new type of bias into our systems about which we are unaware. 4 IR for Supporting Knowledge Goals and Decision-Making
Obstacles and Risks. The conceptual framework is quite general, and while many tasks can be addressed with this framework, this generality might make it difficult to pin down all possible challenges and see which practices can be shared. Some of the engineering structures required to support GIOs are much less developed and investigated than conventional IR system components. Particular challenges lie in an algorithmic understanding of information needs and response text. This requires a representation and inter- action mechanism that allows referring to generated response parts, giving relevance explanations for generated information units, and reasoning about conflicts and trustworthiness of harvested information units. Industrial applications of GIOs for task-specific purposes are likely to push the development of this area quite quickly ahead of the research community. We run the risk of falling behind rather than leading this effort.
Obstacles and Risks. For academics seeking to undertake research in large-scale IR systems there are obvious risks, primarily in regard to achieving genuine scale. Many of the research questions that offer the greatest potential for improvement – and the greatest possibilities for economic savings – involve working with large volumes of data, and hence significant computational investment. Finding ways of collaborating across groups, for example, to share hardware and software resources, and to amortize development costs, is a clear area for improvement. Current practice in academic research in this area tends to revolve around one-off software developments, often by graduate students who are not necessarily software engineers, as convoluted extensions to previous code bases. At the end of each student’s project, their software artifacts may in turn be published to GitHub or the like, but be no less a combination of string and glue (and awk and sed perhaps) than what they started with. Agreeing across research groups on some common data formats, and some common starting implementations, would be an investment that should pay off relatively quickly. If nothing else, it would avoid the ever-increasing burden for every starting graduate student to spend multiple months acquiring, modifying, and extending a code base that will provide baseline outcomes for their experimentation. Harder to address is the question of data scale and hardware scale. Large compute instal- lations are expensive, and while it remains possible, to at least some extent, for a single server to be regarded as a micro-unit of a large server farm, there are also interactions that cannot be adequately handled in this way, including issues associated with the interactions between different parts of what is overall a very complex system. Acquiring a large hardware resource that can be shared across groups might prove difficult. Perhaps a combined approach to, for example, Amazon Web Services might be successful in being granted a large slab of storage and compute time to a genuinely collaborative and international research group. Harder still is to arrange access to large-scale data. Public web crawls such as the Common Crawl can be used as a source of input data, but query logs are inherently proprietary and difficult to share. Whether public logs can be used in a sensible way is an ongoing question. Several prior attempts to build large logs have not been successful: the logs of CiteSeer and DBLP are heavily sk...
