Learning Structured Prediction Sample Clauses

Learning Structured Prediction. An alternative is based on a view of the situation understanding problem as a structured prediction problem, whereby a collection of input objects (observations) need to be mapped to a predicted structured object (event = certain sequences of observations). In its most generic form, this is a typical supervised learning problem; the desired mapping can be learned using a correctly labeled set of input-prediction training data and a loss function that evaluates the quality of a mapping. This monolithic approach, however, does not take advantage of the inherent structure found in the predictions and therefore requires a large number of training data to compensate for this inflexibility. A recent research trend [U7], [U8] in the area of structured prediction reduces the problem to a sequential decision problem, whereby the predicted structured object for a given input is built incrementally making one decision at a time. Therefore, instead of modeling what a good prediction looks like, this sequential approach directly models how to build a good prediction. Finding the optimal policy for the resulting Markov Decision Process (MDP) of this reduction is equivalent to finding the mapping that minimizes the loss function. A illustrative example of this approach is given in [U9]: let Y represent the prediction of the handwritten word X which is the handwritten word STRUCTURE with a mark on the letters UR (making them partly invisible). In the traditional formulation, the 9 hand-written letters are mapped directly to the predicted word STRUCTURE, whereas in the sequential formulation the prediction is built letter-by-letter. The reduction described above is not practical on its own, given that the full model of the MDP is either huge or totally unknown. However, it is important because it opens the door to using a wide variety of successful reinforcement learning (RL) methods [U10], [U11], which can learn approximate policies without a model. Briefly, RL is learning by trial-and-error; the system observes the current state, takes an action, receives a reward or penalty, observes the next state, etc. and through these samples of interaction eventually learns a good policy (choosing actions in each state) that maximizes the expected long-term reward. Our intention is to adapt advanced RL algorithms, such as Least-Squares Policy Iteration (LSPI) [U11] and Rollout-Sampling Policy Iteration (RSPI) [U12], and apply them in the NOPTILUS context. It is expected that both L...