Optimal Feedback Control and Its Relation to Uncontrolled Manifold Analysis

Motor control theories must propose solutions to the degrees of freedom problem, which is the fact that the movement system has more ways to move than are ever required to perform a given task. This creates a problem for action selection (which of the many ways to do something do you choose?) and a problem for action control (how do you create stable, repeatable movements using such a high dimensional system?).

Different theories have different hypotheses about what the system explicitly controls or works to achieve, and what is left to emerge (i.e. happen reliably without explicitly being specified in the control architecture). They are typically about controlling trajectory features such as jerk. Are you working to make movements smooth, or does smoothness pop out as a side effect of controlling something else? This approach solves the degrees of freedom control problem by simply requiring the system to implement a specific trajectory that satisfies some constraint on that feature you are controlling (e.g. by minimising jerk; Flash & Hogan, 1985). They internally replace the solutions afforded by the environment with one desired trajectory. 

Todorov and Jordan (2002a, 2002b; thanks to Andrew Pruszynski for the tip!) propose that the system is not optimising performance, but the control architecture. This is kind of a cool way to frame the problem and it leads them to an analysis that is very similar in spirit to uncontrolled manifold analysis (UCM) and to the framework of motor abundance. In these papers, they apply the mathematics of stochastic optimal feedback control theory and propose that working to produce optimal control strategies is a general principle of motor control from which many common phenomena naturally emerge. They contrast this account (both theoretically and in simulations) to the more typical 'trajectory planning' models.

The reason this ends up in UCM territory is that it turns out, whenever it's possible, the optimal control strategy for solving motor coordination problems is a feedback control system in which control is deployed only as required. Specifically, you only work to control task-relevant variability, noise which is dragging you away from performing the task successfully. The net result is the UCM patterns; task-relevant variability (V-ORT) is clamped down by a feedback control process and task-irrelevant variability (V-UCM) is left alone. The solution to the degrees of freedom control problem is to simply deploy control strategically with respect to the task; no degrees of freedom must be 'frozen out' and the variability can be recruited at any point in the process if it suddenly becomes useful - you can be flexible.

What follows is me working through this paper and trying to figure out how exactly this relates to the conceptually similar UCM. If anyone knows the maths of these methods and can help with this, I would appreciate it!
The optimal controllers Todorov & Jordan build are based on the following assumptions:

• feedback (rather than feed forward) control will be optimal and will be used whenever possible, specifically when the task has redundant solutions, when the movement is happening over longer time scales than any delays in the system, and if the behaviour of the motor system is unpredictable and therefore needs to be monitored
• motor system noise is signal dependent, e.g. faster movements are less accurate in proportion to the change in speed (this is going to be why only controlling when you have to will help stabilise a movement a great deal - every command comes with noise)
• perceptual systems are delayed and noisy, requiring an internal forward model incorporating body dynamics and efference copies of motor commands. The model is used to compute optimal control signals on the fly, and not specify a single trajectory i.e. it's not a motor plan. (This is common in motor control models, and obviously I have some issues with this all! I'd be interested to know if this kind of forward model can be implemented in a Bayesian, Friston kind of way, though. That said, getting more ecological about the role of perception in this model would be something I'd want to do).
• tasks are specified to the model in terms of the instructions about the outcome. Instructions, because they have no perceptual basis for specifying task (another reason to get perception on here!). Outcomes, as opposed to implementations, to distinguish them from motor commands specifying tasks in terms of trajectory features, such as minimal jerk.
• outcomes are subject to an effort penalty

No specific trajectory is ever specified; trajectories emerge as the model runs and optimises control instead of movements. The form of the analysis is to identify the optimal control strategy that produces the relevant values of a performance variable that mean the movement was a success.

Properties of the resulting model
The Minimal Intervention Principle
Optimal control systems follow this principle, specifically deviations from the average trajectory are corrected only when they interfere with task performance (T&J, 2002b). Exerting control to execute a correction entails multiplicative noise, and whatever it is you are correcting (e.g. amount of jerk) will incur an effort-dependent penalty as well. Doing something rather than nothing comes at a cost. 

The net result of this is the UCM result; movement variability is constrained to lie along the manifold, with off-manifold noise actively reduced via control.

Movement flexibility
Optimal control systems are feedback control systems, and this feedback acts whenever required (as per the Minimal Intervention Principle) and whenever possible (see above). This means that movements are controllable in real time, at nearly all times, with respect to the task goal. If anything about the task changes, or if a different trajectory suddenly becomes a better solution, the system can still shift to this new trajectory - it can be flexible. This is not possible if the control system is working to a specified, desired trajectory, and it shows up in data. 

No more planning
Under this analysis, movements are not planned then executed. They are planned, executed, and controlled by the same mechanism. This does away with the need for having internal representations implementing minimisation tricks to get stable movements. This also gets at a lot of fascinating issues with the idea of motor planning in general, which I might get into in future posts. 

Task specificity
This approach does not dictate a 'best movement' but a 'best control strategy'. What's best then depends on exactly what you are trying to do. In other words, the solution will be task-specific which means we need to study a wide range of tasks to uncover the underlying optimality principles, instead of working a single task to death to demonstrate optimal features of particular solutions. 

Summary
The uncontrolled manifold analysis is an implementation of the theoretical claims of the motor abundance hypothesis. The Todorov & Jordan papers are the application of stochastic optimal control theory to the problem of motor coordination. They both essentially arrive at the same place, namely that the optimal solution to the degrees of freedom control problem is the strategic deployment of control processes to only correct errors that bring you away from the task goal. This reduces the control requirements from impossible down to possible, without requiring a single solution (which the data clearly show no-one does anyway). Variation along the manifold then becomes 'the bliss of motor abundance'; it doesn't stop you from achieving the task goal but it creates the opportunity to switch movement trajectories to cope with local task demands.

For the record, I am loving this framework and finding it deeply useful. It is, of course, missing one thing - perception. What specifies the manifold, from the first person perspective of the organism? I have a potential solution, and will develop that in future posts.

References
Flash, T., & Hogan, N. (1985). The coordination of arm movements: an experimentally confirmed mathematical model. The Journal of Neuroscience, 5(7), 1688-1703.

Todorov, E., & Jordan, M. I. (2002a). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5(11), 1226-1235.

Todorov, E., & Jordan, M. I. (2002b). A minimal intervention principle for coordinated movement. In Advances in Neural Information Processing Systems, pp. 27-34.