Improved Local Learning Rule for Information Maximization and Related Applications

For a neural network comprising feedforward and lateral connections, a local learning rule is proposed that causes the lateral connections to learn directly the inverse of a covariance matrix. In contrast to earlier work, the rule involves just one processing pass through the lateral connections for each input presentation, and consists of a simple anti-Hebbian term. This provides an effective and simple method for online network learning algorithms that implement optimization principles, drawn from statistics or from information or control theory, for which a running estimate of the covariance matrix inverse is useful. An application to infomax learning (mutual information maximization) in the presence of input and output noise is used to illustrate the method.