Learning and Representation Capabilities of Echo State Networks

We present results on the capability of echo state networks (ESNs) to generate and learn sinusoidal oscillations, and on the sufficiency condition for ESNs to exhibit the echo state property. In particular, we show analytically and verify numerically that, provided , the adjustable weights of a linear ESN with N reservoir nodes can be chosen so that the ESN generates a linear superposition of K sinusoidal oscillations having arbitrary prescribed periods, including periods long compared with the network dynamics, and multiple incommensurate periods. These weights can be learned using the usual MSE minimization training procedure; however, the accuracy with which they are learned is limited by numerical round-off error. When the network activities (after training) are temporarily perturbed by random noise, the periods of the individual oscillations comprising the signal are recovered correctly by the ESN, but their amplitudes and relative phases are not. We also present numerical results on the ability of nonlinear ESNs to learn individual sinusoidal oscillations and their superpositions. Finally, we demonstrate through explicit example that not all ESNs having reservoir connection matrices with spectral radii less than unity possess the echo state property. The probability of encountering such counterexamples decreases with increasing N, consistent with a conjecture by Jaeger.