Kohonen Networks

Dateline: 11/17/98

Neural networks are probably the hottest topic in AI these days. Every now and than you can hear about new software package that employs some "advanced neural network technology", even when the core algorithm implements only some decades-old and negelected statistical technique. This is the side effect of the "neural network craze": if something can be called a neural network, its chances of attracting attention and sales money are greatly increased.

Basics

Typical neural net tries to simulate the biological human brain. Explaining how the human brain learns is quite difficult for a simple reason: nobody knows it exactly. However, it is known that human brain consists of a large number of neural cells that process informations. Each cell works like a simple processor and only the massive interaction between all cells and their parallel processing makes the brain's abilities possible. These basic cells are called neurons: each neuron consists of a core, dendrites for incoming information and an axon with dendrites for outgoing information that is passed to connected neurons.

If a certain amount of electrical stimulation is received by a neuron, it generates an output to all other connected neurons. Now if the incoming stimulation is too low, no output is generated by the neuron and the information's further transport will be blocked. On the other hand, if a certain amount of stimulation is received by a neuron, it generates an output to all other connected neurons and so information takes its way to its destination where some reaction will occur.
It is supposed that during the learning process the connection structure among the neurons is changed, so that certain stimulations are only accepted by certain neurons. This means that the firm connections exist between the neural cells that once have learned a specific fact, enabling the fast recall of this information.

Today, probably the most "popular" neural net type is backpropagation network, which was first introduced by G.E. Hinton, E. Rumelhart and R.J. Williams in 1986. It employs supervised learning algorithm, where the desired output is known and compared to the actual output pattern. Unlike the biological model, a neural net has an unchangeable structure. It is built of a specified number of neurons and a specified number of connections between them - weights - which have certain values. Depending on the difference between actual output and target, an error value is computed and used for adjusting the weights between neurons.

Unsupervised learning

If we want to simulate the human brain, selforganization is probably the best way to realize this. It is known that the cortex of the human brain is subdivided in different regions, each responsible for certain functions. The neural cells are organizing themselves in groups, according to incoming informations. The most famous neural network model geared towards this type of unsupervised learning is the Kohonen network, named after its inventor, Teuvo Kohonen from University of Helsinki (first presented in 1982). This time, unlike backpropagation networks, the inputs and outputs are not presented at the same time to the network. Kohonen network relies on a type of learning called competitive learning, where neurons "compete" for the privilege of learning, and the "correct" output is not known. It is essentialy a two-layer network, but because special input normalization is required, there is one more, normalization layer, which ensures that inputs lie within symmetric bounds (usually [-1,1]), and the length of input vector is the same in all training and test cases. There are several normalization methods, like simple length adjustment and Z-axis normalization. For each training presentation, output neurons compete with each other. Only the "winner" (and its neighbors in some variations) is allowed to learn. Each output neuron is connected to the normalized inputs by a vector of n weights (or n+1 when normalization requires synthetic input). The activation of a neuron is a dot product of the weight vector with the normalized input vector:

output = sumi=0, ..., n(xi wi)

The neuron having maximum activation for this presentation is declated the winner, and its weights are updated in such a way that it will react to this particular input even more strongly next time, thus strenghtening its winning position. The weight vectors will eventually converge to a point of stability, when the training is said to be complete. Unknown cases are now presented to the network, and classification of these cases can be performed by finding the maximally activated neuron.

There are several methods for updating a neuron's weight vector in response to input pattern. The original algorithm proposed by Kohonen is to add a fraction of the input vector to the weight vector, renormalizing the sum vector after that. This pushes the weight vvector in the direction of the data vector "bit by bit". If x is the input vector presented to the network, and wt presents the weight vector of the winning neuron at time t, than the updated weight vector w t+1 can be calculated as

wt+1 = wt + α x / | wt + α x |

Denominator in this equation calculates the length (Eucledian norm) of the vector, and the constant α is called the learning rate. It is always much less than 1, and is usually decreased as training progresses.

Another, subtractive method relies on the fact that a weight vector can be pushed toward input data vector by subtracting them (thus finding the error difference), and adding the fraction of this difference to the weight vector.

ε = x - wt
wt+1 = wt + α ε

There are many fine details that should be studied in order to successfuly use this type of neural network in your projects. These include consience mechanism, inhibition algorithms, etc. Below are some excellent sites that you can use to gain more knowledge on this subject.

Similar site(s) of interest: