Neural Networks 101
Start exploring the world of Neural Networks, massively parallel systems designed to mimic the architecture and function of our brains.
  Related Resources
• Past issues of weekly features
• AI Sourcebook, Part I: Neural Networks
• Kohonen Networks
 
 Elsewhere on the Web
• Neural Nets by Kevin Gurney
• ANN Technology by Dave Anderson and George McNeil
• Introduction to ANNs by Nimrod Hoofien
• An Introduction to Neural Networks from Syracuse University
• A Brief Tour of the Brain from Syracuse University
• comp.ai.neural-nets FAQ
 
 

Neural networks (NNs) are massively parallel systems designed to mimic the architecture of the human or animal brain. This topic received a lot of the public attention in the past years, as NNs are capable of performing a variety of tasks that are difficult or impossible to do with conventional computers. The reasons why NNs often outperform classical statistical methods lie in their abilities to analyze incomplete, noisy data, to deal with problems that have no clear cut solution and to learn on historical data. Because of these advantages, they have shown success in predictions of data series that have high degree of volatility and fluctuations. It is mathematically proven that three-layer neural networks are capable to approximate any nonlinear function. However, the most fascinating and almost magic-like feature of NNs is that they not programmed like conventional computers, but are literally trained to produce desired results. This time we'll just start exploring the world of Neural Networks: later articles will feature more information and hands-on articles on applying various architectures in the real-world applications.

Basics

NN usually consists of two or more layers or groups of processing elements called neurons. The term neuron denotes a basic unit of a neural network model attended for data processing. Neurons are connected into a network in the way that the output of each neuron represents the input for one or more other neurons. According to its direction, the connection (often called synapse as its biological counterpart) between neurons can be either one-directional or bi-directional, and due to its polarity the connection can be excitatory or inhibitory. Neurons are grouped into layers, with three main types of layers used in practice: input, hidden, and output. The input layer receives input data from external environment, and sends it to the hidden layer(s). In the hidden layer the information is processed and sent to the output layer neurons, where the network output is compared to the desired output and the network error is computed.


Figure 1: even such simple, two layer, fully connected networks can often produce interesting results

In the most typical case, the error information then flows backward through the network and the values of connection weights between the neurons are adjusted using the error term. The process is repeated in the network for a number of iterations that is necessary to achieve the output closest to the desired (actual) output. Finally, the network output is presented to the user. Neural network learning is basically the process by which the system arrives at the values of connection weights between neurons.

Connection types

Connections in the network are either realized between two layers or between neurons in the same layer. There are various types of connections between layers:

  • fully connected - each neuron in the first layer is connected to each neuron in the second layer
  • partially connected - each neuron in the first layer doesn't necessarily have to be connected to every neuron in the second layer
  • hierarchical - neurons in one layer are connected only to the neurons of the next neighbor layer
  • feed-forward - connection between neurons is one-directional, neurons in the first layer send their output to the neurons in the second layer, but they do not receive any feedback
  • bi-directional - there is a feedback when the neurons from the second layer send their output back to the neurons in the first layer
  • resonance - two-directional connection where neurons continue to send information between layers until a certain condition is satisfied

    Here are the examples of some well known NN architectures along with their main features:

  • Perceptron - first NN, two-layered, fully connected
  • ADALINE (Source Code)- two-layered, fully connected
  • Backpropagation (Source codes)- first NN with one or more hidden layers, connection between hidden layers is hierarchical
  • ART (Adaptive Resonance Theory) (Source code) - resonance connection, three-layered network
  • Feedforward Counterpropagation (Source code)- structure similar to the popular backpropagation network, three-layered, but non-hierarchical, there is also a connection between neurons in one layer

    Connections between neurons in the same layer can be:

  • Recurrent: the connection is realized in a way that neurons communicate their outputs with each other after they receive their inputs from another layer. The communication continues until neurons do not reach a stable condition. When the stable condition is reached, neurons are allowed to send their output to the next layer.
  • On-center/off-surround: a neuron in one layer has an excitatory connection toward itself and toward the neighbor neurons, but an inhibitory connection toward other neurons in the layer.

    Some of the "intra-layer" networks with recurrent connection are:

  • Hopfield's network (Source code) - two-layered, fully-connected, neurons of output layer are mutually connected with recurrent intra-layer connection
  • Recurrent Backpropagation network - recurrent intra-layer connection, but one-layered, where a part of neurons receive inputs, and the other part of neurons is fully connected with recurrent intra-layer connection

    Examples of the networks with on-center/off-surround connection are:

  • ART1, ART2, ART3 (Source code)- resonance on-center/off-surround connection
  • Kohonen's self-organizing network (Source code)

    Connection between input and output data

    NNs can also be distinguished according to the connection between input and output that can be:

  • autoassociative - input vector is the same as output (common in pattern recognition problems, where the objective is to obtain the same data in output as they are in input)
  • heteroassociative - output vector differs from the input vector. Autoassociative networks are used in signal processing, noise filtering and similar problems that aim to recognize the patterns of input data.

    Input and transfer functions

    When a neuron receives the input from the previous layer, the value of its input is computed according to an input function, usually called a "summation" function. In the simplest case, input of a neuron is the sum of all weighted outputs that arrive into that neuron. Besides this standard network input, there are other additional specific types of inputs in a network, including independant inputs from external environment and bias inputs. Input values can be normalized to an interval to avoid the extreme influence of high-valued inputs.

    After receiving the input according to the summation function, the output of a neuron is computed and sent to other neurons it is connected to. The output of neurons is computed according to the so-called transfer function. Several most frequently used transfer functions are:

  • step function,
  • signum function,
  • sigmoid function,
  • hyperbolic-tangent function,
  • linear function,
  • threshold linear function.

    Learning

    Every NN goes through three operative phases:

  • learning (training) phase - network learns on the training sample, the weights are being adjusted in order to minimize the objective function (for example RMS - root mean square error),
  • testing phase - network is tested on the testing sample while the weights are fixed,
  • operative (recall) phase - NN is applied to the new cases with unknown results (weights are also fixed).

    There are two main types of learning in a neural network: supervised and unsupervised. The difference between those two types is in the availability of known output in the training sample. In supervised learning, the set of training data consists of previous cases with known input and output values. The neural network system receives the actual output, computes the error and adjusts the weights according to the error.

    On the other hand, the actual outputs are not known in unsupervised learning. Inputs are available to the network, but the weights cannot be adjusted based on the actual output. This type of learning is commonly used for pattern recognition problems and clustering. Kohonen's self-organizing network is based on unsupervised learning.

    Learning rules

    A learning rule represents the formula that is used in NN to adjust the connection weights among neurons. Among various learning rules developed so far, four of them are most commonly used:

  • Delta rule (Widrow/Hoff's rule)
  • Generalized Delta rule,
  • Delta-Bar-Delta and Extended Delta-Bar-Delta rules,
  • Kohonen's rule