Psychology Wiki


Redirected from Parallel Distributed Processing

34,203pages on
this wiki
Add New Page
Talk0 Share

Assessment | Biopsychology | Comparative | Cognitive | Developmental | Language | Individual differences | Personality | Philosophy | Social |
Methods | Statistics | Clinical | Educational | Industrial | Professional items | World psychology |

Biological: Behavioural genetics · Evolutionary psychology · Neuroanatomy · Neurochemistry · Neuroendocrinology · Neuroscience · Psychoneuroimmunology · Physiological Psychology · Psychopharmacology (Index, Outline)

This article is in need of attention from a psychologist/academic expert on the subject.
Please help recruit one, or improve this page yourself if you are qualified.
This banner appears on articles that are weak and whose contents should be approached with academic caution

Connectionism is an approach in the fields of artificial intelligence, cognitive science, neuroscience, psychology and philosophy of mind. Connectionism models mental or behavioral phenomena as the emergent processes of interconnected networks of simple units. It founded on the assumption that all learning and behavior reflects the stimulus-response paradigm and it is these connections that form the basis of larger units.

There are many different forms of connectionism, but the most common forms utilize neural network models.

Basic principlesEdit

The central connectionist principle is that mental phenomena can be described by interconnected networks of simple units. The form of the connections and the units can vary from model to model. For example, units in the network could represent neurons and the connections could represent synapses. Another model might make each unit in the network a word, and each connection an indication of semantic similarity.

Spreading activationEdit

Most connectionist models include time, i.e. the network changes over time. A closely related and extremely common aspect of connectionist models is activation. At any time a unit in the network has an activation, which is a numerical value intended to represent some aspect of the unit. For example, if the units in the model are neurons the activation could represent the probability that the neuron would generate an action potential spike. If the model is a spreading activation model then over time a unit's activation spreads to all the other units connected to it. Spreading activation is always a feature of neural network models, and it is very common in connectionist models used by cognitive psychologists.

Neural networksEdit

Main article: Neural networks

Neural networks are by far the dominant form of connectionist model today. A lot of research utilizing neural networks is carried out under the more general name "connectionist". These connectionist models adhere to two major principles regarding the mind:

  1. Any given mental state can be described as a (N)-dimensional vector of numeric activation values over neural units in a network.
  2. Memory is created by modifying the strength of the connections between neural units. The connection strengths, or "weights", are generally represented as a (N×N)-dimensional matrix.

Though there is a large variety of neural network models, they very rarely stray from these two basic principles. Most of the variety comes from:

  • Interpretation of units—units can be interpreted as neurons or groups of neurons.
  • Definition of activation—activation can be defined in a variety of fashions. For example, in a Boltzmann machine, the activation is interpreted as the probability of generating an action potential spike, and it's determined via a logistic function on the sum of the inputs to a unit.
  • Learning algorithm—different networks modify their connections differently. Generally, any mathematically defined change in connection weights over time is referred to as the "learning algorithm".

Connectionists are in agreement that recurrent neural networks (networks wherein connections of the network can form a directed cycle) are a better model of the brain than feedforward neural networks (networks with no directed cycles). A lot of recurrent connectionist models incorporate dynamical systems theory as well. Many researchers, such as the connectionist Paul Smolensky, have argued that the direction connectionist models will take is towards fully continuous, high-dimensional, non-linear, dynamic systems approaches.

Biological realismEdit

The neural network branch of connectionism suggests that the study of mental activity is really the study of neural systems. This links connectionism to neuroscience, and models involve varying degrees of biological realism. Connectionist work in general need not be biologically realistic, but some neural network researchers try to model the biological aspects of natural neural systems very closely. As well, many authors find the clear link between neural activity and cognition to be an appealing aspect of connectionism. However, this is also a source of criticism, as some people view this as reductionism.


Connectionists generally stress the importance of learning in their models. As a result, many sophisticated learning procedures for neural networks have been developed by connectionists. Learning always involves modifying the connection weights. These generally involve mathematical formulas to determine the change in weights when given sets of data consisting of activation vectors for some subset of the neural units.

By formalizing learning in such a way connectionists have many tools at their hands. A very common tactic in connectionist learning methods is to incorporate gradient descent over an error surface in a space defined by the weight matrix. All gradient descent learning in connectionist models involves changing each weight by the partial derivative of the error surface with respect to the weight. Backpropagation, first made popular in the 1980s, is probably the most commonly known connectionist gradient descent algorithm today.


Connectionism can be traced back to ideas more than a century old. However, connectionist ideas were little more than speculation until the mid-to-late 20th century. It wasn't until the 1980's that connectionism became a popular perspective amongst scientists.

Parallel distributed processingEdit


Complex parallel distributed processing programs, such as PDP++ shown here, can result in powerful simulations

The prevailing connectionist approach today was originally known as Parallel Distributed Processing (PDP). PDP was a neural network approach that stressed the parallel nature of neural processing, and the distributed nature of neural representations.

PDP provided a general mathematical framework for researchers to operate in. The framework involved eight major aspects:

  • A set of processing units, represented by a set of integers.
  • An activation for each unit, represented by a vector of time-dependent functions.
  • An output function for each unit, represented by a vector of functions on the activations.
  • A pattern of connectivity among units, represented by a matrix of real numbers indicating connection strength.
  • A propagation rule spreading the activations via the connections, represented by a function on the output of the units.
  • An activation rule for combining inputs to a unit to determine its new activation, represented by a function on the current activation and propagation.
  • A learning rule for modifying connections based on experience, represented by a change in the weights based on any number of variables.
  • An environment which provides the system with experience, represented by sets of activation vectors for some subset of the units.

These eight aspects are now the foundation for almost all connectionist models.

A lot of the research that led to the development of PDP was done in the 1970s, but PDP became popular in the 1980s with the release of Parallel Distributed Processing: Explorations in the Microstructure of Cognition - Volume 1 (foundations) & Volume 2 (Psychological and Biological Models), by James L. McClelland, David E. Rumelhart, and the PDP Research Group. Although the books are now considered seminal connectionist works, the term "connectionism" was not used by the authors to describe their framework at that point. However it is now common to fully equate PDP and connectionism.

Earlier workEdit

PDP's direct roots were the perceptron theories of researchers such as Frank Rosenblatt from the 1950s and 1960s. However, perceptron models were made very unpopular with the release in 1969 of a book titled Perceptrons by Marvin Minsky and Seymour Papert. Minsky and Papert elegantly demonstrated the limits on the sorts of functions which perceptrons can calculate, showing that even simple functions like the exclusive disjunction could not be handled properly. The PDP books overcame this earlier limitation by showing that multi-level, non-linear neural networks were far more robust and could be used for a vast array of functions.

However, there were many researchers outside of the perceptron theorists who were advocating connectionist style models prior to the 1980s.

In the 1940s and 1950s researchers such as Warren McCulloch, Walter Pitts, Donald Hebb, and Karl Lashley were advocating connectionist style theories. McCullough and Pitts showed how first-order logic could be implemented by neural systems: their classic paper "A Logical Calculus of Ideas Immanent in Nervous Activity" (1943) is important in this development here (they were influenced by the important work of Nicolas Rashevsky in the 1930's). Hebb contributed greatly to speculations about neural functioning, and even proposed a learning principle that is still in use today, known as Hebbian learning. Lashley argued for distributed representations as a result of his failure to find anything like a localized engram in years of lesion experiments.

Connectionism apart from PDPEdit

Though PDP is the dominant form of connectionism, other theorists' work should be classified as connectionist.

Many connectionist principles can be traced back to early work in psychology such as the work of William James, although it should be pointed out that psychological theories based on what was then known about the human brain were quite fashionable at the end of the 19th century. As early as 1869, the neurologist John Hughlings Jackson was arguing for multi-level, distributed systems. Following from this lead, Herbert Spencer's Principles of Psychology, 3rd edition (1872), and Sigmund Freud's Project for a Scientific Psychology (composed 1895) propounded connectionist or proto-connectionist theories. However these tended to be speculative theories. But by the early 20th century Edward Thorndike was carrying out experiments on learning that posited a connectionist type network.

In the 1950s the researcher Friedrich Hayek posited the idea of spontaneous order in the brain arising out of decentralized networks of simple units, but Hayek's work was generally not cited in the PDP literature until recently.

Another form of connectionist model was the relational network framework developed by the linguist Sydney Lamb in the 1960s. Relational networks have only ever been used by linguists, and have never been unified with the PDP approach. As a result, relational networks are used by very few researchers today.

Connectionism vs. computationalism debateEdit

As connectionism became increasingly popular in the late 1980s there was a reaction against connectionism by some researchers, including Jerry Fodor, Steven Pinker, and many others. These theorists argued that connectionism, as it was being developed at that time, was in danger of obliterating the progress made in the fields of cognitive science and psychology by the classical approach of computationalism. Computationalism is a specific form of cognitivism which argues that mental activity is computational, i.e. that the mind is essentially a Turing machine. Many researchers argued that the trend in connectionism was towards a reversion to associationism, and the abandonment of the idea of a language of thought, something they felt was mistaken. On the other hand, it was those very tendencies that made connectionism attractive for other researchers.

Connectionism and computationalism need not be at odds per se, but the debate as it was phrased in the late 1980s and early 1990s certainly led to opposition between the two approaches. However, throughout the debate some researchers have argued that connectionism and computationalism are fully compatible, though nothing like a consensus has ever been reached. The differences between the two approaches that are usually cited are the following:

  • Computationalists posit symbolic models that do not resemble underlying brain structure at all, whereas connectionists engage in "low level" modeling, trying to ensure that their models resemble neurological structures.
  • Computationalists generally focus on the structure of explicit symbols (mental models) and syntactical rules for their internal manipulation, whereas connectionists focus on learning from environmental stimuli and storing this information in a form of connections between neurons.
  • Computationalists believe that internal mental activity consists of manipulation of explicit symbols, whereas connectionists believe that the manipulation of explicit symbols is a poor model of mental activity.

Though these differences do exist, they may not be necessary. For example, it is well known that connectionist models can actually implement symbol manipulation systems of the kind used in computationalist models. So, the differences might be a matter of the personal choices that some connectionist researchers make as opposed to anything fundamental to connectionism.

To make matters more complicated, the recent popularity of dynamical systems in philosophy of mind (due to the works of authors such as Tim Van Gelder) have added a new perspective on the debate. Some authors now argue that any split between connectionism and computationalism is really just a split between computationalism and dynamical systems, suggesting that the original debate was wholly misguided.

All of these opposing views have led to a fair amount of discussion on the issue amongst researchers, and it is likely that the debates will continue.

See alsoEdit


  • Ellis, R. & Humphreys, G. W. (1999). Connectionist Psychology. London: Erlbaum UK.ISBN 0863777872
  • Heinke, D., Humphreys, G. W. & Olson, A. (1999). Connectionist models in cognitive neuroscience - The 5th Neural Computation and Psychology Workshop. London: Springer-Verlag ISBN 185233052X
  • Rumelhart, D.E., J.L. McClelland and the PDP Research Group (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, Cambridge, MA: MIT Press
  • McClelland, J.L., D.E. Rumelhart and the PDP Research Group (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 2: Psychological and Biological Models, Cambridge, MA: MIT Press
  • Pinker, Steven and Mehler, Jacques (1988). Connections and Symbols, Cambridge MA: MIT Press.
  • Jeffrey L. Elman, Elizabeth A. Bates, Mark H. Johnson, Annette Karmiloff-Smith, Domenico Parisi, Kim Plunkett (1996). Rethinking Innateness: A connectionist perspective on development, Cambridge MA: MIT Press.

Johnson, M (1993) (Ed.), Brain Development and Cognition: A Reader. Oxford: Blackwell Publishers.

  • Marcus, Gary F. (2001). The Algebraic Mind: Integrating Connectionism and Cognitive Science (Learning, Development, and Conceptual Change), Cambridge, MA: MIT Press


Bates, E A & Elman, JL (1993) Connectionism and the Study of Change. In Johnson, M (Ed.), Brain Development and Cognition: A Reader. Oxford: Blackwell Publishers. Pp. 623-642.Full text

External linksEdit

Edit General subfields and scientists in Cybernetics
K1 Polycontexturality, Second-order cybernetics
K2 Catastrophe theory, Connectionism, Control theory, Decision theory, Information theory, Semiotics, Synergetics, Sociosynergetics, Systems theory
K3 Biological cybernetics, Biomedical cybernetics, Biorobotics, Computational neuroscience, Homeostasis, Medical cybernetics, Neuro cybernetics, Sociocybernetics
Cyberneticians William Ross Ashby, Claude Bernard, Valentin Braitenberg, Ludwig von Bertalanffy, George S. Chandy, Joseph J. DiStefano III, Heinz von Foerster, Charles François, Jay Forrester, Buckminster Fuller, Ernst von Glasersfeld, Francis Heylighen, Erich von Holst, Stuart Kauffman, Sergei P. Kurdyumov, Niklas Luhmann, Warren McCulloch, Humberto Maturana, Horst Mittelstaedt, Talcott Parsons, Walter Pitts, Alfred Radcliffe-Brown, Robert Trappl, Valentin Turchin, Francisco Varela, Frederic Vester, John N. Warfield, Kevin Warwick, Norbert Wiener
Types of learning
Avoidance conditioning | Classical conditioning | Confidence-based learning | Discrimination learning | Emulation | Experiential learning | Escape conditioning | Incidental learning |Intentional learning | Latent learning | Maze learning | Mastery learning | Mnemonic learning | Nonassociative learning | Nonreversal shift learning | Nonsense syllable learning | Nonverbal learning | Observational learning | Omission training | Operant conditioning | Paired associate learning | Perceptual motor learning | Place conditioning | Probability learning | Rote learning | Reversal shift learning | Second-order conditioning | Sequential learning | Serial anticipation learning | Serial learning | Skill learning | Sidman avoidance conditioning | Social learning | Spatial learning | State dependent learning | Social learning theory | State-dependent learning | Trial and error learning | Verbal learning 
Concepts in learning theory
Chaining | Cognitive hypothesis testing | Conditioning | Conditioned responses | Conditioned stimulus | Conditioned suppression | Constant time delay | Counterconditioning | Covert conditioning | Counterconditioning | Delayed alternation | Delay reduction hypothesis | Discriminative response | Distributed practice |Extinction | Fast mapping | Gagné's hierarchy | Generalization (learning) | Generation effect (learning) | Habits | Habituation | Imitation (learning) | Implicit repetition | Interference (learning) | Interstimulus interval | Intermittent reinforcement | Latent inhibition | Learning schedules | Learning rate | Learning strategies | Massed practice | Modelling | Negative transfer | Overlearning | Practice | Premack principle | Preconditioning | Primacy effect | Primary reinforcement | Principles of learning | Prompting | Punishment | Recall (learning) | Recency effect | Recognition (learning) | Reconstruction (learning) | Reinforcement | Relearning | Rescorla-Wagner model | Response | Reinforcement | Secondary reinforcement | Sensitization | Serial position effect | Serial recall | Shaping | Stimulus | Reinforcement schedule | Spontaneous recovery | State dependent learning | Stimulus control | Stimulus generalization | Transfer of learning | Unconditioned responses | Unconditioned stimulus 
Animal learning
Cat learning | Dog learning  Rat learning 
Neuroanatomy of learning
Neurochemistry of learning
Adenylyl cyclase  
Learning in clinical settings
Applied Behavior Analysis | Behaviour therapy | Behaviour modification | Delay of gratification | CBT | Desensitization | Exposure Therapy | Exposure and response prevention | Flooding | Graded practice | Habituation | Learning disabilities | Reciprocal inhibition therapy | Systematic desensitization | Task analysis | Time out 
Learning in education
Adult learning | Cooperative learning | Constructionist learning | Experiential learning | Foreign language learning | Individualised instruction | Learning ability | Learning disabilities | Learning disorders | Learning Management | Learning styles | Learning theory (education) | Learning through play | School learning | Study habits 
Machine learning
Temporal difference learning | Q-learning 
Philosophical context of learning theory
Behaviourism | Connectionism | Constructivism | Functionalism | Logical positivism | Radical behaviourism 
Prominant workers in Learning Theory|-
Pavlov | Hull | Tolman | Skinner | Bandura | Thorndike | Skinner | Watson 
Category:Learning journals | Melioration theory 
This page uses Creative Commons Licensed content from Wikipedia (view authors).

Ad blocker interference detected!

Wikia is a free-to-use site that makes money from advertising. We have a modified experience for viewers using ad blockers

Wikia is not accessible if you’ve made further modifications. Remove the custom ad blocker rule(s) and the page will load as expected.

Also on Fandom

Random Wiki