“ There are two very compelling reasons why mathematical and physical models of the neuron should be built. Model building, while widely used in the physical sciences, has been largely neglected in biology. However, there can be little doubt that building neuron models will increase our understanding of the function of real neurons, if experience in the physical sciences is any guide. Secondly, neuron models are extremely interesting in their own right as new technological devices. ”

“ The function of nerves is to carry impulses—like telegraph wires. The functions of brains is something else.” But, confronted with such awesome experimental difficulties, with no comprehensive mathematical theory in sight, we are largely limited otherwise to verbal discourses, rationales and theorizing, a hopelessly clumsy tool for the development of an adequate understanding of brain function. ”

“ Each element requires a bias, an error signal, and the desired final output, these things being uniform for all elements in a network. Some external device must manipulate the bias properly, but this is a simple behavior depending only on an error signal and the desired final output—not on the state of individual elements in the network. What one has, then, is a network consisting of elements which are nearly autonomous as regards their decisions to change weights. Such a scheme appears to be the only way to avoid constructing a central weight-change decision apparatus of great complexity. ”

“ The most dramatic aspect of neuron function is, of course, the axon discharge. It is this which gives the neuron its “all-or-nothing” character, and it is this which provides it with a means for propagating its output pulses over a distance. Hodgkin and Huxley (1) have developed a very complete description of this action. Their model is certainly without peer in describing the nature of the real neuron. ”

“ But, if we are to stick by our definition of a condition of semipermanent change of behavior as a criterion for learning, then we may also be misled into considering the development of a neurosis, for example, as learning, or even a deep coma as learning. ”

“ The polarity of an individual input determines whether it shall be excitatory or inhibitory. This much of the circuitry is very similar to Harmon’s model.
Learning is postulated to take place in the following way: when the neuron fires, an outside influence (the environment, or a “trainer”) determines whether or not the result of firing was desirable or not. If it was desirable, the threshold of the neuron is lowered, making it easier to fire the next time. If the result was not desirable, the threshold is raised, making it more difficult for the neuron to fire the next time. ”

“ The central problem in network learning is that of determining, for a given input, the set of elements whose outputs can be altered so as to correct the final element, and which will do the least amount of damage to previous adaptations to other inputs. Once this set has been determined, the incrementing rule given for a single element will apply in this case as well (subject to the restriction of leaving interconnecting weights positive) , since the desired final output coincides with that desired for each of the elements to be changed (because of positive interconnecting weights) . ”

“ It can be demonstrated that if a sufficiently large number of linear threshold elements is used, with the outputs of some being the inputs of others, then a final output can be produced which is any desired logical function of the inputs. The difficulty in such a network lies in the fact that we are no longer provided with a knowledge of the correct output for each element, but only for the final output. If the final output is incorrect there is no obvious way to determine which sets of weights should be altered. ”

“ In the companion article [8] we will define a self-organizing system as one which, after observing the input and output of an unknown phenomenon (transfer relation) , organizes itself into a simulation of the unknown phenomenon.
Within the mathematical model, the aforementioned phenomenon may be represented as a topological space thus omitting for the moment the (arbitrary) designation of input and output which, as will be shown, bears on the question of uniqueness. ”

“ In terms of functional operation, we can think of the synapse as a transducer. The input to this transducer is a spike or series of spikes in the presynaptic axon. The output is an accumulative, long-lasting potential which in some way (perhaps not uniquely) represents the pattern of presynaptic spikes. The pacemaker appears to perform the function of a clock, producing periodic spikes or spike bursts or producing periodic changes in the over-all excitability of the neuron. The graded-response regions appear to act as nonlinear amplifiers and, occasionally, spike initiators. ”

“ In putting forth a model for such an adapting or “learning” network, a requirement is laid down that the complexity of the adaption process in terms of interconnections among elements needed for producing appropriate weight changes, should not greatly exceed that already required to produce outputs from inputs with a static set of weights. In fact, it has been found possible to use the output-from-input computing capacity of the network to help choose proper weight changes by observing the effect on the output of a variety of possible weight changes. ”

“ Thus, since any individual component of a random cartesian product may be studied independently of the others, all that one needs to study is: (1) a Gaussian distribution on a single real axis and (2) the irreducible denumerable groups.
Finally, it should be emphasized that there are only these two ways of decomposing a metrizable topology; (1) if a (statistical) basis is given, use the diagonalization of a symmetric matrix algorithm described earlier (and given in detail in the three channels in cascade problem) , and (2) otherwise use a suitable network of the NPO’s with n0=1. ”

“ It is assumed that in addition to its normal inputs, each element receives a variable input bias which we can call b. The output of every element should then be determined by the sign of the usual weighted sum of its inputs plus this bias quantity. This bias is to be the same for each element of the network. If b = 0 the network will behave as before. However, if b is increased gradually, various elements throughout the network will commence changing from -1 to +1, with one or a few changing at any one time as a rule. If b is decreased, the opposite will occur. ”

“ If the intensity of the stimulus is below this threshold, the neuron does not fire.
c. The neuron is capable of temporal and spatial integration. Many subthreshold stimuli arriving at the neuron from different sources, or at slightly different times, can add up to a sufficient level to fire the neuron.
d. Some inputs are excitatory, some are inhibitory.
e. There is a refractory period. Once fired, there is a subsequent period during which the neuron cannot be fired again, no matter how large the stimulus. This places an upper limit on the pulse rate of any particular neuron. ”

“ In the first place, we have to select the space in which the problem is to be set. The most general space that we feel capable of tackling is a metrizable topology. On the other hand, anything less general would be unnecessarily restrictive. Thus, we choose a metrizable topological space. [Pg 66]
As soon as we have made this choice, we regret it. In order to improve the situation somewhat, we show that there is no (additional) loss of generality in using an orthogonal Euclidean space times [9] a denumerable random cartesian product of irreducible (wrt direct product) denumerable groups. ”

“ On the outer fringes of biology, still less interest has been displayed; the philosophical vitalists would surely be revolted by the idea of such models of mind and memory, and at the other end of the scale, contemporary computer engineers generally assume that a nerve cell operates much too slowly to be of any value. ”

“ Each can make the selections it prefers, because the parameters of real neurons which are necessary and sufficient for learning have not been exhaustively identified.
From the point of view of one whose interests include real neurons, this lack of identification is unfortunate. I once wrote a book which included some guesses about the essential attributes of neurons. Since that time, many neuron simulation programs have been written. But these programs, although interesting and worthwhile in their own right, have done little to answer the question of the necessary parameters. ”

“ The human brain contains approximately 1010 neurons to which the neuron theory assigns the primary role in central nervous activity. These cells occupy, however, a relatively small fraction of the total volume. There are, for example, approximately 10 times that number of neuroglia, cells of relatively indeterminate function. Each neuron (consisting of cell body, dendrites and, sometimes, an axon) comes into close contact with the dendrites of other neurones at some thousands of places, these synapses and “ephapses” being spaced approximately 5μ apart (1) . ”

“ In the absence of a priori knowledge of the environment, the self-organizing machine must resort to a sequence of projections on unit spheres to effect this decomposition. Such a sequence of projections can be provided by repeated use of a nilpotent projection operator (NPO) . An analog computer mechanization of one such NPO is discussed and the signal processing behavior of the NPO is presented in detail using the Euclidean geometrical representation of the metrizable topology provided in the companion paper. ”

“ We should turn our attention to the properties of real neurons, and see which of them are the most important ones for us to imitate. Obviously, we cannot hope to imitate all the properties of a living neuron, since that would require a complete simulation of a living, metabolizing cell, and a highly specialized one at that ”

“ If the stimulus potential is a voltage ramp, for example, the response will depend on the slope of the ramp. If the rate of rise is sufficiently low, the membrane will respond in a passive manner to voltages much greater than the spike threshold for suddenly applied voltages. ”

“ Experiments have been performed on simplifications of the model described so far which had the occasional result that a small number of elements came to a state where they received most of the weight increments, much to the detriment of the learning process. In such cases the network behaves as if it were composed of many fewer adjustable elements. In a sense this is asking each element to maintain a record of the data it is being asked to store so that it does not attempt to exceed its own information capacity. ”

“ The mechanization of the circuitry is rather straight-forward. A portion of the output of the pulse generator is routed through a “pulse-stretcher” or short-term memory which temporarily records the fact that the neuron has recently fired. The pulse-stretcher output controls a gate, which either accepts or rejects the P-R signal. ”

“ Figure 6 shows a sample gold dendrite grown by electrodeposition (actual size, about 1 mm) from a 54% nitric acid solution to which gold chloride was added. When such a dendrite is attached to a piece of iron (both submerged) , activation of the excitable element produces a field in such a direction as to promote further growth of the dendritic structure. Thus, if gold chloride is added to the solution used in the elementary interaction cells described above, all input influence “weights” tend to increase with use and, hence, produce a plasticity of function. ”

“ We choose to impose the additional condition that the resulting metric space be a Euclidean geometry with a rectangular coordinate system. [Pg 67]
Even this always does not yield uniqueness, but we will show the additional restriction that will guarantee uniqueness after the necessary language is developed. Since all metrizations of a given metrizable topology are isomorphic, in the quotient class the orthogonal Euclidean geometry serves the purpose of being a convenient representative of the unique element resulting from a given metrizable topology. ”

“ The spike frequency increases monotonically with increasing leakage current. In addition, if the sodium conductance inactivation is allowed to accumulate over several spikes, periodic spike pairs and spike bursts will result. Subthreshold pacemaker potentials have also been observed in the model, but with parameter values set close to the Hodgkin-Huxley data these are generally higher in frequency than pacemaker potentials in real neurons. It is interesting that a pacemaker mode may exist in the absence of the simulated sodium conductance. ”

“ The neuron responds to a stimulus with an electrical pulse of standard size and shape. If the stimulus continues, the pulses occur at regular intervals with the rate of occurrence dependent on the intensity of stimulation. ”

“ For small values of input voltage, the response curve is linear; the membrane is passive. As the stimulus voltage is increased, however, the response becomes more and more disproportionate. The membrane is actively amplifying the stimulus potential. At even higher values of stimulus potential, the system becomes regenerative ”

“ Strong Coupling
If two touching pieces of iron are placed in a bath of nitric acid, a wave generated on one will ordinarily spread to the other. As is to be expected, a similar result is obtained if the two pieces are connected through an external conducting wire. However, if they are isolated, strong coupling does not ordinarily occur, especially if the elements are small in comparison with a “critical size,” σ/ρ where σ is the surface resistivity of passive iron surface (in Ω-cm2) and ρ is the volume resistivity of the acid (in Ω-cm) . ”