Modeling Neuromorphic Persistent Firing Networks

Neurons are believed to be the brain computational engines of the brain. A recent discovery in neurophysiology reveals that interneurons can slowly integrate spiking, share the output across a coupled network of axons and respond with persistent firing even in the absence of input to the soma or dendrites, which has not been understood and could be very important for exploring the mechanism of human cognition. The conventional models are incapable of simulating the important newly-discovered phenomenon of persistent firing induced by axonal slow integration. In this paper, we propose a computationally efficient model of neurons through modeling the axon as a slow leaky integrator, which captures almost all-known neural behaviors. The model controls the switching of axonal firing dynamics between passive conduction mode and persistent firing mode. The interplay between the axonal integrated potential and its multiple thresholds in axon precisely determines the persistent firing dynamics of neurons. We also present a persistent firing polychronous spiking network which exhibits asynchronous dynamics indicating that this computationally efficient model is not only bio-plausible, but also suitable for large scale spiking network simulations. The implications of this network and the analog circuit design for exploring the relationship between working memory and persistent firing enable developing a spiking network-based memory and bio-inspired computer systems.


Introduction
Understanding how human brain represents, stores, and processes information is one of the greatest unsolved mysteries and fundamental challenges of science today.Over the past century, since Lapicque introduced the integrate-and-fire model of the neuron in 1907 [1], computational neuroscientists have developed several mathematical and computational neural models.Generally, one approach in computational neuroscience involves creating biologically realistic models, where information about the biological details of neurons including their electrochemistry, biochemistry, and detailed morphology and connectivity are also included [2], such as Hodgkin-Huxley [3] and compartment models [4].Another approach involves building qualitative models to capture the spiking nature and the essential elements of the behavior with simplified complexity, for example, leaky integrate-and-fire [5], FitzHugh-Nagumo [6], Morris-Lecar [7], Hindmarsh-Rose [8], Wilson [9], Resonate-and-Fire [10] and Izhikevich [11] neuron models.
However, as neuroscience continues to advance rapidly, more and more complex neuronal behaviors and brain dynamics are revealed.This has posed challenges for neural modeling as conventional neuron models fail to reflect the newly-discovered complexity of neural systems, such as the persistent firing phenomenon recently observed in rodent hippocampal inhibitory neurons [12].In the classic viewpoint about the information flow in the nervous system, synaptic inputs are received and integrated in the dendrites only on a timescale of milliseconds to seconds, and when the depolarized somatic membrane potential exceeds the threshold, action potentials are triggered at the axon hillock and propagate along the axon.However, recent discovery reveals that some action potential began at the distal end of axon instead of at the axon hillock.A much slower form of potential integration is observed which leads to persistent firing in distal axons of rodent hippocampal and neocortical interneurons [12].The slow integration lasts from tens of seconds to minutes in distal axon, so does the persistent firing.During the persistent firing, the somatic depolarization is not observed, implying that axon may perform its own neural computations without any involvement of the soma or dendrites.
In this paper, we propose a new computationally-efficient artificial neuron model that account for all-known neural behaviors.In this model, axon is an independent computational unit complementary to the classic somatic computational unit which evokes action potentials.Compared to the soma which integrates dendritic inputs in a timescale of milliseconds to seconds, the axon integrates the spikes evoked by soma in a timescale of tens of second to minutes, and consequently determines the persistent firing behavior of the axon.Besides the computational model, a neuromorphic model of persistent firing neurons and its analog circuit are also proposed.In addition, a polychronous spiking network [13] with persistent firing inhibitory interneurons is simulated, which may assist the development of spiking network-based memory and bio-inspired computer system.

Persistent Firing Neuron Model
The conventional models such as Izhikevich neuron model [11] are incapable of simulating the important newlydiscovered phenomenon of persistent firing induced by axonal slow integration, as a recent discovery in neurophysiology reveals that interneurons can slowly integrate spiking, share the output across a coupled network of axons and respond with persistent firing even in the absence of input to the soma or dendrites.To this end, we extend the Izhikevich neuron model by modeling the axonal slow integration, which can be mathematically described by Equations ( 1)-( 6): ( ) with the auxiliary resettings: where ' d dt = , t is the time, w describes the hyper theoretical potential of the axonal leaky integrator, v and u are the dimensionless variables that describe the membrane potential, the membrane recovery, respectively.Parameters , , , , a b c d e and f are dimensionless, a describes the timescale of u , b represents the sensitivity of u to the subthreshold fluctuations of v , c is the after-spike reset value of the membrane potential, d describes the after-spike reset of u , e represents the after-somatic-spike axonal accumulation in the passive conduction mode, and f is the value describing the rate of the axonal leak.In Equations ( 5) and ( 6), p w is the high threshold (rising edge) value of w to trigger the persistent firing mode of axon, and n w is the low threshold (falling edge) value of w for the axon to return to passive conduction mode.( ) This model extends the Izhikevich neuron model [11] described by Equations ( 1)-( 2), which well captures the somatic spiking dynamics.By modifying the , , , a b c d variables in Equations ( 1), (2), and ( 4), different firing patterns can be generated.One important concept of our model is the axon-dependent switch of the axonal firing patterns through dynamically selecting different variable set of , , a b c and d .The axon is modeled as a slow leaky integrator, which is capable to alter the axonal functions between passive conduction and persistent firing modes, depending on the potential of axonal integrator.In the passive conduction mode, the axon acts as a transmission cable in which stable propagation occurs once an action potential is evoked by synaptic inputs.In the persistent firing mode, in which the parameters in the model are chosen to set the dynamical system to be self-oscillatory, axon acts as a bistable oscillator which does not require stimulus from dendrites to sustain the persistent firing of action potentials.
In contrast to the somatic leaky integrator which accounts for the integration of dendritic inputs, the axonal leaky integrator has a larger time constant for its integration and leakage, integrating incoming spikes generated in the axon hillock in the timescale from tens of seconds to minutes, due to its slow rate of leakage f , as described in Equation (3).When the potential in the axonal leaky integrator exceeds p w , which denotes the high threshold (rising edge), the persistent firing will be triggered, and the firing pattern is determined by the parameter set of ( ) , , , , p a b c d e .If there are no somatic spikes accumulated in the axon, the potential $w$ decreases at the rate of f .When w reaches the low threshold (falling edge), the axon n w returns to passive conduction mode, and the , , , , a b c d e variables in the model are reset to ( ) , , , , n a b c d e .We have simulated the persistent firing neuronal behaviors under two stimulation protocols, with the parameters summarized in Table 1.The simulation results under step/pause stimulation protocol [12] are shown in Figure 1.We applied 1-second current step of 500 pA during each 10-second sweep to the simulation model.During the sweeps of the step/pause input stimulus, it is observed that the somatically evoked action potentials appear when the synaptic input stimulus is presented and disappear when the stimulus is removed (Figure 1(a)).When w accumulates to a level higher than the high threshold p w , persistent firing occurs (Figure 1(a)), 12th sweep), after which w decreases monotonically due to axonal leakage and no stimulus (Figure 1(d)).The per- sistent firing ends when w is lower than the low threshold n w .It should be noted that in this model, the resting potential of persistent firing spikes is about 20 mV below the one of somatically evoked action potentials, which fits well with the data from current-clamp recordings in the persistent firing neurons [12] [14].We emulate this phenomenon through switching the parameter c in Equation ( 4) between p c and n c .The variation of persistent firing frequency was also simulated (Figure 1(b) and Figure 1(c)).As the firing frequency can be tuned by the parameter b in Equation ( 2), the decrease of persistent firing frequency observed in the current-clamp experiment can be emulated in the model through defining the following relationship between parameter b and f .) The instantaneous firing frequency of axon-evoked action potentials attains its maximum (40 Hz) at the onset of persistent firing, and decreases over time till the end of persistent firing.
We also applied the stimulation with a 40-second long pulse of 15 pA current to the persistent firing neuron model described by Equations ( 1)-( 7) with a different parameter set (Table 1).The simulation results are shown in Figure 2. The repeated somatic current stimuli eventually trigger the persistent firing that outlasts the current stimuli by more than 30 seconds.
By tuning the parameters of , , p e f w and n w , we can set different time scales for the axonal slow integration, allowing the model to accommodate different types of neurons with different persistent firing behaviors.

Model Analysis
In this section, we present the mathematical analysis on the conditions which guarantee the existence of the persistent firing behavior in the neuron.Let [ ] X uvw ′ = . Equations ( 1)-( 3) can be rewritten as: where I is the input current, , , A B C and H are the corresponding matrices determined by Equations ( 1)-(3).When the input current I is not present, the equilibrium points of the dynamic system in Equation ( 8) are the solutions of the Equation ( ) 0 F X = , the two equilibrium points are obtained ( ) ( ) Here, we assume that + − > and it is noted that 1 X is a stable equilibrium point while 2 X is unstable.Based on the analysis of stability of the equilibrium points and the nonlinear dynamic characteristic of Equation ( 8), the following conditions can be summarized to guarantee the existence of the persistent firing behavior.
Condition 1.The input current Condition 1 means that there is no equilibrium point for the equation ( ) Otherwise, X may approach the equilibrium point and not change any more even when I is present.In this case, it is impossible for persistent firing to exist.Condition 2 requires that the neuron fires a sufficient number of spikes to ensure that In the above analysis, we choose b to be such that 0 ∆ > .Now we analyze the case that 0 ∆ ≤ .It can be shown that there is no real equilibrium point or at most a unstable equilibrium point for the equation Then Condition 1 is achieved even if 0 I = .The persistence firing can be always observed no matter whether the Conditions 2 and 3 are satisfied or not.However, in this case, the nonlinear system ( ) is unstable.This is the reason that generally b is suggested to be chosen as a value such that 0 ∆ > .The geometrical approach provides a clear and insightful perspective of investigating the characteristics of dynamical system of neuron [15], and is therefore adopted to analyze the persistent firing model.We choose conveniently to conduct geometrical analysis on the phenomenon of persistent firing due to a long-lasting pulse stimulation (Figure 2 and Figure 3(a)) rather than on that due to a pulse train (Figure 1), since the geometry of the former is less complicated yet retains significant information regarding the time evolution of the dynamical system of neuron, such as the stable equilibrium and the threshold values, as shown in Figure 3(b) where the geometrical trajectory of the state variables u , v , w is plotted in three-dimensional phase space.It is worth mentioning that a new set of values in Table 2 is assigned to the model's parameters to further reduce the complexity of the geometry, as shown in Figure 3(a) and Figure 3(b).For the convenience of analysis, the projections of the 3-D trajectory on the uw , wv , and uv phase planes are illustrated in Figure 3(c), where five points in time (t_0, t_1, t_2, t_3, t_4) are labeled on the curves, so as to partition the whole course of persistent firing into four temporal stages: Stage 1: During the time range of 0 1 t t → , the system begins with its initial state at 0 t , converges to a stable equilibrium, stay there until 1 t .The stable equilibrium can be calculated to be [ ]

Neuromorphic Model of the Artificial Neuron
Since the last two decades, there has been a continuing interest in developing the neuromorphic circuits and systems that mimic the operation of biological neurons and brains, which enables considerably faster and more energy-efficient emulations of the neurons and neural systems.Due to the computational simplicity of our model, it is rather straightforward to implement the proposed neuron model in hardware, either in digital circuits or analog circuits.There have been several circuit implementations of Izhikevich neuron model and its variants [16] [17].Thus in our case it is intuitive to add a leaky integrator emulating the axon, as well as the switching devices for selecting one of two sets of , , , , a b c d e parameters, which may be stored in memory devices, e.g.nonvolatile memory.
A conceptual design of the proposed artificial neuron are shown in Figure 4, where the axonal spikes are originated from the unstable state of the neuron circuit and the artificial neuron is considered as a parameter-dependent dynamic system.The parameters of such dynamic system can be stored in non-volatile memory devices.Through modifying the parameter values in the memory, the spiking neuron unit can alter the axonal dynamics between the passive conduction mode and the persistent firing mode.
We have built the neuromorphic model based on the concept design with SPICE and Verilog-A languages, and simulated the model in Synopsys® HSPICE Simulator.The circuit model is shown in Figure 5(a) and Figure 5(b).The soma membrane circuit (Figure 5(a)) essentially implements the Izhikevich neuron model [10], which includes a SPICE block describing Equations ( 1)-( 2), an adder, a Schmitt trigger and two NMOS transistors (M1, M2) as switches.The stimulus I is the only external input to the soma unit, which also requires to fetch a parameter set of , , p a b V and c from its memory during the operation.The Schmitt trigger imple- mented in Verilog-A, compares the membrane potential v with p V (peak of the spike, typically 30 mV) and c (rest potential, typically −65 mV), and produces the output v C , of VDD only when p V > , and GND only when v c < .When v c is high (VDD), which means a spike is generated, the two NMOS switches (M1, M2) are turned on thus the potentials of u and v are reset to new u and c , respectively.When v C is low (GND), the potentials of $u$ and v remain to be solely determined by Equations ( 1)-( 2) in the SPICE block., and the neuron spiking dynamics.Based on the long-pulse stimulation protocol and the circuit parameters given in Table 3, the simulation results of the circuit are shown in Figure 6.It should be noted that the values of the stimulus current I and simulation time here are different from those discussed in the previous section, with the purpose of achieving simpler implementation in the circuits.Nevertheless, it can be clearly seen that after we continuously applied the stimulus current for a period of time and stopped the input, the persistent firing of spikes was triggered, lasted for a similar duration, and finally stopped, depending on the level of axonal integrator potential w .

Persistent Firing Spiking Network
We have used this model to simulate a spiking network of 1000 randomly connected neurons.The network takes into account of axonal conduction delays [18] and spike-timing-dependent plasticity (STDP), which is considered to be "polychromous" [13].The network consists of 800 excitatory neurons with regular spiking pattern [11] and 200 inhibitory neurons.As around 80% of EGFP-positive interneurons were found to have persistent firing behavior by repeated somatic current injection [12], we set in the 80% simulation, or 160 of the inhibitory neurons to be capable of persistent firing while the remaining 20% are fast spiking inhibitory neurons.
Each excitatory neuron is connected to 100 random neurons, and each inhibitory neuron, including persistent firing neuron, is randomly connected to 100 excitatory neurons only.Each synaptic connection has a fixed integer conduction delay between 1 ms and 20 ms.The conduction delay in the range of 1 -20 ms is randomly as-  signed to all excitatory synapses, while the 1 ms delay are assigned to all inhibitory synapses [13].The synaptic connections are modified according to the STDP rule [19].The implementation of the STDP function is based on Equation ( 9): ( ) e for 0 e for , 0 persisting firing of inhibitory interneurons in the network, leads to reciprocal inhibition in a longer timescale and thus shut down the activities for a longer period (Figure 7(b) middle column).The synchronization of persistent firing interneurons could contribute to the beta and gamma oscillation, whose frequency ranges are close to the persistent firing frequency [20].

Discussion
The possible functions of the persistent firing were suggested to be related to working memory [12].The ability to maintain the persistent firing of action potentials without on-going stimulation provides a mechanism of storing the information for a short period of time.This mechanism is similar to our working memory [21]- [23], which actively holds a limited amount of information [24] in the absence of stimuli.Working memory has been extensively explored from perspectives of highly abstract top levels in the domains of psychology, neuroscience and anatomy, but there are much less work from perspectives of bottom level of biological neurons [25]- [30].It is still unknown how working memory is represented within a population of cortical neurons.
The presented simple model of persistent firing neurons enables further investigation of possible functions of persistent firing, especially the relationship between working memory and persistent firing, and the neural correlate of working memory.Recently, we have proposed a model of short term persistent habituation [31], consisting of a persistent firing neuron and a habituating synapse, to explore the presynaptic learning and memory.The interaction of persistent firing axonal and presynaptic processes increases the retention time of the synaptic conductance and therefore the recovery time, and continues the learning of short term habituation for the duration of persistent firing.This leads to a working memory for habituation.
Through incorporating the persistent firing dynamics in spiking networks with axonal delays and STDP learning rules, we can further investigate the interaction of polychronization and persistent neural activities.In the polychronous spiking network, the number of co-existing polychronous groups far exceeds the number of neurons in the network, resulting in an unprecedented memory capacity of the system [13].Thus it would be interesting to investigate how working memory is presented in the polychronous network, and simulate a bioplausible working memory system with increased memory capacity.To this end, we are working towards the development of artificial cognitive memory with the objective of developing a novel function-driven memory technology in comparison to conventional density-driven storage technology [32].The models of persistent firing neuron and spiking network presented in this paper can be used in the simulation of cognitive memory architectures.
Due to the computational simplicity of our model, it is straightforward to implement the model in hardware.We have developed a neuromorphic model of persistent firing neurons [33], which reproduces the neuronal persistent firing behavior by integrating somatic and axonal computational processes of different timescales.Considering there are many existing VLSI implementations of Izhikevich neuron model and its variants [16] [34]- [36], the proposed neuron model can be conveniently implemented in silicon by incorporating the axonal leaky integrator into the Izhikevich VLSI designs, enabling a considerably faster emulation of the neural systems with persistent neural activity in a highly parallel manner.

Conclusion
Based on the recent discovery in neurophysiology which revealed that interneurons can slowly integrate spiking, share the output across a coupled network of axons and respond with persistent firing even in the absence of input to the soma or dendrites, we proposed a new model of persistent firing neuron to bridge the gap between the conventional models and the newly-discovered phenomenon.In this work, we presented and discussed the mathematical and neuromorphic models of the artificial neuron, as well as a persistent firing polychronous spiking network which exhibits asynchronous dynamics.The artificial neuron we proposed, being computationally efficient yet bio-plausible, would be useful to construct and simulate the large scale models of animal or human cortex, which provides a neuromorphic platform for further investigation of the possible functions of persistent firing and their roles in animal and human brain, especially for exploring the relationship between working memory and persistent firing spiking network-based memory and the bio-inspired computer systems.
d e a b c d e ≤ ← d e describe the parameter sets of , , , , a b c d e variables when the axon is in the persistent firing mode and passive conduction mode, respectively.

Figure 1 .
Figure 1.Simulation of persistent firing neuron model.(a) The waveform of membrane potential.The simulation used the same stimulation protocol as the experiment of whole-cell current-clamp recording by Sheffield et al [12].To evoke persistent firing, 1-s current step of 500 pA was delivered to the neuron during each 10-s sweep (left, sweeps 1 -6; right, sweeps 7 -12).In this simulation, persistent firing was evoked after the 12th sweep, and the total number of evoked action potentials before persistent firing was 1278.(b) The waveform of persistent firing, which lasted over 2 minutes.The total number of evoked axonal action potentials was 3073.The simulation reflects the decrease of instantaneous firing frequency, which was shown in the current-clamp experiment.(c) The instantaneous firing frequency of persistent firing action potentials, which decreased from 40 Hz at the onset of persistent firing to 12 Hz at the end of persistent firing as the parameter b decreased in this simulation. (d) The time response of w , which represents the hypertheoretical potential of the axonal leaky integrator.

Figure 2 .
Figure 2. Simulation of persistent firing neuronal behaviors with long pulse stimulation protocol.(a) The input stimulus.A pulse of current with 15 pA amplitude lasts 40 seconds as the synaptic input to the neuron.(b) Time response of w , the variable describing the potential of the axonal leaky integrator.(c) Time response of v , which describes the membrane or axon potential.The first part is the fast spiking waveform of v when the 15pA current step is present, and the second part is the persistent firing waveform of v .
later than the time when I is removed.Condition 3. When p w w > , the parameter c is denoted as p c in Equation (5) (during the persistent firing period) is chosen to be the value such that before I is removed.Condition 3 implies that, during the persistent firing period,

2 : 3 : 4 :
During the time range of 1 2 t t → , the system exhibits action potential evoked by a rectangular current signal, which also leads to the persistent firing in the next stage.StageDuring the time range of 2 3 t t → , the system exhibits persistent firing, which is triggered at 2 t when w reaches the upper threshold 0During the time range of 3 4 t t → , the system returns to the state of stable equilibrium and rests there.

Figure 6 .
Figure 6. Circuit simulation results of the artificial neuron, showing that after a continuous applied stimulus stops, the slow integration in the axon leads to persistent firing of spikes.Top: the membrane potential v ; Middle : the potential of axonal integrator w ; Bottom: the stimulus current I .

Figure 7 .
Figure 7. (a) A simplified example of the persistent firing spiking network structure.Among the neurons ranging from A to F in the network, (B, C, D, E) represent the excitatory neurons, and (A, F) represent inhibitory neurons in which F is capable for persistent firing.(b) Raster plots of spike activities in the spiking network of 1000 neurons including persistent firing inhibitory interneurons.Each raster plot shows the 1-second segment of the firing activity.Neurons indexed from 1 to 800 are excitatory neurons, and neurons indexed from 801 to 1000 are inhibitory.Within the 200 inhibitory neurons, there are 160 randomly selected neurons with persistent firing capability.Horizontally continuous dotted lines with neuron index in the range of 801 -1000 indicate there are persistent firings of action potentials in inhibitory neurons.