- This page was last modified on 25 June 2014, at 12:09.
- This page has been accessed 8,936 times.
-
![\"Creative](\"http://i.creativecommons.org/l/by-sa/3.0/88x31.png\")
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. -
![\"Powered](\"/CompCogNeuro/skins/common/images/poweredby_mediawiki_88x31.png\")
(click and Save As to download, then open in Emergent)
This simplified basal ganglia (BG) network considers learning and action selection among just two alternative responses (but see Frank (2006) for a model with four alternative choices and also explores the function of the subthalamic nucleus and the 'hyperdirect' pathway).
| ? It is recommended that you click here to undock this document from the main project window. Use the Window menu to find this window if you lose it, and you can always return to this document by browsing to this document from the docs section in the left browser panel of the project's main window. |
The BG circuitry is notoriously complex and counterintuitive, involving a combination between excitatory and inhibitory projections and multiple pathways. Given a sensory stimulus, the pre/motor cortex generates candidate motor actions, and then the BG selectively \"gate\" one of these actions to be executed while suppressing the other action. This gating process occurs via a sequence of steps from the Striatum through two opposing pathways (the direct and indirect pathways) to the output nucleus, the Thalamus, and back up to cortex. The BG learn which actions to select and which to suppress as a function of reinforcement signals encoded by dopamine. That is, unlike the error-driven *supervised* learning mechanisms in other task learning simulations, this network learns to make responses based purely on an unsupervised reinforcement signal conveying the reward value of outcomes that follow actions, and is never 'told' which response it should have made.
| ? As usual begin by inspecting the pattern of weights in the network. |
| ? To get a sense of how the dynamics of action selection work in more detail, step through a single trial a few cycles at a time. Click Init, and then Cycle on the MasterControl panel. (You can choose to step 5 Cycles at a time instead of clicking this button for every cycle by checking off the '5' box next to Step). During the initial cycles of settling you can observe the basic \"default\" function of the BG to suppress responses. A stimulus input pattern is presented, and both competing responses initially become noisily activated (or \"considered\") in pre/motor cortex. (This noise is helpful for producing exploratory motor actions before learning has occurred). In the absence of Striatum activity, neurons in the GP_Int (globus pallidus internal segment, the output nucleus of the BG) are tonically active (this is due to a reverse leak current in which positive ions leak into the cell rather than out of it as usual). Importantly, the projections from GP_Int units to the Thalamus are inhibitory, so that GP_Int activation chronically suppresses the Thalamus. Because bottom-up thalamic-cortical activity is required for a motor response to become sufficiently activated, this thalamic inhibition prevents all responses from getting executed, leading to only noisy cortical activity and no action selection. |
| ? Click a few more cycles until you see activity in the Striatum in response to the Input pattern. The Striatum is divided into two layers, with \"Go\" units in the left layer, and \"NoGo\" units in the right, and separate columns for each response. The first column of Go units receives input from the first column of motor units (which we will call R1), whereas the second column receives input from the second column (R2), and similarly for NoGo units (columns 1 and 2 of the Nogo Layer are NoGo-R1 and NoGo-R2). However, the particular Go or NoGo units within a column that get active depend also on the sensory stimulus Input. Thus the striatal units encode conjunctions between stimulus input and actions that are considered in motor cortex, so that some units represent Go-R1 for a particular stimulus, whereas others may become active for another stimulus. If a particular column of Go units is more active than the NoGo units, it will tend to inhibit the corresponding column of the GP_Int, which ultimately will allow that action to be executed. |
| ? Step through a few more cycles until you see this inhibition of a column in GP_Int. You should see that as the GP_Int units become inhibited, the Thalamus is no longer suppressed. This process is referred to as \"disinhibition\" because the effect of the striatal Go activation is not to directly excite, but only to remove inhibition of GP_Int onto Thalamus. The corresponding column of Thalamus will become excited only if it also receives top down activation from motor units in the same column. It is this property that makes the BG contribution to action selection a *gating* process: noisy striatal activity would not by itself select an action unless the motor cortex was already 'considering' that action as a plausible candidate. Similarly, in some cases both responses may get a similar level of Go activity in striatum initially, and in this case only the one also having greater cortical activity will be gated. |
You should see that once a Thalamic unit is active, the corresponding column of cortical motor units (PMC) quickly becomes maximally active, while the competing motor column is inhibited. (There is lateral inhibition between the competing motor responses, and the thalamic activation gives the winning column sufficient activity so that the other column is completely inhibited).
| ? Once you are finished watching the network activations evolve in this trial, switch to stepping by Settle instead of Cycle. Step through a few trials in the same way. You may also see some NoGo activity (in the right half of striatal units), which would serve to prevent selection of the corresponding response, because NoGo units project to and inhibit the external segment of the globus pallidus (GP_Ext), which in turn send inhibitory projections to the GP_Int. Thus whereas Go activity disinhibits the Thalamus, NoGo activity has the opposite effect, further activating GP_Int so that the Thalamus remains inhibited. Before learning. whether Go or NoGo activity for a given response predominates is somewhat arbitrary (related to random synaptic weights and overall dopamine levels), but informative differences will emerge with learning. |
Each trial consists of two settling phases: one in which the network selects a response, and another in which it receives dopaminergic feedback about whether the outcome was good or bad. You should also see that depending on whether the network responded correctly or not (in the task described below), there will be either a dopamine burst (SNc units completely activated) or dip (complete inhibition). This reinforcement forms the basis for learning in the model, as described next.
Dopamine (DA) from the SNc modulates the relative balance of activity in Go versus NoGo units via simulated D1 and D2 receptors. Dopamine effects are greatest on those striatal units that are already activated by corticostriatal glutamatergic input. Go units activated by the current stimulus and motor response are further excited by D1 receptor stimulation. In contrast, DA is uniformally inhibitory on NoGo units via D2 receptors. This differential effect of DA on Go and NoGo units, via D1 and D2 receptors, affects performance (i.e more tonic DA leads to more Go and associated response vigor, faster reaction times) and, critically, learning.
Specifically, when the network selects the 'correct' response, a dopamine burst subsequently reinforces the response, further exciting Go units and inhibiting NoGo units. Learning occurs during this phasic DA signal, so that synapses between active cortical and striatal Go units are strengthened, whereas those for NoGo units are weakened. This learning allows the striatum to facilitate selection of the rewarding response in future presentations of the same stimulus. In contrast, if the network selects the incorrect response, DA units cease firing, and the associated dip in DA activity allows NoGo units (which were previously inhibited by DA) to become excited. The resulting increase in activity is also associated with strengthening of synapses from active cortical cells. With learning, NoGo cells differentially respond to stimulus-response combinations that have negative value, so that non-rewarding responses are likely to be suppressed. The mechanisms by which DA affects activity and plasticity are motivated by several biological experiments and are consistent with effects of dopamine D1 and D2 pharmacological agents on activity and long term plasticity (see Wiecki & Frank, 2010 for a recent review).
The net result is that the BG selects one response if a particular \"Go\" signal in the striatum is stronger than its corresponding \"NoGo\" signal, while concurrently suppressing alternative responses. Because direct and indirect pathway cells compete at BG output, the action most likely to be gated is a function of the difference in activity in these pathways for each action in parallel.
Now that we've seen how the dynamics of action selection and learning work, let's put the mechanisms to a test, and see how they are sensitive to dopamine manipulations as in empirical studies. Specifically, many studies implicate the basal ganglia dopamine system in reinforcement learning in probabilistic environments. In these tasks, multiple stimuli are presented on different trials, and participants have to learn by trial and error which response to make. The difficulty is not only that there is no clear underlying 'rule' for determining which response to make (it is determined arbitrarily), but also that there is no absolute answer that will always work. However, certain responses are reinforced on a greater proportion of trials than others, so that people have to learn to integrate the reinforcement history across multiple instances to determine the optimal response.
This project simulates the role of the basal ganglia dopamine system in learning and reward-based decision making. We focus here on effects of Parkinson's disease in the probabilistic selection task (Frank et al 2004). The task is simulated here such that when a stimulus S1 is presented, the network can respond by choosing A (selecting Response R1), or choosing B (selecting R2). When A is chosen, it will be rewarded (dopamine burst) on 80% of trials, and punished (dopamine dip) on the remaining 20%. The reverse is true when the network chooses B (it is punished on 80% of trials). Note that as in the human task, choice of A/R1 can result due to the network relying on Go R1 and/or from NoGo R2. The task includes a transfer phase which is designed to test which of these associations was better learned, that is, whether choices are primarily determined by positive or negative outcomes. The below simulations also allow us to investigate separable effects of dopamine manipulations on learning (modulations of dopamine during acquisition of reinforcement contingencies) vs. the expression of learning (decision making after learning has occurred).
Here, we simulate the task in a four response network. The procedure is as follows. When S1 is presented, the model selects between R1 and R2 (these responses are provided with some bias in the cortical activity to ensure that R3 and R4 are not eligible in these trials). These other responses, representing choice of C or D, become eligible in other trials when the corresponding stimulus represention S2 is active in the input. The discrimination to be learned in these trials is more tricky: C is rewarded on 60% of trials, and punished on 40%, and vice-versa for D.
After this learning phase, a \"transfer phase\" ensues in which we allow the model to choose between paired combinations of all the responses (with a composite of S1 and S2 presented in the input layer). The following choices are made available to the network, again by simply biasing the activity of the eligible motor cortical units):
i) R1 vs. either R3 or R4. Thus the network chooses between A (which had been rewarded on 80% of trials) and C or D (on some trials C is eligible and on some trials D is eligible), which had both been rewarded less often (60 or 40 %, so on average these alternative responses have 50% neutral value, as in the human task).
ii) R2 vs either R3 or R4. Here the network chooses between B (which had been punished on 80% of trials), and the same other more neutral ones.
Thus case i) is equivalent to \"choose-A\" in the probabilistic selection task, wheras case ii) is equivalent to \"avoid-B\". No reinforcement feedback is given during this phase, so choices reflect only performance based on past learning.
To run a batch of intact networks, on the MasterControl panel make sure num_intact_snc_units is set to 4. Then just click Init and Run, which will run a batch of 50 networks for 30 epochs of training and 30 epochs of testing (repeated presentations of test trials without learning to get sufficient statistics on choice accuracy for each network). You may want to uncheck the display box on the network so that it runs faster (it takes time to update the cycle by cycle activations on the network view).
| Question BG.1 : Are there differences in the learning curves for the two discriminations? Explain. Also note that if there are differences in the blue curves for RT. Can you think why there might be differences in the time the network takes to make a choice between these conditions? |
Next, check out the Stats bar graph on the right panel, which presents summary statistics of choices made across all networks in the transfer phase, as a function of trial type. On the left side of the graph labeled \"comboA\", the mean percentage choice of R1 is plotted in black across all of the choose-A trials. The networks should choose R1 over R3 or R4. You should see that r1_mean is indeed high (substantially above 0.5), indicating that networks do tend to choose A on most of these trials. The bar for r2_mean will be zero for \"comboA\" as the R2 cannot be selected during these trials.
In the B trials, the network should *avoid* R2 when paired with either R3 and R4. Thus good performance here is indicated by a low red bar: you should see that r2_mean is indeed less than 0.5 i (closer to 0 means better avoidance and more selection of the alternative choices; thus 'avoid-B' accuracy is (1-r2_mean)). (If you want to more closely examine the relative choices of R3 and R4, you can do so in the Stats datatable under AnalysisData, and you should see that choices in suboptimal trials nevertheless occur in proportion to their relative values - r3 should be selected more consistently than r4. This is the basic prediction of the softmax logistic function often used to summarize BG choice -- that choices are made probabilistically as a function of the relative difference in values between alternative responses).
Empirical studies testing the predictions of this model have shown that patients with Parkinson's disease (PD) show different patterns of Go or NoGo learning impairments depending on whether they are medicated or not (Frank et al, 2004; 2007; Palminteri et al, 2009, Bodi et al, 2010, etc).
| ? To simulate Parkinson's disease (PD), set the number of intact DA units to 2 in the .PanelTab.MasterControl control panel. This sets the number of SNc units that are connected to the Striatum to 2 (out of 4, i.e. 50% DA damage - you can also go all the way down to 1 but this can lead to very degraded overall performance). This reduction in number of intact DA units leads to effectively reduced tonic and phasic DA levels in the Striatum. |
You can step through a few trials to examine network dynamics under this dopamine depleted state. You might see overall greater levels of NoGo activation, which also produces general slowing in action selection (i.e. akinesia as in the disease). If you look at the network as it settles, you'll likely see that in the beginning of training the network has difficulty gating a response at all -- a cardinal symptom of PD - but nevertheless, the noise in motor cortex allows it to pick something randomly. The networks can still benefit from reinforcement at this stage and eventually do learn to gate responses with the BG, once the Go weights are sufficiently strong (you can see this if you look at settling later on during training).
| Question BG.2 : Run a full batch of networks in this PD state across the task. What happens to choose-A and avoid-B performance? Explain any differences from the intact state (and between performance in the two test conditiosn) in terms of the mechanisms in this model. |
This avoidance bias is similar to what is observed empirically with this task, and has also been shown in both the model and in animal studies to underly a progressive development of Parkinsonian symptoms in motor skill learning: essentially the network learns to avoid selecting actions with experience, because the DA-depleted BG is biased to interpret them as having negative outcomes (see Wiecki & Frank 2010, Beeler et al 2012 for empirical demonstration).
You can also run an intermediate case of num_intact_snc_units =3. You should see that relative performance in choose-A vs avoid-B is greater than with more damage (snc=2 or 1) and less than that with higher dopamine levels (snc =4). Thus these simulations capture the basic dissociation seen in the empirical studies in terms of choice performance on these test trials as a function of dopamine depletion and stimulation.
You can also simulate effects of DA medication given to PD patients: set the number of intact DA units back to 4 units (to simulate the increase in DA synthesis due to levodopa, the main form of medication). But in addition to allowing for more dopamine availability, medications also continually stimulate dopamine D2 receptors. To capture this effect, increase d2_acq (which reflects the impact of dopamine onto D2 receptors) from the default of 0.1 to 0.15. This will make it harder for NoGo units to respond to negative outcomes during DA dips (which requires less D2 stimulation; see Frank 2005). Note that for this demonstration (to show clearly that it is a learning effect), the D2 stimulation is returned to the intact state during the transfer phase (to rule out an effect of this D2 stimulation during choices in the test phase itself, i.e. a 'performance effect' that could have explained the results).
Hit Init and Run.
| Question BG.3 : What happens to avoid-B performance during the transfer phase after this manipulation (ie increasing D2 stimulation during acquisition only)? Compare this performance to those you observed in intact networks (with default levels of D2 stimulation (0.1)), and explain any differences in terms of the mechanisms in this model |
Note that the ability to choose-A should also depend on 'Go learning' that A is rewarding in the first place, so sufficient phasic DA should be needed during the training phase. Thus in order for a PD network to express any Go activity to rewarding choices, it must have had sufficient phasic increases to drive some Go learning (which may be just prevented from being expressed by the overactive NoGo pathway in the absence of sufficient D2 stimulation).
To test this idea, you can reduce the magnitude of phasic bursts (in addition to the general DA neuronal degradation associated with eliminating half of the units). To do this, change the DA_burst value in the control panel from 1 (maximal 100% firing) to .04 (this translates to roughly 70% maximal firing of SNc units during bursts: .04 is simply the external net input applied to these units which corresponds to that firing rate due to the nonlinearity of the activation function). If you set snc = 2 you are now simulating Parkinson's with depleted tonic DA levels and even more depleted phasic levels than usual.
You should see that now the PD network, regardless of whether D2 agonists are added during the test phase, is impaired (near chance) at choose-A performance while still showing relatively good performance at avoid-B.
Notably, levodopa, the most common medication used to treat PD, by acting as a DA precursor increases the phasic spike-dependent release of dopamine. This contrasts with the D2 agonist mechanism which only tonically stimulates D2 receptors and thus inhibits NoGo units, but does not increase phasic release needed for Go learning.
The corticostriatal pathway and reinforcement learning is not the only form of learning in this model. There is also unsupervised Hebbian learning that occurs directly from sensory to motor cortex. This pathway learns the statistics of its own actions -- when a particular stimulus is presented, on average which action did it select in the past? As long as this learning is slower than that in the BG, the actions that had been most frequently selected for this stimulus will be those that were most often reinforced (Frank, 2005; Frank & Claus 2006). This learning provides a mechanism by which the development and learning of habits is initially dependent on BG and dopamine function, but their later expression is not: once these mappings are strong enough, the network can rapidly activate only the most appropriate response and is not dependent on BG gating. This provides a natural explanation for the following observations: (i) While learning of simple instrumental actions are initially BG and DA-dependent, their later expression is not (Smith-Roe & Kelley, 2000), (ii) in well-learned tasks, striatal activation is sometimes seen *after* motor unit activation and the onset of movement (Alexander & Crutcher, 1990) and (iii) Parkinson's patients have much less difficulty executing well learned motor actions.
To see this in the model, change the Input_Motor_lrate from 0 (this learning has been turned off in this network so far) to a .001 (it is important that Hebbian learning is slow relative to reinforcement learning, so that the habits ingrained in cortex stamp-in the ones that have been selected based on reward!). Make sure the network is in the intact state (4 SNc units, DA burst val =1). With the network display turned on, run a single network from the LeabraTrain program and stop it near the end of training (e.g., after 25 epochs) and just step through a few trials and observe the network activity in the motor cortex. You should see that well before the BG gates an action, the motor cortex now preferentially activates a single response to the stimulus, which will be the one it most often has activated in the past (and if the RL process worked well in the BG, it will also be the more rewarding response). Thus this provides a simple demonstration that the motor cortex can first generate the candidate actions based on their prior probability of selection in the current sensory context, and if more than one of these actions is a suitable candidate, the BG can select between them. Also, if you let the network run a full set of batches, you should see that by the end of training they more reliably learn the difficult 60/40 discrimination (see trl_group panel): corticocortical learning forces it to more robustly stick with the action that works the most frequently and not be subject to a string of a few bad outcomes.
For several other demonstrations using this model and extensions thereof, see the \"BG Projects\" link on Michael Frank's website.
Although previous simulations have shown the effects of dopaminergic manipulations on \"Go\" vs \"NoGo\" learning in this task using a network with 2 motor responses, these have involved recording internal striatal valuation signals (assessed by Go/NoGo activations for stimulus-response combinations that had been primarily associated with positive or negative outcomes). Intact networks showed relatively greater Go than NoGo activity for positive S-R combinations and greater NoGo activity for negative combinations. Simulated DA depletion as in Parkinson's disease resulted in a bias, with stronger NoGo-negative striatal valuation signals than Go-positive ones, and simulated DA medications reversed this bias.
However, these prior simulations did not show that these striatal valuation signals translate to effects on actual choice (action selection). The reason for this is that there is no visual object recognition system in the network and hence it is not possible for the network to separately encode two stimuli presented in parallel (as any number of units presented in the input layer would be interpreted as one representation). Therefore the task is simulated by presenting a single stimulus (made up of four units), and differentially reinforcing responses.
Note that all accuracy analysis occur after filtering out trials in which a response was not gated by the maximum 150 cycles of each trial (otherwise choices selected in these trials are random, due only to motor cortical noise).
Also, unlike the original model, this particular version of the model uses a population of inhibitory interneurons to regulate striatal activity rather than the simplified k-winner take all (kWTA) approximation (the basic effects of DA manipulation apply also in the basic model). (The inhibitory interneurons are active in this network but iconified to minimize the amount you have to look at - feel free to click on that layer and de-iconify it to watch those dynamics!) The original simplification did not permit independent manipulation of the D2 inhibitory projections, because kWTA 'hijacks' the inhibitory currents in each unit. In earlier simulations the effects of DA were therefore always simultaneously affecting GO and NoGo cells -- more DA was associated with more Go excitation via D1 projection -- whereas here we can independently manipulate D2 receptor strength or blockade etc (by varying the strength of the inhibitory weight scale from DA units to NoGo cells) separately from general inhibitory effects within striatum (implemented with inhibitory interneurons that contribute to inhibitory currents in striatal cells but do not hijack them entirely) . Including inhibitory interneurons is somewhat more realistic (GABAergic neurons regulate spiny neuron activity in striatum), but the main reason for this is to enable other manipulations of D2 receptor inhibitory effects independently from inhibitory regulation of striatal cells in general (both Go and NoGo) based on inhibitory neuron activity. (see technical note 4 at the bottom for more details). This allows us to explore effects of D2 receptor sensitivity, blockade etc.
| ? You may now close the project (use the window manager close button on the project window or File/Close Project menu item) and then open a new one, or just quit emergent entirely by doing Quit emergent menu option or clicking the close button on the root window. |