Arbeitsgruppe “Visuelle Aufmerksamkeit und Perzeptuelles Lernen” am Institut für Neurobiologie Magdeburg
Prof. Dr. med. Jens-Max Hopf
Dipl. neurobiol. Hendrik Strumpf
Dr. rer. nat. Mandy Bartsch
Dipl.-Psych. Anja Rautzenberg
MSc. Haydee Guadalupe Garcia-Lazaro
- Dr. rer. nat. Carsten Nicolas Boehler (Department of Experimental Psychology, Ghent University, Belgium)
- Prof. John K. Tsotsos (Centre for Vision Science, York University, Toronto)
- Prof. Steven Hillyard (UCSD, LaJolla, USA)
- Prof. Steven J. Luck (UC Davis, Davis, USA)
- Prof. George R. Mangun (UC Davis, Davis, USA)
- Prof. Mircea A. Schoenfeld (Sektion Experimentelle Neurologie, Universität Magdeburg)
- Prof. Emrah Düzel (Institute for Cognitive Neuroscience, University College London)
- Prof. Jochen Braun (Institut für Biologie, Universität Magdeburg)
Langfristiges Ziel der Arbeitsgruppe ist es, Hirnmechanismen der selektiven Verarbeitung von Objekten und deren elementare Merkmale (z.B. Lokalisation, Farbe) zu verstehen. Um dafür relevante Hirnaktivität dynamisch on lokalisatorisch genau zu erfassen, kombinieren wir zeitlich hochauflösende Messverfahren wie die Elektro- und Magnetoenzephalographie mit räumlich hochauflösenden bildgebenden Verfahren wie das der funktionellen Kernspinresonanztomographie.
» mehr In der Vergangenheit konnte unsere Forschung entscheidende Impulse für das Verständnis von räumlicher Selektivität liefern (vgl. Übersicht in (Hopf et al., 2013)). Dieser Schwerpunkt wird durch Untersuchungen zu Prozessen nicht-räumlicher Merkmalsselektion im visuellen Cortex erweitert. Ein wesentliches Ergebnis dieser Forschung war die erstmalige Demonstration, dass attentionale Merkmalsverarbeitung (Farbe, Orientierung, Bewegung) im visuellen Cortex kein einfacher Einschritt-Prozess ist, sondern auf einer Sequenz und flexiblen zeitlichen Koordination (Schoenfeld et al., 2014) von mehreren Selektionsschritten beruht, die in Umkehrrichtung der Cortexhierarchie erfolgen (Bondarenko et al., 2012; Bartsch et al., 2014). Im Rahmen des SFB 779 (TP1) wurden dieser Forschungsschwerpunkt durch Untersuchungen zu belohnungsabhängigen Determinanten attentionaler Merkmalsselektion im visuellen Cortex (Buschschulte et al., 2014; Hopf et al., 2015) sowie in dopaminergen Mittelhirnarealen und frontalen Kontrollstrukturen erweitert (Boehler et al., 2011d; Boehler et al., 2011c; Boehler et al., 2014).
Unserer Forschung zu attentionalen Prozessen im visuellen System ist entscheidend durch komputationale Vorstellungen des sog. Selektive Tuning Modells (STM) motiviert (Tsotsos, 2011), welches visuelle Aufmerksamkeit als eine Konsequenz von Hirnprozessen ansieht, die der Lösung von Komplexitätsproblemen der Kodierung im visuellen System zugrunde liegen. Ein entscheidender Ansatz ist dabei, dass entsprechende Kodierungsprobleme aus der speziellen funktionellanatomischen Architektur des visuellen Systems resultieren. Solche architekturabhängigen Kodierungsprobleme ergeben sich u.a. aus der konvergenten Verarbeitungshierarchie des visuellen Cortex, durch dass es zu Kodierungsmehrdeutigkeiten infolge des Verlusts der räumlichen Auflösung der Repräsentation in höheren visuellen Arealen kommt. Unsere Forschung hat sich auf die Aufklärung der Mechanismen konzentriert, die die Beseitigung solcher Kodierungsmehrdeutigkeiten im visuellen System ermöglichen. Hier konnten wir zeigen, dass Disambiguierungsprozesse auf sog. top-down gerichteter Selektion im visuellen Cortex beruht, bei der die Selektion von hierarchisch höheren zu niedrigeren Arealen (rückwärts durch die Verarbeitungshierarchie) vonstatten geht (Hopf et al., 2010). Besonders zu erwähnen ist, dass unsere Ergebnisse zeigen, dass die kortikalen Prozesse, die für die Beseitigung von Kodierungsambiguitäten sorgen, gleichzeitig die Prozesse sind, die die (attentionale) Steigerung der räumlichen Auflösung der visuellen Diskrimination vermitteln (Boehler et al., 2011b). Im Rahmen dieser Untersuchungen zu Kodierungsambiguitäten und räumlicher Selektivität haben wir uns auch auf die Rolle subcortikaler Strukturen (Pulvinar des Thalamus) konzentriert. Hier konnten wir wesentlich zur Klärung einer lange Jahre geführten Debatte über die Rolle des Pulvinars bei der attentionalen Selektion beitragen (Strumpf et al., 2013).
Topic 1 - Solving cortical architecture-bound problems of visual selection:
1.1 The problem of spatial sampling and resolution: Neural mechanisms of surround attenuation and distractor competition in visual search
Our research into mechanisms that solve architecture-bound problems of attentional selection we focused on two problems more intensively: (a) the problem of spatial sampling and resolution, and (b) the problem of feature correspondence.
The problem of spatial sampling and resolution arises because the convergent forward hierarchy of the visual cortex entails increasing overlap between representations at progressively higher levels of the visual hierarchy (Hopf et al., 2013). This necessitates mechanisms that effectively eliminate overlap from non-target representations. With our research we could provide broad evidence in support of STM’s proposal that the elimination of irrelevant overlap is accomplished by top-down selection in visual cortex, that is, by selection operating in reversed (coarse-to-fine) direction through the visual cortical hierarchy (Hopf et al., 2010; Boehler et al., 2011b). Previous research has identified electromagnetic correlates of attentional selection processes that serve to resolve ambiguous feature encoding. One such correlate is the N2pc – a component known to reflect the disambiguation of feature selection based on the suppression of distractor information in extrastriate cortex (Luck et al., 1997; Hopf et al., 2000; Hopf et al., 2002; Hopf et al., 2006). Another correlate, hypothesized to reflect the disambiguation of spatial coding, is surround attenuation which had been extensively characterized by our group based on neuromagnetic recordings (Hopf et al., 2010; Hopf et al., 2013). Based on this research, we hypothesized that because surround attenuation attenuates distractor information in the vicinity of relevant input, it may index a mechanism that serves to disambiguate the coding of target information in the focus of attention (Hopf et al., 2006). In a series of EEG/MEG experiments we were able to pinpoint the functional link between both attenuation effects (Boehler et al., 2011b). We could show that the N2pc and surround attenuation do not reflect the same underlying operation. They represent signatures of different cortical selection processes, which operate with maximum effects at opposite levels of the visual cortical hierarchy. Specifically, the N2pc turned out to reflect the operation of biasing competition between feature-representations at levels of the visual hierarchy where coding becomes ambiguous due to the loss of spatial separation in large receptive fields. Figure 1 illustrates this finding based on a simplified three-level model (n, n+1, n+2) of the visual cortex hierarchy. In this model the color-to-item correspondence is unambiguous at the lowest level n, but is lost at higher levels. That is, at levels n+1 and n+2 the cell cannot decide which of the stars is drawn in red and which in blue. Our results indicate that the N2pc arises at hierarchical levels where this form of feature-to-item assignment becomes ambiguous (levels n+1 and n+2). Furthermore, we observed that at the lower hierarchical level (n+1) the N2pc response appeared with a delay (20 ms) relative to the N2pc response elicited at the higher level (n+2), which is clearly in support of the claim that top-down selection propagates in reverse hierarchical direction through the visual cortex hierarchy. Surround attenuation, in contrast, was not reflecting this biasing in higher level visual cortex. Instead, it was found to be linked to the processing consequences of this biasing operation which gains maximum expression in early visual cortex. It reflected the top-down directed elimination of distractor representations with the cortical extension of this elimination process being controlled by the outcome of the feature- biasing process indexed by the N2pc. Beyond this, our research could provide strong evidence for one key hypothesis of STM, namely that the operation of eliminating distractor information is also the operation that underlies the increase of spatial resolution of visual perception (Hopf et al., 2012).
Illustration of feature-coding ambiguities arising in visual cortex due to the convergent architecture (left) and the corresponding locus of source activity underlying the N2pc response in visual cortex. At a low hierarchical level (n), items (the red and the green star) are more likely to be represented by separate receptive fields of feature-selective cells (red and blue circles). At higher levels (n+1, n+2) this spatial separation is progressively lost with the consequence that the feature-to-item correspondence becomes spatially ambiguous. One solution to resolve this ambiguity is to bias the competition between feature-selective cells (i.e. select red over blue) – an operation suggested to be indexed by the N2pc (Luck et al., 1997). In fact, we could show that the spatio-temporal pattern of source activity underlying the N2pc is directly consistent with the locus of ambiguous feature coding in the visual cortical hierarchy. Furthermore, as shown in the lower right, the biasing of competition in higher–level visual areas (upper distribution map) turned out to appear earlier than in lower-level areas (lower distribution map), consistent with the notion that biasing in visual cortex operates in a recurrent coarse-to-fine direction (Boehler et al., 2011b).
1.2 The problem of feature correspondence: Neural mechanisms of object-based feature integration in visual search
A recent investigation of the mechanisms underlying object-based attention revealed another architecture-bound problem – the problem of feature correspondence. The stunning observation was that attention, commonly assumed to solve this problem via feature integration, actually caused the problem to arise (Boehler et al., 2011a).
A key notion of most influential theories on visual attention is that spatial attention serves to integrate features into coherent object representations (Treisman, 1988; Kahneman et al., 1992; Wolfe, 1994). Common to these theories is the hypothesis of a temporary pre-attentional state of representation during which visual features are represented in preliminary form, ’loosely’ filed at locations where they defy conscious access. Such pre-attentional state poses a form of the above mentioned correspondence problem of feature assignment, which is commonly hypothesized to be solved by attention. Specifically, spatial attention is suggested to bind features at locations thereby forming integrated object representations that then can be accessed consciously. Our recent research addressed exactly this operation of attentional selection during the formation of object representations (Boehler et al., 2011a). Much to our surprise, we observed that ~80 ms after the formation of an integrated object by spatial attention, feature attention ’spilled over’ to task-irrelevant features of the attended object, which in turn, biased the selection of that irrelevant feature at unattended object locations (referred to as Irrelevant Feature Effect (IFE) Figure 2B and E (red area between waveform-traces)). (See our corresponding research in the color/motion domain that yielded comparable observations (Schoenfeld et al., 2014)) The puzzling implication of this finding in view of the above mentioned theories is that attention serves to integrate features into objects (assumed to be accomplished within the time-range of the N2pc (blue area between traces in Fig. 2E), just in order to deconstruct this integrated representation within further 80 ms (red area between traces in Fig. 2E). Aside from the fact that this observation conflicts with established views about the role of attention in object integration, it adds important evidence in support of a notion first put forward by our group namely that feature-based attentional selection always operates in a spatially global manner, no matter whether selection concerns task-relevant features or features that gain processing priority by virtue of object-based selection.
Topic 2 - Neural mechanisms of surround attenuation and distractor competition: Subcortical structures
Continuing our research into the neural mechanisms of spatial selectivity and distractor competition, we targeted the role of the (subcortical) pulvinar nucleus of the thalamus (the red structure in Figure 2) during attentional selection in visual search (Strumpf et al., 2013). This research was exclusively based on functional brain imaging (fMRI), because electromagnetic responses cannot be obtained from the pulvinar.
The research was designed to address a very long-standing and debated issue whether the pulvinar’s attentional function is one of subserving spatial shifts of attention or alternatively of eliminating distractor influences during visual search. To this end, we compared search arrays requiring the elimination of distractor interference with search arrays requiring shifts of attention. As summarize in Figure 2, the experiments revealed that the pulvinar is mainly relevant for resolving distractor interference independent of the number of attention shifts to localize the target.
Activation of the pulvinar nucleus when the search array required resolving distractor interference to localize the target (BOLD contrast of search with versus without distractor interference). No such activation at all was seen in the pulvinar when comparing the same search arrays requiring shifts of attention (not shown). This demonstrates that the pulvinar is primarily important for distractor attenuation independent of the number of attention shifts required to localize the search target. The relative location of the activation maxima within the pulvinar is illustrated in the 3D-rendering of the pulvinar (red) on the right. The maxima are shown as yellow spheres. Why we observed two separate activation maxima in the pulvinar is currently under investigation.
Topic 3 - Mechanisms of global feature-based attentional selection in human visual cortex.
One notable property of feature-based attention, previously documented by us (Boehler et al., 2011a; Stoppel et al., 2012b) and by many others, is that it operates in a spatially global way. That is, attending to a feature of an object biases the selection of that feature outside the attended object at any place in the visual field the feature happens to be present.
Our research into the underlying mechanisms - focused on the orientation and color domain - confirmed previous research showing that the effect of feature attention is caused by a gain modulation of neural processing in human ventral extrastriate cortex. Beyond that, we were able to demonstrate for the first time that respective gain modulation is not a single unitary operation as commonly assumed. Instead, it involves a sequence of spatiotemporally separable selection steps progressing in reverse hierarchical direction from higher to lower levels of representation in visual cortex (Figure 3) (Bondarenko et al., 2012; Bartsch et al., 2014). Importantly, the sequence of modulation steps reflects functionally distinct operations, with the early modulations indexing global selection of features that are task relevant no matter whether they are actually presented in the focus the observer’s attention. Later modulations at lower levels, in contrast, were found to reflect feature selection as a consequence of object discrimination. Feature selection in reverse hierarchical direction is an important observation as it may represent the solution to a fundamental coding problem (see the discussion of cortical architecture-bound problems above): How to represent and select every possible task-relevant feature in visual cortex? Computational considerations (Tsotsos 2011) demonstrate that this problem has exponential complexity and is formally intractable without setting proper architectural constraints. Our observations suggest that the key to turn this problem into a solvable one is to constrain feature selection so that it follows the representation hierarchy of the visual cortex in reverse direction. Finally, from a more general perspective, these observations have important implications for understanding performance failure in daily life situations (e.g. in car driving), as they provide one account for why humans cannot fully concentrate on attended objects or processes: There will always be inescapable attraction by irrelevant objects in case they carry attended features.
Figure 3 shows the backward propagation of activity underlying global-feature based attention to color in one selected observer taken from a recent study (Bartsch et al., 2014). Panel (A) illustrates where the visual cortex is situated in the observer’s left hemisphere. The alternating blue-yellow coloration highlights the location of the early retinotopic areas V1 through V4 (V1-yellow, V2-blue, V3-yellow, V4-blue). Panel (B) shows the time course of activity modulations in areas of different hierarchical levels in the visual cortical system (staircase), where LOC (hot-scale areas on the inflated brain) represents a the highest and V1 the lowest level of representation shown here. Notably, source activity reflecting global color selection (red, yellow, and blue waveforms) appears first at the highest levels of representation (LOC), which is then followed by activity at the next lower level (V4), and is again followed by activity modulations at an even lower level (V3).
Topic 4 - Mechanisms of reward-dependent feature selection and its relation to attention
The research reported under Topic 3 shows that feature attention involves multiple steps of gain modulations in ventral extrastriate visual cortex, with early modulations reflecting the selection based on the mere task relevance of a feature.
Reward-relevance is a closely related concept and reward is often used (and in animal research almost the only way) to define taskrelevance and what to attend in attention experiments. It is therefore debated whether effects of attention and reward are separable or just two sides of the same coin. A core issue of the debate is whether performance effects due to attention and reward arise from the same modulatory effects in sensory cortex areas or whether they refer to independent modulatory influences. Our research aimed at addressing this issue with an experimental approach that permitted to separate rewardand attention-defined task parameters. In particular, we made sure that we defined reward-relevance of a feature but did not confound this definition with attention, i.e. with what defines the task-relevant feature of the target. After such paradigmatic separation of reward and attention, we observed that reward-driven effects on feature selection in visual cortex show strong mechanistic overlap (Hopf et al., 2015) but are dissociable from effects of attention regarding top-down modulatory influences from frontal cortex structures like the dorsal anterior cingulate cortex (dACC). Importantly, the processing of task-irrelevant reward-associated features was found to be under strong inhibitory top-down control from the dACC (Buschschulte et al., 2014). Such strong inhibitory control aligns with the results of behavioral studies showing that reward prospect can facilitate response inhibition in tasks where performance success requires the stopping instead of executing responses (Boehler et al., 2012a; Krebs et al., 2015). In (Boehler et al., 2014) we used fMRI to pinpoint the brain networks associated with reward-processing and response-inhibition. We show that enhanced connectivity between reward- and response-inhibition-related areas in medial prefrontal cortex underlies the beneficial effect of reward on response-inhibition.
Hopf JM, Schoenfeld MA, Buschschulte A, Rautzenberg A, Krebs RM, Boehler CN (2015) The modulatory impact of reward and attention on global feature selection in human visual cortex. Visual Cognition 23:229-248.
Krebs RM, Hopf JM, Boehler CN (2015) Within-trial effects of stimulus-reward associations. In: Motivation and Cognitive Control (Braver T, ed). New York: Psychology Press. (in press)
Becke A, Muller N, Vellage A, Schoenfeld MA, Hopf JM (2015) Neural sources of visual working memory maintenance in human parietal and ventral extrastriate visual cortex. Neuroimage 110:78-86.
Bartsch M, Boehler CN, Stoppel C, Merkel C, Heinze HJ, Schoenfeld MA, Hopf JM (2014) Determinants of global colorbased selection in human visual cortex. Cerebral Cortex.
Boehler CN, Schevernels H, Hopf JM, Stoppel CM, Krebs RM (2014) Reward prospect rapidly speeds up response inhibition via reactive control. Cogn Affect Behav Neurosci 14:593-609.
Schoenfeld MA, Hopf JM, Merkel C, Heinze HJ, Hillyard SA (2014) Object-based attention involves the sequential activation of feature-specific cortical modules. Nat Neurosci 17:619-624.
Buschschulte A, Boehler CN, Strumpf H, Stoppel C, Heinze HJ, Schoenfeld MA, Hopf JM (2014) Reward- and Attention-related Biasing of Sensory Selection in Visual Cortex. J Cogn Neurosci 26:1049–1065.
Merkel C, Hopf JM, Heinze HJ, Schoenfeld MA (under revision) Neural correlates of multiple object tracking strategies. NeuroImage.
Merkel C, Stoppel CM, Hillyard SA, Heinze HJ, Hopf JM, Schoenfeld MA (2014) Spatio-temporal patterns of brain activity distinguish strategies of multiple-object tracking. J Cogn Neurosci 26:28-40.
Hopf JM, Heinze HJ, Boehler CN (2013) Profiling the spatial focus of visual attention. In: Cognitive Electrophysiology of Attention and Cognition: Signals of the Mind (Mangun GR, ed), pp 3-15: Academic Press.
Bonath B, Tyll S, Budinger E, Krauel K, Hopf JM, Noesselt T (2013) Task-demands and audio-visual stimulus configurations modulate neural activity in the human thalamus. Neuroimage 66:110-118.
Kau S, Strumpf H, Merkel C, Stoppel CM, Heinze HJ, Hopf JM, Schoenfeld MA (2013) Distinct neural correlates of attending speed vs. coherence of motion. Neuroimage 64:299-307.
Strumpf H, Mangun GR, Boehler CN, Stoppel C, Schoenfeld MA, Heinze HJ, Hopf JM (2013) The role of the pulvinar in distractor processing and visual search. Human brain mapping 34:1115-1132.
Boehler CN, Hopf JM, Stoppel CM, Krebs RM (2012a) Motivating inhibition - reward prospect speeds up response cancellation. Cognition 125:498-503.
Boehler CN, Appelbaum LG, Krebs RM, Hopf JM, Woldorff MG (2012b) The influence of different Stop-signal response time estimation procedures on behavior-behavior and brain-behavior correlations. Behav Brain Res 229:123-130.
Bondarenko R, Boehler CN, Stoppel C, Heinze H-J, Schoenfeld MA, Hopf JM (2012) Separable mechanisms underlying global feature-based attention. Journal of Neuroscience 32:15284-15295.
Hopf J-M, Boehler CN, Schoenfeld MA, Mangun GR, Heinze HJ (2012) Attentional selection for locations, features, and objects in vision. In: Neuroscience of Attention (Mangun GR, ed), pp 3-29. Oxford: Oxford University Press, Inc.
Stoppel CM, Boehler CN, Strumpf H, Krebs RM, Heinze HJ, Hopf JM, Schoenfeld MA (2012a) Distinct Representations of Attentional Control During Voluntary and Stimulus-Driven Shifts Across Objects and Locations. Cerebral Cortex 23:1351-1361.
Stoppel CM, Boehler CN, Strumpf H, Krebs RM, Heinze HJ, Hopf JM, Schoenfeld MA (2012b) Spatiotemporal dynamics of feature-based attention spread: evidence from combined electroencephalographic and magnetoencephalographic recordings. Journal of Neuroscience 32:9671-9676.
Boehler CN, Schoenfeld MA, Heinze H-J, Hopf J-M (2011a) Object-based selection of irrelevant features is not confined to the attended object. Journal of Cognitive Neuroscience 23:2231-2239.
Boehler CN, Tsotsos JK, Schoenfeld MA, Heinze H-J, Hopf J-M (2011b) Neural mechanisms of surround attenuation and distractor competition in visual search. Journal of Neuroscience 31:5213-5224.
Boehler CN, Hopf JM, Krebs RM, Stoppel CM, Schoenfeld MA, Heinze HJ, Noesselt T (2011c) Task-load-dependent activation of dopaminergic midbrain areas in the absence of reward. Journal of Neuroscience 31:4955-4961.
Boehler CN, Bunzeck N, Krebs RM, Noesselt T, Schoenfeld MA, Heinze HJ, Munte TF, Woldorff MG, Hopf JM (2011d) Substantia Nigra Activity Level Predicts Trial-to-Trial Adjustments in Cognitive Control. J Cogn Neurosci 23:362-373.
Stoppel CM, Boehler CN, Strumpf H, Heinze HJ, Hopf JM, Schoenfeld MA (2011a) Neural processing of reward magnitude under varying attentional demands. Brain Res 1383:218-229.
Stoppel CM, Boehler CN, Strumpf H, Heinze HJ, Noesselt T, Hopf JM, Schoenfeld MA (2011b) Feature-based attention modulates direction-selective hemodynamic activity within human MT. Hum Brain Mapp 32:2183-2192.