Visualization of neural activity during motor imagery using common spatial pattern analysis
Electroencephalogram (EEG) data is a recording of the electrical activity of the brain from the scalp and it is usually measured in microvolts. There are different bands of frequencies that are observed in EEG signals: delta, theta, alpha, beta and gamma. Motor Imagery (MI) is the term used to describe the imagination and planning of motor actions like moving one's arm or leg, as opposed to actually performing motor actions. MI is typically characterised by signals spanning from 8 to 30 Hz, which involve the alpha and beta bands. MI is widely studied and tested in EEG-based brain computer interfaces (BCI), since literature shows that unlike some other brain activities, it can be characterised using EEG signals. Raw EEG data is preprocessed to remove noise and artifacts like blinking of the eyes. Features are extracted from the pre-processed data and the most important features for classification are selected. There are several algorithms that perform feature extraction, for example, spatial filtering algorithms that combine several channels into a single one, generally using weighted linear combinations to form lower dimensional features with high information. Common spatial pattern (CSP) algorithm is the most widely used spatial filtering algorithm to transform data in the oscillatory field. The transformations on the data serve to dimensionally reduce the data and to emphasize some features over the others. CSP is usually applied on data that is indiscernible when plotted directly on a 2D plane. CSP algorithm is designed in such a way that filtered signals' variance is maximum for one class while minimum for the other making classification easier. Nevertheless, CSP has some limitations, specifically that the frequency band and the time window for the target action (start and end time) must be known. Once the CSP filters are learnt, one should be able to apply those filters on the data and distinguish between classes of data. Topo-plots of the filters can be plotted to visualise spatially the activation of electrodes for specific motor imagery like movement of left arm versus that of right arm, the pair for which contra lateralisation is observed. This work aims at understanding the neural activity during motor imagery by experimenting with various protocols.
Keywords: EEG, BCI, spatial filters, MI
Electroencephalogram (EEG) remains vastly popular for recording the activity of brain due to its relatively low cost, high time resolution and easy availability in medical facilities and research institutions. With an increasing number of techniques that process EEG data, the protocols under which they are recorded have grown in relevance. The design of an effective protocol that improves the overall quality of results obtained from subjects is a problem that we tackle in this work. This work proposes two approaches - use of somatosensory cues, and the enforcement of a protocol in which motor imagery of a specific limb is followed by the actual movement of the limb, in contrast to just imagining the movement.
It is known from existing literature that for motor imagery, a minimum of 8 channels are needed to obtain optimal performance Benjamin Blankertz, 2010. However, increasing the number of channels introduces the curse of dimensionality. Thus, we resort to using a spatial filtering algorithm that combines several channels into one, usually using weighted linear combinations through which features are extracted.
The common spatial pattern (CSP) algorithm linearly transforms the multi-channel EEG signal into a low-dimensional subspace such that the variance of the EEG signal from one class is maximized while that from the other class is minimized.
This effectively highlights those regions of the brain that are most contributing to a particular action (imagery/actual movement) by comparing it to the activation as obtained by the other action. CSP is especially effective in BCI applications that use oscillatory data like EEG, since their most important features are band-power features. The CSP algorithm is computationally efficient and easy to implement. However, CSP does have some limitations. CSP is not robust to noise or non-stationarities and may not generalize well to new data especially when there is very less data available.
A total of seven healthy subjects participated in the experiments (5 males/2 females, all right handed, with a mean age of 22.5 years). Some of the participants took part in more than one protocol. The protocols were approved by the Institute Human Ethics Committee of Indian Institute of Science, Bangalore. The participants signed an informed consent form before taking part in the experiments. Subjects are labelled 1 through 7. Protocols are labelled A through E. Henceforth, a label 'A3', for example, would refer to subject 3 under protocol A.
The EEG data was sampled at 1000 Hz from an ANT Neuro Eego Mylab amplifier using the EEGCA64-500 montage, and the 10/10 electrode placement system with "CPz" electrode as reference electrode. A 64-channel cap with EOG (electrooculogram) channel was used for acquisition. EOG channel was not used for ICA (independent component analysis) artefact removal, since all the participants were instructed to keep their eyes closed throughout the experiment, except when they were under rest. The subjects were seated on a wooden chair, in a well ventilated room. The subjects were also instructed to rest their feet on a wooden support to ensure that there was no electrical contact with the floor.
The acquired EEG was preprocessed using EEGLAB Scott Makeig, 2004. The 50 Hz line noise was removed using notch filter applied from 49-51 Hz. The data was bandpass filtered from 8-30 Hz. This was done because the changes in EEG signals due to motor imagery are more visible in mu and beta bands F.H. Lopes da Silva, 2006. From literature, it is known that motor imagery activation in mu and beta bands (Rolandic) is visible in the central region of the brain, primarily consisting of "C3", "Cz" and "C4" electrodes. When the arm is moved (or imagined to have moved), mu rhythm changes contralateral to that arm movement (imagination) Juri D. Kropotov, 2016 . The frequency band of Rolandic beta rhythms varies from subject to subject, while falling in the broad range of 14-30 Hz. Juri D. Kropotov, 2009
Toolboxes and Algorithms Used
Since we are most interested in the spatial regions of the brain that are associated with the imagery in consideration, we decided to use the filters produced by the Lagrangian CSP algorithm and implemented it using Fabien Lotte's RCSP-Toolbox for MATLAB Cuntai Guan, 2010 Cuntai Guan, 2011. We also tried the other algorithms in the RCSP toolbox like DLR-CSP, SR-CSP, TR-CSP over multiple subjects. To maintain time we also developed a small web-app that helps us in delivering cues on time removing the hassle of manually keeping time for the experiment coordinators.
Cues and the Tasks
Across all the protocols, EEG data was labeled using the eego software at set intervals. The labels were marked after each cue was delivered, and the subjects were expected to perform a task after the cue was presented.
The audio and somatosensory cues as detailed in TABLE 1 were delivered by one of the experiment conductors seated directly in front of the subject, using the web-app for timing.
After a set of 25-30 trials, the web-app was paused in order to let the subject take a rest for approximately 2-3 min. Depending on their fatigue level, the subject had a choice to decide lesser number of trials for the next epoch of the experiment.
|Auditory||ARM||Imagine closing your right hand into a fist|
|Auditory||FOOT||Imagine wiggling the toes of your right foot|
|Auditory||LEFT||Imagine closing your left hand into a fist|
|Auditory||RIGHT||Imagine closing your right hand into a fist|
|Auditory||START||Perform the actual action of the previously imagined task|
|Somatosensory||Tap/Poke with a pen on the left inner wrist||Imagine closing your left hand into a fist|
|Somatosensory||Tap/Poke with a pen on the right inner wrist||Imagine closing your right hand into a fist|
|Somatosensory||Tap on the left/right knee||Take rest|
The scope of this work was to design a protocol that would return consistent results in terms of the area of activation as expected, based on previous work carried out on similar motor imagery tasks. In the literature, all the reported works either administer audio cues or visual cues to the subjects following which they were expected to imagine the motor action (motor imagery) for the cue given. C. Neuper, 2001 Wojciech Samek, 2015 Wojciech Samek, 2016 The most popular datasets in use in this domain are the BCI Competition Datasets.
In BCI Competition III Dataset, the subjects had to imagine either closing the right hand into a fist or wiggling the toes on the right foot. To validate our work on EEG, we tested our pipeline on the standard datasets available and the obtained results were very comparable to those obtained by Lotte and Guan Cuntai Guan, 2011
Protocol A: RA vs. RF using Auditory Cues
To begin with, the first three subjects (A1,A2,A3) were made to follow the protocol as described in BCI Competition III Dataset IVa's description as closely as possible. This was to serve as a baseline for the research done.
The protocol included three types of events - two imagery tasks of right arm (RA) and right foot (RF), and one rest event. Subjects were given rest for around 5 seconds after every imagery task, which itself lasted for 5 seconds. The order of the imagery tasks was randomized to ensure that the subjects were unable to predict the sequence of cues.
Timings of the protocol : An auditory cue lasting less than 500 ms was delivered to let the subject know that s/he had to start imagining the motor movement. After a period of 5 s, one more auditory cue was delivered to let the subject know that s/he had to stop the imagination.
The feedback obtained from subjects on this protocol was that it was often difficult to imagine the motor action without actually executing it and even harder to focus on the imagery for a duration as long as 5 seconds. The subjects described their imagination of the action merely as a visual image of the action, which does not strictly correspond to the motor imagery/planning that we were looking for.
Protocol B: RA vs. RF with Actual Actions using Auditory Cues
EEG data of two subjects, B4 and B5 was recorded under this protocol. This protocol has 4 types of events - two imagery tasks of right arm (RA) and right foot (RF), two actual motion tasks of right arm and right foot. Subjects were given auditory cues "arm" and "foot" to perform the corresponding imagery tasks repeatedly for 4s. Followed by each imagery task, an actual action of the previously imagined task was asked to be performed before 3s. The auditory cue for the actual action was "start".
Timings of the protocol: An auditory cue lasting less than 500 ms was delivered to let the subject know that s/he had to start imagining the motor movement. After a period of 5s, one more auditory cue was delivered to let the subject know that s/he had to actually perform the action once.
The changes introduced in this protocol were in consideration of the feedback obtained from subjects who participated in protocol A.
Protocol C: LA vs. RA with Actual Actions using Auditory Cues
Using this protocol, five subjects' (C1,C3,C4,C5,C6) EEG data were recorded.
This protocol has 4 types of events - two imagery tasks of left arm (LA) and right arm (RA), two actual motion tasks of left arm and right arm. Subjects were given auditory cues "left" and "right" to perform the corresponding imagery tasks which lasted for 4s. Followed by each imagery task, an actual action of the previously imagined task was asked to be performed once within 3s. The auditory cue for the actual action was "start". Since we know that the sensory cortex coincides with the motor cortex in the central region of the brain, we thought introducing localized somatosensory cues will further improve our results and realized the next protocol.
Timings of the protocol : An auditory cue lasting less than 500s was delivered to let the subject know that s/he had to start imagining the motor movement. After a period of 5s, one more auditory cue was delivered to let the subject know that s/he had to actually perform the action once.
Protocol D: LA vs. RA using Somatosensory Cues
One subject (D2) was recorded using this protocol.
This protocol has 3 types of events - two imagery tasks of left arm (LA) and right arm (RA) and one rest event. Somatosensory cues were given on the subject's outer wrist in the form of a gentle tap to let the subject know that he/she should perform the imagery task of the corresponding hand. The imagination was asked to be carried out for 4s and a tap on the knee of the same side of the body as the previous cue was given to let the participant stop imagination and take rest.
Timing of the protocol: Tap on the wrist was the somatosensory cue for starting the imagination and tap on the knee was the cue for stopping the imagination and taking rest.
This protocol was to test if the introduction of somatosensory cues would affect the quality of results obtained.
Protocol E: LA vs. RA with Actual Actions using Somatosensory Cues
EEG data of two subjects (E1,E6) were recorded using this protocol.
This protocol has 4 event types - two imagery tasks of left arm (LA) and right arm (RA), two actual motion tasks of left arm and right arm. The cues that were administered to indicate the onset of imagination or task of actual motion were somatosensory in nature and given using the blunt side of a pen. Upon inducing somatosensory stimulus on the subject's inner wrist, the subject was asked to perform the corresponding imagery task, which lasted for 4s. The somatosensory cue given on the palm told the subject to perform the actual action.
Timings of the protocol: A poke on the inner wrist was the somatosensory cue for starting the imagination and that on the palm was the cue for performing the actual action.
We compare the performance of the tested protocols by visually inspecting the filters produced by the CSP algorithm. The criteria of evaluation is the correctness of regions highlighted in the filters, with reference to what we know about the expected activation of the brain for the motor imagery in consideration. There are broadly two types of imagery in the protocols tested - arm and leg.
RA vs. RF Based Protocols
For the wiggling of toes on the foot and clenching of hand into a fist, there is a lot of reference material which could be used to validate the baseline protocol we tested. According to literature, the CSP filter plots for right arm when compared to right foot (RA-RF protocol) indicate activation on the left side of the brain along electrodes "C3", "C5" and "CP3". The CSP filter plots for right foot when compared to right arm is along the central electrode "Cz".
Protocol A: RA vs. RF using auditory cues
As shown in Fig 2, the topoplots obtained highlight the regions as expected for the imagery pair RA-RF. However, the regions are not defined very distinctly. This was an observation common to all subjects who participated in this protocol. This could be because the subjects found it difficult to imagine the motor action without actually performing the action. This is also likely affected by the fact that the subjects were unable to stay focused on the imagery for extended periods of time. Taking these into account, changes were made to the protocol.
Protocol B: RA vs. RF with actual actions using auditory cues
As seen in Fig 3, the introduction of actual motion into the protocol, following every single trial of imagery did result in better results. The topoplots exhibit more definitive regions and the subjects also reported that it was easier to stay focused through the trials as they were instructed to perform the actual motor movement.
LA vs RA Based Protocols
For the clenching of a hand into a fist, we expect activation predominantly centered around the central line electrodes ("C5","C3", "C1", "Cz", "C2", "C4" and "C6"). This activation is also known to show contralateralization with respect to the left or right arm in imagery. This is to say that for activities on the left arm, the right side of the brain is activated more and vice-versa for the right hand. This property of lateral symmetry of activity for imagery of the left or right arms helps in evaluating visually, the performance of the protocols in question.
Protocol C: LA vs. RA with Actual Actions using Auditory Cues
For this protocol and those to follow, a reduced electrode set was considered to reduce dimensions. This was decided based on our understanding of the regions within which the two motor imagery (left arm and right arm) are expected to lie. The topoplots of the filters (Fig 4), obtained with this protocol are very close to the ones reported in the literature.
Protocol D: LA vs. RA using Somatosensory Cues
As seen in the topoplots (Fig 5), the introduction of localized somatosensory cues as detailed earlier helps the subject to stay focused during the imagery and leads to better results.
Protocol E: LA vs. RA with Actual Actions using Somatosensory Cues
Considering the benefits of somatosensory cues from protocol D, protocol E was brought about to put protocols C and D together. This was tested on 2 of the 5 subjects who participated in protocol C. The improvements do seem to stack up as seen in the topoplots for this protocol (Fig 6). The activation is exactly where they are expected according to literature and the plots themselves have sharp and defined regions which are considered favorable traits.
The results obtained certainly show promise. Changing the protocol to include actual movement helps improve the results. This arises from the fact that healthy subjects are unlikely to have been in a situation where their brain exercised motor planning, (i.e, attempted to move a certain limb) but was unable to move the limb. Thus, healthy subjects find it difficult to imagine only the planning, without actually performing the associated movement later. On the other hand, allowing the subjects to move after the trial ensures that the trial captures the activity of the brain during planning.
The results also seem to indicate that somatosensory cues stimulate the subjects better which results in more focused motor imagery. Studies show that somatosensory actions induce activity in the same part of the brain as do the motor actions for the corresponding limbs. This helps not only in inducing activity in the same part of the brain, but also makes it easier for the subject to localize. A subject who is touched on his hand would be able to easily resolve where he was touched and will be able to quickly imagine motor actions on the same hand. This resolution based on localization, we speculate, results in better quality of motor imagery.
Claudia Sannelli, Thorsten Dickhaus, Sebastian Halder, Eva-Maria Hammer, Klaus-Robert Müller, Benjamin Blankertz, 2010, On Optimal Channel Configurations for SMR-based Brain–Computer Interfaces, Brain Topography, vol. 23, no. 2, pp. 186-1931
Arnaud Delorme, Scott Makeig, 2004, EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis, Journal of Neuroscience Methods, vol. 134, no. 1, pp. 9-211
G. Pfurtscheller, C. Brunner, A. Schlögl, F.H. Lopes da Silva, 2006, Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks, NeuroImage, vol. 31, no. 1, pp. 153-1591
Juri D. Kropotov, 2016, Introduction, Functional Neuromarkers for Psychiatry, pp. 79-801
Juri D. Kropotov, 2009, Introduction, Quantitative EEG, Event-Related Potentials and Neurotherapy, pp. 181-1891
Fabien Lotte, Cuntai Guan, 2010, Spatially Regularized Common Spatial Patterns for EEG Classification, 2010 20th International Conference on Pattern Recognition1
Fabien Lotte, Cuntai Guan, 2011, Regularizing Common Spatial Patterns to Improve BCI Designs: Unified Theory and New Algorithms, IEEE Transactions on Biomedical Engineering, vol. 58, no. 2, pp. 355-3621
G. Pfurtscheller, C. Neuper, 2001, Motor imagery and direct brain-computer communication, Proceedings of the IEEE, vol. 89, no. 7, pp. 1123-11341
Stephanie Brandl, Johannes Hohne, Klaus-Robert Muller, Wojciech Samek, 2015, Bringing BCI into everyday life: Motor imagery in a pseudo realistic environment, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER)1
Stephanie Brandl, Klaus-Robert Muller, Wojciech Samek, 2016, Alternative CSP approaches for multimodal distributed BCI data, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)1