A Baseline Electroencephalography Motor Imagery Brain-Computer Interface System Using Artificial Intelligence and Deep Learning

This paper presents a baseline or reference (single channel, single subject, single trial) electroencephalography (EEG) motor imagery (MI) brain computer interface (BCI) that harnesses deep learning artificial neural networks (ANNs) for brainwave signal classification. The EEG electrode or sensor is placed on the scalp within the frontal lobe of the right hemisphere of the brain and approximately above the motor cortex. Signal classification discriminates among three MI classes, namely, right first closed event, neutral event and left first closed event and the measured accuracy of the deep learning ANN was 83% which significantly outperforms chance classification. The effectiveness of the system is demonstrated by applying it to the navigation of a virtual environment, specifically, immersive 360-degree panoramas in equirectangular projection.

Introduction

In recent times, brain-computer interface (BCI) prototypes based on electroencephalography (EEG) and the motor imagery (MI) paradigm have become ubiquitous [1]–[11]. These BCIs offer advantages including affordability and well-established protocols and hold out the promise of higher usability in practice.

A plethora of signal classification systems and algorithms have been developed and reported in the literature. These include systems based on principal component analysis (PCA), support vector machines (SVMs), independent component analysis (ICA), linear discriminant analysis (LDA) and their variants [12]–[16]. Other approaches involve the generation of artificial intelligence (AI) models based on a storied range of artificial neural networks (ANN) coupled with a very wide variety of learning methods including deep learning algorithms [17]–[21].

Existing methods typically employ a plurality of EEG electrodes with those systems harnessing a higher number of electrodes generally judged to yield better performance.

Consequently, there exists a need for minimal, cost-effective and functional BCI systems based on the EEG and in particular the motor imagery paradigm that could serve as baseline or reference systems.

This work presents just such a minimal, cost-effective and functional baseline EEG motor imagery BCI with application to environment navigation.

Fig. 1 outlines the design of the system and highlights its application to the generation of navigation commands for the control of a virtual environment.

Fig. 1. Schematic of brain-computer interface system.

The commands gleaned from the classification outcomes of the trained ANN model could also be harnessed in the control of external devices.

Materials and Methods

Participant Recruitment

Healthy participants without any history of neurological conditions volunteered to take part in this study. All participants gave informed consent and were apprised of the benefits of the research. The data analyzed in this paper were recorded from one adult male participant.

Ethical Approval

The studies were approved by the Research Ethics Committee at Topfaith University and were conducted in compliance with all applicable ethical and regulatory requirements.

Electroencephalography (EEG) Brainwave Data Acquisition

Raw EEG data streams were captured using the Emotiv EPOC Wireless EEG headset. This headset has been validated for research-grade brainwave data recordings [22]–[25]. For both training data acquisition and online application, participants wore an Emotiv EPOC Wireless headset (with the electrode, F3, positioned as depicted in Fig. 2, the motor cortex of the frontal lobe within the left hemisphere of the brain believed to be involved in the generation of signals for movement) while sitting in a relaxed position in a calm environment. Readings from Emotiv EPOC electrodes are reference-compensated.

Fig. 2. Arrangement for EEG brainwave data acquisition. The sensor is located above the motor cortex of the frontal lobe within the left hemisphere of the brain in the vicinity indicated generally as F3.

During training data capture, participants performed designated motor imagery activities in response to automatically generated prompts. In contrast, during online utilization, participants were free to carry out any motor imagery activity and the system would detect those activities or events that it had been trained on.

All data capture operations including the generation of automatic prompts and recording of data streams from the Emotiv EPOC Wireless device were coordinated using a 64-bit Python application named EEG Data Studio that was custom developed for this purpose.

This study delineates three classes of motor imagery activities or events, namely, Left Fist Closed, Neutral (both fists open and relaxed) and Right Fist Closed.

In order to capture training data, EEG Data Studio prompts the participant by displaying a specific trigger pattern for one second during which the participant would perform the designated motor imagery activity and EEG Data Studio records the corresponding EEG brainwave data stream as an input vector containing as many elements as there are data samples in one second of EEG data and a matching three-element output vector, clearing the display for a break in recording for the next two seconds for the participant to relax and then repeating the process until the desired number of samples are recorded.

For the Left Fist Closed motor imagery activity, EEG Data Studio displays a square at the center of the left half of the screen as shown in Fig. 3 and the participant closes only the left fist. For the Neutral motor imagery activity, EEG Data Studio displays a circle at the center of the screen as depicted in Fig. 4 and the participant relaxes without closing any fist while for the Right Fist Closed motor imagery activity, EEG Data Studio displays a square at the center of the right half of the screen as illustrated in Fig. 5 and the participant closes only the right fist.

Fig. 3. Left fist closed motor imagery data acquisition in EEG data studio. The captured one-second EEG data stream is displayed at the bottom of the screen with EEG signal voltages in microvolts on the vertical axis and the sample number (0 to 127) on the horizontal axis.

Fig. 4. Neutral (both fists open) motor imagery data acquisition in EEG data studio. The captured one-second EEG data stream is displayed at the bottom of the screen with EEG signal voltages in microvolts on the vertical axis and the sample number (0 to 127) on the horizontal axis.

Fig. 5. Right fist closed data acquisition in EEG data studio. The captured one-second EEG data stream is displayed at the bottom of the screen with EEG signal voltages in microvolts on the vertical axis and the sample number (0 to 127) on the horizontal axis.

The matching three-element output vectors for the three separate output classes using a one-hot encoding configuration are:

Left Fist Closed:	➜	1	0	0
Neutral:	➜	0	1	0
Right Fist Closed:	➜	0	0	1

Since the sampling frequency of the Emotiv EPOC Wireless headset is 128 samples per second, each recorded input vector representing one second of EEG data contains 128 elements or data points.

Three hundred seconds of raw EEG data were collected with one hundred seconds of data recordings for each of the Left Fist Closed, Neutral and Right Fist Closed motor imagery tasks. There were 300 input vectors (one vector for each second of recording and 100 vectors for each class with 128 elements each). Accordingly, a total of 38,400 (300 seconds multiplied by 128 samples per second) input EEG data samples were recorded. Consequently, the recording session lasted 900 seconds (15 minutes) since each one-second recording was followed by a break of two seconds before the next recording.

In summary, the input data comprised 300 vectors each containing 128 elements with values representing the raw EEG voltages measured at the electrode in microvolts. The Emotiv EPOC data is reference-compensated. Raw EEG data were analyzed without preprocessing to avoid distortions as the Artificial Neural Network (ANN) harnessed could discriminate between the relevant features. The output data comprised 300 vectors each containing 3 elements corresponding to the specific motor imagery task performed by the participant when the input data was measured. As indicated earlier, the output vector was {1 0 0} for the Left Fist Closed task, {0 1 0} for the Neutral task and {0 0 1} for the Right Fist Closed task.

Data Availability

The data (including raw EEG data) that support the findings of this study are available from GitHub at  https://github.com/frankekpar/eeg_motor_imagery_bci/blob/main/dataset.zip.

Artificial Neural Network (ANN) Architecture

TensorFlow Framework was harnessed in conjunction with the Keras Application Programming Interface (API) to build, train and deploy the deep learning artificial neural network (ANN) models in the Python programming language [26], [27]. Fundamentally structured as a multi-layer perceptron, the deep learning ANN comprised an input layer with 128 units corresponding to the number of elements in an input vector, two hidden layers each with 512 units and an output layer with 3 units corresponding to the number of elements in an output vector. Network connectivity was characterized by dense sequential layers. Rectified Linear Units (reLU) were utilized throughout except for the output layer where sigmoid activation units were utilized. Fig. 6 is a graphical illustration of the ANN architecture. This architecture was selected after extensive experimentation and was motivated by the fact that a single channel of EEG data was analyzed.

Fig. 6. Artificial neural network (ANN) architecture. X1, X2,…, Xn represent inputs; HL1, HL2, HLp represent hidden layers while Y1, Y2, Ym represent outputs.

Application to Virtual Environment Navigation

360-degree panoramas find applications in a plethora of fields ranging from medicine to virtual and augmented reality [28]–[31] and present excellent virtual environments for immersive user experiences. By wrapping a 360-degree panorama as a texture around a sphere and using a three-dimensional (3D) rendering framework such as OpenGL for tessellation of the sphere via triangle strips or other suitable primitives as illustrated in Figs. 7 and 8, perspective-corrected views of regions of interest within the sphere can be generated for seamless navigation of the environment represented by the 360-degree panorama.

Fig. 7. Tessellation of a sphere representing a 360-degree panorama via triangle strips.

Fig. 8. 360-degree panoramic image in equirectangular that could be textured or wrapped around the sphere with tessellation based on triangle strips depicted in Fig. 7.

In order to effect navigation of the 360-degree panorama, the pan, tilt and zoom (scale) parameters need to be specified. Given the angle representing the pan as θ, the angle representing the tilt as φ and the radius of the sphere used in the 3D rendering framework as r, Fig. 9 illustrates the computation of the 3D cartesian coordinates of the point indicated by the navigation parameters.

Fig. 9. 3D spherical to cartesian coordinate transformation.

As shown in Fig. 9, the computation of the 3D Cartesian coordinates x, y and z of the point with 3D spherical coordinates r, φ and θ can be facilitated by introducing the symbol k to represent the hypotenuse of the right-angled triangle formed by the sides x and z on the X-Z coordinate plane.

Consequently, two right-angled triangles stand out—one with known acute angle φ, hypotenuse r, opposite side y and adjacent side k; the other with known acute angle θ, hypotenuse k, opposite side z and adjacent side x.

The following equations express the coordinate transformation using the symbols in Fig. 9.

(1) $$\begin{array}{r}\text{k}=\text{r}\cos \mathrm{\phi }\end{array}$$

(2) $$\begin{array}{r}\text{x}=\text{k}\cos \mathrm{\theta }=\text{r}\cos \mathrm{\phi }\cos \mathrm{\theta }\end{array}$$

(3) $$\begin{array}{r}\text{y}=\text{r}\sin \mathrm{\phi }\end{array}$$

(4) $$\begin{array}{r}\text{z}=\text{k}\sin \mathrm{\theta }=\text{r}\cos \mathrm{\phi }\sin \mathrm{\theta }\end{array}$$

Note that θ ranges from 0 to 2π radians while φ ranges from −π/2 to π/2 radians. Equations (1)–(4) are applied in the 3D rendering of the panorama in the OpenGL framework.

Actual navigation of the 360-degree panorama is depicted in Fig. 10 and involves performing a perspective projection to a view window with a given width (W) and height (H) in the U-V coordinate system corresponding to the pan (θ₀), tilt (φ₀) and zoom or scale factor (λ) navigation parameters while performing the coordinate transformation described in the foregoing.

Fig. 10. Perspective projection.

Navigation of the 360-degree via thought using this baseline BCI is accomplished by using the trained ANN model to detect the EEG brainwave data signals corresponding to the Left Fist Closed or Right Fist Closed motor imagery activities and decreasing or increasing the value of the pan (θ₀) navigation parameter by a specified quantum depending on the type of motor imagery activity detected or leaving it unchanged if the Neutral motor imagery activity or any other unrecognized activity is detected. The exact value of the increment or decrement is made responsive to the speed of the computer system on which the BCI runs and the desired smoothness of the navigation.

Results

First, the aggregated input and corresponding output vectors were shuffled at random to improve balance and reduce bias. Next, the input vectors were extracted and preprocessed using the min-max scaler in the Scikit-learn Machine Learning Python Toolkit to improve processing efficiency. There was no need to preprocess the output vectors since they had values comprising 1 s and 0 s exclusively. Finally, the data was split into two datasets, a training dataset with 70% of the data and a test or validation dataset with the remaining 30% of the data.

Trained on the training dataset with the Adam deep learning optimization algorithm [32], [33], a default learning rate of 0.001 and a batch size of 32 over a total of 300 epochs with categorical cross-entropy loss function, the ANN achieved a classification accuracy of approximately 83% on both the training and test or validation datasets—vastly outperforming chance classification and demonstrating the effectively of the system.

To navigate 360-degree panoramic images through motor imagery activity and essentially by thought alone, the trained ANN model was used to classify EEG data streams captured as described in this paper and the output of the classification was transformed into an increment operation on the value of the pan navigation parameter for the Right Fist Closed motor imagery activity, a decrement operation on the parameter for the Left Fist Closed motor imagery activity and a retention of the current value of the parameter for the Neutral motor imagery activity or any other unrecognized activity. Perspective-corrected views of the panorama were generated on the basis of the pan navigation parameter (initially set to 0 and modified in accordance with the detected motor imagery activity), a fixed tilt navigation parameter (default set to 0) and a default zoom or scale factor using the coordinate transformations and perspective projection described herein. The tilt and zoom navigation parameters could be set using the mouse or keyboard.

Fig. 11 is a sample 360-degree panorama in equirectangular projection while Figs. 12 and 13 depict perspective-corrected views of the panorama with different navigation parameters.

Fig. 11. Sample 360-degree panorama in equirectangular projection.

Fig. 12. Perspective-corrected view of 360-degree panorama of Fig. 11.

Fig. 13. Another perspective-corrected view of 360-degree panorama of Fig. 11 with different navigation parameters.

Discussion

During training and validation dataset acquisition, participants are prompted to actually close the left or right fist with the recording of the EEG data stream commencing at the time of the display of the prompt on the screen which precedes the actual motor activity, implying that the recorded EEG signals encompass signals emitted during the motor imagery thought processes preceding the actual closure of the fist. In online applications of the resulting BCI such as in the navigation of immersive environments represented by 360-degree panoramic images, participants simply imagine closing the left or right fist.

The EEG electrode is placed above the motor cortex within the frontal lobe of the brain as this region is understood to be associated with motion and motor imagery activity [34]. In this study, the EEG device was placed above the left hemisphere of the brain. Alternative placements could be explored in the future.

Conclusion

This paper introduced a minimal electroencephalography (EEG) motor imagery brain computer interface (BCI) based on deep learning artificial neural networks (ANNs). The effectiveness of the system was demonstrated by applying it to the navigation of immersive 360-degree panoramic images. This BCI could also be utilized for the control of external devices as well as in virtual and augmented reality as illustrated in this paper as the resultant ANN significantly outperformed chance classification. This work could serve as a reference or baseline or benchmark BCI for further research and development.