Interactive Android Application for EEG Biofeedback

//–The Code Is On Github!–//

ABSTRACT

This post details the research and development of a mobile application which receives and annotates neurofeedback data from a commercial electroencephalography (EEG) device with a single dry electrode. The system is designed to convert any Android mobile phone into a portable storage device that passively records the user’s brainwaves while providing an interface to manually annotate the data with daily activities and moods. This application has the potential to provide numerous benefits to medical fields including but not limited to: neuroscience, psychology, psychiatry, and head and neck trauma. While the medical implications seem to be the most immediately prevalent, an application like this could also provide the everyday person with a better understanding of how their daily routine affects their state of mind.

Useful Links

Electroencephalography (EEG), The Frontier Nerds, brain-computer interface (BCI), Arduino, NeuroSky, brainwave, Bluetooth, Google Android, Emotiv, MindFlex, Processing, Java, SD card, neurofeedback, active vs. passive electrode, wet vs. dry electrode, International 10-20 System, The OpenEEG Project, .csv file, baud rate, serial communication, integrated development environment (IDE)

INTRODUCTION

This project began as a personal infatuation with brain-computer interfacing after I discovered some fascinating interactive games and applications that people were developing using EEG technology. Neurofeedback, until the early 2000s, had been predominantly used in medical and academic settings, where expensive equipment was used to test subjects with specific neurological and psychological conditions. This paradigm began to change as open-source initiatives like The OpenEEG Project [1] and companies such as Emotiv [2] and NeuroSky [3] started developing commercial EEG hardware platforms that were available to the general public for testing and development. While these technologies are by no means cheap, they do provide a new medium for developers and artists to create projects that interface the neurological biofeedback of the human body.

Problem Statement

Today, Neuroscientists, Psychologists, and other physicians use neurofeedback to diagnose, predict, and treat certain neurological and pathological conditions. Some of these infirmities include epilepsy, seizures, mood disorders, and trauma. While research in this field has been conducted since the early 1900s, it hasn’t been until recently that large advancements have been made due to the immense impact of technology on understanding the human brain [4]. In lieu of this, doctors who use EEG in their work are seeing an increasing benefit in working with developers and computer scientists to provide new methods of both retrieving and analyzing biofeedback from the brain. While there will always be a need for running EEG experiments in controlled environments, advancements in technology are enabling the creation of new applications that can provide more portable devices for retrieving, storing, and annotating neurofeedback. Such devices could prove to be invaluable for the advancement of medicine and achieving a more thorough understanding of the human brain as a whole.

Additionally, the average person leads their daily life with very little understanding of what is actually going on inside of their own head. Most people have a very qualitative view of why they feel certain emotions or experience different moods. What most people don’t realize is that there are quantitative and measurable data that can be retrieved from the brain that can provide better insight into why we feel and act the way that we do. The biggest obstacle that is preventing the average person from knowing more about his or her own brain is the difficulty of providing a non-invasive yet informative neurofeedback system at an affordable cost.

Objective

The objective of this project is to provide a starting point for a customizable neurofeedback application that can be tailored to the needs of different doctors, researchers, and individuals. The application will help to provide additional insight into the understanding of the brain – both medically and in a general sense. Furthermore, it will serve as a reference for other developers that aspire to contribute to the field of interfacing the brain. It is important to keep in mind that this is an early iteration of an ongoing design process, and that development f this application will continue after further testing and collection of user feedback.

PRECEDENTS

Over the last decade, an increasing number of EEG-related projects have emerged outside of the medical world. They range from open-source collaboration initiatives to commercialized proprietary hardware intended for commercial development of applications. The projects listed below inspired the development of this application and provided invaluable knowledge for its execution.

The OpenEEG Project

EEG began to emerge outside of medical and research settings in 2004 when the non-profit organization known as Creative Commons, founded by Lawrence Lessig, launched an open-source EEG initiative called “The OpenEEG Project” [1]. It became the first online forum for open discussion about EEG technology and it has brought together many experienced professionals who are willing to share their knowledge about EEG hardware and software. The website’s homepage states that:

“The OpenEEG project is about making plans and software for do-it-yourself EEG devices available for free (as in GPL). It is aimed towards amateurs who would like to experiment with EEG. However, if you are a pro in any of the fields of electronics, neurofeedback, software development etc., you are of course welcome to join the mailing-list and share your wisdom.”

The website provides tutorials on how to build your own EEG devices, as well as, examples of code that manage the intricate signal processing side of the technology. Additionally, anyone has the ability to join the OpenEEG mailing list to receive up-to-date information on advancements in EEG technologies.

Frontier Nerds: How to Hack Toy EEGs

In April 2010, a team from NYU’s ITP graduate program comprised of Eric Mika, Arturo Vidich, and Sofy Yuditskaya published a blog post titled “How to Hack Toy EEGs.” It thoroughly documents how they hacked an early EEG toy built by NeuroSky known as the Mind Flex. In the tutorial, they provide a list of commercial EEG devices contemporary in 2010, a brief description of the science behind EEG, sample code, an Arduino library for retrieving and outputting the EEG data, a Processing sketch that visualizes the data channels sent out from the Mind Flex, a corresponding Processing library, and video documentation of the entire process [5]. The project is very well executed and documented, and it served as the primary inspiration for the application that I am currently developing, which uses the same Mind Flex toy to record the EEG data. Figure 2 reveals a generalized diagram of how the Frontier Nerds hacked the EEG data out of the Mind Flex and used Arduino to synthesize the data. My application is an extension of this process and uses their Arduino library to retrieve the signals which are then passed through a Bluetooth device to the Android.

Necomimi: Brainwave Controlled Cat Ears

In early 2012, a company called Neurowear designed a fashionable EEG accessory, known as the Necomimi, which uses NeuroSky’s internal signal processing chip. The accessory makes a pair of artificial cat ears react to the wearer’s brain state. As the wearer gets more excited the ears stand up. Conversely, when the wearer is more relaxed the ears sink down [6].

The device demonstrates the usability of commercial EEG data for real-time interaction. Though this is a very simple demonstration of the data, it serves as a proof of concept for the realm of commercial EEG. Additionally, its immediate popularity is indicative of a shift in society’s opinion on the integration of BCIs into common culture.

TARGET USER GROUPS

The main goal for the current iteration of this application is to provide a more portable and scalable application for medical fields that currently use EEG. With that said, the application is customizable and can be tailored for independent research and annotation of neurofeedback data.

Neurologists and Psychologists

In the field of Neurology, doctors are already examining correlations between EEG and numerous brain disorders, including addiction and other naturally occurring diseases of the brain. EEG is commonly used to study brain ailments such as multiple personality disorder [7], migraines [8], epilepsy [9], and seizures [10]. With regards to the effects of drugs on the human brain, research has also been done into EEG in the classification and evaluation of the pharmacodynamics of psychotropic drugs [11], cerebral blood flow velocity abnormalities in chronic cocaine users [12], as well as other forms of addiction. Additionally, EEG has been used to study many forms of brain trauma including neuronal injury after brain anoxia [13] and impact-related brain trauma [14]. While neurologists and psychologists continue to learn more about the (brain?), portable and easy-to-use neurofeedback applications like this project will provide new methods of attaining vital information for the advancement of neuroscience.

The Average Person

In addition to the immediate benefits that this technology will provide to doctors, there is also a future for personal EEG data tracking. In the same way that doctors use EEG to learn about neurological disorders and how they are triggered, the everyday person may soon be able to use portable neurofeedback devices to augment daily routine by quantitatively juxtaposing common daily activities against personal perception of state of mind.

TECHNOLOGY

This project involves the use of a variety of technologies. At the core of the system is the NeuroSky signal-processing chip. The data from this single dry electrode EEG toy is sent to an Arduino using methods described in the hack by the Frontier Nerds (mentioned above). The data is then encrypted and sent through a Bluetooth module that was purchased from Sparkfun [15]. The encrypted Bluetooth packets are received by, parsed, and time stamped by the application that I developed on an HTC Nexus One Google Android mobile phone. The packets are then methodically stored to a .csv file on the internal SD card of the phone. Additional manually selected inputs are also time-stamped and written to the .csv file on the phone’s internal SD card. The .csv file can then be opened in Microsoft Excel to be analyzed and visualized with charts and graphs. The remainder of this section goes into further detail about the technologies involved in this project.

Figure 1. Flow of EEG data from the Mind Flex to a Bluetooth communication device [5].

Electroencephalography (EEG)

EEG is the recording of changes in electric potential outside the skull, caused by fluctuations in brain activity. The electrical activity originates from current flowing between the neurons of the brain. EEG is commonly used to research evoked potentials (EP) and event-related potentials (EPRs). The former requires averaging the data and connecting it to external stimuli, and the latter refers to averaged EEG responses that correlate to more complex processing of stimuli. EGG signals are separated into different bands that are unique in their frequency. These different frequencies have unique distributions over the scalp and also have individual neural significance that relates to different brain activity. While there are more accurate methods of retrieving neurofeedback data than EEG, they are more invasive and require more equipment. These techniques include electrocorticography (ECG), and magnetic resonance imaging (MRI) [16].

EEG Bands

Standard EEG band frequencies fall in the range of 0 to just over 100 Hertz. These bands differ between adults and children and their locations are dominant in different regions of the brain. The chart below reveals some of the common EEG bands and their respective frequencies, brain states, and wave types.

Figure 2. Common EEG Band Chart [16]

Electrode Types

Electrodes used in EEG devices have two types of classification that affect the quality of data received by the electrode. The first classification is whether the electrode is “dry” or “wet”. Dry electrodes consist of a dry conductive surface that is placed against the scalp. Wet electrodes are coated with a highly conductive solution, often saline, which significantly increases the clarity of the data [17]. The other classification is whether the electrode is “active” or “passive”. Active electrodes have inbuilt circuitry that amplify the electrical current close to where the signal is picked up from the scalp. Due to the extremely small electric potentials (millionths of a volt) that are recorded during EEG, data can be greatly distorted by even the resistance of common conductive wire. Therefore, active electrodes that are able to amplify the signal early in the system produce a much better resolution. In both of these cases, however, the option that provides the stronger signal is also more cumbersome [1].

International 10-20 System

When dealing with multiple electrodes, EEG placement has been standardized to allow for better collaboration between researchers. The standardized system is referred to as the International 10-20 System. The diagram in Figure 3 depicts a top-down perspective of the electrode placement of an apparatus the uses the 10-20 system [18]. Note that this project does not use this system due to the fact that it utilizes a single electrode and in turn does not provide spatial resolution of the EEG signal.

Figure 3. International 10-20 System of EEG Electrode Placement [18]

Hardware Used

I used a variety of hardware in this project that enabled collection and transfer of EEG data.

NeuroSky MindFlex

The NeuroSky MindFlex is a proprietary commercial EEG device that uses a single electrode that is both dry and passive to parse a raw neurofeedback signal into 11 channels. The 11 channels are: connectivity – a reading between 0 and 200 (0 being perfect connectivity), attention (a black box value calculated by NeuroSky’s proprietary signal-processing software), meditation (similar to attention), theta, delta, low alpha, high alpha, low beta, high beta, low gamma, and high gamma [3].

Arduino

“Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It’s intended for artists, designers, hobbyists, and anyone interested in creating interactive objects or environments [19].”

Sparkfun Bluetooth Mate Silver

This Bluetooth module is designed to be used with the Arduino environment. The module’s modems work as a serial (RX/TX) communication. It can be programmed to stream serial data at baud rates between 2,400 and 115,200 bps. The device is described thoroughly on Sparkfun’s website, which contains schematics, data sheets, and tutorials for the product [15].

HTC Nexus One Google Android Phone

The HTC Nexus One is a mobile phone that runs on the Google Android operating system 2.3.6. It was released in 2010, is Bluetooth compatible, has a resolution of 480×800 px with 254 pixels per inch, and contains an internal SD card. Though it is not the most modern Android phone, it was still able to run this version of the EEG application – a good sign for future iterations.

Software Used

A wide variety of software was used to develop this application. I worked with Arduino and Processing primarily. Arduino’s IDE is based on C/C++ and is supported by an extensive online website with learning material, examples, and forums [19]. Processing is a large collection of Java libraries that are primarily graphics-based. It too is supported by an extensive website that allows even novice programmers to start developing right away. Processing is very useful when working with Android development because there is an Android mode built into the Processing IDE that allows developers to test applications on an Android emulator that can mimic various types of Android phones. Additionally, developers are able to access the Android SDK and APIs directly from the Processing environment. Certain aspects of the application required writing raw Java code that had not yet been directly translated into processing libraries. These elements of the code included working with Bluetooth, writing to the SD card, and running the application as a background process so as not to interrupt the data stream [20].

EARLY PROTOTYPE & PLANNING

This project began in December of 2011 after I first familiarized myself with EEG technology and the field of brain-computer interfacing. It started as a predominantly hardware-based project while I was still getting acquainted with the Arduino environment and replicating the hack detailed by the Frontier Nerds of NYU’s ITP department. In addition to the initial hardware, I created an interface concept design for what would eventually evolve into the current application that this paper details. Before beginning work on the current application, I also conducted a generic survey via Facebook to gather input on what activities and moods the “everyday person” would be interested in comparing to their own quantitative EEG data.

BrainCap v1.0

My own work with BCIs began with a project that I dubbed the BrainCap, which eventually became known as BrainCap v1.0 due to later iterations. The device used the Arduino library created by the Frontier Nerds to extract the serial EEG data from the NeuroSky MindFlex. With some simple Arduino code, the system wrote the incoming data to an SD card using an SD Breakout module purchased from Sparkfun. A 9-volt battery was used to power the device, which allowed the user to walk around without any chords attached. Additionally, the devices had built-in buttons and beepers to ensure that the system would start and stop appropriately without deleting the collected data. All of the components of the device were mounted onto a baseball cap that was purchased at a dollar store.

The device was successful in that it collected and time-stamped data, but there was no built-in mechanism for adding context to the data with external stimuli. I tried doing this by taking extensive notes during sessions when the data was recording, but it proved to be tedious. BrainCap v1.0 can be seen in Figure 4 while some of the data that was collected can be seen in Figure 5. A more thorough explanation of the project and its results can be viewed at the following URL:

http://www.digitaldistillation.com/pComp/?p=897

Figure 4. BrainCap v1.0

Figure 5. External Stimuli Recorded by Hand to Add Context to the Recorded EEG Data

Application Interface Concept Design

As the need for a more convenient method for recording external stimuli became apparent, I began to develop concepts for a mobile application interface to accompany the BrainCap. The basic interaction of this interface design was to provide a simple yet comprehensive system for manually inputting both preset and custom activities and moods to contextualize the neurofeedback. Some sketches of the early interface design, which ended up being very similar to the current interface, can be seen in Figure 6.

Figure 6. Early Interface Design

To ensure that the interface received some feedback before I began developing it, a simple survey was distributed via Facebook to get input about common daily activities and moods that people would be interested in learning more about with regards to their own EEG data. The survey questions can be seen in Figure 7.

Figure 7. Facebook Survey for Feedback on Interface Design Features

CURRENT SYSTEM

The current iteration of the application achieves the goals that were set in the early stages of the project. The system successfully records both passively collected EEG data from the MindFlex and manually input activities and moods from the mobile application to the internal SD card of the Nexus One Android mobile phone. The BrainCap was modified to include the bare minimum number of components in order to achieve a longer battery life and also to ensure the clean processing of EEG data with the Arduino.

Portable EEG Device (BrainCap v2.0)

BrainCap v2.0 consists of the following components: the NeuroSky MindFlex, an Arduino Uno, a long-lasting Ultralife 9v battery, a breadboard, the Sparkfun Bluetooth Mate Silver, electrical wires, and the same baseball cap used in v1.0. The only software written for this part of the project was compiled in Arduino and uses the library created by the Frontier Nerds. BrainCap v2.0 can be seen in Figure 8.

Figure 8. BrainCap v2.0

Mobile Application Interface

The interface of the Android application contains four main tabs: Annotate, Your Brain, Share, and Settings. Currently the only tab that has any functionality is the annotate tab. Within this tab the user is prompted to select from two types of manual inputs – activity or mood. Once either of these is selected, the user is taken to a new window where he or she is able to select from an assortment of various preset common activities or moods. In addition the user has the capability to create custom activities and moods to personalize his or her EEG annotation. Once an activity or mood is selected, the user has the ability to decide whether the input should be turned on or off, selected as an instant event, or retroactively added to record past events. If an activity or mood is turned on, it is then highlighted with a green overlay to indicate to the user that it has been activated. After a selection has been made, the application then writes the appropriate subcategory to the internal SD card on the Android and uses the internal clock of the phone to time-stamp the entry so it can be synchronized with the passively recorded EEG data. Additionally, the interface has a narrow text field at the top of the screen that shows the incoming EEG data packets that arrive once per second. A screenshot of the interface can be seen in Figure 9 below.

Figure 9. Screenshot of the Application’s Interface

Data Analysis Techniques

Once data has been recorded to the internal SD card of the phone, the .csv file that contains the data can be easily sent to a computer via a standard USB cable. That file can then be opened in Microsoft Excel to construct charts and graphs for better interpretation of the data. Figure 10 shows a graph of sample data that was recorded using the application detailed in this project. The graph depicts my brain’s attention and meditation activity as I played a FIFA World Cup final in the Xbox game FIFA 2010 with one of my friends. The data points are averaged over 30 second intervals to provide a more coherent visualization of the EEG. Averaging the data over larger intervals is necessary due to the poor quality of the EEG data that is recorded from a single electrode that is both dry and passive.

Figure 10. A Graph That Depicts My Brainwaves as I Played a Video Game with My Friend

CONCLUSION

The many phases of this project have encompassed a wide range of technologies that have come together to form a unique application with some potentially significant implications for various fields of research. Though the project is still in an early stage, the groundwork has been laid and the application can be easily adapted from this point forward. The testing that has been done so far has provided unique insight into what is possible with a system that uses this type of EEG device – a device that provides crude real-time data that can be averaged over longer intervals to identify clear trends in brain activity. However, the lack of data analysis is one of the major shortcomings of this project. Because I have the only working instance of the system I am unable to get large amounts of data from different test subjects.

Findings

At this point no analysis has been done into finding correlations between external stimuli, personal perception of state of mind, and EEG data. This type of research will require an initial user group to test the application. Despite this, the few tests that have been run show clear linear and oscillating trends in brainwaves when averaged over extended periods of time. It was discovered early on that the data is too chaotic when looking at the data points that are received every second. However, when this data is averaged, distinct patterns in brainwave activity can be identified.

Future Directions

Moving forward there are many steps that need to be taken.

Initial User Group

An initial user group that is willing to test this application needs to be selected in order to get user feedback on the interaction of the application. Additionally, this user group will provide the amount of data that is necessary to start juxtaposing EEG data against daily activities and moods. From there, data analysis can commence to see what types of correlations data of this quality can produce between EEG and other information.

Interface Additions

As far as the interface is concerned, a data visualization system will be implemented so that the user can track data in real-time. This data visualization will allow the user to see automated graphs of his or her brain activity without having to upload the .csv file into Microsoft Excel. The data visualization interface? will resemble something similar to the processing sketch (Figure 11) that the Frontier Nerds made to accompany their Arduino library [5].

Figure 11. EEG Data Visualization Developed by The Frontier Nerds [5]

Sharing Capability

The next iteration of this project will also include an option for the user to anonymously share his or her EEG data and inputs with a central database. This database will serve as an aggregation of human brain activity and will provide an invaluable set of data to analyze and use as a baseline to compare to the EEG data of any individual.

Ethical Considerations

When dealing with a technology as new and powerful as brain-computer interfacing, there are obvious and not-so-obvious ethical implications that must be taken into consideration. While the capabilities of this type of application are limited due to the crude quality of the data, technology will continue to advance allowing for clearer data to be acquired more easily. Soon it will not be science fiction to have a portable EEG device that has bi-directional interactivity – in other words, a device that talks back. The implications of this type of technology lead to numerous potential risks that need to be planned for in order to prevent their negative consequences. Some of these risks include health and safety, legal issues, abuse of technology, security and privacy, and social impacts.

ACKNOWLEDGEMENTS

I’d like to give a special thanks to my Parsons Design + Technology instructors from Spring 2012, Jonah Brucker-Cohen, Katherine Moriwaki, Joel Murphy, and Ed Keller, for providing great mentorship throughout the development of this project. I’d also like to thank Neurosky, Sparkfun, Arduino, Google, and Processing for giving me fun toys to play with. Thank you, my good friend, Joe Artuso, for editing the piece. Lastly, thank you too The Frontier Nerds for giving me a place to start.

REFERENCES

I tried my best to give credit to all of the work that I referenced in the development of this project. Please contact me if you’d like me to add/remove anything to/from this blog post. I will not hesitate to do so.

1. The OpenEEG Project. http://openeeg.sourceforge.net/doc/
2. Emotiv. http://www.emotiv.com/
3. NeuroSky. http://neurosky.com/
4. Electroencephalography. http://www.bem.fi/book/13/13.htm#03
5. Mika, E., Vidich, A., Yuditskaya, S. How to Hack Toy EEGs. Frontier Nerds: An ITP Blog. http://frontiernerds.com/brain-hack
6. Necomimi. http://neurowear.com/
7. Arikan, K., et al. EEG Correlates of Startle Reflex with Reactivity to Eye Opening in Psychiatric Disorders: Preliminary Results. Clinical EEG and Neuroscience 37.3 (2006), 230-234.
8. Bjørk, M.H., et al. Interictal Quantitative EEG in Migraine: A Blinded Controlled Study. The Journal of Headache and Pain 10.5 (2009), 331-339.
9. Kennett, R. Modern Electroencephalography. Journal of Neurology 259 (4), 783-789. April 2012.
10. Bubrick, E.J., Bromfield, E.B., Dworetzky, B.A. Utilization of Below-the-Hairline EEG in Detecting Subclinical Seizures. Clinical EEG and Neuroscience 41.1 (2010), 15-18.
11. Saletu, Bernd, Anderer, P., Saletu-Zyhlarz, G. EEG Topography and Tomography (LORETA) in the Classification and Evaluation of the Pharmacodynamics of Psychotropic Drugs. Clinical EEG and Neuroscience 37.2 (2006), 66-80.
12. Copersino, M.L., et al. EEG and Cerebral Blood Flow Velocity Abnormalities in Chronic Cocaine Users. Clinical EEG and Neuroscience 40.1 (2009), 39-42.
13. Rossetti, A.O. Early EEG Correlates of Neuronal Injury After Brain Anoxia. Neurology 78 (11): 796-802. March, 2012.
14. Tezer, I., Dericioglu, N., Saygi, S. Generalized Spike-Wave Discharges with Focal Onset in a Patient with Head Trauma and Diffuse Cerebral Lesions: A Case Report with EEG and Cranial MRI Findings. Clinical EEG and Neuroscience 35.3 (2004): 151-157.
15. Sparkfun. http://www.sparkfun.com/
16. Budzynski, H., Budzynski, T., Evans, J. Introduction to Quantitative EEG and Neurofeedback: Advanced Theory and Applications. 2^nd Edition. Elsevier Inc. 2009.
17. Wang, Wang, Maier, Jung, Cauwenberghs. Dry and Noncontact EEG Sensors for Mobile Brain-Computer Interfaces. IEEE Transaction on Neural Systems and Rehabilitation Engineering 20 (2): 228-235. March, 2012.
18. Gilmore, R.L. American Electroencephalographic SocietyGuidelines in Electroencephalography, Evoked Potentials, and Polysomnography. J. Clin. Neurophysiol (11), 147. January, 1994.
19. Arduino. http://arduino.cc/
20. Processing for Android. http://wiki.processing.org/w/Android
21. Rao, R.P.N. University of Washington. Computer Science. March 7, 2012. http://www.cs.washington.edu/homes/rao/

WANNA GET INVOLVED?!

I’d be happy to share the graphics, code, and system schematics with anyone that wants to help with application development and/or collection of data. If enough interest is generated, I’d also be willing to post a thorough step-by-step procedure for application/device development with all the necessary assets for the Android App. If you are even remotely interested, don’t be afraid to contact me at conor.russomanno@gmail.com AND comment on the blog (interest is contagious!). The step-by-step will eventually get submitted regardless, but support from you guys would definitely help get the ball rolling. Brainiacs, assemble!