Search

C World

Conor's Blog/Portfolio

Category

Parsons

M.F.A. Design and Technology

ROB3115 – A Neuro-Immersive Narrative

In-experience screenshot

ROB3115 is an interactive graphic novel that is influenced by the reader’s brainwaves. The experience is driven by the reader’s ability to cognitively engage with the story. ROB3115′s narrative and its fundamental interactive mechanic – the reader’s ability to focus – are tightly intertwined by virtue of a philosophical supposition linking consciousness with attention.

ROB3115 explores the intersection of interactive narrative, visual storytelling, and brain-computer interfacing. The experience, designed for an individual, puts the reader in the shoes of a highly intelligent artificial being that begins to perceive a sense of consciousness. By using a NeuroSky brainwave sensor, the reader’s brain activity directly affects the internal dialogue of the main character, in turn, dictating the outcome of his series of psychosomatic realizations. The system is an adaptation of the traditional choose-your-own-adventure. However, instead of actively making decisions at critical points in the narrative, the reader subconsciously affects the story via their level of cognitive engagement. This piece makes use of new media devices while, at the same time, commenting on the seemingly inevitable implications of their introduction into society.

This project was my thesis in graduating from Parsons with an M.F.A. in Design & Technology.

Brain Interface Lab

I recently founded the Brain Interface Lab with some colleagues from Parsons MFA Design & Technology and Columbia University. The lab is dedicated to supporting the open-source software and hardware development of brain-computer interfaces. Check out our website and all of the awesome stuff that was created during our first big event titled Hack-A-Brain:

Plasma Ball Concentration Game (openFrameworks + Neurosky’s EEG Mindset)

Project Summary

This project relates to the brain-computer interface work I’ve been doing for my thesis. As I will soon be creating generative animations that responds to brain activity, which are part of a digital graphic novel, I wanted to do a prototype of a visually complex animation that was dependent on a person’s brain activity. This project was written in openFrameworks and uses a Neurosky Mindset to link a player’s attention level to the intensity of electricity being generated from a sphere in the middle of the screen. The meat of the code is a recursive function that creates individual lightning strikes at a frequency inversely proportional to the attention parameter calculated by the Neurosky EEG headset. The project was visually inspired by the tesla coil and those cool electricity lamps that were really popular in the 90s (see below).

Once the connection between the Neurosky headset and the user’s computer has strong connectivity, the user can press the ‘b’ key (for brain) to link their EEG with the plasma ball. At any point the user can press the ‘g’ key (for graph) to see a HUD that displays a bar graph of their attention value on a scale from 0-100. The graph also shows the connectivity value of the device and the average attention value, calculated over the previous 5 seconds, being used to dictate the frequency of the electricity.

In order to get this application working on your computer, you must first download and install the Neurosky Thinkgear connector. You should be able to get it working with any bluetooth enabled Neurosky device; I’ve documented how to do so in the readme file on my github. You can get my code for the project on my Github page here: https://github.com/crussoma/conorRussomanno_algo2012/tree/master/Conors_Final

Also, if you just want to see the recursive electricity code working independent of a person’s EEG, download and install the app lightningBall (not lightnightBall_brain) from my github.

Project Video

To see this project in action check out my demo reel and jump to 35s.

Visual Inspiration

lightnightBall

lightnightLamp

Screenshots

Screen Shot 2013-03-06 at 5.29.23 PM

Screen Shot 2013-03-06 at 5.29.00 PM

References

My code uses some of the logic and algorithms Esteban Hufstedler’s processing sketch:http://www.openprocessing.org/sketch/2924

Additionally, a big shout out to Akira Hayasaka for writing the Neurosky openFrameworks addon that I used to pull this off:  https://github.com/Akira-Hayasaka/ofxThinkGear

‘Wetlands’ Architectural Renders

Project Summary

I spent the past 6 weeks working with the amazing and progressive artist Mary Mattingly on her project titled Wetlands. Most of her work explores the complex relationship between people and the Earth. Wetlands, currently in the design phase, is a self-sustained living environment that floats in the rivers outside of Philadelphia. The structure will be a low-cost floating barge with various components that explore DIY techniques of sustainability.

My Role

I worked with 2 other artists to create an architectural design for the structure that optimized the functional and design constraints. I helped with the concept drawings and took the lead on creating 3D renders of the design.

Renders

Project Presenation PDF: wetlands

Please Vote For An Awesome EEG Project!

Please take 10 seconds to vote for my New Challenge application:

Despite being rather silent on this blog recently, I’ve actually been quite busy. My ongoing thesis at Parsons MFA Design & Technology is an exploration of practical applications of wearable brain-computer interfaces. More on that to come.

Recently, some fellow designers, engineers, researchers, and myself applied for an award of up to 10K to explore if wearable BCIs could be used to find complimentary or alternative solutions for people suffering from attention disorders such as ADHD. If you support this cause, please click on the image above or the following link and click the “vote” button. You could comment here, but it would be better for you to comment on the application page itself in order to prove to the judges that people truly do care about this cause.

The application is as follows:

Project Title: Brain Design Lab – Finding Alternative Approaches to Addressing ADHD

People Involved:

  • Conor Russomanno (Director) – Conor is currently a 2nd Year in Parsons School of Design MFA Design & Technology Program. Conor did his undergraduate degree in engineering at Columbia University, and has been working with brain-computer interfaces for the past year. Check out his website at conorrussomanno.me.
  • Kristen Kersh – Candidate for MFA in Design & Technology at Parsons School of Design, Masters in Neuroscience and Education from Harvard University
  • James Ramadan – Received dual majors in biology and statistics from University of Virginia, currently does research in statistical analysis of quantitative EEG.
  • Amy Burns – Award winning reporter, spent more than 17 years in the multi-media industry, covering a diverse range of topics through the written word, social media, and the power of video.
  • Other members of the Brain Design Lab (our website is currently being built, braindesignlab.com)

The Problem

Our brains are dependent on the stimuli provided by our environment. Neuroplasticity is the notion that our neurons can be molded and re-purposed based on our experiences, even after critical stages of development. Currently, elite academic institutions such as Harvard, Columbia, and MIT are using functional magnetic resonance imaging (FMRI), magnetic resonance imaging (MRI), and electroencephalography (EEG) to research the brain’s ability to develop and change in response to stimuli. These studies have produced important findings with regards to a wide range of neurological diseases, traumatic brain injuries, and learning. In turn, these findings are being translated and applied to improved techniques in medicine, therapy, and education.

One of the main shortcomings of interfacing the brain is the ability to attain data outside of the confinement of a laboratory setting. There are very few studies done with a patient within the context of their normal environment, looking at how their home, what they eat, smell, see, hear, and touch affects the activity within their brain. Understandably, this is a very large challenge to address. If we are honored with receiving funds from the New Challenge competition, we intend to contribute to this pervasive challenge by addressing the issues of one of its sub communities, people suffering from attention disorders that affect their ability to focus and learn.

In 2007, the Center for Disease Control reported that 8.4% of American children aged 3-17 were at one point diagnosed with ADHD. Roughly 50% of children with attention disorders continue to experience issues as they progress into adulthood, and almost 60% of people diagnosed with these disorders are prescribed medication in an attempt to address the symptoms. It is vital that researchers continue to explore alternative and complimentary methods for solving attention-related disorders, and do not rely entirely on prescription medication to resolve the issue. Additionally, we believe that solutions to these problems have the potential to extend beyond the scope of individuals diagnosed with ADHD, and could be implemented by undiagnosed individuals trying to enhance their level of focus, learning ability, and productivity. It is this ubiquitous issue that we intend to examine.

Our Solution

To address this problem, my team of designers, engineers, and researchers has come together to found the Brain Design Laboratory (BDL). The goal of this community is to design, build, test, and rebuild non-invasive neurofeedback platforms that allow users to record environmental conditions over prolonged periods of time, while simultaneously tracking brain activity. In order to explore alternative techniques to addressing ADHD, we want to analyze the data that is recorded by these systems.

The systems will be comprised of a non-invasive headset that wirelessly sends brainwave data to a mobile phone and a central server, as well as a mobile application that tracks environmental stimuli both actively and passively. Passive stimuli will include variables such as location, noise, and movement, using GPS, audio inputs, and accelerometers. Actively recorded stimuli will include variables such as diet, activities, and moods, and will be input manually by the user. We believe this system will provide invaluable insight into how environmental stimuli correlate to variations in levels of attention. We will reach out to find user groups willing to test the platform. Eventually we hope to be able to provide real-time feedback to the user about how their environment is affecting their level of attention.

Currently, the potentials of commercial EEG have been used primarily for stationary recording and interaction, and do not serve as a good system for prolonged recording of brain activity. Some of the major shortcomings include comfort and attention to aesthetics. We believe that our diverse team of designers and engineers with experience in neuroscience, electrical engineering, as well as fashionable technology, can provide a new outlook on these problems, creating a system that is both wearable and functional. Lastly, we don’t want to just build technology; we strive to turn BDL into an open community of designers, researchers, patients, parents, and other organizations who are dealing with this problem.

Rough Budget

Item Cost Rationale
20xNeuroskyThinkgearChip $35 each The Neurosky Thinkgear chips (http://neurosky.com/Business/ThinkGearChipsets.aspx) are commercial
Electronics $2000 Bluetooth modules, Android testing platforms, electrodes, wires,
Materials $500 Garments, materials, and accessories for designing and building wearable devices. Including fabric, sewing equipment, hats, etc.
Website $500 We will use this money to establish our validity as an organization so that we can reach out to potential user groups for testing
Contingency Cost $1000 Miscellaneous expenditures

Our Qualifications

I first began trying to address this issue last spring when I designed and built a baseball cap with a sensor for recording brainwaves. To accompany the hat I developed a mobile application for Android that received and recorded the user’s EEG allowing for retroactive analysis of the data. The application also allowed the user to record a variety of moods and daily activities, the intention being to see how quantitative brain activity could be used to find new comparisons between the two. For more information about the project refer to: http://conorrussomanno.me/2012/06/19/interactive-android-application-for-eeg-biofeedback/

This Fall, with the support of former dean of Parsons The New School of Design’s Art Media and Technology department, Sven Travis, I founded the Brain Design Lab (BDL), a community focused on finding practical applications for brain-computer interfaces. Since it’s inception the community has grown and now has members both inside and outside of the New School community. Some of BDLs most prominent members include a recent graduate of Harvard’s Neuroscience and Education M.S. program, a University of Virginia graduate with a double major in biology and statistics, an award-winning journalist whose son suffers from an incredibly rare undiagnosed neurological disorder, and a graduate of Columbia University’s engineering program.

Recently we received $1,500 from the New School Student Activities Finance Committee to host a development jam titled Hack-A-Brain. The goal of the event is to explore the potentials of various front-line commercial EEG devices, while introducing New School students to the emerging industry of brain-computer interfacing (BCI). The Brain Design Lab has already connected with a number of individuals and organizations involved in the industry. Now we are looking to find additional support, make new connections, and apply novel design techniques to address problems related to the brain. We want to start by attempting to build user feedback applications for addressing attention disorders such as ADHD.

 

ABC No Rio – An Illustrated Short Story Prototype

I collaborated with two other artists, Tharit Firm Tothong and Giselle Wynn, on the creation of this illustrated short story for a class project. The piece pays tribute to ABC No Rio, an art gallery and concert space in the Lower East Side that has been in operation since the early 80s and was very active politically active during the late 80s and early 90s, acting as a sanctuary for society’s misfit demographics, as well as taking a strong stance of opposition to NYC’s heavy gentrification at the time. The piece speculates a fictional narrative from the point of view of a poor musician living in the slums of an overpopulated and depressed urban setting.

The piece is comprised of unpolished illustrations, done by myself and Giselle, as well as a collection of photographs of authentic artwork from within the walls ABC No Rio itself, taken by Firm. Firm also oversaw the design and layout of the composition.

Bull’s Eye – Hand-drawn Animation

This hand-drawn animation is of an archer readying and firing his bow:

[SHARED] Post-Planetary Design (Class Blog)

I’m not sure why I haven’t shared the below link yet. It’s skipped my mind for long enough. The site is centered around one of my favorite Parsons D+T courses to date, Post-Planetary Design (yes, I stole the name for one of my blog’s categories).  The course, taught by the great thinker Ed Keller, is a fantastic combination of Philosophy, Technology, Sci-Fi, Religion, Film, Literature, and much more. If you are interested in any of those topics, I urge you pay it a visit. My personal contribution to the class was an exploration on the ethics of brain-computer interfacing.

Much of the discussion and course content from Post-Planetary Design is inspiring one of my current ventures, a sci-fi graphic novel about the future of mankind that explores the question of what it means to be human.  More on that to come!

Post-Planetary Design

Interactive Android Application for EEG Biofeedback

//–The Code Is On Github!–//

ABSTRACT

This post details the research and development of a mobile application which receives and annotates neurofeedback data from a commercial electroencephalography (EEG) device with a single dry electrode. The system is designed to convert any Android mobile phone into a portable storage device that passively records the user’s brainwaves while providing an interface to manually annotate the data with daily activities and moods. This application has the potential to provide numerous benefits to medical fields including but not limited to: neuroscience, psychology, psychiatry, and head and neck trauma. While the medical implications seem to be the most immediately prevalent, an application like this could also provide the everyday person with a better understanding of how their daily routine affects their state of mind.

Useful Links

Electroencephalography (EEG), The Frontier Nerds, brain-computer interface (BCI), Arduino, NeuroSky, brainwave, Bluetooth, Google Android, Emotiv, MindFlex, Processing, Java, SD card, neurofeedback, active vs. passive electrode, wet vs. dry electrode, International 10-20 System, The OpenEEG Project, .csv file, baud rate, serial communication, integrated development environment (IDE)

INTRODUCTION

This project began as a personal infatuation with brain-computer interfacing after I discovered some fascinating interactive games and applications that people were developing using EEG technology. Neurofeedback, until the early 2000s, had been predominantly used in medical and academic settings, where expensive equipment was used to test subjects with specific neurological and psychological conditions. This paradigm began to change as open-source initiatives like The OpenEEG Project [1] and companies such as Emotiv [2] and NeuroSky [3] started developing commercial EEG hardware platforms that were available to the general public for testing and development. While these technologies are by no means cheap, they do provide a new medium for developers and artists to create projects that interface the neurological biofeedback of the human body.

Problem Statement

Today, Neuroscientists, Psychologists, and other physicians use neurofeedback to diagnose, predict, and treat certain neurological and pathological conditions. Some of these infirmities include epilepsy, seizures, mood disorders, and trauma. While research in this field has been conducted since the early 1900s, it hasn’t been until recently that large advancements have been made due to the immense impact of technology on understanding the human brain [4]. In lieu of this, doctors who use EEG in their work are seeing an increasing benefit in working with developers and computer scientists to provide new methods of both retrieving and analyzing biofeedback from the brain. While there will always be a need for running EEG experiments in controlled environments, advancements in technology are enabling the creation of new applications that can provide more portable devices for retrieving, storing, and annotating neurofeedback. Such devices could prove to be invaluable for the advancement of medicine and achieving a more thorough understanding of the human brain as a whole.

Additionally, the average person leads their daily life with very little understanding of what is actually going on inside of their own head. Most people have a very qualitative view of why they feel certain emotions or experience different moods. What most people don’t realize is that there are quantitative and measurable data that can be retrieved from the brain that can provide better insight into why we feel and act the way that we do. The biggest obstacle that is preventing the average person from knowing more about his or her own brain is the difficulty of providing a non-invasive yet informative neurofeedback system at an affordable cost.

Objective

The objective of this project is to provide a starting point for a customizable neurofeedback application that can be tailored to the needs of different doctors, researchers, and individuals. The application will help to provide additional insight into the understanding of the brain – both medically and in a general sense. Furthermore, it will serve as a reference for other developers that aspire to contribute to the field of interfacing the brain. It is important to keep in mind that this is an early iteration of an ongoing design process, and that development f this application will continue after further testing and collection of user feedback.

PRECEDENTS

Over the last decade, an increasing number of EEG-related projects have emerged outside of the medical world. They range from open-source collaboration initiatives to commercialized proprietary hardware intended for commercial development of applications. The projects listed below inspired the development of this application and provided invaluable knowledge for its execution.

The OpenEEG Project

EEG began to emerge outside of medical and research settings in 2004 when the non-profit organization known as Creative Commons, founded by Lawrence Lessig, launched an open-source EEG initiative called “The OpenEEG Project” [1]. It became the first online forum for open discussion about EEG technology and it has brought together many experienced professionals who are willing to share their knowledge about EEG hardware and software. The website’s homepage states that:

“The OpenEEG project is about making plans and software for do-it-yourself EEG devices available for free (as in GPL). It is aimed towards amateurs who would like to experiment with EEG. However, if you are a pro in any of the fields of electronics, neurofeedback, software development etc., you are of course welcome to join the mailing-list and share your wisdom.”

The website provides tutorials on how to build your own EEG devices, as well as, examples of code that manage the intricate signal processing side of the technology. Additionally, anyone has the ability to join the OpenEEG mailing list to receive up-to-date information on advancements in EEG technologies.

Frontier Nerds: How to Hack Toy EEGs

In April 2010, a team from NYU’s ITP graduate program comprised of Eric Mika, Arturo Vidich, and Sofy Yuditskaya published a blog post titled “How to Hack Toy EEGs.” It thoroughly documents how they hacked an early EEG toy built by NeuroSky known as the Mind Flex. In the tutorial, they provide a list of commercial EEG devices contemporary in 2010, a brief description of the science behind EEG, sample code, an Arduino library for retrieving and outputting the EEG data, a Processing sketch that visualizes the data channels sent out from the Mind Flex, a corresponding Processing library, and video documentation of the entire process [5]. The project is very well executed and documented, and it served as the primary inspiration for the application that I am currently developing, which uses the same Mind Flex toy to record the EEG data. Figure 2 reveals a generalized diagram of how the Frontier Nerds hacked the EEG data out of the Mind Flex and used Arduino to synthesize the data. My application is an extension of this process and uses their Arduino library to retrieve the signals which are then passed through a Bluetooth device to the Android.

Necomimi: Brainwave Controlled Cat Ears

In early 2012, a company called Neurowear designed a fashionable EEG accessory, known as the Necomimi, which uses NeuroSky’s internal signal processing chip. The accessory makes a pair of artificial cat ears react to the wearer’s brain state. As the wearer gets more excited the ears stand up. Conversely, when the wearer is more relaxed the ears sink down [6].

The device demonstrates the usability of commercial EEG data for real-time interaction. Though this is a very simple demonstration of the data, it serves as a proof of concept for the realm of commercial EEG. Additionally, its immediate popularity is indicative of a shift in society’s opinion on the integration of BCIs into common culture.

TARGET USER GROUPS

The main goal for the current iteration of this application is to provide a more portable and scalable application for medical fields that currently use EEG. With that said, the application is customizable and can be tailored for independent research and annotation of neurofeedback data.

Neurologists and Psychologists

In the field of Neurology, doctors are already examining correlations between EEG and numerous brain disorders, including addiction and other naturally occurring diseases of the brain. EEG is commonly used to study brain ailments such as multiple personality disorder [7], migraines [8], epilepsy [9], and seizures [10]. With regards to the effects of drugs on the human brain, research has also been done into EEG in the classification and evaluation of the pharmacodynamics of psychotropic drugs [11], cerebral blood flow velocity abnormalities in chronic cocaine users [12], as well as other forms of addiction. Additionally, EEG has been used to study many forms of brain trauma including neuronal injury after brain anoxia [13] and impact-related brain trauma [14]. While neurologists and psychologists continue to learn more about the (brain?), portable and easy-to-use neurofeedback applications like this project will provide new methods of attaining vital information for the advancement of neuroscience.

The Average Person

In addition to the immediate benefits that this technology will provide to doctors, there is also a future for personal EEG data tracking. In the same way that doctors use EEG to learn about neurological disorders and how they are triggered, the everyday person may soon be able to use portable neurofeedback devices to augment daily routine by quantitatively juxtaposing common daily activities against personal perception of state of mind.

TECHNOLOGY

This project involves the use of a variety of technologies. At the core of the system is the NeuroSky signal-processing chip. The data from this single dry electrode EEG toy is sent to an Arduino using methods described in the hack by the Frontier Nerds (mentioned above). The data is then encrypted and sent through a Bluetooth module that was purchased from Sparkfun [15]. The encrypted Bluetooth packets are received by, parsed, and time stamped by the application that I developed on an HTC Nexus One Google Android mobile phone. The packets are then methodically stored to a .csv file on the internal SD card of the phone. Additional manually selected inputs are also time-stamped and written to the .csv file on the phone’s internal SD card. The .csv file can then be opened in Microsoft Excel to be analyzed and visualized with charts and graphs. The remainder of this section goes into further detail about the technologies involved in this project.

Figure 1. Flow of EEG data from the Mind Flex to a Bluetooth communication device [5].

Electroencephalography (EEG)

EEG is the recording of changes in electric potential outside the skull, caused by fluctuations in brain activity. The electrical activity originates from current flowing between the neurons of the brain. EEG is commonly used to research evoked potentials (EP) and event-related potentials (EPRs). The former requires averaging the data and connecting it to external stimuli, and the latter refers to averaged EEG responses that correlate to more complex processing of stimuli. EGG signals are separated into different bands that are unique in their frequency. These different frequencies have unique distributions over the scalp and also have individual neural significance that relates to different brain activity. While there are more accurate methods of retrieving neurofeedback data than EEG, they are more invasive and require more equipment. These techniques include electrocorticography (ECG), and magnetic resonance imaging (MRI) [16].

EEG Bands

Standard EEG band frequencies fall in the range of 0 to just over 100 Hertz. These bands differ between adults and children and their locations are dominant in different regions of the brain. The chart below reveals some of the common EEG bands and their respective frequencies, brain states, and wave types.

Figure 2. Common EEG Band Chart [16]

Electrode Types

Electrodes used in EEG devices have two types of classification that affect the quality of data received by the electrode. The first classification is whether the electrode is “dry” or “wet”. Dry electrodes consist of a dry conductive surface that is placed against the scalp. Wet electrodes are coated with a highly conductive solution, often saline, which significantly increases the clarity of the data [17]. The other classification is whether the electrode is “active” or “passive”. Active electrodes have inbuilt circuitry that amplify the electrical current close to where the signal is picked up from the scalp. Due to the extremely small electric potentials (millionths of a volt) that are recorded during EEG, data can be greatly distorted by even the resistance of common conductive wire. Therefore, active electrodes that are able to amplify the signal early in the system produce a much better resolution. In both of these cases, however, the option that provides the stronger signal is also more cumbersome [1].

International 10-20 System

When dealing with multiple electrodes, EEG placement has been standardized to allow for better collaboration between researchers. The standardized system is referred to as the International 10-20 System. The diagram in Figure 3 depicts a top-down perspective of the electrode placement of an apparatus the uses the 10-20 system [18]. Note that this project does not use this system due to the fact that it utilizes a single electrode and in turn does not provide spatial resolution of the EEG signal.

Figure 3. International 10-20 System of EEG Electrode Placement [18]

Hardware Used

I used a variety of hardware in this project that enabled collection and transfer of EEG data.

NeuroSky MindFlex

The NeuroSky MindFlex is a proprietary commercial EEG device that uses a single electrode that is both dry and passive to parse a raw neurofeedback signal into 11 channels. The 11 channels are: connectivity – a reading between 0 and 200 (0 being perfect connectivity), attention (a black box value calculated by NeuroSky’s proprietary signal-processing software), meditation (similar to attention), theta, delta, low alpha, high alpha, low beta, high beta, low gamma, and high gamma [3].

Arduino

“Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It’s intended for artists, designers, hobbyists, and anyone interested in creating interactive objects or environments [19].”

Sparkfun Bluetooth Mate Silver

This Bluetooth module is designed to be used with the Arduino environment. The module’s modems work as a serial (RX/TX) communication. It can be programmed to stream serial data at baud rates between 2,400 and 115,200 bps. The device is described thoroughly on Sparkfun’s website, which contains schematics, data sheets, and tutorials for the product [15].

HTC Nexus One Google Android Phone

The HTC Nexus One is a mobile phone that runs on the Google Android operating system 2.3.6. It was released in 2010, is Bluetooth compatible, has a resolution of 480×800 px with 254 pixels per inch, and contains an internal SD card. Though it is not the most modern Android phone, it was still able to run this version of the EEG application – a good sign for future iterations.

Software Used

A wide variety of software was used to develop this application. I worked with Arduino and Processing primarily. Arduino’s IDE is based on C/C++ and is supported by an extensive online website with learning material, examples, and forums [19]. Processing is a large collection of Java libraries that are primarily graphics-based. It too is supported by an extensive website that allows even novice programmers to start developing right away. Processing is very useful when working with Android development because there is an Android mode built into the Processing IDE that allows developers to test applications on an Android emulator that can mimic various types of Android phones. Additionally, developers are able to access the Android SDK and APIs directly from the Processing environment. Certain aspects of the application required writing raw Java code that had not yet been directly translated into processing libraries. These elements of the code included working with Bluetooth, writing to the SD card, and running the application as a background process so as not to interrupt the data stream [20].

EARLY PROTOTYPE & PLANNING

This project began in December of 2011 after I first familiarized myself with EEG technology and the field of brain-computer interfacing. It started as a predominantly hardware-based project while I was still getting acquainted with the Arduino environment and replicating the hack detailed by the Frontier Nerds of NYU’s ITP department. In addition to the initial hardware, I created an interface concept design for what would eventually evolve into the current application that this paper details. Before beginning work on the current application, I also conducted a generic survey via Facebook to gather input on what activities and moods the “everyday person” would be interested in comparing to their own quantitative EEG data.

BrainCap v1.0

My own work with BCIs began with a project that I dubbed the BrainCap, which eventually became known as BrainCap v1.0 due to later iterations. The device used the Arduino library created by the Frontier Nerds to extract the serial EEG data from the NeuroSky MindFlex. With some simple Arduino code, the system wrote the incoming data to an SD card using an SD Breakout module purchased from Sparkfun. A 9-volt battery was used to power the device, which allowed the user to walk around without any chords attached. Additionally, the devices had built-in buttons and beepers to ensure that the system would start and stop appropriately without deleting the collected data. All of the components of the device were mounted onto a baseball cap that was purchased at a dollar store.

The device was successful in that it collected and time-stamped data, but there was no built-in mechanism for adding context to the data with external stimuli. I tried doing this by taking extensive notes during sessions when the data was recording, but it proved to be tedious. BrainCap v1.0 can be seen in Figure 4 while some of the data that was collected can be seen in Figure 5. A more thorough explanation of the project and its results can be viewed at the following URL:

http://www.digitaldistillation.com/pComp/?p=897

Figure 4. BrainCap v1.0
Figure 5. External Stimuli Recorded by Hand to Add Context to the Recorded EEG Data

Application Interface Concept Design

As the need for a more convenient method for recording external stimuli became apparent, I began to develop concepts for a mobile application interface to accompany the BrainCap. The basic interaction of this interface design was to provide a simple yet comprehensive system for manually inputting both preset and custom activities and moods to contextualize the neurofeedback. Some sketches of the early interface design, which ended up being very similar to the current interface, can be seen in Figure 6.

Figure 6. Early Interface Design

To ensure that the interface received some feedback before I began developing it, a simple survey was distributed via Facebook to get input about common daily activities and moods that people would be interested in learning more about with regards to their own EEG data. The survey questions can be seen in Figure 7.

Figure 7. Facebook Survey for Feedback on Interface Design Features

CURRENT SYSTEM

The current iteration of the application achieves the goals that were set in the early stages of the project. The system successfully records both passively collected EEG data from the MindFlex and manually input activities and moods from the mobile application to the internal SD card of the Nexus One Android mobile phone. The BrainCap was modified to include the bare minimum number of components in order to achieve a longer battery life and also to ensure the clean processing of EEG data with the Arduino.

Portable EEG Device (BrainCap v2.0)

BrainCap v2.0 consists of the following components: the NeuroSky MindFlex, an Arduino Uno, a long-lasting Ultralife 9v battery, a breadboard, the Sparkfun Bluetooth Mate Silver, electrical wires, and the same baseball cap used in v1.0. The only software written for this part of the project was compiled in Arduino and uses the library created by the Frontier Nerds. BrainCap v2.0 can be seen in Figure 8.

Figure 8. BrainCap v2.0

Mobile Application Interface

The interface of the Android application contains four main tabs: Annotate, Your Brain, Share, and Settings. Currently the only tab that has any functionality is the annotate tab. Within this tab the user is prompted to select from two types of manual inputs – activity or mood. Once either of these is selected, the user is taken to a new window where he or she is able to select from an assortment of various preset common activities or moods. In addition the user has the capability to create custom activities and moods to personalize his or her EEG annotation. Once an activity or mood is selected, the user has the ability to decide whether the input should be turned on or off, selected as an instant event, or retroactively added to record past events. If an activity or mood is turned on, it is then highlighted with a green overlay to indicate to the user that it has been activated. After a selection has been made, the application then writes the appropriate subcategory to the internal SD card on the Android and uses the internal clock of the phone to time-stamp the entry so it can be synchronized with the passively recorded EEG data. Additionally, the interface has a narrow text field at the top of the screen that shows the incoming EEG data packets that arrive once per second. A screenshot of the interface can be seen in Figure 9 below.

Figure 9. Screenshot of the Application’s Interface

Data Analysis Techniques

Once data has been recorded to the internal SD card of the phone, the .csv file that contains the data can be easily sent to a computer via a standard USB cable. That file can then be opened in Microsoft Excel to construct charts and graphs for better interpretation of the data. Figure 10 shows a graph of sample data that was recorded using the application detailed in this project. The graph depicts my brain’s attention and meditation activity as I played a FIFA World Cup final in the Xbox game FIFA 2010 with one of my friends. The data points are averaged over 30 second intervals to provide a more coherent visualization of the EEG. Averaging the data over larger intervals is necessary due to the poor quality of the EEG data that is recorded from a single electrode that is both dry and passive.

Figure 10. A Graph That Depicts My Brainwaves as I Played a Video Game with My Friend

CONCLUSION

The many phases of this project have encompassed a wide range of technologies that have come together to form a unique application with some potentially significant implications for various fields of research. Though the project is still in an early stage, the groundwork has been laid and the application can be easily adapted from this point forward. The testing that has been done so far has provided unique insight into what is possible with a system that uses this type of EEG device – a device that provides crude real-time data that can be averaged over longer intervals to identify clear trends in brain activity. However, the lack of data analysis is one of the major shortcomings of this project. Because I have the only working instance of the system I am unable to get large amounts of data from different test subjects.

Findings

At this point no analysis has been done into finding correlations between external stimuli, personal perception of state of mind, and EEG data. This type of research will require an initial user group to test the application. Despite this, the few tests that have been run show clear linear and oscillating trends in brainwaves when averaged over extended periods of time. It was discovered early on that the data is too chaotic when looking at the data points that are received every second. However, when this data is averaged, distinct patterns in brainwave activity can be identified.

Future Directions

Moving forward there are many steps that need to be taken.

Initial User Group

An initial user group that is willing to test this application needs to be selected in order to get user feedback on the interaction of the application. Additionally, this user group will provide the amount of data that is necessary to start juxtaposing EEG data against daily activities and moods. From there, data analysis can commence to see what types of correlations data of this quality can produce between EEG and other information.

Interface Additions

As far as the interface is concerned, a data visualization system will be implemented so that the user can track data in real-time. This data visualization will allow the user to see automated graphs of his or her brain activity without having to upload the .csv file into Microsoft Excel. The data visualization interface? will resemble something similar to the processing sketch (Figure 11) that the Frontier Nerds made to accompany their Arduino library [5].

Figure 11. EEG Data Visualization Developed by The Frontier Nerds [5]

Sharing Capability

The next iteration of this project will also include an option for the user to anonymously share his or her EEG data and inputs with a central database. This database will serve as an aggregation of human brain activity and will provide an invaluable set of data to analyze and use as a baseline to compare to the EEG data of any individual.

Ethical Considerations

When dealing with a technology as new and powerful as brain-computer interfacing, there are obvious and not-so-obvious ethical implications that must be taken into consideration. While the capabilities of this type of application are limited due to the crude quality of the data, technology will continue to advance allowing for clearer data to be acquired more easily. Soon it will not be science fiction to have a portable EEG device that has bi-directional interactivity – in other words, a device that talks back. The implications of this type of technology lead to numerous potential risks that need to be planned for in order to prevent their negative consequences. Some of these risks include health and safety, legal issues, abuse of technology, security and privacy, and social impacts.

ACKNOWLEDGEMENTS

I’d like to give a special thanks to my Parsons Design + Technology instructors from Spring 2012, Jonah Brucker-Cohen, Katherine Moriwaki, Joel Murphy, and Ed Keller, for providing great mentorship throughout the development of this project. I’d also like to thank Neurosky, Sparkfun, Arduino, Google, and Processing for giving me fun toys to play with. Thank you, my good friend, Joe Artuso, for editing the piece. Lastly, thank you too The Frontier Nerds for giving me a place to start.

REFERENCES

I tried my best to give credit to all of the work that I referenced in the development of this project. Please contact me if you’d like me to add/remove anything to/from this blog post. I will not hesitate to do so.

  1. The OpenEEG Project. http://openeeg.sourceforge.net/doc/
  2. Emotiv. http://www.emotiv.com/
  3. NeuroSky. http://neurosky.com/
  4. Electroencephalography. http://www.bem.fi/book/13/13.htm#03
  5. Mika, E., Vidich, A., Yuditskaya, S. How to Hack Toy EEGs. Frontier Nerds: An ITP Blog. http://frontiernerds.com/brain-hack
  6. Necomimi. http://neurowear.com/
  7. Arikan, K., et al. EEG Correlates of Startle Reflex with Reactivity to Eye Opening in Psychiatric Disorders: Preliminary Results. Clinical EEG and Neuroscience 37.3 (2006), 230-234.
  8. Bjørk, M.H., et al. Interictal Quantitative EEG in Migraine: A Blinded Controlled Study. The Journal of Headache and Pain 10.5 (2009), 331-339.
  9. Kennett, R. Modern Electroencephalography. Journal of Neurology 259 (4), 783-789. April 2012.
  10. Bubrick, E.J., Bromfield, E.B., Dworetzky, B.A. Utilization of Below-the-Hairline EEG in Detecting Subclinical Seizures. Clinical EEG and Neuroscience 41.1 (2010), 15-18.
  11. Saletu, Bernd, Anderer, P., Saletu-Zyhlarz, G. EEG Topography and Tomography (LORETA) in the Classification and Evaluation of the Pharmacodynamics of Psychotropic Drugs. Clinical EEG and Neuroscience 37.2 (2006), 66-80.
  12. Copersino, M.L., et al. EEG and Cerebral Blood Flow Velocity Abnormalities in Chronic Cocaine Users. Clinical EEG and Neuroscience 40.1 (2009), 39-42.
  13. Rossetti, A.O. Early EEG Correlates of Neuronal Injury After Brain Anoxia. Neurology 78 (11): 796-802. March, 2012.
  14. Tezer, I., Dericioglu, N., Saygi, S. Generalized Spike-Wave Discharges with Focal Onset in a Patient with Head Trauma and Diffuse Cerebral Lesions: A Case Report with EEG and Cranial MRI Findings. Clinical EEG and Neuroscience 35.3 (2004): 151-157.
  15. Sparkfun. http://www.sparkfun.com/
  16. Budzynski, H., Budzynski, T., Evans, J. Introduction to Quantitative EEG and Neurofeedback: Advanced Theory and Applications. 2nd Edition. Elsevier Inc. 2009.
  17. Wang, Wang, Maier, Jung, Cauwenberghs. Dry and Noncontact EEG Sensors for Mobile Brain-Computer Interfaces. IEEE Transaction on Neural Systems and Rehabilitation Engineering 20 (2): 228-235. March, 2012.
  18. Gilmore, R.L. American Electroencephalographic SocietyGuidelines in Electroencephalography, Evoked Potentials, and Polysomnography. J. Clin. Neurophysiol (11), 147. January, 1994.
  19. Arduino. http://arduino.cc/
  20. Processing for Android. http://wiki.processing.org/w/Android
  21. Rao, R.P.N. University of Washington. Computer Science. March 7, 2012. http://www.cs.washington.edu/homes/rao/

WANNA GET INVOLVED?!

I’d be happy to share the graphics, code, and system schematics with anyone that wants to help with application development and/or collection of data. If enough interest is generated, I’d also be willing to post a thorough step-by-step procedure for application/device development with all the necessary assets for the Android App. If you are even remotely interested, don’t be afraid to contact me at conor.russomanno@gmail.com AND comment on the blog (interest is contagious!). The step-by-step will eventually get submitted regardless, but support from you guys would definitely help get the ball rolling. Brainiacs, assemble!


Blog at WordPress.com.

Up ↑