Search

C World

Conor's Blog/Portfolio

Category

Interaction

ROB3115 – A Neuro-Immersive Narrative

In-experience screenshot

ROB3115 is an interactive graphic novel that is influenced by the reader’s brainwaves. The experience is driven by the reader’s ability to cognitively engage with the story. ROB3115′s narrative and its fundamental interactive mechanic – the reader’s ability to focus – are tightly intertwined by virtue of a philosophical supposition linking consciousness with attention.

ROB3115 explores the intersection of interactive narrative, visual storytelling, and brain-computer interfacing. The experience, designed for an individual, puts the reader in the shoes of a highly intelligent artificial being that begins to perceive a sense of consciousness. By using a NeuroSky brainwave sensor, the reader’s brain activity directly affects the internal dialogue of the main character, in turn, dictating the outcome of his series of psychosomatic realizations. The system is an adaptation of the traditional choose-your-own-adventure. However, instead of actively making decisions at critical points in the narrative, the reader subconsciously affects the story via their level of cognitive engagement. This piece makes use of new media devices while, at the same time, commenting on the seemingly inevitable implications of their introduction into society.

This project was my thesis in graduating from Parsons with an M.F.A. in Design & Technology.

BrainSYNC – Neurofeedback vs. Social Media

Check out the post that I made below to read about my ideas of syncing personal EEG feedback with the masses via a smartphone application.

http://interaction2012.coin-operated.com/?p=547

Project Notes (EEG to Mobile App)

Notes on my ongoing EEG project (with semester plan):

A Response to Emerging Technologies

This piece is a response to the following articles:

The evolution of new systems that undermine the physical boundaries of our current world is the theme that stands out when comparing and analyzing these three articles.  Each of these articles identifies emerging synthetic life forms that are redefining the world’s power structures. These life forms are systems that humans – as individuals – can create, but once created have little to no control over.

In the article Why We Twitter: Understanding Microblogging Usage and Communities by Akshay Java, Tim Finin, Xiaodan Song, and Belle Tseng, the authors identify a new system of communication referred to as micro-blogging. This system has evolved as a result of rapidly advancing Internet technologies in addition to humanity’s slow evolution into and acceptance of a joint physical-virtual existence. In the last 20 years, the emergence of virtual social media platforms has redefined people’s understanding of community. Platforms such as Twitter, Facebook, and Xbox Live have remapped our community of friends and family from a demographic of people within our close physical proximity to a community of individuals that could be anywhere in the world but who share a similar virtual demographic as ourselves. Amidst this evolution there has been a drastic transformation in how information is disseminated from “the source” to “the recipient.” The world in which exist linear channels of information transfer dominated the media no longer exists. Instead information flow resembles more of a spider web pattern where anyone and everyone can and does contribute.  The result of this shift is a world where control of information is virtually impossible.

In the Wired article Great Wall of Facebook: The Social Network’s Plan to Dominate the Internet – and Keep Google Out, the author, Fred Vogelstein, brings up some very daunting facts about how much control Facebook and Google have over our personal information. Even more foreboding is how much our personal information is worth. After reading this article two things stood out to me.

First, I found it incredible how one individual, in this case Mark Zuckerberg, could create a system that had the ability to infect the entire planet in a matter of a few years. At this point, it would be impossible to stop the Facebook pandemic.  Even if Facebook were to destroy all of its servers today, Google would immediately fill the void with Google+ – and if not Google+, then somebody else.  The demand for personal information of the masses is just too high.

The second idea that stood out to me – and this is a personal opinion, not a fact – is the notion that governments no longer monitor and control their citizens.  Instead, it’s the corporations housing our personal information and virtual identities that truly govern us.  In the same way that communication is evolving so is the concept of governance and patriotism. Governments are losing their authority to corporations as a result of new forms of taxation. Corporations are able to tax our virtual identities, properties, and businesses whereas governments can only tax our physical ones. Just a little fact: Apple surpassed the United States government in liquid cash flow in late 2011 (http://www.denverpost.com/business/ci_18590939).  I would be interested to see a study on the distribution of people who would rather be seen wearing an American flag on their T-Shirt compared to those who would rather have a glowing white apple on their laptop.

I found the third article, Why The Future Doesn’t Need Us, by Bill Joy, to be the most thought provoking of the bunch. Joy raises the issue of the looming threat of human intelligence and the “desire to know” leading to our eventual demise – his biggest concern being emerging genetic, nano-biotechnological, and robotic (GNR) advancements. After reading this article I was disappointed in myself for not having read it in the first 12 years of its existence. I recommend it as a must-read for anybody interested in technology and innovation and who also has an appreciation for ethics and the further existence of the human race. What fascinated me most about this article was how critically Joy examined the existence and importance of the systems of GNR advancements and how keen he was to the volatility of they’re potentials. The viral nature of social media platforms such as Twitter and Facebook are proof of the disastrous capabilities of technologies that attempt to revolutionize fields such as genetics, cells, and robots that augment human life.  It is important that humans proceed with caution before we see the Mark Zuckerberg of genetic engineering create the Facebook of gene-modifications that can’t be reversed.

Just as a single spark can ignite a container of gasoline, an individual today is able to create a self-replicating system that can infect the entire world. It requires cooperation and careful planning to be able to control and harness these systems to be used for the proper augmentation of human life.

Kinematics-to-Color Conversion Game

[VIDEO TO COME]

I. Introduction

This project was very experimental in nature.  I wanted to create an interface that uniquely translated the mind’s understanding of one common system onto a very different common system through a simple switch interface.  Both the physics of kinematics and the science of color have always intrigued me.  With this project I attempted to interface the two systems by means of an alternative method that the human mind would not typically think of. It was my hope that this odd translation would provide a new lens for understanding both systems, as well as, shed a new light on the human mind’s perception of a system.  In the end I decided to turn the interface into a game that tracked the player’s progress of how well he or she was able to translate between the two systems of calculus-based kinematics and the RGB color system.

II. How It Works

This game assumes that the player or viewer has a basic understanding of the RGB color system in addition to grasp of the calculus-based relationship between position, velocity, acceleration, and jerk (the rate of change of acceleration).  Through 4 clicks of the button (or switch) at the center of the application the system tracks the absolute value of the velocity, acceleration, and jerk of this interaction.  It does this by assuming a “distance” of 3 units – 1 unit for each click after the first one which is supposed to represent the starting line.  It then generates an average velocity (v), average acceleration (a), and average jerk (j) based on the timing between each of the 4 clicks.  These three values are then mapped onto the R, G, and B values respectively.  The algorithm behind the conversion assumes that the user’s 4-click timespan (total time) will be between 0.1 seconds and 30 seconds.

Method for Calculation

(Note: the values calculated for v, a, and j are rough averages due to the fact that there are only 4 data points recorded.  Additionally, this is one method of averaging the data but there are numerous other ways these averages could have been calculated.)

Referenced Variables:  T12 = time between the first and second click, T23 = time between second and third click, T34 = time between third and fourth click; additionally T1 = 0 (first click starts timer), T2 = time of second click referencing the timer that starts with the first click, T2.5 = the time halfway between the 2nd and 3rd click, etc.

Velocity:  This is calculated as the total distance, 3, divided by total time, T14:  x/t

v = 3 / (T12 + T23 + T34)

Acceleration:  This is calculated as the of the average of the local accelerations between click 1 and click 3, and click 2 and click 4.  In other words the change in velocity from T12 to T23 is averaged with the change in velocity from T23 to T34.  I used this method because in order to calculate an acceleration there needs to be a flux in velocity and with the recorded data there are actually two fluxes in velocity (at click 2 and click 3).  Note: t

a = ((A123 + A234) / 2)

A123 = (V23 – V12) / (T2.5 – T1.5)    and     A234 = (V34 – V23) / (T3.5 – T2.5)

Jerk:  This is calculated as the of the rate of change of acceleration between A123 and A234

 j = |(A432 – A321) / (T3 – T2)

Method for Conversion

Once the user has flipped the game’s switch 4 times each of the absolute values, or magnitudes, of the above calculations (v, a, and j) are respectively mapped onto a 0-255 range of the red (R), green (G), and blue (B) values of a color.  Initially the system mapped the lowest extremes of negative ( – ) acceleration and negative jerk to values of 0 for G and B respectively.  This was not necessary for velocity-to-R bc it was impossible to generate a negative value for velocity.  After some testing, however, I decided to change the conversion to where it used the absolute values of velocity and acceleration to map to the 0-255 ranges of G and B.  My rationale for this change was that the interaction between the two systems is confusing enough and shouldn’t include variable ranges for (a and j), with both negative and positive possibilites, that map onto the G and B variables that only have positive ranges.  Therefore, as it stands now acceleration values of 0 to 10 are mapped onto G-values of 0-255.  The original method had acceleration values of -10 to 10 being onto G-values of 0-255.

III. Process

Concept

Early Sketches and Calculations

These sketches and calculations show some of my thought processes in the development of my system to generate the average values for v, a, and j.  This was the first phase of my pseudocode before jumping into Openframeworks.

This is the first sketch for the layout of the game’s interface.

Manifestation

The final aesthetic was done in Photoshop and the back-end coding was done with Openframeworks.  Below is a screenshot and sample code of the variable I used:

IV. Conclusion

The game is not too difficult to pick up if you have a good understanding of calculus and/or kinematics, but it is very difficult to master due to the limited number of player inputs.  I found it hardest to produce a deep green color.  This entailed producing a high but constant acceleration without a high velocity or jerk.

Moving forward, I would like to get this game up on a website so that more people could test it.  Additionally I would like to make it social by adding a high scores database that all players can view and compete for.  In terms of practical applications for this type of conversion, a similar system might be beneficial to do data visualization in industries where calculating jerk is important.  Examples where people consider jerk include: boxing (the higher the jerk, the more devastating the punch), car accidents, etc.

Blog at WordPress.com.

Up ↑