Portraying The Terran Condition is a 7D (backwards compatible to 2D) world simulation which depicts six different key events in the history of Terra (“Earth”), a low-tech civilization that self-destructed several aeons ago. Based on the relatively few biological and cultural artifacts, a team of multi-AI minds was able to recreate a stunningly accurate depiction of this ancient civilization.

The game is available for Windows and Mac.

Additional thanks are due to Eric Wenske, my teammate from Cosmic DJ, who made a couple models, and to Heather Kelley, fellow Kokoromi member, whose wise words led me to do procedural level generation for the game. So, I guess it’s also a rogue-like!

I recently purchased a Meeblip because I wanted a project to work on with my maker friends that I get together with every week. I decided to buy the full build-it-yourself kit, which meant I was in for a fair amount of soldering and so forth. Anyway, that all finally came together and I plugged it into my computer to see if I could control it with a MIDI track in Reaper. Yup! You sure can!

However, what you can’t do is create an envelope for any old MIDI CC message. You can put CC messages into the MIDI clips themselves, but that’s not very conducive to doing the kind of automation I’m used to doing with bass synth VSTs. It turns out that Reaper comes with a JS effect called MIDI_CCRider, which allows you to put an LFO on any CC message you want. I was able to look up which CC message was filter cutoff for the Meeblip and automate it using that effect. But this was not great because it meant that I’d need an instance of the effect for every parameter I wanted to control and that I’d have to constantly remind myself which CC messages controlled which parameters of the synth. So, instead, I decided to use that effect as the basis for a new JS effect that would have all that information baked into it and simply give me a bunch of sliders and combo boxes labeled the same as on the Meeblip. And voila:

The effect only sends a CC message when the value of a slider changes. This means that you can use it to set some initial values and then play with the knobs on the Meeblip while you loop a bassline. Only parameters of the effect that you choose to automate with an envelope will be constantly overridden. If you’d like to try it out with your Meeblip, you can grab it from Github: https://gist.github.com/3606021

Now, if you Google the title of this post, you’ll find quite a few people who are seeing spikes in the Unity3D profiler for something called Overhead and if you follow those links you’ll find that for the most part people aren’t incredibly certain how to make them go away. The Unity3D docs don’t really explain what Overhead represents, but you can see it in the very first image on the page. Some people on the internet seem to think that Overhead includes garbage collection, but I’m pretty good about not creating tons of new objects every frame and others point out GC.Collect is what will show up for that. Also, if you look closely at the legend in the image you’ll see GarbageCollector has its own color.

No, it turned out that the culprit was using UnitySendMessage in a native plugin code! My app has a music system and in order to keep good metronomic time, I run the metronome with native code. This is important because if the metronome is only updated every frame, then you can wind up with wonky rhythm when there are framerate spikes. Lots of things are tied to the metronome in this app, so I needed a way to tell script about “ticks”, which simply means calling script on every sixteenth note. I used a line of code that looked like this:

UnitySendMessage( "Metronome", "Broadcast", m_tickString );

This function tells the Unity runtime to find the object named “Metronome” in the scene and then call a function named “Broadcast” that takes a string. In my Metronome.cs script I then converted the string to a float and told all of the objects listening to the metronome that a tick just happened. I wrote it this way because the very first tip given for iOS plugins is “Managed-to-unmanaged calls are quite processor intensive on iOS. Try to avoid calling multiple native methods per frame.” So I figured having Metronome.cs call a native function every frame to retrieve ticks would probably be a bad idea, but it turns out it’s better than using UnitySendMessage.

I changed the way the metronome works so that when the native code gets to each tick it pushes it onto the back of a list, then in the Update function for my C# Metronome, I pass a float array to a native Metronome_GetTicks function, which then fills the array with all of the ticks in the tick list and returns how many ticks it put into the array. Then, for each tick in the array, I call the same Broadcast(float) method that I called from Broadcast(string). Now profiling the exact same level that gave me spikes looks like this:

The Chambre Sonore is a room that was custom built for the Gaîté, though not specifically for my piece, that contains a 10 channel sound system (8 mounted in the walls around the room, 1 in the ceiling, and a sub behind one of the walls), 8 pressure sensitive floor panels, and 8 color LED lights that ring the ceiling. All of these can be interfaced with from a computer under the stairs next to the room.

Elephant in the Room Wall Text

High Concept

The high concept of the piece is that the room has a digital creature living in it and that visitors can interact with the creature by stepping on the different floor panels in the room. Over time this interaction with the creature changes how it sounds so that if someone visits it more than once in a day, or comes back several days later, it should sound noticeably different than their previous visit.

Implementation

The actual implementation of the piece differs slightly from the high concept, however. I had originally envisioned creating a single complex sound generating system that would have lots of different values that could be slowly changed by stepping on the various floor panels. In this way, it would have been a very slowly evolving soundscape, without a readily discernible way for visitors to tell that they were making a difference by being in the room. Ultimately, I decided that it would make more sense within the context of the exhibition as a whole if visitors were able to easily figure out that the piece was reacting to them.

So, the piece contains five different modes that it might be in when a visitor enters and by paying close attention to the feedback provided by the lights and sounds, they should be able to figure out the “puzzle” of the mode and then intentionally push the mode to its end, triggering a transition to the next mode. The modes have the very unpoetic names of Rainstorm, Pulsing Tones, Soothe, Chirps, and Tune Maker.

Hyrax footprints stuck to the floor of the Chambre Sonore to indicate where visitors can stand to interact with the piece.

Each mode is a very different soundscape, helping to meet the goal of visitors hearing something different if they visit the room a second time. Rainstorm sounds like digital rain with a whooshing wind and if visitors stand in the correct place, they will anger a creature that will roar at them. Pulsing Tones is a large pulsing chord that visitors can turn into high-pitched static by increasing the amplitude modulation frequency applied to each note in the chord. Soothe is a gentle environment with a low frequency noise wash, long glassy tones taken from a harmonic series, and a soft heartbeat emanating from the wall. Visitors can create marimba-like tones by stepping on floor panels. Chirps emits periodic descending or ascending tones that are patched through a modulated delay effect. Visitors can use floor panels to increase and decrease the duration of the tones, or to increase and decrease the speed of the delay modulation. Tune Maker is a generative dance tune that visitors can add musical elements to simply by dancing around the floor. If they dance fast enough they are rewarded with a prerecorded dance tune that has choreographed lighting. Here’s a short video of what Rainstorm sounds and looks like.

Technology

As mentioned, there are three systems in the room that the computer interfaces with: 10 channel audio, 8 pressure sensitive floor panels, and 8 color LED lights. The audio runs through an audio card installed in the computer, out to two amplifiers. The floor panels send MIDI messages and are plugged into a USB MIDI interface. The lights are controlled using DMX and I chose to use the ENTTEC DMX USB PRO to control them. I was able to interface with all of this hardware using Processing.

Visual proof that Elephant in the Room was built with Processing.

For the audio, I used Minim, my sound library. Almost all of the sound is generated in real-time using the UGen framework. However, to handle the 10 channels, I had to bypass using an AudioOutput object and write a custom class that could deal with the sound card on the computer. JavaSound gives access to audio devices through the Mixer class. Typically, for a multi-channel sound card, the outputs will be made available as stereo pairs. So, there’s a Mixer for channels 1/2, 3/4, 5/6, and so on. Once you have a Mixer, you can ask it for a SourceDataLine that is used to write out audio data, but if a Mixer represents a stereo pair, you’re never going to be able to ask for a 10 channel SourceDataLine. In fact, even though two of the Mixers listed for the sound card were named 5.1 and 7.1, I was not able to ask those for 6 and 8 channel outputs. Instead, I gather up all the stereo pairs representing channels 1 through 10 and ask each of them for a stereo output. Then I generate 10 channels of audio from my root UGen and write out two channels of audio to each stereo output in turn. This lets me think about generating audio in 10 channel terms and makes it easy to write UGens that can pan sound around the room, send sound to just one speaker, or expand a stereo signal across several speakers.

Flashing disco lights during Tune Maker.

For the lights, I used a library for Processing called dmxP512, which made dealing with the lights super easy. At first, I thought I would have to use MIDI and a light cue building program that had been used with previous installations for the Chambre Sonore, but the direct control that dmxP512 gave me was much better. I wound up writing a wrapper class for it that let me set colors using a color variable in Processing and also let me fade colors over time. A very useful idea I came up with was to allow for animating color and intensity separately. So I can set a light to color(255,15,23), for instance, but then set the intensity to 0.2 and wind up with that color at 20% brightness. I simply multiply the RGB components by the intensity before sending them to the lights and this allows me to do stuff like blink a light by animating the intensity while at the same time slowly shifting the color from blue to pink.

The floor communicates with the computer by sending MIDI messages and I was able to receive these by writing some pretty straightforward JavaSound MIDI code. Basically, you just ask for the Transmitter object of the MIDI device that the floor is plugged into, write a class that implements the Receiver interface, and set an instance of that class as the Receiver for the Transmitter from the MIDI device. So I wrote a FloorReceiver class that I can add a FloorListener to so that the MidiMessage parsing can be in one place and other code can simply receive notifications when a floor panel goes down or up. I believe the floor is made by a French company called Interface-Z and it appears to send only one kind of MIDI message: a NOTE_ON message where the note number indicates which pad sent the message and the velocity indicates whether the panel went down (64) or up (0). I did find the floor to be a little bit frustrating to work with because some panels respond in only a small area and others a comparatively large one. This is the reason for the footprint stickers: to show people where they can stand to best interact with the room and also which direction to face so they will hopefully see a connection between where they are standing and the light that is blinking at them.

The Chambre Sonore lit with cool cyan lights.

Reception

From what I have seen, most people enjoy interacting with the room and seem to understand that they are having some kind of effect on the sound. But aside from several people I know personally who really made an effort to figure out the room, I haven’t seen anyone actually clue in to how to get a mode to an end point. Certainly, part of this is due to the fact that nothing explicitly states that this is possible, but I think it also has to do with the fact that the only feedback is from lights and sound. Each floor panel is associated with a particular light and speaker, and this relationship is maintained throughout all the modes, but unfortunately the layout of the room doesn’t physically reinforce the relationship. This isn’t a problem per se, since the piece is meant to work as simply an interactive ambient soundscape, and the piece has been switching between modes despite people not quite getting it. I had hoped to do more observation of the general public interacting with the room, but if I’m in the room, it sort of changes the experience for people, so I’ve mostly stayed out of sight. All in all, I’m pretty happy with how it’s going so far.

Elephant in the Room shows at the Gaîté Lyrique in Paris as part of Joue le Jeu – Play Along until August 12th, 2012.

Also coming reasonably soon will be version 2.0, which will include Dropbox integration and Audio Copy/Paste.

The reason it doesn’t work is because (I’m pretty sure) openGL’s coordinate system is not the same as Apple’s. Part of what openFrameworks does is translate the coordinates of incoming touches from Apple’s coordinate system to openGL’s before sending touch events. This allows you to always think about the drawing surface in the “natural” way for doing openGL apps: upper left corner is always (0,0), positive X is to the right, positive Y is down. You can use this knowledge to simply transform your openGL coordinates to Apple coordinates when setting the position and size of UIKit widgets, but if you are using ofxiPhoneSetOrientation to reorient your openGL surface, the mapping between openGL coordinates and UIKit coordinates changes depending on the orientation. This means you have to reposition all of your UIKit objects every time the device is reoriented. Even worse, what I discovered is that in one of the landscape modes, even though the UIKit widgets looked correct, I actually had the device orientation set to the opposite of what it should have been, so when I displayed popovers, they were upside-down. When I presented modal view controllers, they would appear in the correct orientation, but then not rotate when device orientation changed.

After much hair-pulling, I finally came up with a solution that I think is pretty solid. Essentially, what I have done is embed the EAGLView in a view hierarchy that is well behaved with respect to Apple’s system and I do not call ofxiPhoneSetOrientation to change the orientation of the app. What this means is that from openFrameworks’ perspective the app never changes orientation, which is true in some sense because the EAGLView is added as a subview to a vanilla UIView that is strictly responsible for handling the rotation.

Ok, onto some actual code. Before setting up any UI, I set the openFrameworks orientation to landscape:

ofxiPhoneSetOrientation( OF_ORIENTATION_90_RIGHT );

Then I rearrange the view hierarchy that openFrameworks creates so that I can set the position and size of UIKit widgets using openFrameworks coordinates. Included in this is creating a root view controller for the window, so that I can take advantage of the auto-rotation behavior they provide (and because Apple expects your app to have a root view controller):

// remove the glview from the UIWindow, which is where openFrameworks puts it
[ofxiPhoneGetGLView() removeFromSuperview];
            
// create a plain old UIView that is the same size as the app
// just use OF sizes as our reference, everything should work out
CGRect mainFrame    = CGRectMake(0, 0, ofGetWidth(), ofGetHeight());
mMainView           = [[UIView alloc] initWithFrame:mainFrame];
            
// grab the glView and rotate/position it so it looks correct
UIView* glView      = ofxiPhoneGetGLView();
glView.transform    = CGAffineTransformRotate(CGAffineTransformIdentity, M_PI_2);
glView.center       = mMainView.center;
            
// glView goes on our main view to prevent wonky resizing
[mMainView addSubview:glView];
            
// root controller has main view as its view
mRootViewController      = [[LandscapeViewController alloc] init];
mRootViewController.view = mMainView;
            
// and finally we set the root view controller
// which will make mMainView the window's primary view
// all other views are then added as subviews of mMainView
// NOT as subviews of the UIWindow or the EAGLView
ofxiPhoneGetUIWindow().rootViewController = mRootViewController;

Here’s the LandscapeViewController class, which limits supported orientations to landscape orientations and also turns off animations during rotation so that the orientation snaps. If you like the animated orientation change, you can simply remove those lines.

// barebones UIViewController so we can control allowed orientations of SoundCloud interface.
@interface LandscapeViewController : UIViewController 
{
}
@end

@implementation LandscapeViewController
        
- (BOOL) shouldAutorotateToInterfaceOrientation:(UIInterfaceOrientation)toInterfaceOrientation
{
    if ( UIInterfaceOrientationIsLandscape(toInterfaceOrientation) )
    {
        return YES;
    }
    
    return NO;
}

- (void) willRotateToInterfaceOrientation:(UIInterfaceOrientation)toInterfaceOrientation duration:(NSTimeInterval)duration
{
    // snap to new rotation
    [UIView setAnimationsEnabled:NO];
}

- (void)didRotateFromInterfaceOrientation:(UIInterfaceOrientation)fromInterfaceOrientation 
{
    // animations back on so we get the nice slide-ons for the keyboard and so forth
    [UIView setAnimationsEnabled:YES];
}

@end

Finally, the class that registers to receive ofxiPhoneAlerts needs to handle device orientation changes like this:

void App::deviceOrientationChanged(int newOrientation)
{
    UIApplication * app = [UIApplication sharedApplication];	
    // status bar orientations are reversed from device orientations
    // so that we can use OF coordinates to layout UIKit widgets.
	if ( newOrientation == UIDeviceOrientationLandscapeLeft )
	{
        [app setStatusBarOrientation: UIInterfaceOrientationLandscapeRight 
                            animated:NO];
	}
	else if ( newOrientation == UIDeviceOrientationLandscapeRight )
	{
        [app setStatusBarOrientation: UIInterfaceOrientationLandscapeLeft 
                            animated:NO];
	}
}

And again, WaveShaper only displays in landscape, which is why those are the only orientations I handle.

Once all of this is in place, you can happily position buttons, sliders, popovers, and whatever else using the openFrameworks coordinate system and they will all look correct and rotate with the screen.

The update should be available on December 30th. This is later than I’d like it to be available, but the iTunes Connect holiday shutdown began today and lasts through the 29th.

My sincerest apologies to all who have bought the app and experienced these crashes.