0x09

I decided to add a rudimentary Eyesweb section since it's very useful with PICs and Pure Data. It's not going to be anything special.

In case you're not privy to what Eyesweb is, Eyesweb is a music and video program very similar to Max and Pd. The primary difference from Pd (from my perspective) is that it's closed source (!), though it's still free as in beer. The reason I use it is that it has a couple objects that make it very easy to get detailed motion capture information from a camera in realtime. I think it wouldn't be too hard to add this functionality to Pd, which would be nice so that I could run everything under Linux and with Pd only, but I'll save that for when I have more time and knowledge. In general, Eyesweb seems to have a higher-level interface than Pd, it's more "plug and play", which has its upsides and downsides

I found the tutorials available for Eyesweb a bit scarce for what I wanted to do, thus this.

This tutorial will show you how to take an avi file or DV cam input and get the cartesian coordinates of a single subject's various bodyparts.

First Up, Linkage

The way I learned this was by cutting and pasting from other people's patches, specifically the ones that came with Eyesweb and these, which are quite excellent though way over my head.

And you can look at the Eyesweb patch I wrote for this tutorial if you're interested.

And Now, Ladies and Gentlemen, Eyesweb.

Here's the basics: read the help files to familiarize yourself with the interface. This is what we in the biz call passing the buck.

Now. Create a Multimedia File Reader, it's under Input->Video. This block needs you to set its file property: for the moment, choose the microdance.avi that came with Eyesweb. You might want to copy it to your project directory. Then create a button (under tools) and set its event to be triggered on startup, this is just like Pd's [loadbang]. Connect the start button to the top of your avi reader (you'll need to export the block's start property). Now the avi will start playing when you run the patch.

Create an Output->Display and connect the avi reader to it so you can monitor the raw video stream.

Background Elimination, or, How I Learned to Extract the Foreground

This is pretty simple, but since I cut and pasted some of it from a sample I found on the web, I don't entirely understand each little bit. Here's the general breakdown: We convert the stream to one channel, compare that channel to a snapshot of the background without any dancers in front of it, apply some convolution filters, add it all up and there we go.

Since I don't fully understand it, rather than going step by step, I'll show what the finished product looks like, and then explain it bit by bit.

We already covered the start button and the avi reader. After that we convert the stream to monochrome and feed the files into a 5 member qeue so that we're doing a time-based comparison further down the line. Everything on the main line then is delayed 5 frames. Adjusting the qeue's length can help adjust for different environments

The next block is a buffer, that, when banged, holds a frame. In this case it grabs a frame at the beginning (assumably the empty stage), and whenever we push the trigger button above it, in case we want to reset. This buffer goes out to a display for reference. The buffer is then subtracted from the live image, so essentially we have the live image with the background subtracted

The greyscale image is then converted to binary so that the blob detector will be able to play with it. The image now only has a 2 bit depth. The median filter then removes noise from the stream. Next come three time-based convolution filters which help to smooth out the moving pixels. These filters are all the same block with a different filter property selected. I forget at the moment what the block name is, so look at the patch linked above if you need to know. Playing with their values is a good idea to adjust for different conditions. After the second erode the image goes to the blob analysis, but we do a couple more things for debugging.

Next we convert the binary image to color (which has no visual effect) and feed it to a display so we can tell what the blob analyzer is seeing. The silhouette is then added to the raw stream and then the raw stream is subtracted from that so that we can see the blob (the area where the silhouette was) in color. It then needs to be inverted. This serves no other purpose but for display and debugging, which we'll do to it next.

Blob Analysis

This bit took me the longest because I assumed it was very difficult to do and looked everywhere on the internet for information on it. The reason there's no information on it is that it's so frickin' easy. This is where I was impressed by Eyesweb.

We start with the binary silhouette we got from the second erode filter and feed it into the Centroid Calc block. That's it, we're done.

To see what we just did, use the mocap draw block to draw the calculated points onto the image we got after the inversion above. Put the product into a fourth display and you should see a colored silohuette (I have no idea how to spell that) with a point each for head, torso, hands, elbows, shoulders, knees, and feet.

Getting that all into Pure Data, or, How I Got My OSC On

This part takes a long time, but is simple. Just tedious. All we're doing is taking the information from the centroid calc block and sending it via OSC to Pure Data. The centroid calc block sends out all its information in one stream, so we'll use the Centroid Extractor block to extract each of the centroids we want to send. Each of these Extractors then sends its data to an OSC Scalar block. You'll need two sets of these extractors and OSC senders: one for x and one for y coordinates. It might be useful to use a single ParamChanger to set the IP address for all of the OSC blocks. In the picture below I used two 0 length qeues as nops for ease of connecting blocks. The general OSC command structure I used was [x or y]/[side of body]/[limb]/[centroid] This made it very easy to route in Pd

The Other Side. Pd

Constructing the OSC reception is much easier in Pd because you can use nested OSC routes. I wonder if there's some easier way to construct commands in Eyesweb. This patch is as straightforward as they come. It requires the OSCx external.