This project was originally intended to familiarize myself with Wiimote functions in Macromedia Director MX 2004. I wanted explore the Wiimote by creating a 3D game for my senior thesis, and Director was known for its ability to create 3D games and other large applications. As the summer went on, I realized that I was spending more time trying to code games than discovering new things with the Wiimote. I decided to broaden the project visions to include Flash functionality, a more popular multimedia authoring application that had a better documented Wiimote API.
After switching to Flash, not only did I discover new Wiimote opportunities within Flash with the help of my advisors, but found the existing use of 3D engines with Flash that had created functional, practical, and aesthetic web content. With project ideas from reading HCI papers, I was able to get a better sense of what I wanted to create for my thesis.
Abstract
Project blog, for a more detailed account of this research project.
Poster
Last spring, the only things I knew were that 1. Wiimotes existed, and 2. They could be connected to the computer. After making sure that it was compatible with Director MX 2004, I decided to spend the summer learning more about it. My major advisor Takis was fully supportive, and introduced me to my co-advisor Orit, who has experience in interaction techniques and tangible user interfaces.
When I say "compatible," I mean that the Wiimote could somehow be connected to Director and work within it. Initial searches turned up just one Director Xtra by Glisferox based on a C++ Wiimote library. The download package included a simple Director application and the Xtra. To use the Wiimote with the computer, I had to buy a Bluetooth dongle and use the Toshiba stack to connect the Wiimote to the computer as an HID device (for more instructions, see Wiimote setup). Once connected, I could run the application from within Director and it would work with the Wiimote.
The first couple of weeks brought an influx of information. I perused ACM articles online to see what research groups were working on, and why they chose the Wiimote. It is used in many fields of research, including HCI, CS, AI, medical imaging, and art. All of these groups acknowledged the Wiimote for its usability, availability, and potential for novel methods of human-computer interaction. I also found many websites and wikis devoted to Wii hacks, Wiimote hardware, C++ libraries, and personal Wiimote projects. This helped me get a sense of what information and programs were available for me to use.
After exploring the web for resources and successfully connecting the Wiimote to Director, I set out to understand the code so that I could start creating my own demos. The demo that was included with Glisferox's package made use of the Wiimote's buttons and accelerometer values. Unable to connect two Wiimotes to the application, I emailed the creator, Jamie, who sent me several new demos that he had been working on. One of these demos showed how to connect 2 Wiimotes to the application, and the other demos included IR motion sensing and Nunchuk support. Since there was no documentation or API for this Xtra, I had to read the code Jamie used and parse it to create my own demos.
Up until this point, I was still set on creating games. While brainstorming for demos, I researched good game programming and physics books online, and ordered a few through InterLibrary Loan. I decided it was a good idea to create a basic demo showcasing the feature I wanted to use first, then program an actual game. For example, my first Director demo extracted the Wiimote's pitch and roll values to create visual feedback of the Wiimote's rotation, and used its accelerometer values as the acceleration for a ball. Using basic physics, I was able to move the ball based on the position of the Wiimote. This demo was meant as practice for creating a Labyrinth game, where the player tilts a board to guide a ball around a maze, avoiding the holes in the board.
My second demo "set" polled for IR values. The practice demo included two points that moved with the Wiimotes, embellished by ripples when the A button was pressed. The game I created to go along with that demo was Pong, which used only the Y-axis IR sensors and A button.
In preparation for the next demo, I did some research on current Wii game controls to see how it was used and what worked best. These interaction methods gave me an idea of the wide range of possibilities within the Wiimote. Meanwhile, I had still been working on the Labyrinth demo when I came to the important realization that I was spending more time doing game programming than making progress with Wiimote applications. I decided that it was not worth it to complete it, because in that time I could create something novel and interesting.
That week, I met with my co-advisors and explained to them this new change of direction. During the exploration phase earlier, I had discovered that the Wiimote was compatible with Flash too, but didn't think much of it because I thought Director's 3D capabilities were superior than Flash's. Orit gave me some great advice on the kind of demos that I could create. The Wiimote was known for its uses in competitive games and applications. What about collaborative ones? Further, instead of games, why not make something that anyone can use, something creative, like an art demo?
The new possibilities from that meeting gave me new inspiration and motivation. After singling out WiiFlash as the best API, I set out to install it on my computer. I use Vista Ultimate, which I found out had issues with the WiiFlash server. After reading error reports, I tried replacing my Toshiba Bluetooth stack with BlueSoleil, which did not solve the problem. Luckily, there was an old laptop in the house that I could use with Windows XP installed, which should be compatible with WiiFlash. Yes, it did work, but, being old, I spent more time waiting for everything to load than get any work done.
My only options left were to dualboot my Vista laptop with XP, or try installing a virtual pc, neither of which I'd done before. I decided dualbooting wasn't worth it for just WiiFlash, so downloaded Microsoft's free Virtual PC 2007 since it was the only one I had heard about. After an agonizingly long day of installing XP and testing all the important applications, it turns out that VPC does not support removable drives, meaning it would not recognize my USB Bluetooth adapter. Well, that wouldn't do. I searched more online and discovered VMWare Workstation, which promised better performance. Sure enough, after some long hours of reinstalling everything, it worked flawlessly.
When I first saw the WiiFlash documentation, it blew me away. The API was clean, easy to understand, and had all the functions that I needed to use. This was clearly better supported and documented than the Director Xtra.
I tested its functionality in my first WiiFlash demo, named WiiPaint. It was a simple drawing application that used two Wiimotes, one that controlled the color and shape of the brush, and one that controlled the drawing. It is collaborative; two people can work together to create a drawing.
Encouraged by the idea of WiiPaint, I created another one the next week, WiiPaint 2. Both Wiimotes were given complete brush functionality, so that two people were able to draw at the same time. To emulate simple gesture recognition, I created a Semaphore demo that recognizes the Wiimotes' positions and translates them to a letter.
During this time, my CS companion, Nick Winter, came to visit. During a meeting with Orit, we talked about adding gesture recognition, something Nick would be able to help me with, having experience working with gesture recognition in Flash. For my last demo, Nick added a recognition algorithm to WiiPaint 3 (also inherited in WiiPaint 2) that could recognize lines, triangles, and squares. I completed the demo by changing one of the brushes into a locking brush. When both Wiimotes are "locked" on to a shape, it becomes scaleable by moving the Wiimotes towards and away from each other.
I did not get to do full Mac testing since I had limited access to a Mac. However, while exploring, I did find a few noteworthy programs to try out.
DarwiinRemote reads in data from the Wiimote and sends back data. It's a simple
application that can emulate the mouse and the Apple Remote.
MoteDaemon connects to Wiimotes using a TCP socket and can link between
Flash/Flex applications. I haven't tried it so I do not know if it's compatible with WiiFlash.
WiiFlashServerJ is compatible with
WiiFlash, written in Java for Mac OS X. I have not tried this program.
There are many possibilities to consider for my thesis project. Over the rest of summer, I hope to gain more familiarity with Papervision 3D so I can gauge the feasibility of creating 3D applications in Flash for my thesis. Possible applicationsks include 3D anatomy visualization, search visualization, origami applications, and interface design.
A huge thank you to my advisors Takis Metaxas and Orit Shaer for supporting me and giving me great advice on this project.
Nick Winter, for helping me with the recognition algorithm and being awesome in general.
Jamie Campbell, for taking an interest in this project and sending me his Director demos.