8/7/14 Update

Things I did today:

  • Increased the possible number of flower from 200 to 500.  This will allow for more exercises in one session, and a more filled garden.
  • Completed a final demonstration for Sharon.  It went pretty well, but there are still some bugs, especially with the legs, that I sort of think are inherent to the Kinect and our cluttered space.  However, I think she was pleased with the final result.  I certainly was.

Feel free to contact me if you have any questions about what I did that this blog and Sharon can’t answer.  My email address is rmb1952@rit.edu.

Thanks to Adeesha, Scott, Toby, and Sharon for creating this project and making my internship possible.  I wish Sharon the best of luck in pulling all these games together and getting them to work with new hardware and software.  Hopefully, we’ll see this project out there in the not-so-distant future.

7/29/14 Update

Things I did today:

Began research on how to update our program for compatibility with new hardware and software.

There isn’t much information about Unity 5 yet; it won’t come out for at least another few months.  However, it seems like the only way to bridge the gap between the newer versions of Unity and Kinect (as they’ve stopped supporting Zigfu), short of writing everything ourselves, is to use the Kinect Wrapper Package for Unity3D.  Though this may not be as accurate as the way we’ve been doing things, it doesn’t seem like there’s really a choice to make.  Currently, this package is supported through Unity 4.2(ish), so I’d imagine it’ll continue to be updated as new Unity versions are released.  Continuing to monitor this situation would be great, though, as new information about Unity 5 is being released frequently.

As for Xbox One Kinect, it seems like that’ll be an easier transition to make.  As you can see here (http://blogs.unity3d.com/2013/11/05/big-xbox-one-news-for-unity-developers/), Microsoft and Unity have come to an agreement that will allow Unity3D users to develop for the Xbox One Kinect beginning this year.  However, that doesn’t say anything about Unity 5; in fact, nobody says anything about Unity 5 yet.  The newer Kinect is supposed to have more accurate tracking, so maybe, combined with the new SDK, we’ll be able to make something that works well.

In terms of Windows 8, it seems that the new Kinect SDK doesn’t even support Windows 7, so we’ll have to upgrade no matter what to get it working.  That shouldn’t be our major concern when trying to update our games.

Based on what Sharon is thinking of doing this fall, it doesn’t seem like it’ll make sense to update any of our hardware or software just yet.  Working with Zigfu has been the easiest solution, and we need to get something tested with the IC Physical Therapy Department to make sure what we’re doing will even be useful before worrying about upgrading to new software and hardware.

7/24/14 Update

Things I did today:

  • Added a piece of MoveFlowers() that makes sure Unity isn’t looping endlessly through random positions because the garden is completely full (there are no unique positions left).  If 5 positions have already been tried and the randomly generated position is within the bounds of the garden, the position is treated as acceptable and unique, no matter how close it is to other flowers.  This prevents an annoying error where Unity appears to freeze, but is actually just infinitely trying to find a unique position.
  • Added a feature to TranslationLayer that increases the gestureRecognitionThreshold after a failed attempt at a gesture.  This will make it easier for the user to complete the gesture on the next attempt.  The gestureRecognitionThreshold goes back to its normal value when the user completes the gesture or when the program marks the gesture as unsuccessful.
  • Doubled the maximum number of flowers from 100 to 200.  This makes it easier to test the top bullet.

The rest of my summer will be focused on thinking and research concerning where to go with this project in the future.  I’ll only use a bulleted list if I’m changing things about my game.  Otherwise, I’ll write in paragraphs, as this allows for a clearer representation of in-depth thought processes.

Today, I discussed with Sharon a concept suggested during a conversation I had with Bill Chew of West Lafayette, Indiana.  Chew has done some research into determining whether humans are in correct positions using computer algorithms.  He brought to my attention that all joints in a specific posture are not created equal; that is, a shoulder exercise might put great importance on arm positioning, but might not care so much about where the patient’s knees are, provided they are not wildly out of position.  The problem of whether the patient is correctly matching the indicated position becomes much more difficult when one considers these factors.

Currently, my game–and those created before mine at IC–uses an overall difference formula to determine whether or not the user is matching the indicated position.  This means that the differences between corresponding joints of the indicator and avatar models are summed; if that sum is less than an overall gesture recognition threshold, the user has matched the position adequately.  What Chew would suggest (and I agree that this really makes sense) is that each joint be given a weight based on how important it is to the exercise.  If a joint is weighted more highly, it must be closer to the indicated position to make a match; if a joint is weighted less highly, the tolerance for that joint is somewhat higher–it can be a bit farther out of position and still match.  Though this greatly ups the complexity of our checking algorithm, it seems to me that it’s important enough that we’ll at least want to consider it in the future.

One method of implementation would be to let the therapists decide what the weights for each joint should be.  This ensures the most accuracy when assigning the importance of joints, which is excellent.  However, it remains to be seen whether this method would be too costly in terms of time spent by therapists.  It seems to me that if a therapist has to assign weights for each and every posture, it might feel like a time-consuming burden.  However, the therapist certainly knows best which parts of the body are most important in each exercise.

Another idea would be to have a “base” position and to weight the indicator joints based on how far they are from the base.  Joints that are farther from their counterparts in the base model are probably more important, and therefore should be held to a lower tolerance than those that are closer to where they started.  This absolutely doesn’t ensure the same type of accuracy with weights that manual input by a physical therapist would, but I think it would be much simpler to implement.

I’m sure there are other, more complicated ways to do it too, but these are a start.  As I said, I think the issue of “not all joints are created equal” is one that future researchers on this project should definitely think about, as it would add a layer of depth to our research that could be extremely valuable.

The focus of my next few days of research will be how to upgrade our games to run on new systems that are appearing everywhere.  Unity 5 will be released soon, and Zigfu isn’t even supported on Unity 4.  Unless they pick it back up for 5 (which seems unlikely), we’ll have to find a new way of bridging the gap between the computer and Kinect.  Speaking of Kinect, they have a new version as well.  Though you can still buy an Xbox 360 Kinect (as of right now), Microsoft isn’t making any more of that model.  Instead, they’re putting out a whole bunch of Xbox One Kinects.  I’ll be looking at what sorts of skeleton possibilities we have with that new version.  Finally, we’ve tried to get our games to work on Windows 8, but we get an error having to do with differences between 32- and 64-bit versions of the OS.  Currently, everything runs on Windows 7, but I’ll be researching what we’ll have to do to get Windows 8 to cooperate.

7/22/14 Update

Things I did today:

  • Created a complex boundary for flowers.  This way, the flowers won’t be constrained to a rectangular shape.  Instead, they spread out around the models and text in a U-shape.  This makes the garden look more natural; rounded in stead of sharp.
  • Added a bit of fuzz around the edges of the garden.  The program will add a random value between -2 and 2 to the X and Z values of the new flower position.
  • Broke the code in MoveFlowers() up into more methods, bool CheckForAcceptablePosition(float x, float z) and bool CheckForUniquePosition(float x, float z).  These methods determine whether the incoming position is acceptable (within the complex boundary of the garden) and unique (far enough away from all existing flowers).

7/17/14 Update

Things I did today:

  • Recorded Sharon speaking sentences that will be used for feedback.  This includes things like, “Try moving your left leg,” and, “Try moving your head or neck.”
  • Added these sounds to Unity and hooked them up in the TranslationLayer
  • Got rid of an annoying dinging sound that played every time the scene was run.  Moral: be careful when dragging GameObjects.
  • Added a feedback timer that makes sure that more feedback can’t be given for two seconds after feedback is given.  This helps avoid confusion.
  • Raised Carl and Dana with respect to the PoseModelCamera so that feedback messages can be displayed completely underneath them
  • Widened the garden to account for the possibility of more flowers

7/15/14 Update

Things I did today:

  • Created a feedback system in TranslationLayer’s CheckForGesture() function!  Once a gesture is failed, the system calculates which joint is farthest away from where it should be and gives text feedback telling the user which part of their body to move.  Next up will be adding audio feedback.
  • Added a piece of the GrowFlowers() function in GardenGameScript which makes the flowers rotate while they are growing.  This was mainly an exercise to get me familiar with Quaternions, but it looks kind of cool, so I’ll leave it in.
  • Added 50 more flowers, bringing the total to 100

7/3/14 Update

Things I did today:

  • Created a gestureTimer in TranslationLayer that counts down each TickEvent().  When the timer reaches 0, the user is out of time for that gesture, and the indicator moves on to the next one.
  • Removed the whole score aspect from the game.  Score has no place in a garden.
  • Removed a bunch of other unnecessary code.  Hopefully everything is easier to read now.
  • Figured out a really annoying bug, which was causing Unity to stop working without displaying an error message.  It turned out that I had narrowed the area where flowers could be placed and increased the minimum distance between flowers, so there simply wasn’t enough room for more flowers.  Difficult to figure out what was going on, but easy to fix once I did.

7/1/14 Update

Things I did today:

  • Added background music.  I couldn’t get the previously-made Sounds prefab to work, so I just made my own game object and added an audio source to it.
  • Added success and failure sounds.  Same thing as the background music.
  • Replaced all cacti with flowers!  They look nice!
  • Added a piece of the MoveFlowers() function that adds a random Y rotation to each flower when it gets moved to the garden, for variety

Note: The purchased flower Flower_01 needed to be higher in the Y direction than other flowers for some reason, so I created the HigherFlower layer and added Flower_01 to it.  MoveFlowers() and GrowFlowers() both have special cases to check for this layer and move that flower to a higher Y value than the others.  

6/26/14 Update

Things I did today:

  • Got the cacti to move!  Instead of dragging a prefab into the Unity scene, I needed to drag only the mesh.  This is because the prefab came in marked static and couldn’t be moved.
  • Changed values so that cacti only show up in the garden, in front of the indicator and avatar
  • Changed values so that cacti show up underneath the garden, so they can grow up into it
  • Added an ArrayList called growingFlowers and a GrowFlowers() function in gardenGameScript.  Once the user completes a position, flowers are added to the growingFlowers list.  GrowFlowers() is called each Update(), and loops through the growingFlowers list.  If a flower has reached the height of the garden, it is removed from the list.  Otherwise, it grows a small amount.
  • Commented a lot.  Hopefully my code is intelligible now.

6/24/14 Update

Things I did today:

  • Downloaded a bunch of free plant prefabs and decided that the cacti would be best for testing
  • Created an empty FlowerHolder GameObject and attached 20 cacti (6 different models randomly duplicated and distributed)
  • Created some variables that I’ll need in order to move flowers, such as the garden X, Y, and Z min and max positions, the maximum number of flowers that can be moved, and arrays of flower X and Z positions (which will be used to make sure flowers don’t overlap)
  • Created the MoveFlowers(int numberOfFlowers) method.  This will be called from FlowerGained() and will be in charge of moving flowers to the garden.  The parameter is the number of flowers to move.
  • Had trouble getting cacti to move when the user completed a position.  As it stands, some piece of the cactus is moving (maybe its transform?), but you can’t see it if you don’t have the cactus selected before you run the scene.  Maybe this is because each cactus is a prefab?  Not sure yet.  Moving works perfectly if you use a simple cube.
  • Next time, I’ll try to fix this issue and maybe get some flowers, rather than cacti