Claire

Link to the sketch: https://editor.p5js.org/Claire_Z/sketches/8xqKf7_FS

This is a project which you can interact with a little creature with your hand.

Inspiration

I build this project on top of my last week project, which to pick and drag the bug with hand pose. The figure of the little bug is inspired by the coal spirit in the film Spirited Away. It’s very cute and I always want to pet one. This idea reminds me of the desktop pet, which are some figures Then I come up with this little creature lived inside the screen. It will wandering on canvas, and you can pick it up and drag and drop it to elsewhere. It can also walk on you head and shoulder.

Process

• Put the little creature into the canvas and move it:

  I used random to make the creature wandering on the screen. In the very beginning, I randomly let the x position + or - 1. Instead of walking, the creature is waggling. In this final version, I separately randomize the direction and distance. And update the variables until it moves to the destination.

  if(!dragging){
        creature = image(walk, preX, preY, 40, 40)
  
        preX = constrain(preX, 0, width-40); 
        preY = constrain(preY, 0, height-40);
        
        // get the position of the bottom center point
        anchorX = preX
        anchorY = preY + 20
  
        ifOnGround(anchorX, anchorY)
  
        // control the movement direction and disntance on x axis
        // to make sure that the pet moved some distance before turn around
        if(movedDistance >= targetDistance){
          targetDistance = random(20, 50)
          direction = random([-1, 1])
          movedDistance = 0
        }
  
        // if touch the ground, then only move horizontally
        if(touchedGround){
          preX += direction * stepWidth
          movedDistance += stepWidth
        }
        else{
          // if not on the ground, then falling down
          accelerate += 0.1
          preY += accelerate 
        }
      }
  

  function ifOnGround(x, y){
    pixColor = segmentation.mask.get(x, y)
    
    if(pixColor[3] == 0){
      touchedGround = false
    }
    else if (pixColor[3] != 0 || preY >= height-40){
      touchedGround = true
    }
  }
  

• Pinch and drag with hands:

  Detect the top point of thumb and index finger. When the index finger point is in the range of the gif image, it’s classified as touched with the pet. Then I calculate the distance between two points. When the distance is less than a certain number, the finger is touched. And when the hand pose satisfied the both condition, you can pick up the bug, and drag it. When you dragging the bug, it’s x and y positions are the position of the top point on index finger.

  if(hands.length > 0){
        thumbTip = hands[0].keypoints[4]
        indexTip = hands[0].keypoints[8]
  
        ifTouched()
  
        fingerDistance = dist(thumbTip.x, thumbTip.y, indexTip.x, indexTip.y)
  
        if(fingerDistance < 30 && fingerTouched){
          dragging = true
          creature = image(picked, preX, preY, 40, 40)
          preX = indexTip.x
          preY = indexTip.y
        }
        else{
          dragging = false
        }
      }
  

  function gotHands(results) {
    // hand pose results
    hands = results;
  }
  
  function ifTouched(){
    if (indexTip && preX-15 < indexTip.x && indexTip.x < preX+15 && preY-15 < indexTip.y && indexTip.y < preY+15){
      fingerTouched = true
    }
    else{
      fingerTouched = false
    }
  }
  

• Body segmentation and find the ground:

  I want to create the effect of the pet walking on your body, like head or shoulder. Iuse bodySegmentation to mask the background. Then get the pixel from the bodySegmentation.mask. The alpha values are different to tell apart the background and the person (0 and 255). I use the bottom middle point of the gif image of the creature as the anchor point. When it’s on the pixel with the alpha value of 0, it’ll falls down (y position increase), until the alpha value is not 0. Not equals to 0 means that it touches the boundary of person.

  function gotResults(result) {
    // body segmentation results
    segmentation = result;
  }
  
  function ifOnGround(x, y){
    // getting the pixel color of the segmentation mask at the current position
    pixColor = segmentation.mask.get(x, y)
    
    if(pixColor[3] == 0){
      touchedGround = false
    }
    else if (pixColor[3] != 0 || preY >= height-40){
      touchedGround = true
    }
  }
  

  // if touch the ground, then only move horizontally
        if(touchedGround){
          preX += direction * stepWidth
          movedDistance += stepWidth
        }
        else{
          // if not on the ground, then falling down
          accelerate += 0.1
          preY += accelerate 
        }
  

Code references

I use the handPose and bodySegmentation code examples on the class page. Also thanks to Daniel helped me with detecting the boundary of the person.

Next Step

I’d like to refine the interface. I want to make it more like an accomplished digital pet app. In the next step, I want to add the feature of changing background, and choose different pets or upload your unique one. Also, the sketch becomes laggy when running both models at the same time. It’ll be better if there are some ways to make the program runs smooth.