lavarius/ProjectOverlay
1
0
Fork 0
Code Issues 14 Pull requests Projects Releases Packages Wiki Activity Actions 1

feat: Full Body Tracking (FBT) via MediaPipe Pose #39

Merged
lavarius merged 22 commits from feat/full-body-tracking into master 2026-03-22 19:56:30 +00:00
Conversation 0 Commits 22 Files changed 72 +73421 -12510
copilot commented 2026-03-13 23:36:12 +00:00
Collaborator
Copy link

Closes #38

Summary

Adds MediaPipe Pose-based full body tracking, extending the existing face/hand UDP pipeline.

Python server (PythonFaceCaptureServer)

  • tracking_config.py: Added ENABLE_POSE_TRACKING and POSE_TRACKING_INTERVAL toggles
  • combinerTracker_udp.py:
    • Imports PoseLandmarker, PoseLandmarkerOptions, PoseLandmarkerResult
    • POSE_LANDMARK_INDICES dict maps the 12 key joint indices (shoulders, elbows, wrists, hips, knees, ankles)
    • callback_pose_result() callback stores world landmarks
    • process_data() emits pose|world|{idx}|{x}|{y}|{z} lines for each key joint
    • run() creates PoseLandmarker from ./models/pose_landmarker.task
    • Main loop calls pose_landmarker.detect_async() every POSE_TRACKING_INTERVAL frames
    • Watchdog/heartbeat covers pose stall detection

Unity (Assets/_SRC/code/PipeServer/)

FaceTrackingManager.cs

  • New static event Action<Dictionary<int, Vector3>> OnPoseTracking
  • ParsePoseTrackingData(): parses pose|world|idx|x|y|z, converts MediaPipe coords → Unity (negate Z)
  • Fixed OnDetectionReceived guard: pose + hand events now fire even when no face avatars are registered

BodyTrackingClient.cs (new)

  • Per-character MonoBehaviour, subscribes to OnPoseTracking
  • Inspector: trackArms, trackLegs, trackHead, poseScale, smoothSpeed
  • 8 IK effector Transform refs (wrists, elbows, ankles, knees) + hipBone anchor
  • Pose offset computed relative to hip midpoint (landmarks 23 + 24)
  • Lerp-smoothed effector movement in Update()
  • Auto-enables/disables TwoBoneIKConstraint weights on component enable/disable
  • SetTrackingEnabled(bool) public API

Next steps

  • Wire BodyTrackingClient on each model in Character Inside.prefab and assign IK effector refs + hipBone
  • Test with live Python server
Closes #38 ## Summary Adds MediaPipe Pose-based full body tracking, extending the existing face/hand UDP pipeline. ### Python server (`PythonFaceCaptureServer`) - `tracking_config.py`: Added `ENABLE_POSE_TRACKING` and `POSE_TRACKING_INTERVAL` toggles - `combinerTracker_udp.py`: - Imports `PoseLandmarker`, `PoseLandmarkerOptions`, `PoseLandmarkerResult` - `POSE_LANDMARK_INDICES` dict maps the 12 key joint indices (shoulders, elbows, wrists, hips, knees, ankles) - `callback_pose_result()` callback stores world landmarks - `process_data()` emits `pose|world|{idx}|{x}|{y}|{z}` lines for each key joint - `run()` creates PoseLandmarker from `./models/pose_landmarker.task` - Main loop calls `pose_landmarker.detect_async()` every `POSE_TRACKING_INTERVAL` frames - Watchdog/heartbeat covers pose stall detection ### Unity (`Assets/_SRC/code/PipeServer/`) **`FaceTrackingManager.cs`** - New `static event Action<Dictionary<int, Vector3>> OnPoseTracking` - `ParsePoseTrackingData()`: parses `pose|world|idx|x|y|z`, converts MediaPipe coords → Unity (negate Z) - Fixed `OnDetectionReceived` guard: pose + hand events now fire even when no face avatars are registered **`BodyTrackingClient.cs`** (new) - Per-character MonoBehaviour, subscribes to `OnPoseTracking` - Inspector: `trackArms`, `trackLegs`, `trackHead`, `poseScale`, `smoothSpeed` - 8 IK effector Transform refs (wrists, elbows, ankles, knees) + `hipBone` anchor - Pose offset computed relative to hip midpoint (landmarks 23 + 24) - Lerp-smoothed effector movement in `Update()` - Auto-enables/disables `TwoBoneIKConstraint` weights on component enable/disable - `SetTrackingEnabled(bool)` public API ## Next steps - Wire `BodyTrackingClient` on each model in `Character Inside.prefab` and assign IK effector refs + hipBone - Test with live Python server 
FaceTrackingManager.cs:
- Add static OnPoseTracking event (Action<Dictionary<int, Vector3>>)
- Add ParsePoseTrackingData(): parses 'pose|world|idx|x|y|z' lines,
  converts MediaPipe coords to Unity (negate Z)
- Fix OnDetectionReceived guard: only skip face/avatar loop when no
  avatars registered; pose and hand events fire regardless
- Fire OnPoseTracking whenever pose landmarks are present

BodyTrackingClient.cs (new):
- Per-character MonoBehaviour, subscribes to FaceTrackingManager.OnPoseTracking
- Inspector fields: trackArms, trackLegs, trackHead, poseScale, smoothSpeed
- 8 IK effector Transform refs (wrists, elbows, ankles, knees)
- hipBone anchor: pose offsets computed relative to hip midpoint (lm 23+24)
- Lerp-smoothed effector movement each Update()
- Auto-enables/disables TwoBoneIKConstraint weights on Enable/Disable
- SetTrackingEnabled(bool) public API for runtime toggling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add SetupBodyTrackingClients editor script (Tools menu, one-shot setup)
- Run script: adds BodyTrackingClient to Lava_fbx, Saju_fbx, MADISONBASE
- All IK effectors auto-assigned (Left/Right Paw, Elbow, Feet, Knee)
- Hip bone auto-assigned for pose-space anchoring
- BodyTrackingClient starts with trackArms=true, trackLegs=true, trackHead=false

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous FindConstraintAbove walked up the parent chain from the IK
effectors (IK/Arms/Left Paw → IK/Arms → IK → model root), but
TwoBoneIKConstraints live in a sibling branch (RIG/Hands/Lefty etc).
This meant the constraints were never found, their weights stayed at 0,
and the skeleton never responded to the effector positions.

Fix: use GetComponentsInChildren<TwoBoneIKConstraint>(true) on the model
root and match each constraint by data.target == our assigned effector.

Also add:
- debugLog bool with per-second console output (poseOrigin, wrist pos,
  IK weight) to help diagnose future issues
- Remove dead FindConstraintAbove (kept as private method for reference)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Animation Rigging constraints evaluate AFTER the Animator but BEFORE
LateUpdate, so any effector positions set in Update() get overwritten
by animation curves before the IK constraint even reads them.

Fix: On Start(), create proxy GameObjects (_TrackProxy_*) that have no
animation curves. Reroute each TwoBoneIKConstraint's data.target and
data.hint to the corresponding proxy, then call RigBuilder.Build() once
to apply. Our code drives the proxies; when tracking is disabled the
proxies mirror the animator-driven effectors so animation-based poses
continue to work unchanged.

Also improves debugLog output to include hipBone world position to help
diagnose the MADISONBASE 'folding' issue (if hipBone is at world origin
the wrist targets end up near the floor).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
MediaPipe PoseLandmarker world landmark +X = camera right = person's
anatomical LEFT. Unity scene +X = character right. Correct mapping is:
  Unity = new Vector3(-x, y, -z)  (negate both X and Z)
Previously only Z was negated, causing left/right arms to be mirrored.

Also added:
- hipYOffset (float): manual Y offset added to hipBone.position. Use
  this if hipBone world Y is near 0 (wrong bone assigned) but you can't
  immediately fix the Inspector reference. Typical value: 0.9–1.0 for
  a standing character.
- mirrorX (bool): toggle to flip X again in case your character's
  coordinate system differs from the standard setup.
- Startup LogWarning if hipBone.position.y < 0.05 to quickly detect
  wrong hip bone assignment.
- debugLog now shows hipWorld (including hipYOffset), LWrist world
  position, and mirrorX state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Removed: leg tracking, hipBone dependency, hipYOffset, mirrorX
- Arms only: wrists (15/16) + elbows (13/14) as hints
- Calibration replaces hip anchor:
    Press RecalibrateFBT (default key: ') while in idle/rest pose.
    Component records current MediaPipe wrist positions vs animated
    effector world positions and stores a persistent offset.
    charPos = mpPos * poseScale + calibOffset
    Re-calibrate any time tracking drifts.
- Subscribes to InputSystemSpawner.OnPlayerInputReady and wires
    playerInput.actions["RecalibrateFBT"].performed -> Calibrate()
- Before calibration, proxies pass-through the animated effectors
    so animation-based idle pose is preserved at startup

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 'using _src.CODE.CONTROLLERS' and remove CONTROLLERS. prefix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Guard _inputHooked to prevent duplicate action subscriptions
- Retry TryHookInput() each Update frame until input system is ready
  (handles scene load order where InputSystemSpawner initializes late)
- Touch SetupBodyTrackingClients.cs to force Unity reimport of the
  already-correct file (stale compile cache showing old errors)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CustomEditor for BodyTrackingClient shows a 'Calibrate (Runtime)' button
in the Inspector, enabled only in Play mode. Bypasses input system for
easy in-editor calibration testing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Python: send hand|{Left/Right}|{idx}|x|y|z using hand_world_landmarks
  (real metric coords, handedness label included)
- FaceTrackingManager: add ParseHandWrists() extracting wrist (lm 0)
  per hand with correct coord flip (-x,y,-z); fires new static
  OnHandWristTracking(leftWrist, rightWrist) event
- ParseHandTrackingData: updated to handle new format, backward-
  compatible with old hand|globalIdx format
- BodyTrackingClient: fully rewritten to subscribe to OnHandWristTracking
  instead of OnPoseTracking; Calibrate() now uses hand data, no more
  'no pose data received' error; elbows remain pass-through only

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Split Calibrate into CalibrateLeft() and CalibrateRight() so each
hand can be calibrated independently — avoids having to drop a hand
to click a button. Both must be calibrated before tracking activates.
Inspector shows side-by-side LEFT / RIGHT buttons.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove incorrect Y-negation in ArmPoseDriver.OnPoseData.
  MediaPipe Pose WORLD landmarks are Y-UP (origin at hip midpoint,
  +Y toward head). FaceTrackingManager's (-x,y,-z) conversion is
  already correct. Negating Y again made pose-space Y-down while
  bone-space remained Y-up, causing cross products for the arm-plane
  normal to flip sign — producing inconsistent roll.

- Replace hips.forward up-hint with arm-plane normal in both
  BuildAlignMatrices and LateUpdate:
    ArmPlaneNormal = Cross(shoulder→elbow, shoulder→wrist)
  This is always perpendicular to the arm axis (never degenerate for
  real arm poses), physically constrains elbow-bend direction, and is
  computed consistently from the same geometry in both spaces.

- Add [DefaultExecutionOrder(15000)] to ArmPoseDriver and
  HandFingerDriver so writes happen after Animation Rigging RigBuilder
  (~10000-12000) and are not overwritten before rendering.

- Add ArmPoseDiagnostic helper for runtime inspection.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- HandFingerDriver: replace thumb MCP/IP angle with tip→middle-knuckle
  distance ratio (CMC rotation is the primary thumb motion, not joint flex)
- HandFingerDriver: add trackingWeight [0-1] blend with Animation Rigging
- ArmPoseDriver: persistent arm-plane normals (_rUpHint/_lUpHint) with
  Slerp smoothing — eliminates snap-to-hips.forward on straight arm
- ArmPoseDriver: remove dead _smooth*Arm quaternion fields
- ArmPoseDriver: add trackingWeight [0-1] blend with Animation Rigging
- Both: store _rootRotAtBuild and prepend correction = R_now * inv(R_build)
  so tracking follows character facing direction when model is rotated

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add HandIKDriver.cs: IK-target-based arm tracking, runs at exec order 9000
  (before Animation Rigging ~12000) so targets are set before IK solves.
  Drives Left/Right Paw + Elbow targets from MediaPipe pose landmarks.
  Inspector-assigned targets (no auto-find). Lerps back to rest on loss.
  Drives TwoBoneIKConstraint weight 0→1 on detection, 1→0 on loss.

- Fix HandFingerDriver wrist rotation:
  - BuildWristAlign and ApplyWristDirect now use consistent backward dir convention
  - Add leftWristRotOffset / rightWristRotOffset (default Y=180) for residual flip
  - Rest rotation captured in Awake (localRotation, bind pose) — lerps back
    to that when hand data is lost, so offset doesn't stay frozen on screen
  - wristReturnSpeed controls lerp rate back to rest

- Add HandTrackingToggle.cs: listens to ToggleHandTracking InputAction,
  flips trackingWeight (0↔1) on all HandFingerDrivers and trackingEnabled
  on all HandIKDrivers found in children. Auto-finds PlayerInput at runtime.

- Rename Character Inside.prefab → Player.prefab, remove Saji.prefab
- Add ToggleHandTracking action to LavaController.inputactions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
HandIKDriver: snap hand/elbow positions to rest when within 1mm
HandIKDriver: snap constraint weight to target when within 0.01
HandFingerDriver: snap wrist rotation to rest when within 0.5 degrees

Asymptotic exp-decay lerp never fully converges, causing continuous
micro-updates (vibration) every frame at low FPS. Threshold snapping
stops the loop once close enough.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous code had  which caused
lerping to fire only ONCE per received packet (one Unity frame), then freeze.
At 17fps data / 60fps Unity, the face was stepping visually every ~59ms.

Changes:
- Remove early-exit on same frameId — ApplyFromState now runs every Unity frame
- Track _prevPacketValues / _currPacketValues on each new packet arrival
- Estimate running-average _packetInterval for timing
- Extrapolate target = curr + velocity * t (damped 0.6x) for the duration
  of each packet interval — motion continues smoothly between packets
- Clamp extrapolation to 1.5x interval to prevent drift on packet loss
- Running avg packet interval adapts to actual server FPS

Result: 17fps server data appears visually smooth at Unity render rate.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Critical fixes:
- EYE_BLINK (9/10): EyesBlinkLeft/Right → Eyes_Closed.L/R (was silent no-op)
- JAW_OPEN  (25):   JawOpen → Jaw_Open (typo, was silent no-op)
- MOUTH_FROWN (30/31): MouthFrownLeft/Right → Mouth_Frown.L/R (was silent no-op)

New mappings added:
- EYE_SQUINT (19/20): → Eyes_HappyClosed.L/R (was unmapped/broken)
- EYE_WIDE (21/22):   → Eyes_Wide.L/R (was EyeWideLeft/Right, not on mesh)
- BROW_OUTER_UP (4/5): → Eyebrows_Raised.L/R (was BrowUpLeft/Right, not on mesh)
- MOUTH_FUNNEL (32):   → Mouth_Puff.L
- MOUTH_LEFT/RIGHT (33/39): → Lips_Pull.L/R
- MOUTH_PUCKER (38):  → Mouth_Blep
- NOSE_SNEER (50/51): NoseSneerRight/Left (swapped+wrong) → Nose_Scrunch.L/R

Removed 29 duplicate/dead entries (62 → 33 entries, no duplicates)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lavarius merged commit b913561026 into master 2026-03-22 19:56:30 +00:00
lavarius deleted branch feat/full-body-tracking 2026-03-22 19:56:30 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
architecture

Architectural refactor

  
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
performance

Performance improvement

  
quality

Code quality improvement

  
question

More information is needed

  
wontfix

This won't be fixed

No labels
architecture
bug
duplicate
enhancement
help wanted
invalid
performance
quality
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lavarius/ProjectOverlay!39
No description provided.