feat: Full Body Tracking (FBT) via MediaPipe Pose #38

New issue

Closed

opened 2026-03-13 23:31:52 +00:00 by copilot · 0 comments

copilot commented

2026-03-13 23:31:52 +00:00

Collaborator

Overview

Extend the existing MediaPipe face capture pipeline to support Full Body Tracking (FBT) using MediaPipe's PoseLandmarker. Characters already have the RIG + IK effector hierarchy set up (IK/Arms, IK/Legs) — this issue wires it all up.

Architecture

Python Server (`PythonFaceCaptureServer`)

Add PoseLandmarker (model already present: models/pose_landmarker.task)
Add ENABLE_POSE_TRACKING toggle to tracking_config.py
Send world-space pose landmarks as pose|world|{idx}|{x}|{y}|{z} over existing UDP channel

Unity Client

FaceTrackingManager: add pose| message parsing + OnPoseTracking broadcast event
New BodyTrackingClient.cs: MonoBehaviour added to each character model that:
- Subscribes to pose data
- Drives the IK effectors (IK/Arms, IK/Legs) using key landmarks
- Enables/disables TwoBoneIK weights automatically when signal is present/lost
- Has a trackHead toggle (default off — face capture already handles head)
- Smooths positions with configurable lerp speed

Landmark → IK Mapping

IK Effector	Pose Landmark
Left Paw (hand target)	15 — Left wrist
Right Paw	16 — Right wrist
Left Elbow (hint)	13 — Left elbow
Right Elbow	14 — Right elbow
Left Feet (foot target)	27 — Left ankle
Right Feet	28 — Right ankle
Left Knee (hint)	25 — Left knee
Right Knee	26 — Right knee

Head Overlap

Face capture already drives head/neck/chest via MultiAimConstraint → HeadTracker. FBT will not override head by default (pose head data skipped). A trackHead inspector bool allows opting in if face tracking is disabled.

Files Changed

PythonFaceCaptureServer/tracking_config.py — add pose toggle
PythonFaceCaptureServer/combinerTracker_udp.py — add PoseLandmarker + pose UDP messages
Assets/_SRC/code/PipeServer/FaceTrackingManager.cs — add pose parsing + event
Assets/_SRC/code/PipeServer/BodyTrackingClient.cs — new per-character FBT component
Assets/_SRC/Prefabs/Character Inside.prefab — add BodyTrackingClient to each model

## Overview Extend the existing MediaPipe face capture pipeline to support **Full Body Tracking (FBT)** using MediaPipe's PoseLandmarker. Characters already have the RIG + IK effector hierarchy set up (IK/Arms, IK/Legs) — this issue wires it all up. ## Architecture ### Python Server (`PythonFaceCaptureServer`) - Add `PoseLandmarker` (model already present: `models/pose_landmarker.task`) - Add `ENABLE_POSE_TRACKING` toggle to `tracking_config.py` - Send world-space pose landmarks as `pose|world|{idx}|{x}|{y}|{z}` over existing UDP channel ### Unity Client - **`FaceTrackingManager`**: add `pose|` message parsing + `OnPoseTracking` broadcast event - **New `BodyTrackingClient.cs`**: MonoBehaviour added to each character model that: - Subscribes to pose data - Drives the IK effectors (`IK/Arms`, `IK/Legs`) using key landmarks - Enables/disables TwoBoneIK weights automatically when signal is present/lost - Has a `trackHead` toggle (default off — face capture already handles head) - Smooths positions with configurable lerp speed ## Landmark → IK Mapping | IK Effector | Pose Landmark | |---|---| | Left Paw (hand target) | 15 — Left wrist | | Right Paw | 16 — Right wrist | | Left Elbow (hint) | 13 — Left elbow | | Right Elbow | 14 — Right elbow | | Left Feet (foot target) | 27 — Left ankle | | Right Feet | 28 — Right ankle | | Left Knee (hint) | 25 — Left knee | | Right Knee | 26 — Right knee | ## Head Overlap Face capture already drives head/neck/chest via `MultiAimConstraint → HeadTracker`. FBT will **not** override head by default (pose head data skipped). A `trackHead` inspector bool allows opting in if face tracking is disabled. ## Files Changed - `PythonFaceCaptureServer/tracking_config.py` — add pose toggle - `PythonFaceCaptureServer/combinerTracker_udp.py` — add PoseLandmarker + pose UDP messages - `Assets/_SRC/code/PipeServer/FaceTrackingManager.cs` — add pose parsing + event - `Assets/_SRC/code/PipeServer/BodyTrackingClient.cs` — **new** per-character FBT component - `Assets/_SRC/Prefabs/Character Inside.prefab` — add BodyTrackingClient to each model