# Video Canvas

Video Canvas [#video-canvas]

The **Video Canvas** is a mirrored 16:9 webcam surface: a `<video>` element with a **pixel-aligned transparent `<canvas>` overlay** for drawing landmark/skeleton annotations, an absolutely-positioned FPS-counter badge, and a child slot for status/badges. The canvas is kept sized to the video's intrinsic resolution, so the coordinates a tracker returns map 1:1 onto it; both layers share the same mirror transform, so the overlay stays aligned.

It is a **shell, not an acquisition engine** — the consuming app supplies the `stream` (`getUserMedia`) and wires a MediaPipe streaming tracker (`createHandTracker` / `createPoseTracker` / `createFaceTracker` / `createGestureTracker` from `@localmode/mediapipe`) via the experimental [`useStreamingTracker`](https://localmode.dev/docs/react) hook. The trackers own their own video frame loop — the app just draws each `results` batch onto the exposed canvas. This keeps the primitive presentational and free of device-permission logic.

**When to use it:** real-time hand/pose/face tracking, gesture recognition, any live-webcam annotation UI (MediaPipe Studio-style apps).

Preview [#preview]

```tsx
'use client';

import * as React from 'react';
import { VideoCanvas } from '@/components/video-canvas';
import type { VideoCanvasHandle } from '@/components/video-canvas';

/**
 * Demo for the VideoCanvas shell, used by the docs live preview.
 *
 * To keep the preview hardware-free it does NOT request the camera; instead it
 * animates a moving marker on the overlay canvas and a simulated FPS counter,
 * showing the canvas-over-video composition and the FPS badge + child slot. In
 * a real app you pass a `getUserMedia` stream and draw MediaPipe landmarks.
 */
export default function VideoCanvasDemo() {
  const ref = React.useRef<VideoCanvasHandle>(null);
  const [fps, setFps] = React.useState(0);

  React.useEffect(() => {
    let raf = 0;
    let frame = 0;
    let last = performance.now();
    const ctx = ref.current?.canvas?.getContext('2d');
    // Give the canvas a fixed size since there is no video to size it from.
    if (ref.current?.canvas) {
      ref.current.canvas.width = 640;
      ref.current.canvas.height = 360;
    }

    function draw() {
      frame += 1;
      const now = performance.now();
      if (now - last >= 500) {
        setFps((frame * 1000) / (now - last));
        frame = 0;
        last = now;
      }
      const c = ref.current?.canvas;
      const context = c?.getContext('2d');
      if (c && context) {
        context.clearRect(0, 0, c.width, c.height);
        const t = now / 600;
        const x = c.width / 2 + Math.cos(t) * 120;
        const y = c.height / 2 + Math.sin(t * 1.3) * 70;
        context.strokeStyle = '#10b981';
        context.lineWidth = 3;
        context.beginPath();
        context.arc(x, y, 28, 0, Math.PI * 2);
        context.stroke();
        context.fillStyle = '#10b981';
        context.beginPath();
        context.arc(x, y, 5, 0, Math.PI * 2);
        context.fill();
      }
      raf = requestAnimationFrame(draw);
    }
    raf = requestAnimationFrame(draw);
    return () => cancelAnimationFrame(raf);
  }, []);

  return (
    <div className="w-full max-w-md">
      <VideoCanvas ref={ref} fps={fps}>
        <span className="rounded-full bg-emerald-500/90 px-2.5 py-1 text-xs font-medium text-white">
          tracking landmark
        </span>
      </VideoCanvas>
      <p className="mt-2 text-xs text-muted-foreground">
        Preview draws a simulated landmark (no camera). Pass a real{' '}
        <code>getUserMedia</code> stream + MediaPipe tracker in your app.
      </p>
    </div>
  );
}
```

Installation [#installation]

```bash
npx shadcn@latest add @localmode/ui/media-vision/video-canvas
```

Dependencies [#dependencies]

* **Data source:** a presentational shell — you supply the `stream` (`getUserMedia`) and draw each tracker batch onto the exposed `<canvas>`; works with any frame source. Recommended LocalMode producer: the experimental `useStreamingTracker` from `@localmode/react` wired to a `@localmode/mediapipe` `create*Tracker` factory (optional).

* `clsx` + `tailwind-merge` — via the shared `cn()` util (installed automatically as a registry dependency)

Files installed [#files-installed]

* `video-canvas.tsx` — the component
* `lib/utils.ts` — the `cn()` helper (if not already present)

Props [#props]

**VideoCanvas**

| Prop | Type | Default | Description |
| --- | --- | --- | --- |
| `stream` | `object \| null` | — | The webcam `MediaStream` (from `getUserMedia`). When set, it is attached to the `<video>` and playback starts. The app owns acquisition + permissions. |
| `onCanvasReady` | `function` | — | Called once the `<canvas>` 2D context is ready and sized to the video. Use it to grab the context you draw MediaPipe streaming-tracker results onto (e.g. from `useStreamingTracker`'s `onResults`). |
| `fps` | `number` | — | Measured frames-per-second to show in the badge. Hidden when undefined. |
| `mirrored` | `boolean` | `true` | Mirror the video + canvas horizontally (selfie view). Most webcam UIs want this. |
| `hideFps` | `boolean` | `false` | Hide the FPS badge even when `fps` is provided. |
| `children` | `ReactNode` | — | Status/badge content rendered in an absolutely-positioned slot over the video (e.g. a recognized-gesture chip). Not mirrored. |

Backing hooks [#backing-hooks]

Wire the experimental [`useStreamingTracker`](https://localmode.dev/docs/react) hook from `@localmode/react` to a `@localmode/mediapipe` streaming tracker factory (`createHandTracker`, `createPoseTracker`, `createFaceTracker`, `createGestureTracker`) running against the exposed `<video>` (`ref.current.video`). The tracker owns its own frame loop; the hook surfaces `{ status, results, fps, start, stop }` — draw each `results` batch onto the exposed `<canvas>` (`ref.current.canvas`) and feed `fps` to the badge. (The single-shot `useDetectHands` / `useDetectPose` / `useDetectFace` / `useRecognizeGesture` hooks are for still images, not the live loop.)

Examples [#examples]

Wire a webcam stream and tracker [#wire-a-webcam-stream-and-tracker]

```tsx
import { VideoCanvas, type VideoCanvasHandle } from '@/components/video-canvas';
import { useStreamingTracker } from '@localmode/react';
import { createHandTracker } from '@localmode/mediapipe';

export function Example() {
  const ref = useRef<VideoCanvasHandle>(null);
  const [stream, setStream] = useState<MediaStream | null>(null);

  const { results, fps, start } = useStreamingTracker({
    video: () => ref.current?.video ?? null,
    create: ({ video, onResults, onError }) =>
      createHandTracker({ video, onResults, onError }),
    onResults: (hands) => {
      const ctx = ref.current?.canvas?.getContext('2d');
      if (ctx) drawHandLandmarks(ctx, hands); // your draw helper
    },
  });

  useEffect(() => {
    navigator.mediaDevices.getUserMedia({ video: true }).then((s) => {
      setStream(s);
      start(); // tracker owns the frame loop from here
    });
  }, []);

  return (
    <VideoCanvas ref={ref} stream={stream} fps={fps}>
      <span className="rounded-full bg-emerald-500/90 px-2.5 py-1 text-xs text-white">
        ✋ Open palm
      </span>
    </VideoCanvas>
  );
}
```

`useStreamingTracker` is experimental: it creates the tracker lazily on the first `start()`, reuses it across start/stop cycles (the model stays warm), measures `fps` over a one-second window, and disposes the tracker on unmount.

Customization [#customization]

The component owns only the video/canvas composition, the mirror transform, and the FPS badge. Set `mirrored={false}` for non-selfie views, hide the badge with `hideFps`, and restyle the 16:9 shell in the copied `video-canvas.tsx`. You retain full control over acquisition and the draw loop.

> **Real-hardware dependency:** the live preview here uses a simulated marker. A real webcam stream requires `getUserMedia` and a user permission grant in the browser — wire that in your app.