Motion tracking

Kinetograph tracks points through a video with Blender's tracker (libmv). You run it once, before rendering. It writes a small JSON file of positions, and your composition reads it with useTrack(). Rendering never tracks, so every render comes out the same.

Track a pointCopy Anchor Link

Find the point in a still of the first frame. You can make one with ffmpeg -i clip.mp4 -frames:v 1 first.png. Then run:

npx kinetograph track public/video/clip.mp4 --point 980,800

This writes public/video/clip.mp4.track.json. Repeat --point for more points. Use name=x,y to name one:

npx kinetograph track public/video/clip.mp4 --point anvil=980,800 --point sign=1690,640

Points are in the video's own pixels. x goes right, y goes down, and 0,0 is the top-left corner.

Use it in a compositionCopy Anchor Link

import { useTrack } from '@kinetographdev/core';

const Label = () => {
  const { x, y, rotation, scale } = useTrack('video/clip.mp4.track.json', 'sign');
  return (
    <div style={{ position: 'absolute', left: x, top: y, transform: `translate(-50%, -50%) rotate(${rotation}deg) scale(${scale})` }}>
      Look here
    </div>
  );
};

Draw <Label /> over your <OffthreadVideo>. useTrack returns:

The clip starts at frame 0 of the Sequence. If the composition and the clip have different frame rates, frames are matched by time. Before the first tracked frame, the first position is used. After the last frame, or after a track is lost, the last position holds.

If you draw the video smaller, larger or moved, apply the same scale and offset to the positions.

OptionsCopy Anchor Link

All defaults are Blender's defaults for a new track.

Find points automaticallyCopy Anchor Link

--detect finds corners with plenty of texture, at least 120 pixels apart, on the first frame tracked. Then it tracks them all, as Blender's Detect Features does.

npx kinetograph track public/video/clip.mp4 --detect

The points are named p1, p2 and so on, after any --point you also give. The defaults find only the strongest few corners. To find more, lower the threshold or the distance.

Pin a picture to a surfaceCopy Anchor Link

A plane track follows a flat surface, such as a screen, a sign or a wall. You make one from four or more tracked points on that surface, as Blender's Create Plane Track does:

npx kinetograph track public/video/clip.mp4 --point 820,540 --point 1250,430 --point 820,360 --point 200,600 \
  --plane screen=p1,p2,p3,p4 --plane-corners screen=700,300,1300,320,1280,700,720,680

--plane-corners sets the plane's corners on the first frame tracked. The order is top-left, top-right, bottom-right, bottom-left, as x,y pairs in video pixels. Without it, the plane starts as the box around the points' patterns. --plane all uses every tracked point.

Pick points that are on the surface and spread out. Do not pick three on one line.

In the composition, usePlaneTrack() gives the corners for the current frame, ready for the cornerPin effect:

import { Image, cornerPin, usePlaneTrack } from '@kinetographdev/core';

const Screen = () => {
  const { pin } = usePlaneTrack('video/clip.mp4.track.json', 'screen');
  return <Image src={staticFile('images/poster.png')} width={1280} height={720} effects={[cornerPin(pin)]} />;
};

The picture's four corners go to the plane's four corners, whatever its size. Draw <Screen /> in a full-frame layer over the video, with the video at its own size.

Steady a shotCopy Anchor Link

There is no stabiliser, but a track can make one. Apply the opposite of the tracked point's motion to the video: translate(-x, -y), rotate(-rotation) and scale(1 / scale), then move the result back to where the point started. Get that start with trackAt(file, 0, 'anchor').

Track with --model locrotscale to undo rotation and zoom too. Wrap the video in a scale(1.1) layer so the frame edges stay out of view. That hides a shake, but a large camera move still shows the edges.

When a track is lostCopy Anchor Link

A track stops where Blender's would. That is when the best match is below the minimum correlation, falls outside the search area, or reaches the frame edge. The command prints the frame, the file lists it in lost, and the position holds from there.

To track further, try a bigger pattern, a bigger search area, a lower --min-correlation, or a point with more texture.

Match with BlenderCopy Anchor Link

Kinetograph drives libmv the way Blender's "Track Markers" does, with Blender's defaults. On the same clip with the same settings, the same frames are tracked and lost. scripts/track-parity/ in the repository runs that comparison.