6. Real-Capture Video

Zero-shot generation from the first frame of 20 real handheld scenes.

Motivation

Reviewer KgbX: "Could the authors clarify whether the method can be trained and evaluated on real-world video data, and discuss the potential challenges?"

Results

Zero-shot on 20 real handheld scenes, from one first frame and the scene's own camera path. The camera path is followed; pixel fidelity is not claimed.

Table 1: Zero-shot generation from real-capture first frames: camera following, and agreement with the real frames over the first 3 s.

20 scenes, first frame only	direction agreement	correlation	PSNR / LPIPS vs real, first 3 s
Memorizon 400 s	0.93	0.79	11.7 / 0.58
Memorizon 200 s	0.90	0.73	11.8 / 0.58
Repeat the first frame	–	–	12.0 / 0.55

Implied FOV is 111–118° against the 90° assumed; cause not identified.

Examples

Left: the real capture along the same camera path. Right: generated from the first frame only (400 s model).

Clip 1

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 2

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 3

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 4

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Show the other 16 scenes

Clip 5

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 6

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 7

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 8

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 9

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 10

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 11

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 12

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 13

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 14

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 15

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 16

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 17

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 18

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 19

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Clip 20

Real capture along the same camera path (4 fps)

Generated from the first frame only (400 s model)

Setup

20 handheld scenes from DL3DV-10K; the model sees the first frame and the scene's own camera path.
Translations rescaled to the training speed; intrinsics fixed at 90° horizontal FOV. Models trained on synthetic video only.

Conclusion

The camera path is followed on real captures (direction agreement 0.93 / 0.90, correlation 0.79 / 0.73 for 400 s / 200 s), without any real-video training.
Pixel fidelity is not claimed: PSNR / LPIPS against the real frames is no better than repeating the first frame.
Real-video training is future work: estimated poses have ambiguous scale and noise, and our retrieval uses pose only.