Welcome to ImgToVid AI

or
🎁Get 5 free credits after sign in

SeedWorld: An AI World Model

SeedWorld is a world model. It keeps a learned picture of a scene and predicts how that scene changes when something acts: a step, a camera turn, a grasp, a brake. Public world models are introduced in the same way, from Ha and Schmidhuber’s 2018 World Models and Dreamer to Genie, Cosmos, and V-JEPA. SeedWorld is described here in that language. The next state depends on the action, not only on making the next frame look plausible.

SeedWorld, a world model, illustrated as a drawing becoming a coastal town
A drawing becoming a place. SeedWorld is concerned with keeping that place consistent as the viewpoint and the objects change.

01 — The model

SeedWorld predicts the next state

A world model builds an internal representation of an environment and uses it to predict how that environment changes. The useful input is a scene together with an action. SeedWorld is a model of that kind: the action is part of the question, and the answer is the next state of the scene.

Language models predict the next word. Video generators predict a plausible run of pixels from a prompt. The 2018 World Models paper, and later Dreamer, treat the model as a compact simulator an agent can roll forward in imagination. SeedWorld is introduced on that definition. It is not presented as a clip generator that happens to show motion.

The public literature approaches the same definition from three directions. SeedWorld’s description draws on all three:

  • Latent dynamics. Ha and Schmidhuber’s World Models, PlaNet, and Dreamer learn a small hidden state and a transition in that state. An agent can plan there before acting outside the model. SeedWorld uses this idea of a compact state that can be rolled forward, rather than treating every pixel as the state.
  • Action-conditioned video. DeepMind’s Genie series learns interactive environments from video, so a key or a camera move changes the next frames. NVIDIA Cosmos is built as world foundation models for physical scenes: driving, robots, and industrial settings. SeedWorld’s visible output is read the same way: frames that are supposed to follow the action.
  • A representation, not only pixels. Yann LeCun’s joint-embedding predictive architecture, and Meta’s V-JEPA, predict the next abstract representation of a video instead of reconstructing every pixel. SeedWorld shares that judgment: much of the pixel detail does not decide what happens next.

The test SeedWorld is written against

A convincing video can still ignore the action. A brick wall and a painted wall can look the same if a model only matches textures. Papers on world models use a stricter test, and it is the test used for SeedWorld: if the action changes, does the future change in the corresponding way? Object permanence, contact, gravity, and the layout of a room are part of that test. Looking realistic is not the whole description.

Physical prediction in SeedWorld, illustrated with a glass sphere, water, and depth
SeedWorld’s physical prediction is about mass, contact, and light, not only about a pretty next frame.

02 — What it predicts

Four parts of how SeedWorld is described

These are the questions used to describe other world models. They are the questions used to describe SeedWorld.

A state, latent or visible

Some world models predict the next pixels. Others compress the scene and predict the latent state, which is how Dreamer rolls the future forward. SeedWorld is described as doing both jobs: a state that can be continued, and frames a person can look at. The state is what gets predicted. The frame is one way to see it.

An action that the future depends on

In Genie-style models the action may be a key or a camera pose. In driving and robotics papers it may be a trajectory or a joint command. SeedWorld takes the action as an input in that same sense. The same history with two different actions is two different questions, and the predicted futures are supposed to diverge.

Memory when the view moves

A frame does not store a map. Weaker video models invent a new room when the camera turns away. Exploration-style world models are discussed in terms of object permanence: footprints, an open door, a moved box. SeedWorld is described with that memory. What was behind the camera is part of the state.

Physics learned from data

A game engine writes mass, friction, and collision as rules. Learned world models, including those trained for Cosmos-style physical scenes, pick the regularities up from video, robot logs, or driving data. SeedWorld is in that learned camp. It can cover scenes the rules never listed, and like other learned models it can also invent contact that never happened.

Prediction
The next state of the scene, carried as a latent state and shown as frames
Condition
An action on top of the current observation: a move, a camera change, a trajectory, or a grasp
Rollout
The predicted state fed back in, so the future can be continued for more than one step
What is checked
Whether the action changes the future, whether objects persist, and whether contact stays plausible

03 — The predictions

Predictions SeedWorld is asked to make

SeedWorld spatial memory, illustrated by a snowy path and glass after the viewpoint moves
Turning around is one of the checks. SeedWorld is supposed to still know what was behind the camera.

Weather on a surface

From a dry cobblestone courtyard, the change is “it starts to rain.” A video model can paint streaks on the frame. SeedWorld is also responsible for whether the stones darken, whether water pools in the low spots, and whether light begins to reflect on the wet ground.

An object and a cause

In a workshop, the action is to push a heavy crate off a bench. SeedWorld has to continue the crate’s path, the bounce, and whatever the crate hits. Looking back at the doorway should show the same room. That is the object-permanence check used for other explorable world models.

Three scenes, read as tests

A place and its weather

Scene: A cliffside library with glass doors opening onto a snowy pine forest.

What SeedWorld has to get right: After a walk outside and a turn back, the footprints and the room that was left still agree with the path.

Contact between objects

Scene: A glass table, a ceramic cup, and a pitcher of water.

What SeedWorld has to get right: A push moves the cup rather than the table, and a spill follows the surface.

A street and a vehicle

Scene: A wet city crossing at dusk, with traffic and a delivery robot.

What SeedWorld has to get right: Two different steering actions produce two futures a driver would recognize as different.

04 — The setting

The settings world models are written for

A humanoid robot on a wet crossing, the kind of scene used when world models are discussed for machines
Robot and driving papers use a predicted future so a machine can consider an action before it takes one. SeedWorld is described in that setting.

Scenes a person can look through

  • Interactive environments. Genie-style models turn video into a place that responds to keys or camera motion. SeedWorld’s scenes are described in that register: the next frames are a response, not a finished clip played back.
  • Moving through a layout. Another line of world models starts from one image or a short description and tries to keep the layout stable as a viewpoint moves. SeedWorld is described as keeping that layout in its state.
  • Several agents in one scene. If more than one agent acts, each predicted future has to include the changes the others cause. SeedWorld’s state is shared in that sense: one scene, more than one action.
  • A shot considered before it is built. Filmmakers already block scenes in rough 3D. A learned world model is discussed there when light, weather, and camera moves stay consistent from one continuation to the next. That consistency is part of how SeedWorld is described.

Futures a machine can roll forward

  • Imagination, as in Dreamer. Dreamer updates behavior on predicted rollouts. SeedWorld’s rollout is the same kind of object: a future continued inside the model, before any action outside it.
  • Rare driving scenes. Logged driving data under-represents ice, glare, and someone stepping out. World models are studied as a way to continue those scenes. SeedWorld’s street predictions are read against that use: the motion has to follow the action.
  • Comparing alternatives. Model-predictive control samples many action sequences and rolls each one forward. SeedWorld is the model that would supply those next states. The page does not describe a controller built on top of it.

05 — Alongside other systems

SeedWorld next to other kinds of systems

These systems can all show a moving scene. They answer different questions. SeedWorld is the column that predicts a next state after a specific action.

Hand-built simulatorVideo generatorRepresentation predictorSeedWorld
What it predictsThe next state from written physicsA plausible pixel sequenceThe next embedding of the sceneThe next state of the scene after a specific action
Where the rules come fromEquations and collision codeStatistics of training videoA learned embedding spaceLearned regularities, in the manner of other world models
Does the action change the future?Yes, through the written rulesOnly if the prompt is rewritten and the clip is regeneratedYes, in representation spaceYes. That response is what SeedWorld is for
What can be looked atA rendered 3D sceneA videoUsually nothing visualFrames, and the state those frames come from
How it is discussedGames and robotics sandboxesA finished shotPlanning and video understandingAs a world model: imagination of the next state

06 — One step

How SeedWorld makes one prediction

  1. 01

    Read the current observation

    A frame, a short video, a depth map, or the latent state already carried. SeedWorld treats that as now.

  2. 02

    Take the action as a condition

    The action is an input, as it is for Genie, for driving world models, and for action-conditioned V-JEPA. Two actions from the same observation are two questions.

  3. 03

    Predict the next state

    SeedWorld predicts the next state and can feed that prediction back in. A long continuation is where memory and learned physics either hold or come apart.

07 — Questions

Questions about SeedWorld

What kind of model is SeedWorld?

SeedWorld is a world model. It predicts how a scene changes after an action. That is the definition used for Dreamer, for Genie, and for other action-conditioned models: the next state is conditional on what was just done.

Is SeedWorld a video generator?

No. A video generator samples a clip that looks plausible. SeedWorld is described by whether the future follows the action. A realistic clip with no action in the input is a video. SeedWorld’s prediction includes the action.

How is SeedWorld different from a game engine?

A game engine steps a scene with rules someone wrote: collision, gravity, materials. SeedWorld, like other learned world models, picks those tendencies up from data. Engines are exact inside the rules they have. A learned model can cover scenes nobody scripted, and it can also be wrong.

Why mention V-JEPA and latent states?

V-JEPA predicts a representation of the next moment instead of every pixel, because much of the pixel detail does not change the decision. SeedWorld is described with a latent state for the same reason. The frames are a view of that state, not a substitute for it.

What is a rollout for SeedWorld?

SeedWorld predicts one step, treats that prediction as the new present, and predicts again. Dreamer and model-predictive control use imagined futures of this kind so different actions can be compared. The rollout is part of the model, not a separate product step.

Why would SeedWorld forget a room?

A frame only shows what the camera sees now. Without a memory of objects and layout, the next frame is free to invent a new background. Long-horizon consistency is an open problem for world models in general, and it is one of the checks applied to SeedWorld.

What is SeedWorld not?

SeedWorld is not a hand-written physics engine, and it is not a prompt-to-video model. It is also not described here as something to operate, subscribe to, or connect to another system. The page only says what kind of model it is, using the terms already used for other world models.

SeedWorld is a model of the next state

SeedWorld takes the scene and the action and predicts what the scene becomes. Video generators, game engines, and representation models each answer a nearby question. The question that places SeedWorld with other world models is the action: the future is conditional on it.