Viggle open-sources the weights of Remake, a video agent for swapping characters

The company Viggle AI has released Viggle Remake, an agent for reworking video that has already been shot. It replaces every character in a scene, changes objects and the background, restyles the image and adds visual effects in clips up to two minutes long. At the same time the company has made the model weights publicly available, so the model itself is on offer alongside the cloud service.

The minimum price in the cloud is $0.96 for 10 seconds of video at 720p resolution. At that rate a minute costs about $5.76, and a clip of the maximum two-minute length about $11.52. Viggle AI claims this is less than half the price of Higgsfield Genjutsu.

What the agent can do

Judging by the announcement, Remake is built for the typical tasks of reworking footage:

• Replacing every character in the scene. The company promises that the new characters stay consistent across edit cuts. For a dialogue scene cut together from a wide shot and close-ups, that is the key requirement: otherwise the character will change from shot to shot.
• Replacing objects and the background. You can swap out an item in the frame or move the action to a different location.
• Restyling and VFX. A clip can be converted to a different visual style or have effects added to it.
• Refining in conversation. You can keep writing to the agent until a specific shot looks the way you need it to.

The company puts the last point like this:

“Keep chatting with the agent to fix any shot until it is the way it should be”.
— Viggle AI

How this differs from Viggle's earlier tools

Viggle already has separate animation tools aimed at narrow tasks. Mix “blends” the image of one character into a video with motion, Multi does the same for several characters at once, and Motion Control brings a still image to life using the motion from a clip. They run on credits, and one credit covers 15 seconds of animation.

Serial content has already been made with these tools. The creator Mr. WAS, as the Viggle blog recounts, made a five-episode series and used Mix for the most important shots, the close-ups in dialogue scenes. Shots with food and exterior locations he made in Runway Gen 2. That kind of process meant assembling material from different services. Remake, judging by the description, brings the replacement of characters, objects, background and style together in a single tool and accepts a scene up to two minutes long with edit cuts as input.

How fair the comparison with Genjutsu is

Higgsfield Genjutsu tackles a similar task: it rebuilds part of a clip without a reshoot. It has two modes. Motion Transfer takes the motion from a video and builds a new scene around it. Object Swap replaces the specified element and leaves the rest of the frame untouched. As described in the Higgsfield guide, per generation the service accepts a clip of up to 30 seconds and up to 30 reference photos, and delivers the result at resolutions up to 1080p. A prompt is optional; the job can be done with presets. There is also a lighter version, Genjutsu Lite, likewise up to 30 seconds, at 720p or 1080p.

On length Remake wins by a factor of four: two minutes against 30 seconds per generation. Price is trickier. Viggle has quoted a cost only for 720p, while Genjutsu also outputs 1080p. The company has not given Remake's rates for higher resolutions, so the claim of less than half the price of Genjutsu is only worth checking for comparable 720p output.

What is still unknown

The open weights are available via the link in the post, and the cloud version launches here. The announcement lacks what matters most to a studio that wants to run the model in-house: the licence of the weights, the terms of commercial use and the GPU requirements. Nor is it disclosed exactly how the agent holds a character across cuts and where that mechanism breaks down.

In our assessment, these answers decide whom Remake will actually be useful to. Creators of short clips and social media content will probably be fine with the cloud and pay-per-second billing. For post-production studios and VFX teams that cannot send a client's material to an outside service, the open weights are interesting for the possibility of running them locally. Until the licence is known, building the model into a commercial pipeline is risky. Also open is the question of quality on close-ups at 1080p and above, where character-replacement errors are most noticeable.

Why did Viggle open the weights of Remake: pressure from open competitors, a bet on the developer community, a fight for market share?