Lightricks brings highlights and shadows back to 8-bit video
Researchers at Lightricks have demonstrated a way to obtain HDR video from an ordinary 8-bit SDR clip. The generative video model is not retrained for this: its weights and encoder are frozen. Frames are fed in the logarithmic ACEScct encoding, familiar to colorists from film pipelines, and only one small adapter is trained. It rebuilds the details in highlights and shadows that were lost when the footage was recorded in 8 bits.
The method is called LumiVid. The output is a sequence of scene-linear EXR in 16-bit half-float, that is, material intended for color grading and compositing. Exposure can then be changed in post, without regenerating anything.
“One small adapter learns to fill in the highlights and shadows that 8-bit video clips away”.
That is how Lightricks describes the essence of the approach in its technical blog.
Why generative video grades badly
The authors name the problem directly: video diffusion models work in 8-bit SDR. HDR stores the continuous brightness of the scene, and a colorist can bring down the exposure on a blown-out window or pull up a dark corner. In an 8-bit frame there is already a white patch where the window was, and a drop into black in the shadows. There is nothing to pull there. That is why a generated clip cuts poorly against shot HDR material and withstands serious correction poorly.
The authors divide existing ways of adapting models to HDR into two groups. Some change the encoder's latent space: a new representation is trained for HDR, and that is extra complexity and heavy data requirements. Others rely on auxiliary inputs: brightness masks, text prompts, external reference images. According to the researchers, this adds manual work, a precise prompt is hard to find, and the result is less stable and often diverges from the source material. LumiVid makes do with a single SDR clip as input.
How a logarithm replaces a new model
The key observation of the work is this: logarithmic encoding compresses HDR into a distribution that matches well what the video model has already been trained on. The model does not have to master a new data type. A light fine-tuning of the adapter is enough, and the encoder stays as it was. According to the authors, about 300 synthetic clip pairs of 49 frames each, assembled on Poly Haven scenes, were enough for training.
The second part of the method concerns training. To make the model infer the missing details itself, the authors degrade the input SDR frames the way a camera degrades them: they clip contrast, blur highlights and shadows, add MP4 compression artifacts and shift exposure. The model cannot copy pixels and restores the dynamic range by relying on what it already knows about the world. The project page has a comparison of three variants posted: without such augmentation, with blurring only, and the full model.
The same place shows a control experiment that is useful for understanding the method. If you run the material through the decoder alone, without the video model itself, exposure can be raised, but details in the highlights are not restored, and there is no stability between frames. It follows from this that the generative model is what rebuilds the lost information.
What you can check for yourself
The project page collects SDR→HDR results for video and for individual frames. An interactive viewer lets you shift exposure down to −3 EV to check the highlights, and up to +3 EV to look at the shadows. LumiVid is compared there with three methods:
- Tedla et al. following the official recipe: 17 frames at 704×1280 resolution, after which this method holds the last frame while LumiVid continues;
- HDRTVNet++: processes frames one by one, then they are stitched into a video;
- DiffHDR: following the official recipe, with linear EXR in Rec.709 as output.
For display, the LumiVid result is converted from ACEScct EXR to Rec.709 with a 2 EV lift and then to BT.2020 HLG. The source SDR stays in Rec.709. Besides the visual comparison, the authors also give a measurement: on cinematic material from ARRI cameras, LumiVid is ahead of the best existing methods by 5.5 dB on the PU21-PSNR metric. A full description of the method is in the paper on arXiv. Its authors are the Lightricks team, Mohamed Oumoumad of Gear Productions, and also Or Patashnik and Daniel Cohen-Or of Tel Aviv University.
Who will find this useful and what is still missing
Judging by the description, the method is addressed first of all to colorists and compositors who insert generated shots into real material. Right now such a shot often behaves like a stranger in the edit: highlights clip, shadows do not pull. Scene-linear EXR as output will probably allow grading it with the same tools as shot footage. Since an ordinary SDR clip is what goes in, the approach is theoretically applicable to already existing SDR material as well, but the authors do not examine that scenario separately.
In Lightricks products the ground for this is already prepared. The LTX documentation has a section on working with EXR and HDR: the Native HDR mode accepts scene-linear or ACEScct OpenEXR, frames and sequences, and returns HDR with color management so that the material carries on into compositing and finishing. For SDR sources, a separate SDR to HDR scenario with its own models is mentioned there. Whether LumiVid specifically is connected to it is not specified in the published materials.
There are limitations too. The published materials do not state graphics card requirements and processing time, nor information on whether the code of LumiVid itself will be released. Details in blown highlights are rebuilt by the model from learned representations: those data are not in the source frame. The result is a plausible but invented detail, and for VFX work, where the factual fidelity of the frame matters, that is worth taking into account.
Will colorists and VFX supervisors accept a detail invented by a model in the highlights, or will it remain too risky for finishing?