EdgeRelight360: Text-Conditioned 360-Degree HDR Image Generation for Real-Time On-Device Video Portrait Relighting

In this paper, we present EdgeRelight360, an approach for real-time videoportrait relighting on mobile devices, utilizing text-conditioned generation of360-degree high dynamic range image (HDRI) maps. Our method proposes adiffusion-based text-to-360-degree image generation in the HDR domain, takingadvantage of the HDR10 standard. This technique facilitates the generation ofhigh-quality, realistic lighting conditions from textual descriptions, offeringflexibility and control in portrait video relighting task. Unlike the previousrelighting frameworks, our proposed system performs video relighting directlyon-device, enabling real-time inference with real 360-degree HDRI maps. Thison-device processing ensures both privacy and guarantees low runtime, providingan immediate response to changes in lighting conditions or user inputs. Ourapproach paves the way for new possibilities in real-time video applications,including video conferencing, gaming, and augmented reality, by allowingdynamic, text-based control of lighting conditions.