Research in our labs - COMPENSATION OF GLASS-TO-GLASS LATENESS VIA VIDEO-FLOW EXTRAPOLATION
Have you ever had a neighbor spoiling a goal by shouting at you in front of a soccer match shortly before the images appear on your TV set? This discrepancy between the actual action and its on-screen display is at the heart of the work of the IEMN UMR CNRS 8520 laboratory.
Presentation of this research work led by François-Xavier COUDOUX, teacher-researcher IEMN UMR CNRS 8520 at INSA Hauts-de-France, Mohamed GHARBI, teacher-researcher IEMN UMR CNRS 8520 at INSA Hauts-de-France as well as Patrick CORLAY, teacher-researcher IEMN UMR CNRS 8520 at UPHF and Hind KANJ, PhD student IEMN UMR CNRS 8520.
As part of the ANR ZL-LVC1 research project, its researchers are investigating innovative techniques to reduce glass-to-glass (G2G) latency, which represents the duration of the end-to-end video broadcast chain, from the moment the scene is captured by the camera to its final restitution on the user's screen. In the case of a tele-driving application, all the elements in this chain add a certain delay to the feedback for the remote driver. G2G latency can then have disastrous effects on interaction with the moving, camera-equipped vehicle. Consequently, reducing G2G delay is essential for such tele-driving or tele-presence applications, in order to guarantee real-time interaction with a satisfactory quality of experience. Several solutions are available to reduce each source of latency. To reduce acquisition delay, analog cameras are traditionally used, as they offer low latency due to the absence of buffering and data processing.
.In video coding, certain configurations make it possible to reduce coding latency by avoiding frame reorganization delay. The most common approach is to reduce video bitrate to decrease the amount of data transmitted per frame, and consequently, latency. However, in this case, the reduction in latency is accompanied by a degradation in the quality of the reconstructed video. Indeed, the reduction in data rate causes strong coding artifacts. The ZL-LVC project proposes another original approach based on video extrapolation to reduce G2G delay. Video extrapolation exploits deep learning techniques by extracting deep features from already acquired images to predict future images. If the extrapolation horizon is judiciously chosen, the extrapolated image can be transmitted before the acquired image and then displayed by the recipient, resulting in a drastic reduction in G2G latency.
.1ZL-LVC ANR project website: https://www.iemn.fr/les-collaborations/projets-collaboratifs/anr-zl-lvc - https://zllvc.wp.imt.fr/
Quality-latency trade-off study
IEMN researchers have investigated the quality-latency trade-off by comparing the two methods of G2G latency compensation: the conventional method by encoding rate reduction, and the new method by image extrapolation. They showed that video extrapolation outperforms bitrate reduction in terms of latency compensation, and can achieve zero G2G latency with a satisfactory level of video quality, particularly when transmitting content with low temporal information. Nevertheless, extrapolation delay reduction is a necessary step when low-capacity transmission channels are considered.
At present, extrapolation is a promising technique, but it is still in its initial phase concerning image quality and extrapolation delay. Studies currently underway in the laboratory aim to propose adaptive mechanisms that take into account the variability of video broadcast chain parameters (channel capacity, encoding rate, etc.) in order to offer the best quality-latency compromise for the various use cases targeted.
At present, extrapolation is a promising technique, but it is still in its initial phase with regard to image quality and extrapolation delay.