Extracting Motion Annotations from MPEG-2 Compressed Video for HDTV Content Management Applications

Broadcast of digital television (commonly referred to as high definition television) was launched in the US in November 1998 and is expected to phase out analog television transmission by the year 2006. The ATSC digital television standard (A/53) that has been adopted for transmission of high definition (HD) and standard definition (SD) digital television by terrestrial broadcasters utilizes the ISO/IEC MPEG-2 standard for video coding and broadcast. Our research concentrates on developing a novel HDTV video content management system that enables end users to search, retrieve, and browse archived SD and HD television material for program production, content repurposing, and other interactive services in a digital television studio. The core of a video content management system is the technology to automatically annotate videos based on available metadata about the programs and on the video content itself such as color, motion, and objects in video such that the archived material will be easily queryable and retrievable. Since video compression for digital television is based on the main profile syntax of the MPEG-2 standard, it is critically important that all content management systems that are to be deployed in digital studios readily annotate, search, and browse MPEG-2 encoded (HDTV) video streams without decompressing them.

We have developed the first system to date that automatically analyses motion occurring in MPEG-2 coded video within the compressed domain itself, and produces video descriptors characterising the global visual motion, for content-based retrieval applications. In this paper, we describe our robust and efficient scheme for automatically generating a flow characterization of a video bitstream which is a frame-type-independent uniform motion representation amenable for consistent interpretation and computed from the raw motion vectors encoded in the MPEG-2 bitstreams. In particular, we propose novel techniques to handle all the different prediction modes that are employed with different frame types and picture structures of MPEG-2 during the motion compensation process to deal with interlaced and progressive source video. We present new and efficient algorithms for generating reliable flow vectors that can be analyzed easily to result in both quantitative and qualitative annotations of video motion useful in content-based retrieval. Experiments with thousands of frames from SD and HD video streams demonstrate the accuracy of our flow estimation process and the effectiveness of utilising flow vectors for global motion annotation of video streams.