A fine-grain scalable video data apparatus (fig. 2), system, method and data structure is disclosed. An encoder (110) for encoding input video data (112) as minimum bitrate macroblock data to produce DCT data having DCT coefficients representing a minimum bitrate version of the macroblock data. The encoder (110) also encodes the input video data (112) as intermediate bitrate macroblock data to produce DCT data having DCT coefficients representing an intermediate bitrate version of the macroblock data. An adaptive motion compensator (132) communicates with the encoder (110) for predicting whether a decoded version of the intermediate bitrate macroblock data has an accumulated predicted error frame energy exceeding a maximum threshold (228).
