Figure 1. Picture comparison test setup. Click here to see an enlarged diagram.
The new digital architectures, like MPEG compression, pose a new set of problems and failure modes. The tools developed for the evaluation of analog video are inadequate for the compressed digital world. The multitude of independent factors that contribute to picture quality in the digital and analog domains differ greatly. Among the MPEG variables affecting the picture quality are:
Intra-frame coding impairments
- The data rate.
- The use of I, P and B frames.
- The number and types of frames between I frames.
- The field/frame adaptive prediction.
- The method of motion estimation. and compensation
- The slice size.
- The buffer size.
CCIR Report 1089 proposes a classification of the impairments associated with bit-rate reduction techniques. It is possible to classify the types of impairments associated with intra-frame coding broadly as follows:
Inter-frame coding impairments
- Slope overload: The rise time of the original signal cannot be matched and, therefore, edges are blurred.
- Edge business: The precise continuity of an edge in the original signal cannot be matched and, therefore, the edges appear noisy.
- Contouring: The uniformity or the monotonicity of the original signal cannot be matched and, therefore, a layering effect occurs (contouring effects).
- Granular noise: Finely detailed portions of the picture — for example, those below a threshold level — are not available, and the picture has a noisy appearance as a result.
- Blocking: The underlying block structure appears.
Figure 2. Feature extraction test setup. Click here to see an enlarged diagram.
The following types of impairment may occur due to temporal prediction inaccuracy:
Subjective tests
- Temporal slope overload: Edges of fast-moving objects cannot be matched and, therefore, become blurred during movement.
- Granularity and edge business: Fine-detailed areas in movement exhibit granular noise effects and edge business.The following types of impairment may occur because of temporal subsampling:
- Jerkiness: The smoothness of movement cannot be matched by the compression system, resulting in discontinuities in moving sequences.
- Temporal aliasing: High temporal frequency components are folded back.
- Loss of resolution in moving pictures: Spatial resolution is reduced during movement.
Figure 3. Single-ended test setup. Click here to see an enlarged diagram.
ITU-R BT.500 sets the standard for subjective measurements. It introduces several new concepts detailed below:
Objective test concepts
- Double stimulus impairment scale (DSIS): The observers are shown multiple reference scene/degraded scene pairs. The reference scene is always shown first, followed by the degraded scene. The picture quality is assessed using a five-level impairment grading as follows:
- Very annoying
- Annoying
- Slightly annoying
- Perceptible but not annoying
- Imperceptible
- Double stimulus continuous quality scale (DSCQS): The observers are shown multiple scene pairs with the reference and degraded scenes randomly first. The picture quality is assessed on a continuous quality scale from excellent to bad. Each scene of the pair is assessed separately but with reference to the other scene in the pair. Analysis is based on the difference in rating for each pair rather than the absolute values.
- Single stimulus methods: Multiple scenes are shown separately. There are two approaches: SS: With no repetition of test scenes. SSMR: The test scenes are repeated multiple times. Three different assessment methods are used as follows: Adjectival: The five impairment grading levels are used, but half-grades may also be allowed. Numerical: An 11 grade numerical scale is used. Non-categorical: A continuous scale with no numbers or a large range of 0 to 100 is used.
- Stimulus comparison method: This method uses two well-matched and calibrated monitors. The differences between scene pairs are scored in one of two ways: Adjectival: A seven-grade (+3 to -3 scale) is used. It is labeled as follows: +3 Much better +2 Better +1 Slightly better 0 The same -1 Slightly worse -2 Worse -3 Much worse Non-categorical: A continuous scale with no numbers is used.
- Single stimulus continuous quality evaluation (SSCQE): Instead of separate test scenes, a program is continuously evaluated over a period of 10 to 20 minutes. Data is taken on a continuous scale every few seconds.Subjective tests work well in development laboratories and pre-purchase system evaluations. Among the advantages of subjective tests are:
- Valid tests results are obtained for both conventional and compressed systems.
- A scalar mean opinion score (MOS) is obtained, which works well over a wide range of still and moving picture applications.Subjective tests, however, have several important disadvantages. Such tests:
- require the selection and screening of many observers;
- require knowledgeable and meticulous setup and control of equipment and demonstrations;
- require a wide variety of possible methods;
- are time-consuming;
- do not lend themselves to operational monitoring, production testing or troubleshooting; and
- produce variable results.
Because, in the end, it is the observer's opinion of picture quality that counts, any objective test system must have a good correlation with subjective test results of the same video system and test sequences. Various manufacturers have developed proprietary objective picture-quality analysis (PQA) methods and equipment. The available devices usually compare a signal before compression with the resulting signal after decompression and assign a quality rating similar to the five-level CCIR quality assessment method. There are three agreed-upon objective methods of picture-quality measurements, resulting in three levels of measurement accuracy:
Looking ahead
- Picture comparison method: As shown in Figure 1 on page 12, this approach compares original and degraded source video signals (for instance, SDI 4:2:2@270Mb/s), feeding the input of the system under test (MPEG codec) with the degraded signal obtained at the output of the MPEG codec. This method is accurate because it has complete information about the original and the degraded signal. This obviously requires that the test equipment as well as the codec be available to the test personnel, a situation encountered in development laboratories. This test method could be extrapolated to a test situation where the MPEG encoder and the MPEG decoder are not side-by-side and, indeed, not in the same geographical location. In this hypothetical case, it is obvious that the source signal would have to be made available to the measuring equipment. This would require an additional, separate signal path that would, by necessity, introduce its own impairments. These impairments would affect the measurement reliability.
- Feature extraction method: As shown in Figure 2, this approach extracts a reduced amount of data from the input signal and compares it with a similarly reduced amount of data extracted from the degraded signal obtained at the output of the MPEG codec. The test equipment compares the extracted “features” and generates an impairment measurement result. The input feature data (several hundred bytes) can be delivered from a remote location to the measurement site using a low-bandwidth data channel, thus allowing the MPEG encoder and the MPEG decoder to be in separate locations.
- Single-ended testing method: As shown in Figure 3, this approach analyzes the received signal for known artifacts and other defects resulting from transmission or the encoding process. Among the impairments tested are “blockiness” resulting from the requantizing and variable length coding (VLC) of the discrete cosine transform (DCT) coefficients.
As more compressed digital systems are installed, the need for fast, accurate and reliable subjective and objective test methods will increase. This need is, and will continue to be, met by ever more sophisticated objective test equipment relying on human vision models to evaluate the picture quality.
Michael Robin, a fellow of the SMPTE and former engineer with the Canadian Broadcasting Corp.'s engineering headquarters, is an independent broadcast consultant located in Montreal, Canada. He is co-author of Digital Television Fundamentals, published by McGraw-Hill and translated into Chinese and Japanese.
Send questions and comments to:michael_robin@primediabusiness.com
