Analog video performance tests are usually carried out using static waveforms. Signal quality measurements are a reasonably good way to determine the picture quality. This is due to the fact that there is a strong correlation between the shape of the video test waveforms and the perceived picture quality with very few exceptions.
The reason signal quality measurements work with analog and full bandwidth digital systems is that uncompressed systems are linear. Signal quality testing does not work well with the compression/decompression processes. Traditional test signals are easily compressed with little distortion or loss.
Figure 1. Equipment evaluation test setup. Click here to see an enlarged diagram.
Compression-related impairments
Due to the lack of distortion, these signals do not evaluate the compression/decompression process. For example, classical signal-to-noise ratio (SNR) measurements using a flatfield or a ramp are not a reliable measure of compressed picture quality. They are not constant and can give misleading results.
Video compression is a nonlinear process. The picture quality is a function of bit rate, picture complexity and encoding algorithm capabilities. MPEG-2 is essentially a lossy compression method. The compressor analyzes a video image, decides what is important information and discards the unimportant information.
Compression algorithms are very good but not perfect. Sometimes certain areas of the image are considered to be less important than others. If important information is discarded, this becomes evident after decompression. Typically, the background of the image, particularly the darker areas, are affected. These distortions have a noise-like appearance and can be categorized as:
- Edge business: Distortion concentrated at edges of objects, moving artifacts or noise patterns superimposed over objects.
- Mosquito noise: Edge business associated with movements.
- Quantization noise: “Snow” or “salt & pepper” similar to random noise but not uniform across the image.
Figure 2. Bidirectional test setup with loop-through in Paris. Click here to see an enlarged diagram.
None of the standard analog SNR tests are adequate. Compressed video tests compare the picture changes to a reference. The tests are useful only if they have a good correlation with subjective tests. One of the testing methods was developed by Tektronix and uses the PQA200 picture quality analysis system. This system expresses the compress/decompress performance of a system in picture quality rating (PQR) and peak signal to noise ratio (PSNR). The system is based on the Just Noticeable Difference (JND) concept developed by the Sarnoff Research Institute.
The PQR performance measurement compares the reconstructed image present at the output of the system with that present at the input of the system, pixel-by-pixel, and expresses the difference in numbers representing the deterioration perceived by the human vision system (HVS). The comparison is carried out by a computer that uses an algorithm simulating the HVS. The performance levels expressed in PQR on a scale of one to 10 are interpreted as follows:
PQR = 1: The picture degradation is hardly perceptible.
PQR = 3: The picture degradation is slightly visible.
PQR = 10: The picture degradation is highly visible.
Table 1. Test results of the MPEG codec. Click here to see an enlarged diagram.
The PQR measurement can be carried out on the luminance signal (PQRy) or on the luminance and chrominance signals (PQRyc). The PQRy measurements are carried out only on the Y signal to speed the process. The results permit the comparison of various technologies or equipment in most cases. The PQRyc measurements are carried out on the luminance and chrominance components. They last longer and allow a more complete analysis. The result is a single number that characterizes the quality of the reconstructed image.
The PSNR measurements are estimates of the quality of the reconstructed image compared to the original image and are expressed in dB. Acceptable PSNR figures vary between 20dB (acceptable) and 40dB (excellent).
Double-ended systems have access to the pre- and post-compression program material. By comparing the original image and the post-compression image or the level of pre- and post-compression impairment, an indication of picture quality can be gained. This obviously has the limitation of the need to have access to the original signal.
Various video test sequences are available on CD-ROMs. A sequence lasts five seconds, of which two seconds are used for analysis. I have been using three sequences, known respectively as “Diva,” “BBC” and “Mobile with calendar.” These sequences have different image details and movements complexity. “Diva” is the least stressful, and “Mobile with calendar” is the most stressful.
Table 2. Test results of the MPEG codec with the intercontinental loop. Click here to see an enlarged diagram.
It is important to note that manufacturers of MPEG compression and decompression equipment (at least the ones I know of) do not offer performance specifications other than as PAL or NTSC analog composite linear distortions, nonlinear distortion and noise. These performance figures are relevant to the analog composite input-decoder/output-encoder (if any) performance and, therefore, have nothing to do with the MPEG-2 compression/decompression performance.
Some time ago, I was involved in the specification and acceptance tests of a bidirectional (Canada>Europe> Canada) MPEG-2 system with input and output SDI 270Mb/s and a 4:2:2 compressed video bit-rate of 7.3Mb/s. The distribution network was fiber-optics, and the transport mechanism was ATM. I carried out several sets of tests using the three video sequences known as “Diva”, “BBC” and “Mobile with calendar.” I am presenting the results of two tests:
Test #1: A local equipment evaluation consisting of a PQA200 analyzer, an MPEG-2 compressor and an MPEG-2 decompressor. The test setup is shown in Figure 1 and the test results in Table 1.
Typical measurement results
Test #2: A bidirectional (loop-through) test comprising the transatlantic bidirectional ATM transport with a loop-through in Paris. The test setup is shown in Figure 2 and the test results in Table 2.
Comparing the results in Table 1 with those in Table 2, you will notice an almost imperceptible picture degradation. This reinforced the expectation that under normal circumstances, the picture quality is essentially determined at the beginning (compressing) and the end (decompressing) of the transmission link. Given error-free transmission of the data signal, the picture quality remains unchanged over the entire path.
The compressed picture quality affects everyone in the broadcast chain. The most effective approach to assessing the performance is a combination of subjective and objective tests. The test setups and the video sequences used need to be consistent, and the measurement results should be referenced to an industry standard.
Broadcasters still have the problem of ensuring adequate picture quality through the transmission chain. This requires the maintenance of the highest standards, including the interoperability of concatenated compression/decompression systems using different technologies. An often neglected aspect is the statistical multiplexing of several programs, which generates a time-varying quality factor. As you undoubtedly noticed, we are not quite there yet.
Conclusion
Michael Robin, a fellow of the SMPTE and former engineer with the Canadian Broadcasting Corp.'s engineering headquarters, is an independent broadcast consultant located in Montreal, Canada. He is co-author of “Digital Television Fundamentals,” published by McGraw-Hill and translated into Chinese and Japanese.
Send questions and comments to:michael_robin@primediabusiness.com
