To most people, video legalization means ensuring that the levels in a baseband digital video signal are legal — that is, they are within the legal range. For SD video, the analog waveform is represented by 8-bit digital values in the range 0 to 255, either in RGB or YUV/YPrPb color spaces. Depending on the color space, some of these values and combinations of values are outside the range of full black to full white; they are sync signals or over-white, or simply cannot be converted from one color space to another.
As an example, for SD video conforming to BT-601, the value of the Y component of the YUV signal should be within the range of 16 to 235. This is because the values of 0 to 15 are below black or within the range of sync values. Likewise, there are upper limits as well as limits on the U and V components, both in their own values and in combination — the combination values being relevant when conversion to the RGB color space occurs (where specific YUV values can generate values outside the legal RGB color space). Video legalization or auto-correction is where these signal levels are monitored, and if they lie outside the valid/legal ranges, then the values are clipped to ensure they are within the ranges required.
Legalization alters the data values — generally losing detail — and affects the video signal in a way that the content provider did not intend. This aside, there are many reasons why video legalization won't work for file-based video.
In effect, legalizing afterward is a bit like papering over cracks.
Types of errors
File-based video is by definition digital files that store the video and audio. In the majority of cases, the video is compressed in some way (usually the audio is compressed as well), and there is transport stream data (or a transport layer/mechanism) and metadata. There is a large increase in abstraction from the baseband signals, as the video/audio data is compressed and metadata is added — and video legalization occurs only at the lowest level. (See Figure 1.)
Therefore, there are many problems that video/audio legalization does not address. In fact, as file-based video is relatively new compared with the well-understood old analog video signal levels, the vast majority of problems are completely unrelated to video legalization. Therefore, it is vital that any test/checking system can detect these.
Problems that occur in file-based video include:
- Transport stream errors, such as incorrect PIDs, PATs, PMTs and PCRs.
- Multiplexing errors, for example, where the video and audio have been truncated when extracted from a multiple program transport stream.
- Missing required data, for example, when closed captioning or teletext are not present.
- Metadata errors, such as missing copyright information or other data used by an automation system.
- Simple factors, such as incorrect play time. Other examples include when the audio has been put on channels 3 and 4 instead of 1 and 2 (or omitted altogether) or the wrong version of the content has been provided.
- Incorrect bit-rate for the video or audio.
- Incorrect stream set-up, such as when three seconds of audio silence is required at the start but is not present.
- Compliance to various industry de-facto standards, such as CableLabs 1.1 compliance.
- Encoding quality errors where the encoder produces a series of blocky video frames, for example, when there is lots of movement.
- MPEG encoding syntax errors, which can occur due to multiple mux/de-mux operations, or an encoder blip.
- Errors in the syntax of the video and audio elementary streams.
- The stream is correct and legal, but still not what the broadcaster needs. For example, it should be NTSC but is PAL, or it should be 4.5Mb/s peak but goes to a peak of 4.6Mb/s. Typically, a broadcaster will have many such constraints/requirements.
- Errors due to the way the data is split out and put onto a video server. Some servers separate video, audio and metadata, and if there are some errors in these elementary streams or other parts of the data, then this process of splitting up can generate errors. Baseband test systems cannot detect these types of errors, and video legalizers cannot to fix them.
In order to do the testing of the baseband as required for video legalization, the compressed video file must be fully decoded to baseband. If there is then a gamut/legality problem and the video is then legalized, it must also be recompressed to the same video standard (MPEG-2, MPEG-4/AVC, VC-1, etc.) and remultiplexed with the audio and metadata. (See Figure 2.)
However, all the encoding schemes use lossy compression, meaning that some of the quality is always lost. The original compressed file had some loss due to the first encoding, but the content provider would (likely) have done a careful and painstaking quality control to ensure that the picture quality was as required.
An automatic re-encoding as done by a legalizer would add enormously to the compression artifacts. It may well be that artifacts not visible on first encoding become visible on re-encoding after legalization. In addition, there would not be the careful quality control afterward, so the results of the legalization may be video with unacceptable artifacts.
Previous research has indicated a 5dB loss in visual quality from doing a second-generation re-encode.
Testing
In order to do the testing of the baseband as required for video legalization, the compressed video file must be fully decoded to baseband. If there is then a gamut/legality problem and the video is then legalized, the following may be needed:
- The file must be recompressed to the same video standard — MPEG-2, MPEG-4/AVC, VC-1, etc.
- It must keep the same parameters, which are sometimes set manually over a range of frames to get the optimum appearance.
- The compressed video will need to be remultiplexed with the audio and metadata.
- The metadata might need to be updated as well.
This is not easy to do, and there is a great chance that this process will introduce errors. As a result, rather than fixing a minor video legality problem, more serious errors have been introduced.
In addition, typically a content provider or broadcaster would have carefully assessed and chosen specific encoders to be optimal for their requirements. With automatic legalization, this will likely use whatever encoder the legalizer has — whether it is good, bad or indifferent. Plus, the encoder in the legalizer would have to be able to deal well with all the different video standards and be able to remultiplex these seamlessly.
The content provider will have all the correct tools and setup to encode correctly and check the video. It is, therefore, far better for a broadcaster to do a comprehensive check at ingest and go back to the content provider in the event of problems. This then means that the content is resupplied with the visual quality that the content provider intended. Also, reporting the problems back to the content provider may mean that future content is perfectly OK.
Crushing
In SD digital video terms, black is assigned a value of 16, and white is assigned a value of 235 (in 8-bit systems like DVD and DV). Legalizers will clip the video signal at those levels. There will never be a sub-black or over-white signal on a DVD, though the format is capable of carrying the entire 0-255 range. (The dynamic range is limited to these values, but it's not relevant to this point.) The legalizer controls can be driven to ensure that the video signal coming off tape and being color-corrected lies between 16 and 235 and is not crushing. Of course, there's always a margin of error in any kind of process that is controlled by a human operator, but it wouldn't be expected that this crushing would exceed 1 percent, which is negligible.
So, if there was a sequence of video bytes, say a luminance ramp from black to white, which was coming in as 16, 17, 18, 19, 20, 21, 22, 23 … 233, 234, 235, and the lift control was turned down so these values became 13, 14, 15, 16, 17, 18, 19, 20 … 230, 231, 232, then at the output of the legalizer, the signal would be 16, 16, 16, 16, 17, 18, 19, 20 … 230, 231, 232.
Thus, some original detail has been clipped off or crushed out and could never subsequently be recovered. If the lift control is later turned back up on this modified signal, the sequence would be 19, 19, 19, 19, 20, 21, 22, 23 … 233, 234, 235. Most of the picture would be returned to its original value, but the blacks would now be raised up, and black would be a dark gray; the original near-black detail is gone forever.
Conclusion
Video legalization can have a role in quality checking of file-based video, but this method only deals with a small subset of the errors that can occur with the content. There are two key points about video legalization of file-based video:
- Although the color gamut can be corrected, legalizing the video can degrade video quality badly and can result in a file that has been re-encoded in a way that was not intended.
- The video can be legally compliant. It can have the correct gamut but still have incorrect syntax, which can cause the set-top box to crash.
The most effective way to check the health of file-based content prior to transmission is by checking that the syntax of the file is correct. It is useless to check gamut if the syntax is not correct, so syntax must be the first check.
File-based video generally comprises one or more complex digital files with many elements, all of which must be correctly decoded for the file to play. A large proportion of file-based video has some syntax errors, so it's important to look for tools that can automatically check for correct syntax, enabling you to find the errors before you get complaints that the end-consumer's set-top box has crashed.
Thomas Dove is senior manager, compressed video, Tektronix.
