Qualify User Guide

Qualify User Documentation

116116

Introduction

Telestream Cloud Qualify is a Cloud-Native, file-based QC service. The service can be used to test a wide variety of media file types for common video, audio and file-based issues. You can interact with the service via our UI, or via our API. Reports can be delivered on screen (with proxy video preview), or as a PDF document for onward consumption.

Projects

Projects are a convenient way to group your Qualify Jobs, as well as specifying key technical details such as the Cloud Provider and Region.

Here’s some information about the different fields you can expect to see when configuring your Project.

  • Project Name – You can give your project a memorable name.

  • Default Template – Choose a Default Template for this Project. This means that any job submitted to this project will, by default, be started with this template. Note: You can always choose the template when starting a Qualify Job.

  • File Upload Store – You can pick which of your Telestream Cloud “Stores” (see the User dropdown menu in the top-right hand corner) to use. This is only relevant if you plan on uploading files from your desktop via the UI.

  • Cloud Provider – Choose which Cloud compute provider you’d like to use for this Project.

  • Cloud Region – Choose which Cloud compute region you’d like to use for this Project.

  • Proxy Video Generation Service – By default, Qualify will use our “Flip” service behind the scenes to create a proxy review video, which you’ll see when viewing a Qualify report. However, if you have our “Transform” service enabled on your account, you can choose to use Transform to create the proxy video and benefit from wider input file support.

Audio Layouts

Audio layouts allow you to choose how Qualify deals with different track layouts in your source media. Use the "Add Next Channel Map" button to specify a new use case for source media with an expected number of tracks and channels. You can use the “Scissors” button to divide audio groups in order to match the expected layout of the source media and choose the expected channel labels from the dropdown list. Add as many channel maps as you require, to ensure Qualify can process your audio correctly. Each Audio group then becomes “Track 1”, “Track 2” etc when building a template in Qualify.

Package Layouts

Package Layouts allow you to choose how Qualify will handle packaged content, such as HLS/Dash. Use the controls to choose which video, audio and subtitle variants will be loaded and tested. This is only relevant to packaged media formats - If you load a non-packaged, monolithic media file (e.g. a standard MOV, MP4, MXF etc) then the Package Layout will be ignored.

Templates

A Template is used to specify what tests you’d like to run against your media. You can choose any combination of tests and use the in-depth settings to tune the thresholds to yield the results you need. It’s worth bearing in mind that the more tests you choose, the longer the test will take to complete. Some tests also command an additional Multiplier.

When you’re creating a template, you’ll also need to be aware of some concepts which are used consistently throughout the product:

  • Add Next Test – This allows you to add multiples of the same test, to test multiple audio tracks simultaneously. If a test is added that references an audio track that does not exist, then an alert will be raised for that test.

  • Test Whole File – This will test the entirety of the file for the artifact in question. You can specify how many seconds/frames are allowed, and if this is exceeded, an alert will be raised.

  • Layout – This gives you extra flexibility in terms of finding artifacts at specific points in the file.

  • For example, you may require that a file has 10s of black at the beginning and 30s of black at the end. You can add 2 Layouts – the first of which tests a fixed duration, starting at 0 and ending at 10, and specifying this “Must Be Black”. Add a second Layout to “Test End Of File” and choose to “Test Last” 30s “With Duration” 30s, and “Must Be Black”. Any deviation from these parameters in the inspected file will raise an alert.

QC Test Reference

The tables below contain a list of all of the tests included with Qualify, and information about what each test does and how it can be configured.

File Checks

CheckIdentifierDescription
File Bit RateFC-101Verifies that the calculated bitrate of the file (video audio bitrate/duration) falls within the defined Min/Max parameters.
DurationFC-102Checks that the duration of the file (video and audio) falls within the defined Min/Max parameters.
Container FormatFC-103Verifies the 'wrapper' or container file used to hold the video and/or audio matches the user defined value. Examples would be 'TS' (MPEG-2 Transport Stream), 'MXF', 'MOV' etc.
SizeFC-104Checks that the total file size falls within the Min/Max parameters.
Container Start TimecodeFC-105Verifies that the Container Start Timecode matches the user specified value, with an adjustable tolerance.
Container End TimecodeFC-106Verifies that the Container End Timecode matches the user specified value, with an adjustable tolerance.
Compare Stream LengthsFC-108Checks whether the audio track is present throughout the duration of the video. An alert is raised if the audio track is shorter than the video track by more than 2 video frames. An alert is also raised if the audio track is found to be significantly longer (more than 2 seconds) than the video.

Deep MXF Tests

CheckIdentifierDescription
Deep MXF TestsMX-101A range of in-depth MXF checks, to ensure technical conformance with a variety of MXF constraints.
MXF Operational PatternMX-102Tests the Operational Pattern of an MXF file. Currently only OP-1a and Op-Atom are supported.
AVC SPS/PPSMX-103This test ensures that every frame of an AVC stream contains an SPS and PPS header. This is mandated by SMPTE RP2027. This test is only available when the 'Video Codec' test is enabled and set to 'H.264' and the 'Container' test is enabled and set to 'MXF'.

Syntax Checks

Enabling syntax checking will test the appropriate codec based on the file provided.

Check

Identifier

Description

MPEG-2 Video Codec Syntax

SC-101

For MPEG-2 streams, section 8.5 (Table 8-13) defines upper limits for bit rate, in Mbps, for each combination of Profile and Level.

For MPEG-2 streams, section 8.5 (Table 8-14) of ISO/IEC 13818-2 defines the vbv_buffer size requirements (in bits) for each combination of Profile and Level.

AVC/H.264/MPEG-4 Video Codec Syntax

SC-102

Tests that the sequence should always start with an IDR picture as per ITU-T H.264. Sequences that start with a non-IDR picture can cause playability problems, including tearing, stuttering and freeze frames.

Tests for a valid version number for VC-3 streams as per section 7.2.1 of SMPTE ST 2019. The valid version numbers are 01(Compression ID 1235, 1237, 1238, 1241, 1242, 1243, 1244, 1250, 1251, 1252, 1253),02(Compression ID 1256, 1258, 1259, 1260),03(Compression ID 1270, 1271, 1272, 1273, 1274).Values 01 and 02 are HD profile.value 03 is RI profile.

Caption Checks

CheckIdentifierDescription
Caption/Subtitle TestCC-101Captions are verified using Telestream's caption decoder to check for deviations from the relevant specifications. For this to work, captions are extracted from either a caption track in the input file or detached from the video essence. C608, C708 and Teletext caption streams are supported and sent to the relevant decoder. If an error is found, it is reported with a video frame index and the channel, service, or page with which the caption is associated. An example of an error might be when too many characters are written to a single row, exceeding the maximum number of characters.
Discrete Subtitle TestCC-102Discrete subtitle data (e.g. sidecar TTML) is parsed and checked for presence and continuity.

Video Metadata

CheckIdentifierDescription
Bit DepthVM-101The number of bits per video sample. For cases where luma and chroma sample depths are different, this check will test the luma bit depth.
Video Bit RateVM-103Check that the bit rate of the video track is within the given range, expressed in Mbit/s.
CodecVM-105Tests the format of the video elementary stream.
Color Space InformationVM-106This tests that the video has certain color space properties (e.g. color primaries, transfer characteristics and matrix coefficients). If a property is not found in the video, a Warning will be generated
Frame RateVM-107Frame rate in video frames per second. "NTSC" denotes a drop frame rate, e.g. "30fps (NTSC)" corresponds to 29.97fps.
Drop FrameVM-108Specify whether the file must have drop/non-drop frame timecode.
Frame SizeVM-109Choose from a selection of standard frame sizes, or specify a Custom frame size (e.g. 2000x1000).
Frame Aspect RatioVM-111Choose from a selection of standard frame aspect ratios, or specify a custom value as an integer or floating point value (e.g. 3 or 1.4).
Pixel Aspect RatioVM-112The aspect ratio of each luminance sample in the coded stream, either as coded directly or implied by the display aspect ratio and frame size.
Chroma Subsampling FormatVM-114Checks that the chroma subsampling format matches the user specified value. Only applicable for YUV-based video.
Clean ApertureVM-115Checks that the Clean Aperture of the video is equal to the Production Aperture. This implies that no cropping of the encoded picture is indicated by the container. This test also verifies that no Clean Aperture offset is indicated. Clean Aperture is a QuickTime concept, thus this test will only be performed on MOV containers.
GOP LengthVM-116Check the distance between successive I-frames in the encoded video. For example, this distance will be 1 for I-frame only video and 12 for a long GOP sequence coded as 'IBBPBBPBBPBBI'. This test is suitable for GOP coded video, such as MPEG-2 and H.264.
Advanced GOP LengthVM-117A more thorough GOP test for MPEG-2 and H.264 that can test for a range of GOP lengths, test for open and closed GOPs (MPEG-2 only), and allow special handling of first and last GOPs. The "First GOP" and "Last 2 GOPs" sections, when ticked, enable specific values to be given for the first and last two GOPs. The "Other GOPs" section will apply to any other GOP (possibly including the first and last GOPs if their specific sections are not ticked). The input boxes for I-frame distance and P-frame distance are comma-separated lists of distances or ranges of distances. For example, "12", "1-12", "1,12,15' and "1,12-15" are all valid inputs. The syntax ">5" and "<5" mean "greater than" and "less than" 5 frames, respectively. If the input is empty, then no check is performed (i.e., any distance is valid). It is possible to choose whether the test operates on the coded frame order or the displayed frame order. This distinction is only important when B-frames are present. The test can be configured to either show unique GOP length errors (default) or to show all of them.
Essence Start TimecodeVM-119Verifies that the Essence Start Timecode matches the user specified input, with an adjustable tolerance.
Essence End TimecodeVM-120Verifies that the Essence End Timecode matches the user specified input, with an adjustable tolerance.

Video Quality

CheckIdentifierDescription
Black FrameVQ-101Checks for instances of both intentional/unintentional black frames, using a luma level specified in the template (default value varies with different video bit depths). The Percentage of Frame option acts as a tolerance control, and the user can set the maximum number of consecutive black frames that are allowed before an alert is raised. This test also has the ability to check certain parts of the file using the "Layout" controls.
Color BarsVQ-102Checks for instances of both intentional/unintentional color bars. The user can specify which type of color bars to search for, and set a tolerance to account for non-standard encoding. This test also has the ability to check certain parts of the file using the "Layout" controls.
Field OrderVQ-103Tests whether the encoded flag matches the user specified input, and optionally tests that the baseband video also has the desired encoded field order. The user can specify a time threshold to ensure relevant alerts are caught.
CadenceVQ-104Checks that the baseband cadence is correct based on the user selected criteria of what is expected in the stream. The user can select from the following expected cadences:

  • Progressive
  • 2:2 (TFF) Top field first interlaced
  • 2:2 (BFF) Bottom field first interlaced
  • 2:3 (TFF) 2:3 cadence mix of progressive and top field first (Prog/Prog/TFF/TFF/Prog)
  • 2:3 (BFF) 2:3 cadence mix of progressive and bottom field first(Prog/Prog/BFF/BFF/Prog)
  • 24 -> 25 fps (TFF) Mix of progressive and top field first used in conversion from 24 fps to 25 fps (12 progressive frames then one TFF)
  • 24 -> 25 fps (BFF) Mix of progressive and bottom field first used in conversion from 24 fps to 25 fps (12 progressive frames then one BFF).

The user can optionally look for broken 2:3 cadence; opt to report all detected cadences or check for poor patterns that could cause excessive stutter.
Freeze FrameVQ-105Checks for instances of both intentional/unintentional freeze frames. The user can specify the maximum number of allowed consecutive frozen frames before an alert is raised. There is also the ability to ignore black frames, using a user specified luma level (default value varies with different bit depths). This test also has the ability to check certain parts of the file using the "Layout" controls.
HDR AreaVQ-106This test measures the number of HDR pixels in a PQ or HLG-1000 graded frame. As per ITU-R B.2408-1 recommendations, a pixel with luminance more than 203 nits is considered HDR. An alert is generated when the percentage of HDR pixels in a frame exceeds the Area Threshold and when this continues for the user specified number of measurements.
HDR Change DetectionVQ-107Checks for changes to the SEI-encoded HDR metadata on a frame-by-frame basis. Any changes to this metadata cause an alert to be raised.
HDR LimitVQ-108This test measures how many HDR pixels lie above the brightest two thresholds, and how many lie below the darkest, giving the user a reliable and useful metric as to how much their content lies within "HDR" boundaries.
HDR MeasurementVQ-109This test measures analyses each decoded frame, in its native bit depth, to calculate the MaxCLL (Maximum Content Light Level) and MaxFALL (Maximum Frame Average Light Level). The measured values are compared against the HDR metadata within the file.
Luma/Chroma LevelsVQ-112This tests that the luma component (Y) of the YUB signal is within valid limits. The Low Luma Limit and High Luma Limit fields allow for stricter or more lenient rules for when a pixel is considered a violation. The defaults of -1% to 103% comply with EBU r103.
RGB GamutVQ-113This test ensures that the YUV components of the decoded video lie within the legal range given by ITU-R- BT.601-5, 709-5 or 2020-1. The default values of -5% to 105% comply with EBU r103.
Media OfflineVQ-114Checks for the presence of the "Media Offline" slate, which can be produced by applications such as Adobe Premiere, After Effects, Avid Media Composer or DaVinci Resolve.
PSE Test (Proprietary)VQ-115Tests for rapid changes in luminance which could trigger photosensitive epilepsy (PSE). This test is in accordance with ITU-R BT 1702-2 (supporting both SDR and HDR content) but can be switched to be run in accordance with ITU-R BT 1702-1 by choosing "Use Legacy". The test will alert if a sequence has more than three and a half harmful flashed per second. Harmful flashes could be caused by both rapid changes of lighting in a scene, or by rapid editorial cuts that cause the screen to flash.
Video Segment DetectionVQ-116Detects video segments within the content and reports the start and end timecode of each segment found. This test works by identifying consecutive black frames, and assuming that anything which is not black is an active video segment. The user can request that digital silence is also present during black sections to ensure accuracy of detection.
Single Color FrameVQ-117Detects sequences where the entire frame is a single color. This can sometimes occur because of syntax errors, missing codecs, or erroneous media encoding. The user can specify.
TekMOSVQ-118TekMOS is a non-referenced perceptual video quality measurement for natural video content. The content is rated on a scale of 0-5, with 5 being the highest quality, in line with the ITU 5-point scale. The Program Threshold computes the percentage of frames within the entire program that fall below the user defined threshold. The short-term threshold searches and reports for short sections of poor video quality, according to a user specified rolling window duration.
Letterbox/PillarboxVQ-120This test is used to detect whether video content has the correct letterboxing (black bars at the top and bottom of the frame). If any content fails the test, then a single error alert is raised for each continuous frame sequence. Changes to letterboxing/pillarboxing are also reported. The user can specify the desired active picture aspect ratio, or a requirement for a specific number of black lines at the edges of the picture. A tolerance can also be set to mitigate false positive alerts, and the black level used to detect the black bars can also be adjusted.
MacroblockingVQ-131Checks for macroblocking artifacts where the macroblock edge is clearly visible. This test is a subjective measure of video quality where each frame is assigned a quality metric and an alert is raised if the video quality drops below the given threshold.

Audio Metadata

CheckIdentifierDescription
Bit DepthAM-101Verifies that the audio bit depth matches the user specified value.
CodecAM-102Checks that the audio codec matches the user specified value(s).
Codec ProfileAM-103Verifies that the audio codec profile matches the user specified value(s).
Sample RateAM-104Ensures that the audio sample rate (KHz) matches the user specified value.
Number of TracksAM-105Checks that the number of audio tracks found matches the user specified value.
Bit RateAM-106Ensures that the audio bit rate (kbps) matches the user specified value.
DialnormAM-107Verifies that the Dialnorm value matches the user specified range. This test is only relevant to assets containing a Dolby encoded audio stream.
Number of ChannelsAM-110Checks that the number of audio channels found matches the user specified range.
Language IDAM-111Checks that the language ID found matches the user specified value.

Audio Quality

CheckIdentifierDescription
Peak LevelAQ-101Measures the Peak audio level throughout the program material, and will alert if it is in violation of the user specified value. The user can choose whether to measure as dBFS or dBTP.
Phase CoherenceAQ-102Measures the phase coherence between channel pairs. Phase is measured on a scale of -1 to 1, where -1 means "completely out of phase" and 1 means "completely in phase/mono".
Dual MonoAQ-103Compares two audio streams and will alert if dual mono sections are detected between them.
Clicks and PopsAQ-104Detects transients or sudden changes in amplitude, such as those caused by noise on a digital or analog signal. The user can choose between low, medium or high sensitivity; high sensitivity is more likely to generate false positive results, but may also detect smaller transients which are deemed erroneous.
ClippingAQ-105Detects audio clipping by testing for successive samples of equal value. Quiet samples (those below a fixed small value) are supressed and do not count towards the clipping test.
Integrated Loudness (inc Dialog Gating)AQ-106Integrated Loudness is measured over the duration of the file. In calculating the loudness, the arrangement of channels is assumed as found in the Audio Layout settings. Dialog gating can be applied, which will measure loudness only when dialog is present.
Short-term LoudnessAQ-107Short-term Loudness is measured over a sliding rectangular time window. The measurement is not gated. The default window length of 3 seconds corresponds to EBU-Tech 3341 short-term mode. The Max Violations setting will report only the worst violation, in accordance with EBU-Tech 3341/EBU r128. The All Violations setting will report all violations.
Momentary LoudnessAQ-108Momentary Loudness is measured over a 400ms sliding rectangular time window as per EBU-Tech 3341 Momentary mode. The measurement is not gated. The Max Violations setting will report only the worst violation, in accordance with EBU-Tech 3341/EBU r128. The All Violations setting will report all violations.
Loudness Range (LUFS)AQ-109Loudness Range is a measurement of the variation in loudness over the duration of a file. The algorithm is specified in EBU-Tech 3342 and produces an output Loudness Range (LRA) in terms of Loudness Units (LU).
Minimum LevelAQ-110Measures the audio RMS level per channel using a sliding window of the user specified length. If the RMS level for a given channel drops below this level for a contiguous period equal to or longer than the window length, an alert will be raised. Due to the nature of the windowing function, if the period is longer than the window length, only the minimum measurement during the period will be given.
SilenceAQ-111Detects both intentional and unintentional Silence. The threshold indicates the threshold below which an audio track is considered silent. If Digital Silence is selected, instead of using the threshold, the track is checked for the presence of absence of audio samples.
Channel PositionAQ-113Checks whether 5.1 channels are in the correct place by using know frequencies to analyze the baseband of the center and LFE channel and validating against what should be present.

Distribution and Broadcast

CheckIdentifierDescription
iTunes ComplianceDC-107Tests several aspects of MOV file atom structure which are relevant for iTunes submission:

  • The video track should have a sample description atom of type 'apch'.
  • The video width and height in the video sample description atom should be either 1920×1080 or 1280×720.
  • The 'vendor' code should be 'appl'.
  • The 'fields' entry should be set to 1 to indicate progressive scan.
  • The 'colr' atom should be present and have the type 'nclc' to indicate non-constant luminance coding.
  • The color primaries, color transfer function and color matrix should all be set to 1.
  • No clean aperture (clap) information should occur after a field (fiel) atom.
  • The edit list (elst atom) for the video track should be exactly 1 entry in size.

Tables

MPEG-2 streams, section 8.5 (Table 8-13)

LevelProfile
SimpleMainSNRSpatialHigh
High80100 all layers 80 mid+base layer 25 base layer
High-14406060 all layers 40 mid+base layer 15 base layer80 all layers 60 mid+base layer 20 base layer
Main151515 both layers 10 base layer20 all layers 15 mid+base layer 4 base layer
Low44 both layers 3 base layer

MPEG-2 streams, section 8.5 (Table 8-14)

LevelLayerProfile
SimpleMainSNRSpatialHigh
HighEnh.2 Enh.1 Base9,781,24812,222,464 9,781,248 3,047,424
High-1440Enh.2 Enh.1 Base7,340,0327,340,032 4,882,432 1,835,0089,781,248 7,340,032 2,441,216

Did this page help you?