There is a price to be paid for the multiple benefits of an IT infrastructure that supports media over IP—one not best measured in dollars but in the effort to plan for and deploy an effective way to manage the complex nature of an IT network and endpoint devices, according to Felix Poulin, director, Media Transport Architecture Lab at CBC/Radio-Canada.
“The greater flexibility that media-over-IP technology will provide us comes with a greater complexity compared to SDI,” he said Nov. 11 during the “Continuous Deployment of CBC’s Media-over-IP Data Center” presentation at the SMPTE2020 virtual event.
Poulin, joined by Alexandre Dugas, project leader for Core Systems Engineering at CBC/Radio Canada, discussed the Canadian national public broadcaster’s new IP-based production center in Montreal, which is targeted for completion in 2021, and other planned national deployments.
“…[W]e have each endpoint that requires thousands of parameters to be configured, including IP and multicast address, PTP parameters and media specific configurations,” said Poulin.
Recognizing the tedious, error-prone nature of manually configuring or reconfiguring each piece of equipment on its network, CBC looked for and found in IT data centers the tools and strategies needed to sustain making initial and subsequent configuration changes over the long haul, he said.
CBC’s approach relies on applying “many of the concepts from continuous integration and continuous deployment (CI/CD) and Dev Ops practices that have [a] proven track record of operating [at] data centers throughout their lifecycles,” said Poulin.
Those concepts include scripted provisioning and configuration, updating and changes, performing sanity checks and monitoring. “This is indeed a major shift for broadcast operations,” he said.
THE PROJECT
The new 420,000-square-foot CBC/Radio Canada headquarters in Montreal is dedicated to the broadcaster’s French language network. It operates a continuous news channel with 10 studio floors under the control of four automated control rooms. There are also two TV studios and two multifunction studios with two control rooms, multiple radio studios and post-production facilities, according to Poulin.
The facility’s centralized master control room, which controls all the French-language stations throughout Canada, is designed to handle 20 TV channels and 80 radio stations delivered via OTA and streaming, he added.
CBC/Radio Canada is also preparing to move the machine room of its Toronto headquarters to a new data center and is at the beginning of a technology refresh for all of its facilities in Canada.
The Montreal facility is the broadcaster’s first major project to be designed around IP technology. The architecture is based on JT-NM TR-1001-1 and uses SMPTE ST 2110 (-20, -30 and -40) for transport and a dual spine-leaf COTS network with SMPTE ST 2022-7 redundancy, said Poulin.
IEEE 1588 Precision Time Protocol operating with the SMPTE ST 2059 profile maintains accurate timing. The broadcaster also is using AMWA NMOS, Ember+ and proprietary API protocols for control, he said.
“With this… [size] of project to deliver over the coming years, we believe that investigating and automating the deployment is the only way to achieve… [this] huge task,” said Poulin.
AUTOMATING DEPLOYMENT
Poulin laid out three steps to automating deployment, including proper network configuration, which eliminates the need for separate staging stations and the risk of errors from human manipulation; initial staging of endpoint devices, which centers on programming the minimum set of device parameters to make it possible to reach the devices’ management interfaces; and device configuration, entailing “software programming all of the media-specific parameters to make the endpoint usable and ready to be provisioned in the broadcast control room,” he said.
While Poulin tackled the deployment from a high-level, Dugas laid out the details of the solution.
Several components make up the architecture of the solution. They include the continuous delivery pipeline, an IP address manager (IPAM), a Dynamic Host Configuration Protocol (DHCP) server to automatically provide and assign IP addresses; a firmware repository running on a file server; a network infrastructure physical path (NIPP) generator; and a sanity check pipeline that runs on a daily basis. “All of these components… [revolve] around IT automation tools,” he said.
“The role of the continuous delivery pipeline is to organize sets of tasks into what the broadcaster calls workflows—which are involved in the initial staging process, can play a role in network configuration and “perform function calls in an organized manner with proper metadata that are fetched to the IT automation tool,” said Dugas. “Continuous delivery pipeline tools provide a level of abstraction to the end user that is not necessarily familiar with scripting language.”
The architecture has three “sources of truth” to ensure error-free automated configuration—a schematic database providing a view and clear understanding of the physical network, a source version control tool that regularly saves the state of the IT automation tool and the IPAM, which provides “a source of truth for all of the logical network components,” said Dugas.
At the heart of the automated deployment system is an agentless IT automation tool that translates staging parameters into a series of comments, synchronizes various components of the system and provides current information on endpoints and network infrastructure, he said.
An “idempotency criterion” is the other core principle of the automated deployment system. “We can say a task is idempotent if it provides the exact same results regardless of the initial stage of the IT automation tool and also regardless of the number of times it was executed,” he explained.
Closing the loop on the deployment is the automated sanity check pipeline, which runs after a new device is enrolled and then again once a day to report any differences between the current state of devices as well as the network and the IPAM, NIPP and source version control sources of truth, he said.
LESSONS TO SHARE
Poulin closed out the presentation by offering some of the lessons CBC/Radio Canada has learned during the deployment.
“We realized that media-over-IP devices take five to 10 times as long to test, stage, configure, troubleshoot… [as does the] equivalent equipment based on SDI,” said Poulin. “It became obvious that we needed automaton tools like those found in data centers to… [make deployment and continuous operation] manageable.”
Broadcasters looking to transition to IT should decide on a “single source of truth” for each aspect of the system, including IP and multicast addresses, VLAN assignments, device inventory list and network configuration, before ever beginning a deployment, he said.
“Build your deployment tools before actually building your system,” he said. “That way you can recover your investment by automating the initial deployment and avoid having to reconcile the state of your system with the configuration scripts [which would be necessary] if you waited to automate at a later stage of the project.”
Poulin also advised broadcasters to involve all key stakeholders, including architect, system designer and operation. “Using a continuity of deployment methodology is a major change of practice, and you need to bring everybody on board so they can implement efficiently the new methodologies,” he said.
The director of the CBC/Radio Canada Media Transport Architecture Lab concluded by reminding those attending the presentation that this approach isn’t business as usual for broadcasters.
“This is a major shift for a broadcast media organization,” he said. “But it is a change that is necessary to adopt the IT technology at the core of our media infrastructure and get the expected benefits of flexibility and agility promised by the complex transition.”
SMPTE 2020 content is available on-demand at 2020.smpte.org.
