Keeping the Show going

An overview of DirectOut’s aids to resilience in a live environment

Stuff happens: we all know that. And sometimes bad stuff happens at the worst possible time. Live events are very unforgiving of technical failures, and many audio engineers will regale you with their favourite, personal, horror story of the day when everything suddenly went very, very quiet……

Happily, DirectOut is run by people who know all about what can go wrong at a big festival, an outside broadcast of a global sporting event, or a major artist’s one-off performance that’s being recorded for posterity.

Here’s a summary of how practical features have been designed into DirectOut products to prevent such an occasion turning into your “worst day”.

Input Managers

Duplication of vital audio feeds via different physical routes is an obvious and well-established method of insuring against accidental cable damage or inadvertent disconnection at some remote point. Using different audio formats for the separate feeds gives further protection as it removes total reliance on the infrastructure behind any one format.

DirectOut has adopted the concept of Input Managers, which allow a priority-based hierarchal “stack” of alternative inputs to be created for each managed channel. An Input Manager constantly monitors all the nominated sources for each input and compares them. In the event of what appears to be a loss of signal integrity in the currently selected input, switchover to a backup is only performed if the next backup signal in the stack is declared “safe” and usable. In this way, the selection process works down the stack, which may be defined with up to six sources. Depending on the DirectOut product type, the sources could be in AES3, MADI, Dante, Ravenna/AES67, Soundgrid or analogue formats.

Input Managers are intelligent, monitoring both the active channel and the nominated alternatives for silence detection or signal coherence/difference to check the continued validity of the inputs. The operator can define exactly how degraded the signal needs to be to initiate switchover: parameters available include the minimum threshold for activation of changeover, the degree of permitted variance between the current source and the next in the stack and times for both changeover and reselection of the original signal, should it become available again.

Input switching is entirely automatic, instant and generally inaudible. The change of source is indicated in globcon.


EARSTM

Enhanced Automatic Redundancy Switching (EARS) uses a BLDS (Buffer Loop Detection System) signal on one channel of a multichannel source to provide seamless changeover in the event of failure of the primary source. EARS can be applied to MADI, Dante, AES67/Ravenna and Soundgrid sources: importantly, primary and backup sources do not need to use the same format.

EARS monitors the primary data stream, and in the event of a data discontinuity, switchover from primary to secondary source is performed within 1 sample. Provided the programme material from both sources is identical and externally synchronised, changeover is inaudible.

A typical application is with dual redundant multitrack playback systems (typically DAWs), as used in many modern live music shows. One track on each DAW carries the BLDS track, which is either generated by the DirectOut device itself or by using a simple Windows or Mac application.


FastSRC

SRC (Sample Rate Conversion) is needed when dealing with multiple digital sources that are not using exactly the same sample rate. In some situations, common unanticipated problems are clock accuracy and stability (drift) or sync waveform distortion; also of course, the simple human error where a device on the system is set to the wrong sample rate. All these will cause intermittent audible glitches. FastSRC, which comes as standard on certain DirectOut devices, is an automatic, low-latency “fixer” solution to this problem, and allows system set-up to proceed rapidly, allowing for further investigation and correction at a later time. For situations such as a major live recording, when it is necessary to decouple the sample rates of two systems while retaining the very best audio performance, DirectOut’s reference quality HD-SRC (see below) is the ideal choice.


HD-SRC: Reference quality SRC

While FastSRC (see above) is a great tool for rapidly resolving any sample rate problems that might crop up during system setup, superior audio performance is delivered by DirectOut’s HD-SRC designs. HD-SRC can be enabled permanently in critical signal paths to provide high quality audio with a very low noise floor.

HD-SRC will effectively decouple two systems running at different or asynchronous sample rates without compromising audio quality. It also provides the ability to bridge between two systems running at different sample rates (typically 48 kHz and 96 kHz): although 96 kHz has become the standard sample rate for high-end live sound mixing, some digital audio devices are not 96 kHz-capable.

The current implementation of HD-SRC is FPGA-based and is available optionally for all PRODIGY AoIP I/O modules: Dante, Ravenna and Waves Soundgrid; and as standard on the 128ch dual-MADI MADI2.SFP.IO


Mirror Mode

As the name suggests, this allows two DirectOut PRODIGY units with the same specification to be operated in tandem, with all – or maybe just some – of the settings from the “primary” being automatically copied to the second.

From a resilience viewpoint, this gives you protection against the failure or sudden unavailability of the primary unit from power failure or major signal loss. It means that the second unit is always “ready to go”, as it will be configured identically to the primary, right up to the moment of primary failure. While Mirror Mode is enabled, any change made to the primary (or just some changes, read on) such as routings, level settings, mutes, etc., will be replicated in the second unit. Downtime is therefore minimised, as all that needs to be done is to reroute the audio inputs and outputs from one unit to the other.

It’s unlikely that you would want every single unit setting to be copied to the backup unit: in this case, Mirror Mode can be set to only copy a defined subset of the primary’s settings. The engineer can choose which settings and parameters will be copied to the backup using Mirror Mode Filter. For example, it probably would not be desirable to copy the primary’s IP address to the backup. But you might only want to copy Output Mute statuses, in which case a single Mute command at the primary will be mirrored in the backup.

An important aspect of Mirror Mode is that it is entirely self-contained within each of the two units and does not rely on any external computers or other equipment.


Sync Priority

All digital audio devices must have a reliable sync source for glitch-free operation. The normal choice for a device with a single, constant audio source will be to derive its clock from the input signal itself, but for a unit such as a router or format converter, the decision is less straightforward, as signals of varying formats and/or sample rates may be in use, and the audio system itself is likely to be much more complex.

The Sync Priority feature in DirectOut devices provides a convenient system of sync management. It gives the engineer a simple method of specifying a preferred set of sync sources, in priority order. The unit will default to the highest priority sync source, until a problem, such as signal loss or excessive jitter, is detected. The unit’s sync selection then switches automatically to the next in the priority list. Sync Priority may also be set to enable continued monitoring of the higher-priority sync source after switchover, switching back to it if the original problem appears to be resolved, after a temporary disconnection, or a device being power-cycled, for example.


PSU redundancy

All rackmounting DirectOut devices have dual, internal power supply modules. Both modules are always active, so that there is no interruption in operation should one fail or lose its AC supply. To enhance resilience further, each module has its own AC power inlet socket, mains switch and fuse, allowing each module to be powered from independent AC circuits where available: supplying one module via a UPS will clearly provide additional protection.

All modules are over-rated for the unit they are fitted to, to ensure that they remain cool, thus extending their operational life.

Smaller DirectOut devices, such as the EXBOX range, have dual, locking DC power input connectors. An external AC PSU is supplied as standard; the second one may be ordered separately if required. PoE (Power-over-Ethernet) is available on some units, which may be used for additional power resilience.


MADI redundancy

Most DirectOut products with MADI capability may be fitted with two independent MADI ports. The “second” port can either be configured for doubling the MADI channel count, or for redundant operation. Units can be easily configured to auto-switch between ports in the event of the active MADI stream failing.

PRODIGY router/format converters have a modular port arrangement, so the two MADI sources in use can connect using coaxial cable, single or multi-mode fibre via an SC optical connector, or fibre via an SFP module. Using different connectivity methods provides extra resilience against external system issues.


Depacketised Network Bridge

Inadvertent intrusion or deliberate hacking into a network is, unfortunately, a concern for users of any IP network. While firewalls can be very effective in general IT applications, they don’t provide a full solution when Audio over IP is involved, due to the depth of integration with core processing required by media streams.

A DirectOut PRODIGY unit inserts a depacketised bridge between the AoIP network interface and the internal router. This “unwraps” the incoming audio from its external packeted format, transforming it to baseband audio. This is then passed to the router, which operates in the baseband domain.

Dispensing with the IP packets – which are a necessary consequence of using existing IP structures for media transfer – completely removes the danger of unwanted or malicious embedded data from interfering with the internal processing or any downstream devices connected to it. This is particularly relevant if the incoming stream has been routed via a public or other potentially vulnerable network.


Dedicated Control Ports

This is a simple design feature implemented on DirectOut’s PRODIGY and ANDIAMO products, which provides a separate, dedicated network port for the unit’s control functions. This allows the system engineer to employ completely independent IP networks for audio and control.

Separation of audio and control IP networks removes the possibility of control data being corrupted by problems with the audio network, which sometimes occur due to the high data rates involved, or issues such as intrusion or hacking. If network audio suffers an outage, the control network will still be available to interrogate the system, change settings, adjust levels or even effect a remote reboot.


ST2022-7 dual network redundancy

In a non-media network, it is normal for switches and other devices to request and/or resend data packets if any arrive corrupted, or are simply missing. However, with live media streams, this is highly undesirable, as resending packets inevitably increases latency and can produce audible glitches. An alternative approach is needed

The ST2022-7 standard is the solution. It describes an intelligent dual topology for use in audio media networks. Devices meeting ST2022-7 communicate using two separate networks, and send identical data over each. The receiving device uses only one of the networks as its primary source, but continually examines both data streams. If a packet is missing from the primary stream, it simply “pastes in” the duplicate packet from the other stream, thus ensuring an unbroken and complete network feed.

DirectOut’s Ravenna/AES67 AoIP card implements ST2022-7. The Dante card offers the same level of resilience, but does so using a proprietary technique.


Best Master Clock Algorithm

The use of a Grand Master Clock as the sync source for all devices in a network-based audio system is highly desirable in most situations. However, the price to pay for the assurance of accurate synchronisation is that the clock source itself then becomes a single point of failure.

This issue is addressed in IEEE1588-2008, which is specified by the AES67 interoperability standard, and is therefore applicable to the Ravenna/AES67 AoIP format (also encompassing ST2110). It is not applicable to Dante. The standard defines PTPv2 which uses a BMCA (Best Master Clock Algorithm) to automatically determine and select the best Grand Master sync source for the whole network.

Under PTPv2, all potential Grand Master sources self-declare their performance quality to the network, so compliant network devices can make an intelligent decision about which of the qualifying network devices should be the active Grand Master, and most importantly, which device should be assigned as an alternative should the first-choice Grand Master become unavailable or suffer degradation in quality.


Dual AoIP network ports

Apart from the resilience afforded by the redundancy provisions of ST2022-7 described above, the provision on some DirectOut products of two independent Dante network ports permits total separation of distinct areas, or “islands”, of the audio network.

An example of this might be to use independent networks for the on-stage sources and the main FOH system. Such an arrangement would guard against a disruption to the sound system caused by inadvertent connection of a “rogue” device to the on-stage system. The two networks remain completely isolated and the FOH engineer retains full control of channel routing.