In terms of sound, this initially meant creating opportunities for the broadest possible coverage of input. In order to also create data input for light, the decision was made to use an experimental approach instead of classic microfoning with radio links.

In the first approach, an interactive concept was planned that should enable the exchange between performer and sound artist without being restrictive. Technically, this led to the idea of ​​capturing all movements and vibrations of the structure as far as possible and making them available to the sound artist as raw material from which he can then shape the music / sound. The desired real-time transmission should enable direct interaction between the artists.

This consideration led to two first technical approaches, a conservative approach and an experimental concept.

The starting point of the two approaches was the technical problem of transmitting as flexible and numerous signals as possible from the moving structure without having a restrictive effect. This required a wireless transmission of the signals in real time as well as a mobile energy supply. The conservative approach envisaged wireless miking of the structure, whereby the resonances in the cavities of the structure as well as the vibrations on the surfaces and in the framework were to be picked up with different microphones.

The experimental approach also provided for microphones, but the processing of the signals should in part already take place inside the structure and enable transmission via wireless LAN. For this, microprocessors should be housed inside the structure, which take over the analog / digital conversion and forward the data to a central server.

Compared to the conservative approach, the experimental approach offered more flexibility at significantly lower costs. At the same time, there was the possibility of combining both approaches in a later phase as well as the use of additional sensors on the microcontrollers to provide further data for musicians and light artists. Based on these considerations, we decided on the experimental approach.

In implementing the experimental approach, the Sound Designers decided on the Raspberry Pi 3B + (hereinafter RPi) as the microcontroller platform / computer in the structure. This should be accommodated in module form, each with power supply and analog / digital conversion and the possibility of connecting min. offer two microphones.

The problem here was the automatic connection of several such modules to the server and, in particular, the transmission of the signals over the wireless network with as little latency as possible. Due to the nature of the network, the distribution of the network capacity over several processes as well as the conversion and data packet management, (almost) real-time transmission is difficult to achieve here. The aim was therefore to minimize the transmission latency as much as possible without impairing the transmission’s quality or stability too much. At the same time, the system should be developed from the outset with the aim of making it as easy to use as possible, so that it can be used in the absence of a specialized technician.

This phase turned out to be relatively complex and took a lot of development time, as there was no ready-made technical solution. In particular, it was important to find a solution that would run reliably on multiple systems.

As an audio solution, the Musicians used the JACK Audio Connection Kit. This offers several integrated / module solutions for the network transmission of audio signals. In particular, these modules turned out to be very platform-dependent, which meant that a reliable wireless connection between the Linux-based RPis and the Win / Mac OSX-based workstations of the artists could not be established. To cope with this problem, an additional Linux server was interposed, which bundled the signals from the modules and passed them on to the workstations via a cable connection. This enabled a connection with relatively low latency and sufficient stability to be established.

This first solution initially only provided for the transmission of audio signals as such. The further processing and conversion of the audio signals should only take place in this form on the part of the artist. Forwarding the audio signals from the sound artist to the light artist should give them the opportunity to further interpret the signals for their work.

On the one hand, however, to enable more flexible use of the data for several artists and at the same time further minimize latencies, we developed a hybrid solution at this point: While time-critical, especially percussive audio signals are passed on exclusively to the sound artist via direct wireless audio connections, the RPi Modules in the structure directly translate the translation of the signals present into sensor data, which in turn can be transmitted more easily and quickly. For this purpose, software was developed that analyzes the signals from the microphones and converts them into data pairs (pitch / intensity) and then streams them over the network. This data can then be accessed at various workstations as required.

In the course of technical development, the idea arose of integrating further sensors in addition to the previously mentioned inputs, which, in addition to the resonances and vibrations of the structure, can also display its movement and acceleration in space. The Team is currently working on the integration of these sensors into the existing system. It is also planned to integrate further objects equipped with sensors to enable the artists to manipulate the sound space. In this way, not only the structure, but the entire stage becomes an interactive instrument. Thanks to the modularization of the sensors, the system can be expanded as required within certain limits. The integration of additional electrical actuators, which can be controlled by the sound artist and thus can directly influence the acoustic events on stage, is also planned.