hylynyiv    //

Notation in VR

Interactive voice controlled system for musical expression in VR.
Commissioned by Tomomi Adachi.
Berlin, 2019-2021.
The project is supported by Berliner Arbeitsstipendien für Neue Musik und Klangkunst 2019.



Movement / Voice / Agency
"Notation in VR" is an interactive virtual environment developed and programmed by Hylynyiv for the experimental voice performer Tomomi Adachi. The environment is envisioned as a 3dimensional score with a novel interactive concept, which Hylynyiv designed and implemented specifically for this purpose, allowing the movement to be controlled entirely by the voice. Guided by the ambition to harness the intrinsic and unique possibilities of VR, it appeared only logical to translate parameters of voice into parameters of movement. “Moved by the voice” becomes a trope when performers physically navigate the notation through their interpretation, manifesting their vocal energy as a force in the virtual space - a poetic configuration that offers a manifold of possible readings.

Symbolic 3D
The environment is populated by a multitude of different objects: letters, handwritten signs, and phonemes, which have emanated from Tomomi’s imagination. They are 3dimensional themselves, and can hence be approached and interpreted “from different angles”, in a non-linear way, really pushing the boundaries of the conventional understanding of performer and score.

Interactivity / Script / Anima
Another distinctive feature sets the system apart from the conventional approch to music interpretation by harnessing the genuine possibilities of technology: the score does not remain in a static state but interacts with the performer, mimicking his utterances in what begins to appear as dialogue, interpreting him in return, as if an imaginary hybrid counterfeit, half symbol, half creature, had emerged from the imagination of the performer, materializing as living and breathing entity in this alternate reality.

∆ºª∆º @∂‚@~π∑€ ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi^·™ÍÁˇ™©ª‚flÍ–∂@ç∆µ∞≈µ~Í∏™ÍÅ°∏Á≠¡}¶∂πø‚π GYRO = new Vector3 (gyro_x,gyro_y,gyro_z); // ACCEL = new Vector3 (
acc_x,acc_y,acc_z);
// MAGN = new Vector3 (mag_x,mag_y,mag_z); // //// //// transform.localPosition = new Vector3 (transform.position.x+1,transform. πø⁄ €π }€{ ∑][«“≠¡}position.y,transform.position.z); // // gameObject.BroadcastMessage ("rotateØ∏تº©@√¥≈…å⁄~¢¶¡¿''€ ∫ƒC_CCW", ACCEL); //
// gameObject.BroadcastMessage ("changØ∏تº©@√¥≈…å⁄~¢¶¡¿''€ ∫ƒeFreq", mag_y);˜\·∑†∂fi^·™ÍÁˇ™©ª‚flÍ–∂@ç∆µ∞≈µ~Í∏™ÍÅ°∏Á §C%Y£fi^ÍŒÅfl„›Ï°ÍÆ°∏Ø’÷—©ª∂«•π{| ‚«“≠}¶{≠“ œ«å~‚…∂ •πœ–…∞€¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓÍŒ // // gameObject.BroadcastMessage ("changeFreq3", mag_z);‚~∞˜\
·∑†∂fi∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞Ï°ÍÆ°∏Ø’÷—Ø∏تº©@√¥≈…å⁄~¢¶¡¿''€ ∫ƒ // // gameObject.BroadcastMessage ("rotation_", ACCEL); // // gameObject.BroadcastMessage ("changeFreq_Sine", mag_y); //// gameObject.BroadcastMessage ("changeGain_Sine", acc_x);«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºª∆º @∂‚@~π∑€ ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶ // // gameObject.BroadcastMessage ("moveY", y_num); //// ga
meObject.BroadcastMessage ("moveZ", z_num); // }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €πgameObject.BroadcastMessage ("changeFreq2", mag_x); }€{ ∑][€ πø⁄ €π }€{ ∑][«“≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºœø⁄•∂ºƒç≈©º‚† §C%Y£fi^ÍŒÅfl„›Ï°ÍÆ°∏Ø’÷—Ø∏تº©@√¥≈…å⁄
~¢¶¡¿''€ ∫ƒ©ª∂«•π{|«“≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚º@ø¨©≠}[–…∞º√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi ≠“ œ{≠“ transform.localEulerØ∏تº©@√¥≈…å⁄~¢¶¡Angles= new Vector3(-gyro_y, gyro_z, 0.0f);œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∂‚ºº√‚∞ç≈…≈Æ°∏Ø’÷—Ø∏تº©@√¥≈…å⁄~¢¶¡¿''€ ∫ƒ©ª∂«•π{|
 ‚«“≠¡}¶∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ gameObject.BroadcastM¡}¶∂πø‚π essage ("rotateC_CW", ACCEL);πø⁄ €π }€{ ∑][
∆ºª∆º @∂‚@~π∑€ public double increment; public double ºƒç≈©º‚†®ΩΩ⁄π }¶phase; private double sampling_frequency = 48000;fi^·™ÍÁˇ™©ª‚flÍ–∂@ç∆µ∞≈µ~Í∏™ÍÅ°∏Á≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][€ πø⁄ €π }€{ ∑][«“≠¡}¶∂πø‚π }¶{∆º
œø⁄•∂ºƒç≈ void changeFreq_Sine(int input_acc){⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈ } // void changeGain_Sine(int input_∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi ≠“∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi ≠“acc2){ // gain += input_acc2/10; // }§C%Y£fi^ÍŒÅfl„›Ï°ÍÆ°∏Ø’÷—Ø√¥≈…å⁄~¢¶¡¿''€ ∫ƒ©ª∂«•π{| ‚« }¶{≠“ œ«å~‚…∂ •πœ void OnAudioFilterRead(float[] data, int channels) }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi ∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|] increment = Math.Round ( frequen
cy * 2 * Math.PI / ∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓsampling_frequency ,3); for (var i = 0; i < data.Lengt∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓh; i = i + channels) { phase = Math.Round ( phase + increª∂«•π{| ‚«“≠¡}¶ÍŒØ™‚~∞˜\·∑†∂fi^·™ÍÁˇ™©ª‚flÍ–∂@ç∆µ∞≈µ~Í∏™ÍÅ°∏Á §C%Y£fi^ÍŒÅfl„›Ï°ÍÆ°∏Ø’÷—Ø∏Øment ,3) ; data[i] = (float)(gain*Math.Sin(phas©ª∂«•π{|«“≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|] œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºª∆º @∂‚@~π∑€ ∫ƒ©ªº©@√¥≈…å⁄~¢¶¡¿'
'€ ∫ƒ©ª∂«•π{|e)); if (channels == 2) data[i + ∏تº©@√¥≈…å⁄~¢¶¡¿''€ ∫ƒ ‚«“≠¡}¶∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][«“≠¡}¶∂πø‚π ∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞Ï°ÍÆ°∏Ø’÷—Ø∏تº©@1] = data[i]; } } ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{∆ºœøåflıÓؙ͌‚~∞˜\·∑†∂©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞Ï°ÍÆ°∏Ø’÷—Ø∏تº©@√¥≈…å⁄~¢¶¡¿''€ ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{≠frequency = (input_acc+100)*5;“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºœø⁄•∂ºƒç≈©º‚† –…∞€ πø⁄ €π }€{ ∑][if (phase > 2 * Math.PI) phase = 0;“≠¡void changeGain_Sine(int input_∂‚ºº√}¶∂πø‚π }
∆ºª∆º @∂‚@~π∑€ ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{∆ºœø⁄…∞•π{| ‚«“≠¡}¶∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ §C%Y£fi^ÍŒÅfl„›Ï°ÍÆ°∏Ø’÷—Ø∏تº©@√¥≈…å⁄~¢¶¡¿''€ ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][«“≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi ∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞Ï°Í
Æ°∏Ø’÷—Ø∏تº©@√¥≈…å⁄~¢¶SerialPort sp = new SerialPort("COM3", 9600); // // void Start () { // sp.Open ();∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi^·™ÍÁˇ™©ª‚flÍ–∂@ç∆µ∞≈µ~Í∏™ÍÅ°∏Á // sp.ReadTimeout = 250; //√¥≈…å⁄~¢¶¡¿''€ ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºœø⁄•∂ºƒç≈©º‚† §C%Y£fi^ÍŒÅfl„›Ï°ÍÆ°∏Ø’÷—Ø∏تº©@√¥≈…å⁄~¢¶¡ //// session_id += 1; // // } // void Update () { // strArr = null; // // currentI€ πø⁄ €π }€{ ∑][€ πø⁄ €π }€{ ∑][«“≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi∆ºœø⁄•∂ºƒç≈©
ºªº©@¿''€ ∫ƒ©ª∂«•π{|«“≠¡}¶∂πø‚π }¶{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄|]“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][ ∆ºª∆º @∂‚@~π∑€ ∫ƒ©ª∂«nput = sp.ReadLine(); // strArr = currentInput.Split (","•∂ºƒç≈©º‚†®ΩΩ⁄|]∂‚ºº√‚∞ç≈…≈åflıÓؙ͌‚~∞˜\·∑†∂fi^·™ÍÁˇ™©ª‚flÍ–∂@ç∆µ∞≈µ~Í∏™ÍÅ°∏Á≠¡}¶∂πø‚π }¶
{∆ºœø⁄•∂ºƒç≈©º‚†®ΩΩ⁄π }¶{≠“ œ«å~‚…∂ •πœ– [0]); // // // // s1 = strArr[0]; // s2 = strArr[1];¡¿''€ ∫ƒ©ª∂«•π{| ‚«“≠¡}¶∂πø‚π }¶{≠“ œ«å~‚…∂ •πœ–…∞€ πø⁄ €π }€{ ∑][print (currentInput);‚†®ΩΩ⁄|]°∏Ø’÷—Ø∏Ø