GendynGAN
Position : Research Assitant
Workplace : AI Media Lab, Art&Technology Department, Sogang University
Category : media technology research, audio generation, deep learning, virtual machines, cloud services
Timeline : 3 months
Skills : software testing, programming, sound generation
Concept
This project combines dynamic stochastic synthesis and deep learning for audio generation. Using a Generative Adversarial Network (GAN) and Csound, a dataset made of songs from the Free Music Archive is used against sounds generated with the dynamic stochastic method.
How it works
The generator network is fed sounds generated with the dynamic stochastic synthesis in Csound, with the Gendy code available in Csound documentation. The discriminator is fed a dataset of sounds from the Free Music Archive, and its role is to reject data from the generator when not similar enough to its dataset. The audio is fed to the models as raw audio, no preprocessing steps are added.
Testing was done on an AWS EC2 deep learning instance, as GPU resources were necessary to speed up training. Using a virtual machine meant to install a virtual audio device, as none was originally installed in the environment.
Tools I used
-
AWS EC2 deep learning instance (Linux, Bash)
-
Python : Pytorch, librosa, fastai, ctcsound
-
Pulse audio virtual sound card
-
Csound
-
Google Colab, Jupyter notebooks
-
Github, Git
Results
Sounds generated with this algorithm were not the most musical and had a lot of noise.
This could be changed by testing different parameters such as batch size, audio segment size, sample rate, stereo or mono, using different forms of audio input instead of raw audio (spectrograms for example).
What could be done also is instead of optimizing the audio generated by Csound we could optimize the parameters of the Gendyn Csound score, which I might give a try in the future and will update here if results are conclusive.
Code is available on Github.