I just completed a new art-toy: PlayWithMozart. To play, click on young Mozart’s image, below.
Be sure to come back here and leave a comment if you enjoy the program of if you have any questions or suggestions. I plan to make the code available in a few weeks, after I have a chance to comment it.
The idea
The idea was simply to try to adapt Mozart’s Musical Dice Game, so that the choice of measures is not random, but deterministic in some fashion. Thus came the idea of drawing a melody line and have the program choose the measure that best fits the melody line.Mozart wrote a minuet with 176 measures. He also created two matrices of 11 rows and 8 columns (look at it as one matrix with 16 columns). Each cell in the matrix contains a serial number referring to one of the 176 measures in the minuet. In the Musical Dice Game, the player rolls two die, thus producing a number between 2 and 12 — corresponding to one of the 11 rows of the matrix. For the first measure, one throws the die and selects the measure in the appropriate row of the first column. For the second measure, one throws the die again and selects the measure in the appropriate row of the second column, and so on. The resulting composition is a re-arrangement of the measures of the original minuet. The brilliance of Mozart was writing 16 sets of 11 measures, such that any of the 11 measures of a set can succeed any 11 measures of the previous set, while sounding good musically (actually Mozart cheated a little bit; the last set only has 3 unique measures, as some measures are repeated in the set). Several sites explain this game and its history; I found this site and this one very useful.Instead of “playing the dice”, the program PlayWithMozart selects the measure that best fits the outline of the melody you have drawn, taking into account whether the melody line is rising or falling, and the average pitch of the melody line for that measure. It does constrain the choices to the set of 11 measures available at any point. In addition, it also penalizes measures previously chosen, so as to force more variety, so that all 176 measures will eventually be used. So, how satisfactory are the melodies resulting from this collaboration with Mozart? Try it for yourself!
Coding tools
After playing with ActionScript a few years ago, I switched to Java and later Processing, because of their then faster execution speed, wonderful IDE (eclipse) and wide variety of open-source libraries. While I find that Java really works well on the desktop, I have grown a little impatient with the time it takes to load applets to the browser and the fact that the browser locks while the applet is being loaded. With ActionScript 3.0, the recent improvements in the Flash Player, and he ability to use Eclipse with Flex, I decided to try Flex. I found the experience of programming Flex/ActionScript quite enjoyable, and was pleasantly surprised by the speed and responsiveness of the result (PlayWithMozart). I do miss the more advanced syntax checkers, style checkers, code formatters, and refactoring tools available for Java – though I expect Flex to catch up.
Producing sound with Flash/ActionScript
I first tried to use Flash’s built-in sound library for PlayWithMozart and was quite disappointed with my inability to control timing precisely. I tried a variety of strategies, but I found it difficult to synchronize separate sound clips. It sounded like timing was off by a significant amount. To confirm this, I wrote a quick Flash program to produce a repetitive beat and recorded the output. Looking at the attacks on the waveforms produced, I found that the beats were off by up to +/- 16 ms. I was about to abandon my project when I stumbled upon Andre Michelle and Joa Ebert’s wonderful Popforge library. With it, I am able to play and apply sound effects to 8 simultaneous stereo tracks, at 20500 Hz, using about 10 to 25% of the CPU. If you want to dynamically sequence sounds, or even synthesize them on the fly, using Flash, I recommend you look at the Popforge library.On a related subject – the piano sounds. Originally, I had planned to embed the waveforms for all 88 piano keys. However the resulting sound file was large and the Flex application was taking too long to load in the browser. So I decided to only use one waveform per octave and have the program pitch-shift that waveform dynamically, as needed, for the other keys of the octave. Some richness of the sound was lost, because all keys in the same octave a clones of each other, but this approach reduced the loading speed by about 90%. On reflection, I think a better compromise would have been to use two keys (e.g. C and G) per octave.

56 Comments
Great work!
Not a big entusiast of mozart, but the programming behind this is amazing. I am working with the popforge library right now and feel a little stuck. Do you have any plans for releasing the source for this application?
Fredrik – thanks for your kind comment. Yes I plan to release the source; I started cleaning up the comments and then got sidetracked (sounds familiar?). Meanwhile I am sending you the source without the final comments directly to your email address, so that you do not have to wait.
hi there, this is really nice, well impressed. I am also trying to work with popforge, am also finding it quite difficult, is there any chance you could send me the source as well? Cheers
Tom – I sent you the source. Thanks for the feedback!
Nice work JM. Pretty amazing what can be done with just a few parameter tweaks. I too am working with PopForge sources, specifically related to pitch shift controls. Any insight you could give would be appreciated!
Thanks for your comment, Jason. Below is how I handle the pitch shift control in PlayWithMozart. The relevant variable in the code below is “pitchMod”. First are excerpts of the Audio loop; then the structure of the class “PianoWav”.
I hope this is helpful.
===========================================
// AUDIO LOOP
===========================================
private var pianoWav: PianoWav;
….
var in0: Sample; // one of 8 Samples (voices)
var outL: Number; // audio amplitude, Left channel
var outR: Number;
private var w0: WavFormat; // waves
private var s0: Number; // speed
private var a0L: Number; // amplitude
private var p0: Number; // phase
// PITCH MODULATION; SET BY SLIDER
var pitchMod: Number = KUI.uiSliders.paramPitch.getValue();
// INSIDE AUDIO OUTER LOOP
// … when new note:
s0 = pianoWav.aDPhase[ currNoteMidi ] * pitchMod;
uLen0 = w0.samples.length – stepsInnerLoop * s0;
….
w0 = pianoWav.aWavPtr[ currNoteMidi ];
s0 = pianoWav.aDPhase[ currNoteMidi ] * pitchMod;
// INSIDE AUDIO INNER LOOP
in0 = w0.samples[ int( p0 ) ]; // phase changes every sample
p0 += s0;
outL += in0.left * a0L;
outR += in0.right * a0R;
< -- insert audio effects here -->
//– store local
out = samples[i]; // for every sample
out.left = outL; // write to sample
out.right = outR;
< -- end of inner loop -->
< -- end of outer loop -->
===========================================
// STRUCTURE OF CLASS PianoWav … index of arrays is Midi note 0..108
===========================================
public class PianoWav {
[ArrayElementType("int")]
// points to wav array of “f” pitch piano notes; one for each of 7 octaves
public var aWavPtr: Array;
[ArrayElementType("int")]
public var aDPhase: Array; // populates step in samples (or “speed”)
….
< -- for each iMidi note -->
aWavPtr[iMidi] = aFa0to6[ixFa0to6] ; // just pointer
var midiFA: int = midiF1 + ixFa0to6 * 12;
// Creates the phase increment for each midi note
var dPhase: Number = dPhase0 * Math.pow( 2.0, (iMidi – midiFA) / 12.0);
aDPhase[iMidi] = dPhase;
…..
Hi Jmbara,
I’m working with popforge, and have ran into a similar roadblock described on your site. By any chance you can send us your version on the code? Thanks very much.
Kenny
Kenny – I sent you the source code. Hope it helps.
Hi! That is really cool.
I have been messing arround with the popforge liberary myself lately. I am having some difficultes playing two samples at the one time. I am sure this is a very simple problem but it is just confusing me.
Did you have a similar problem? Also, could pass on the source code if you wouldn’t mind?
Well done again!
Hi John F – Thanks for your feedback!
PlayWithMozart plays up to 8 samples concurrently (new notes/chords plus the tail end of previous notes/chords). You can actually see the number of voices playing at any one time if you click the “Audio” tab, after you have drawn the outline of your melody.
Once you get your code to play one voice, it should be relatively easy to get it to play more than one voice at one time. The basic idea for the code comes from the examples in popforge’s library: fetch the current value of every sample (adjusting the amplitude as needed) and add all sample values (additive synthesis) before storing the result in the buffer. Separately, I am sending you the source code and you will be able to see how I did it — look in the file MainLoop.as
I hope this helps
jmbara
Thanks so much!
Hi JMB,
this application is very cool. hope you are well.
spencer
Hi Spencer – glad you enjoyed it. Thanks for your feedback!
Hi Jimbara!
Very interesting!
I’m also one of them who would be very happy for a copy of the code…!
I’ve done some notation tools in Delphi, looking for a way to convert them to AS3.
Popforge seems to be the way to go for playback…!
Also looking for solutions for synching score graphics and multi channel mp3 playback.
Here’s my old (very first Flash!) try:
http://www.resurs.folkbildning.net/projekt/mumma/scorxFlash/__seeAmid.html
More examples at korakademin.se -> Fri körrepertoar (swedish language)
Regards / Jonas Nyström
Jonas – Thanks for your comments. I sent you the source code for PlayWithMozart via email. I hope it is useful.
I tried scorxFlash and I like the elegant interface. The synching of the score-sweeping needle with the voices (beautiful voices, by the way) looks quite precise. While I do not understand Swedish, I can see how scorxFlash would be a useful teaching tool.
In PlayWithMozart, the synching of audio and graphics is easy since they are both triggered by the same time source (I use the sound buffer playback loop as the clock). However, I have not yet tried to synch long recorded audio segments with graphics or video.
wow, this is very, very cool! nice work. if you have time to send i’d love to take a look at the source. i’m very interested in seeing how you used popforge to create the clock. timing with actionscript has always driven me nuts!
Hi Jmbara,
this is very beautiful – i too am looking into the generative possibilities and work with 3d flash engines and lsystems – trying to bring the sounds of all this to life and would love to learn from your source on this – thankyou for being a real teacher here – the popforge library is very powerful – love it – but supercollider is just amazing too – keep in touch and lets share some knowledge thoughts and code!! jez. great work!
this is awesome, it seems like your disc drum program keeps sync well, is that using flash’s Sound Library, or does it use popforge as well? I am trying to write a sequencer in Flash similar to your boxes program. I’m struggling with popforge, seems like the way to go, i’d love to check out your source if possible.
Hi,
your app is awesome… i am developing an app where i need to play two or more sound files.please can you share your code…
Hi Prajwal!
I’m working on the multi sound thing myself. Please drop me a mail. Maybe we could share ideas and code.
Jonas
jonasnys@gmail.com
Hello, Codemuse fans!
POPFORGE and OGG VORBIS?
Jmbara has shown what’s possible using the popforge library.
There are solutions for loading ogg in as3 showing up, for example the one created by Tor-Einar Jarnbjo (http://lists.xiph.org/pipermail/vorbis-dev/2008-October/019587.html) Anyone interested in joining me in the effort of combining these two?
What about you, Jmbara?
Regards / Jonas
Prajwal, since with Popforge you are dealing with sounds at the byte level, it is easy to have more than one voice. PlayWithMozart adds the samples (additive synthesis) of up to 8 simultaneous stereo voices (in this case stereo piano sounds) before they are stored in the audio playback buffer. Alternatively one could generate FM synthesis by multiplying two streams of bytes (if I remember correctly the Popforge distribution also has an example of FM synthesis).
Jonas, if you can develop a way to access the ogg sounds at byte level, there should be no reason one could not use these sounds with Popforge. This would circumvent the need to package the sounds in furnace format. I am not volunteering to participate in this effort because I do not have the time right now but I would be very interested in seeing the results.
Chris, Jez, David and Prajwal, I sent you PlayWithMozart’s source code earlier today.
Jmbara
Tor-Einar Jarnbjo is decoding ogg in FP9 (http://lists.xiph.org/pipermail/vorbis-dev/2008-October/019587.html) and also now in FP10. He uses PCMAudioLibrary for playback.
He also confirmed that it should be possible.
Any other numberchrunching bravehearts?
Jonas
Jmbara,
I’m so excited about flash player 10 – we’ll finally be able to unleash the real audio power in flash.
You did an excellent job!
I’d love to see how you handle the BPM variation.
So when is the source code coming out? Thanks.
Wags, thanks for your feedback. Here is how the tempo works:
The heart of PlayWithMozart is the audio player — it follows closely the PlayWavFile example in the popforge library. Given my choice of buffer size, the player performs one full loop every 11.6 milliseconds (a constant).
The internal clock is a counter that increments every time an audio loop is completed. This clock in turn drives the pointer that reads the score. The rate at which the score pointer advances at every clock “tick” is proportional to the tempo (which in turn can be varied by the slider in the Audio tab of the GUI).
On the other hand, the pitch (and tuning) is controlled by the rate at which the stored sound samples are traversed. Therefore tempo and tuning are independent of each other, e.g. changing one does not change the other (unlike a tape recorder/player, where both the sampling rate and the tempo increase when the speed of the tape increases).
I hope this gives an overview of the clock mechanism. I have sent you the source code via email, so that you can see how I did it in more detail.
Hi, I am a junior programmer in Hong Kong. I am currently working in a sound system for flash but don’t know how to make effects to sound.
For the reason that your flash involves the popforge library, would you like to share your code with me? I will really happy to you.
Thank you very much.
Best Regards,
Cody
Hi Cody – I sent you the source code. I hope this helps with your project – I would be interested in seeing what you do with it.
This is absolutely amazing. You have done some great work here. I would love to see the code for this and your implementation of the popforge library.
Jeff
Jeff, thanks for your feedback. I have sent you the source code – I hope it is helpful.
Congratulations for your work, Jmbara!
I found your game while trying to find some samples of popforge library. I would like to make some easy music educational games using flash but i didn’t know how to start using popforge. Could you help me by sending me your source?
Sandra
Sandra – Thanks for your comment! Before you start on your project I suggest you check for the latest status of the FlashPlayer 10 sound capabilities; I believe that there are enhancements that may simplify your task. I have not yet had time to explore the new player, but you will find some information about the changes here
http://www.make-some-noise.info/
I sent you the source code for PlayWithMozart. I hope it helps you in your educational game project. I would love to see the results.
Hi there,
This is really cool!
I’m trying to get into popforge programming and there aren’t a lot of examples out there. Could you send me the source code?
Jan-Karl – I sent you the source code. Thanks for your comments.
Dear jmbara,
your application is really nice and impressive.
Is there any chance to get the source code in order to been able to study it? May you send it to me too?
Thank you very much,
Bruno
Bruno – I sent you the source code – it should be pretty easy to follow, especially if you have played with the popforge library.
Many thanks for your feedback.
Hi,
Stumbled upon your experiment while looking for some popforge examples; looks really nice! I was wondering if I could take a look at the source code, as I’m finding popforge a little daunting…
Cheers,
Bastiaan
Hi Bastiaan – I just sent you the source code via email – enjoy!
This is a really impressive app. I’m working on an educational music site and popforge looks like it would simplify my task greatly. Would you consider sending me the source for this?
Kudos,
Bill
Bill
Am glad to share the source code – I have sent it to you. I hope it helps you with your educational music site. I would be very interested in seeing the results. Thanks for your feedback!
jmBara, very nice and creative project you made ! I’m working on a ‘random loop mixer’ project in which i need to play a few loops simultaneously and seamlessly. mp3 format doesn’t work exact enough because of the enocded headers, even if i loop with timers and without sound_complete events. Then i found popforge and I am very happy with the wavdecoder. But now with flash10 out i would like to loop wav pcm data and use the new sampledataevent to manipulate the pcm byteArray. But without using the popforge audiobuffer, just the regular fp10 sound object. Do you have any ideas how to get the decoded wavformat samples from popforge into the regular byteArray to use in the sampledataevent? and how to hereby effectively loop wav-files? I am also quite interested in your source files, thanks al lot and good luck ! Victor.
Hi Victor – I sent you the source code for PlayWithMozart – I hope it is helpful. I have not yet had a chance to play with the Flash 10 new sound capabilities so I am sorry I cannot help you with your question. My only thought is that if you are already using the popforge library to decode the wav data, why not also use popforge to play the audio loops. In that case the looping of the wav arrays can be done right inside the main loop of the popforge’s audio player (see MainLoop.as in the PlayWithMozart source) by updating (position and speed) the indexes to the loop arrays and then mixing the samples read from the different loops into one single stereo sample (additive synthesis) before the resulting stereo sample is pushed into the popforge audio buffer. This is done one sample at a time, providing great real time control. I assume that with the new fp10 there is a way to do all of this sidestepping the popforge library but the few posts I have read on the subject suggest that latency may still be a little bit of a problem. Good luck with your project!
Hi, been playing with popforge for a while but as I don’t have Flex is kind of difficult for me to understand it and get it going, as the others here I’d like to take a peek at your source files if possible
thanks!
Hi Freddy – I sent you the source. To understand popforge you might want to start by examining the file MainLoop.as. I hope you find it useful.
Love your motzart example, the playback is fantastic. Is there anyway i can have a look at the source code, i quite fancy trying to get a few other musical intruments to work as well. Keep it up. Aden.
Hi JMBara – I know, again someone who wants the source code
I need to know how I can manipulate multiple sounds realtime. (eventually in MP3 format, but that is phase 2) I hope to look in your source code to see how you solved it with popforge if it\’s possible.
Thanks in advance!
Patrick and Aden – I have sent you both the source (Aden, sorry for the delay). Patrick — You will see in the source that PlayWithMozart can manipulate and play 8 voices simultaneously. The number of voices is only limited by the host computer CPU. I don’t know how to use MP3 though; I used Andre Michelle and Joa Ebert’s furnace format and all soundfiles get loaded into memory at the start. To save memory, I only load seven piano sounds, about one per octave; all the other 101 MIDI pitches are produced on the fly by varying the sampling rate. Using more sounds would enrich the dispersion of timbres, but one per octave seemed good enough to fool the ear (at least my ear!).
wow your application is really cool, .. i was searching for pitching and popforge and so on… and i’ve found you!
i have no clue to pitch audio files with popforge, can you please send me your code, i think that could help me a lot!!
greetz from the Mozart-country austria (:
Betty
Betty – I sent you the source code. I hope this helps you with your question on how to pitch sounds with popforge. You will see in the source code that I generate all semitones of the grand piano keyboard by pitch shifting only 7 original sounds (notes). This economy of stored sounds also helps speed up the initial sound loading times.
When you have PlayWithMozart running, after you draw the outline of the melody, try clicking on the “Audio” tab and then moving the BPM (Beats Per Minute without pitch shifting) and TUN (pitch shifting) sliders. This illustrates the flexibility of the popforge framework.
Good luck with your project — I would love to see the results.
Thanks for your comments!
Hello, I also recently stumbled upon PopForge and was having the exact same problem coming up with pitch shifts… and I don’t mean to be a bum but it would be great if I could look at the source too 0:)
I’m a student and I’ve been thinking of doing a simple version of Korg’s KAOSSillator using the XY Plane, and I’m pitch shifting correctly but I don’t know how the scales / pitches work…
and your PlayWithMozart is amazing, by the way!
Jan – I sent you the source code. PlayWithMozart uses sampled sounds; of course, to emulate the KAOS you will want to synthesize your own sounds rather than using samples — if you haven’t already, you can find examples of simple software synths in the popforge library. Good luck with your project!
Great project! I’ve been creating a BPM timer/keeper… my approach is to create an empty sound sample using bytearray and popforge’s SoundFactory, and using sound_complete. So far it’s somewhat accurate with the ms intervals being quite constant.
I’d like to know how you did yours. I can send you mine for comparison.
Gerald – I sent you the source code. To see how I implemented the timer look for the file KClock.as. The timer gets incremented by the main loop in MainLoop.as. I look forward to seeing your BPM timer keeper.
Hello!
I know it’s been asked a few dozen times before – but I’m also beginning to work with the Popforge library and would be very interested in seeing your code to analyze as a basis for a piano song composition program I’m in the process of working on!
Best Regards,
Adam
Adam – Am glad to share the source code — I’ve sent it to you. I’d be interested in seeing your piano song composition program when it is finished.
Hey, any chance you could share your source? I am struggling synching multiple sound and it sounds like you’ve got it sorted!
Hi Chris – I sent you the source code for PlayWithMozart – I hope it helps. Just to make sure credit goes where it is due: Andre Michelle and Joa Ebert sorted out the sound timing issues with their Popforge library, so I did not have to worry about that. Good luck with your project!
2 Trackbacks
[...] CODEMUSE » PlayWithMozart (tags: as3 flash flex audio music) [...]
[...] a young Mozart and are contained in a manuscript owned by the Mozarteum for more than 100 years. …CODEMUSE PlayWithMozartTo play, click on young Mozart’s image, below. Be sure to come back here and leave a comment if you [...]