Flashcoders take on MIDI
conversation ported directly from the flashcoders mailing list and other conversations
Tyler Wright
The Flash Player has evolved through the ages to provide the most needed functionality. Through each version there have always remained a few common goals. What I have found is that:
Flash is small — from the player itself to the swf file format to the assets it is optimized to load, focus has been placed on small file sizes (this of course is not as apparent in many websites that are heavy in multimedia)
Flash supports standards — the player supports many web and multimedia formats standard in the industry, such as jpg, mp3 and xml
Flash is interactive — the players greatest strength is the dynamic behavoir through ActionScript to allow user interactivity
MIDI, a music standard format that most computers support today, fits all of these categories (like a glove). In fact there’s an opensource project being developed to allow MIDI through Flash, though it requires an additional download and install to the user apart from the Flash Player itself (seen at osflash.org)
I’d like to take a poll. Do you think MIDI should be included in the Flash Player? Why or why not? I want both votes and opinions as I’ll organize the results and send them off to ” Adobe, formerly known as Macromedia”. Please respond with some sort of opinion whether it’s pro or con. I’ll list the pros/cons I can think of below (you don’t have to read the rest of this email if you already have your opinion).
A little more on MIDI:
MIDI is a standard music format (some will argue that it’s the only standard) that represents pitches and instruments to be played as a song. It’s extremely small, being the vector of music, and has to be interpreted by a users soundcard. Almost all computers these days support standard MIDI, though it sounds synthesized (especially on the voice and string instruments). Some soundcards or additional software transform the common MIDI into amazing orchestrations, but most users don’t have this advanced playback.
MIDI pros:
can be generated dynamically and played through a sequencer to allow complete on-the-fly customization of sound.
very small in filesize
supported by almost all soundcards
numerous applications for the creation of MIDI songs (many are free)
it’s a standard that has been around for a long time (so there is a lot of support for it)
a small implementation (wouldn’t increase the Flash Player size by more than 50K)
MIDI cons:
most people will have a more synthesized sound
user experience isn’t guarenteed to be consistant (for those with higher quality soundcards)
as with all advancements, could make it really easy for developers to have annoying sounds playing on their sites 
In short, if the Flash Player had a midi sequencer built in it would allow developers to create lightweight interactive music applications, such as this sheet music rendering application or music creation applications. It could also allow users to experience a website that contained sound effects or decent background music at very little bandwidth cost. Formerly know as Macromedia has always been good about listening to the developer community and will surely make efforts to build the features we need, if we tell them. This is your forum.
Tyler
_______________________________________________
Marc Hoffman
I think it would be a great idea for Flash to support MIDI. As you point out, major advantages over wav and mp3 are the tiny file size and the potential for allowing Flash to create music dynamically, even interactively. The biggest drawback (which you also recognized) is lack of quality control — somewhat akin to using system fonts with which there’s no guarantee of consistent text layout, only with MIDI the synthesizers in most sound cards are cheesy to say the least. Whatever happened to Beatnik, which used a MIDI-like command set to control small sound fonts consisting of real, not synthesized, sounds?
It might also be of interest to readers here that MIDI can be used for more than the creation, reproduction, and scoring of music. It has been used for controlling robots and similar applications. Not sure how much of that has been done, but it might offer a new, ready-made interface for Flash to communicate with peripheral hardware.
Back when Flash sound was truly terrible (version 3) I built a site (http://www.silent-music.com) for a pianist who accompanies silent films. I used an 11 Kb MIDI file in a hidden frame to replace about 100Kb of very poor-sounding ADPCM in Flash. Once Flash supported MP3 compression, we redid the site with real recordings, but the MIDI version was not bad — better-sounding than the ADPCM that Flash 3 used.
- Marc Hoffman
_______________________________________________
Martin Wood
Interesting question, although ive made use of midi everyday for, well, too long now :), im really not sure if making it part of the player as a single functional unit would be the best move.
Personally i would rather see a more open, low level approach to sound in the player upon which MIDI and other implementations could be developed.
Now that the player (8.5 and onwards) will have much better binary data handling, theres nothing to stop me or you from creating a MIDI file parser.
Thats one side of the equation, obviously the more intricate side is audio playback.
I’ve been moaning about the audio capabilities of the flash player for longer than i can remember, and would dearly love for it to become much more capable.
The options i can think of so far are (and they arent exclusive)
1. An api for accessing midi devices on the user system, much like accessing a webcam, where you can query for which devices are present, then get a handle to a device and start manipulating it, something like:
var midiDevice:MidiDevice = Audio.getMidiDevice(1);
var instrument:Instrument = midiDevice.createInstrument();
instrument.setChannel(1);
instrument.setProgramNumber(34);
instrument.setController(23,44);
instrument.noteOn(velocity);
etc…
2. ACCESS TO THE SOUND BUFFER. Please. 
This would open up a lot more possibilities for audio generation, manipulation etc..
The 8.5 player already has a Loader.loadBytes feature where you can send binary data locally, i.e. you can create a jpg in the player and then load it into a movieclip without sending it to a server. This is great, but I think a similar scheme for audio would be fantastic.
With a simple sound.setBuffer(binaryData) you could do a huge amount of interesting things.
Sound synthesis, generation.
From musical applications, to game sound effects, or just sound notifications within applications. All of this could be done with a minimal impact on filesize. No need for .wav’s , mp3’s etc..
Sound capture
You could capture audio from mic and allow the user to edit it and process it. You could build annotation tools, voice messaging, musical applications etc..
Also if you can get a handle on the audio stream before it hits the audio device you could have live control over streamed audio, tone controls, reverb, delays, echo cancellation, noise reduction. etc..
So, personally I would rather see the components available to us as developers, upon which we can build a variety of applications. MIDI playback being just one particular application of the feature set. Also other similar systems like OSC could be used.
anyway, im glad you are asking and i’ll happily contribute anything to a document you will put forward requesting audio related capabilities.
thanks,
Martin
_______________________________________________
A3D
Pros: Midi gives Flash / you an interface to communicate with the world out site your computer…and you can even make (sequens) music with it Cons: Lots of bad tunes.
A3
_______________________________________________
Johan Lopes
Very interesting thread. I recall a post on FlashCoders by Alexis Isaac a while back and I relayed it on my blogsite here:
http://www.statik1.com/mt-weblog/archives/000084.html
He’s the direct link to it:
http://www.alexisisaac.net/products/flashMidi/, although as Martin mentioned you can also find it on http://osflash.org/flashmidi.
I think this is a great effort and something we should all support (I know he was looking for Mac beta testers). I haven’t had much time to play with it but I will now that you’ve reminded me!
Cheers,
/Johan
_______________________________________________
hank williams
I think midi support would be nice. In truth though, the question is a little broader than midi. It would be great if there were a sandbox safe way to access USB and/or firewire devices. Midi is but one. It would need to work something like the way that the webcam access works where the software asks the user if it is ok for the user to access a given device.
I certainly think midi should be an important part of any such initiative, but I see it kinda like giving us binary sockets. We need the machine internal equivalent to binary sockets.
On another note I also think, particularly with the speed of 8.5, that it would be great to give us direct access to the sound buffer. I am not sure how fast 8.5 math will be but if you can do a good FFT we could be making actual synthesizers and audio processors in flash. To me that would be cool.
Regards
Hank
_______________________________________________
Martin Wood
> On another note I also think, particularly with the speed of 8.5, that
> it would be great to give us direct access to the sound buffer. I am
> not sure how fast 8.5 math will be but if you can do a good FFT we
> could be making actual synthesizers and audio processors in flash. To
> me that would be cool.
absolutely,
the 8.5 player already has an FFT (aka sound spectrum) function built in, but i think its buggy (this is what i have read, ive not tried it yet)
Now, if they would add an inverse FFT as well then it would be fantastic. It would definitely take a load off the actionscript processing required if they were implemented natively.
but at the very least access to the sound buffer would open up a whole new world.
martin
_______________________________________________
hank williams
> the 8.5 player already has an FFT (aka sound spectrum) function built
> in, but i think its buggy (this is what i have read, ive not tried it yet)
>
Are people currently trying to do sound related stuff?
Hank
_______________________________________________
Robert A. Colvin
http://www.richapps.de/?p=23
check it out..;)
_______________________________________________
hank williams
Thanks, but could you give me a little overview of what that is. There is only code with no description.
Thanks
Hank
_______________________________________________
Martin Wood
being able to read the data is cool, but just to say it one more time, being able to create audio data would be great.
I do have one idea which i’ll attempt if it doesnt look like we will get sound buffer access when the next player is released.
As i mentioned earlier there is a Loader.loadBytes function which im hoping allows you to create and load swf’s entirely in the player.
So with a bit of trickery it might be possible to synthesize audio into a buffer, write it into a swf and then use the Loader.loadBytes to play it, all done in the player.
I think one of the biggest challenges for this idea would be encoding the sound into the created swf, im not really up for writing an mp3 encoder in actionscript. But if there is a more simple encoding scheme that can be used then it would be one route to synthesizing and processing audio entirely in the client.
Of course, it may even be possible to do something like create the audio in the swf as a streaming sound and just keep pumping bytes into the loader, but im getting way ahead of myself with that
martin.
_______________________________________________
hank williams
> I think one of the biggest challenges for this idea would be encoding
> the sound into the created swf, im not really up for writing an mp3
> encoder in actionscript. But if there is a more simple encoding scheme
> that can be used then it would be one route to synthesizing and
> processing audio entirely in the client.
>
Why bother with compression. This just creates more processor overhead. a minute of cd quality sound should be 10mbytes. This is not so bad. I dont know if flash would have any memory issues with this but I doubt it. In any case it could be chunked. But then the issues is whether it is possible to effectively sync separate tracks. Because if it is then You dont have to worry about doing one big sound file which would reduce memory needs.
But I am really curious about this loader thing. I am not really aware of it. It sounds like a huge deal.
Regards
Hank
_______________________________________________
bryan.rice
> But I am really curious about this loader thing. I am not really aware
> of it. It sounds like a huge deal.
It is a very cool AS3 feature:
http://www.flashguru.co.uk/actionscript-3-new-capabilities/#more-606
blue skies,
bryan
_______________________________________________
Martin Wood
> Why bother with compression. This just creates more processor
> overhead. a minute of cd quality sound should be 10mbytes. This is not
> so bad. I dont know if flash would have any memory issues with this
> but I doubt it. In any case it could be chunked. But then the issues
> is whether it is possible to effectively sync separate tracks. Because
> if it is then You dont have to worry about doing one big sound file
> which would reduce memory needs.
good point, for some mistaken reason i was thinking that you couldnt put uncompressed audio data in a swf. im getting old and my memory isnt so good these days.
> But I am really curious about this loader thing. I am not really aware
> of it. It sounds like a huge deal.
see the link to Guy’s page, also from the docs :
loadBytes Method
public loadBytes(bytes:ByteArray) : Void
Loads from binary data stored in a ByteArray object.
Parameters
bytes:ByteArray — A ByteArray object. The contents of the ByteArray can be any of the file formats supported by the Loader class: SWF, GIF, JPEG, PNG.
from http://livedocs.macromedia.com/labs/1/flex/langref/index.html
i think its a really significant addition, lots of possibilities
now all we need is an actionscript API for creating SWF’s in the player.
_______________________________________________
ryanm
> Personally i would rather see a more open, low level approach to sound in
> the player upon which MIDI and other implementations could be developed.
>
Normally, I am all about giving flexibility to the developer, but raw sockets + byteArray + laccess to the sound buffer would be a security nightmare. It’s like a big welcome mat for viruses. I wouldn’t hold my breath.
ryanm
_______________________________________________
hank williams
I dont understand. Are you saying that raw sockets is a security nightmare? or bytearrays? or the sound buffer access? Raw sockets and bytearrays are already in flash 8 so are you saying there is already a security nightmare? Or are you saying that sound buffer access would somehow tip the scales into a security nightmare? If so, how? It seems to me that that is no more dangerous than giving developers direct access to the screen display buffer which they now do already in flash 8.
Regards
Hank
_______________________________________________
ryanm
> I dont understand. Are you saying that raw sockets is a security
> nightmare? or bytearrays? or the sound buffer access? Raw sockets and
> bytearrays are already in flash 8 so are you saying there is already a
> security nightmare? Or are you saying that sound buffer access would
> somehow tip the scales into a security nightmare? If so, how? It seems
> to me that that is no more dangerous than giving developers direct
> access to the screen display buffer which they now do already in flash
> 8.
>
It is the combination. With all 3 of those, you would have unrestricted access to download and execute code on the client machine without any way for the VM (flash player) to determine if the code was malicous or not. Pushing malicous code into the sound buffer is an old and well known way to get virus code onto a client machine, because audio streams can’t be virus checked without causing latency, which is an unacceptable cost in fields like gaming and other types of distributed interactive media.
The screen buffer access you have is only for the Flash renderer, it doesn’t give you direct access to the system-level screen buffer (no hardware or even OS level access). Audio goes directly to the systel-level sound buffer to minimize latency, making it a prime target for a virus entry point.
ryanm
_______________________________________________
hank williams
How exactly does this audio buffer virus stuff work?
Honestly I cant imagine that it is not possible to create a buffer that stores numbers that represent sound waves where the flash engine routes those numbers to the audio out driver. This would be no different than the way flash handles the display buffer using exactly the same methodology. The audio and sound techniques would be 100% analagous.
Regards
Hank
_______________________________________________
David Rorex
I agree…bytearrays aren’t anything bad, its just another way to manipulate data. The only way it could cause problems, is if there are any exploits in the flash runtime, it would make it slightly (only slightly) easier to take advantage of them.
Raw sockets, are still restricted by the same cross-domain restrictions as exist in flash 7 & 8, so sites have to specifically allow flash clients to connect (opt-in)
direct access to sound buffer? how could there be any security problems with this? flash already can play sounds, so the most direct access would let you do, is create very strange sounds that maybe you couldn’t compress in mp3? maybe you could write some kind of nerual virus, that when people hear it, in infects their brain?
I don’t see any security issues with these features. But I understand the position, I don’t want flash to become another attack vector, like signed java applets. Right now, the worst a flash movie can do is pop up lots of browser windows. The worst a signed java applet can do is ANYTHING (if the user presses Yes in the “accept applet?” dialog). I could for example write a java applet that deletes all your files.
-David R
_______________________________________________
ryanm
> Raw sockets, are still restricted by the same cross-domain
> restrictions as exist in flash 7 & 8, so sites have to specifically
> allow flash clients to connect (opt-in)
>
Well, obviously, a virus author would allow connections from other domains, to allow his virus to spread. The cross-domain restrictions protect the *server* from your flash, not the client.
> direct access to sound buffer? how could there be any security
> problems with this? flash already can play sounds, so the most direct
> access would let you do, is create very strange sounds that maybe you
> couldn’t compress in mp3? maybe you could write some kind of nerual
> virus, that when people hear it, in infects their brain?
>
No, direct access would let you write *any* kind of binary data to the sound buffer, and when the sound buffer overflows, that data gets dumped into a predictable place (in memory or to disk, depending on the OS, etc). If they can find a way to execute that code they can install a virus on the client, bypassing both the virus scanner and the firewall. It’s old school, I know, but it is still being used by viruses all the time. Add into the equation file system access and you’ve got a whole list of ways to compromise the client with an innocent-looking Flash x-mas card.
ryanm
_______________________________________________
ryanm
> Honestly I cant imagine that it is not possible to create a buffer
> that stores numbers that represent sound waves where the flash engine
> routes those numbers to the audio out driver. This would be no
> different than the way flash handles the display buffer using exactly
> the same methodology. The audio and sound techniques would be 100%
> analagous.
>
*Some* kind of API could and should be exposed for developers, for sure. But only the Flash player should be writing directly to the sound buffer. Now, maybe what they should do is add a synth to the player, and shoot for something unique and interesting instead of just pushing midi through. For example, people are buying up old Commodore 64 computers to pull the SID synth chip out because it is unique and is capable of producing sounds that modern synths just can’t match without the help of samplers. Building a “soft-SID” into Flash would open up a whole new use for Flash: the production of software sequencers and sequenced-synth playback tools. It would also allow great depth in your soundtracks without heavy wav or mp3 files.
Midi could be extremely useful if there were low-level interfaces for it, so that you could attach devices that use midi to it and use the midi commands to control Flash, and vice versa. For example, this is a bottom of the line midi light controller:
http://www.behringer.com/LC2412/index.cfm?lang=ENG
It only handles 24 channels, but they make some that handle up to 2048 midi channels. I don’t have any idea what kind of Flash content could put such controls to use, but the best way to find out would be to make it possible, and then sit back and see what people do.
ryanm
_______________________________________________
David Rorex
> No, direct access would let you write *any* kind of binary data to the
> sound buffer, and when the sound buffer overflows, that data gets dumped
> into a predictable place (in memory or to disk, depending on the OS, etc).
> If they can find a way to execute that code they can install a virus on the
> client, bypassing both the virus scanner and the firewall. It’s old school,
> I know, but it is still being used by viruses all the time. Add into the
> equation file system access and you’ve got a whole list of ways to
> compromise the client with an innocent-looking Flash x-mas card.
>
> ryanm
All the time? Can you provide any examples? I’m somewhat curious about this now.
Even if this were true, all MM would have to do to prevent this, is just add some checks in the flash runtime so that it doesn’t let you write past the end of the buffer. Like:
// pseudo code of internal flash player logic
function SoundDevice.fillBuffer(data:ByteArray):Boolean {
if(data.length > _internalSoundBuffer.length)
return false;
_internalSoundBuffer.copyData(data, data.length);
return true;
}
Not exactly rocket science. Buffer overflows are preventable.
-David R
_______________________________________________
Weyert de Boer
The availability of MIDI would be nice! It will avoid the use of a simple server/client system. Most of the time at school we use a old keyboard for it. For example, earlier this month some class mates used it to make a music painting canvas. This means you use a old wireless keyboard, and use a lot of buttons which are all associated with a unique key. Now if you press a button with your brush, the computer receives a specific letter. Because Flash doesn’t have any problem with this — you can associate a specific key range with a specific sound. It was really nice.
_______________________________________________
Judah Frangipane
+1
I’ve programmed a software sampler back in the day called Vision Sampler Software. It was one of the funnest projects I did and got me into programming. I would love to have more access to the soundbuffer and midi controller.
Judah
_______________________________________________
hank williams
> > Honestly I cant imagine that it is not possible to create a buffer
> > that stores numbers that represent sound waves where the flash engine
> > routes those numbers to the audio out driver. This would be no
> > different than the way flash handles the display buffer using exactly
> > the same methodology. The audio and sound techniques would be 100%
> > analagous.
> >
> *Some* kind of API could and should be exposed for developers, for sure.
> But only the Flash player should be writing directly to the sound buffer.
You keep refering to *the* sound buffer like it is some magical thing. It is not. There isnt even an “it”. The chips that do this stuff can read from anyplace and convert that data into sound. And the drivers provide a whole other level of indirection. Then flash opens up a whole other level of indirection.
So I am not talking about “some” kind of API. I am talking about a memory area that allows you to create sound waves. It is a sound buffer (which can be anywhere in memory including inside a flash sandbox) which is no different than a buffer that holds strings, or bitmaps or whatever. In fact just like a string buffer, one can have lots of sound buffers. Its just a memory area that holds data in 16bit (or whatever) wave amplitudes.
Regards
Hank
_______________________________________________
Moses Ong
Making MIDI available through Flash will open another market for Adobe. Right now graphic designers and programmers are using Flash, so having musicians on board would definately make Flash a universal tool. So, I’m all for it!
While we’re at it, I suggest take the effort to the next level by consider Roland GS and Yamaha XG format, and MIDI2 if you will. For those who’re not familiar, Roland GS and Yamaha XG are richer format yet is backword compatible with GM MIDI, which has only 128 instrument sampler and 1 drum kit. Oh, supporting SYSEX would be fun too…. Am I asking for too much? He he…
From,
Moses
_______________________________________________
Stan Vassilev
If Flash will have anything it won’t be general MIDI banks embeded in the player (= huge size).
SF2 player would be more than enough with ability to embed SF2 files like we do wijth fonts.. But that.. later.
_______________________________________________
Jason Cunliffe
> Who wants MIDI in the Flash Player?
YES PLEASE - I WANT MIDI in the Flash Player…
Too many people assume MIDI is just for music, musicians and for musical instruments. It is not.
Arguably MIDI’s greatest success has been in interoperative hardware of many kinds.
http://www.google.com/search?q=show+control
In addition to keyboards, drum boxes, synths, FX devices and PCs, there is a valuable world of MIDI control surfaces which include Mixers. The control surfaces are rally a subset of the “industrial” uses of MIDI which fall under the rubrik “MIDI SHOW CONTROL” used by major entertainment developers for show biz extravaganzas, such as Rock and Broadway shows, Las Vegas themed developments and Theme parks like Disney World etc.
Midi Show Control was at origin a broad and visionary protocol extension anticipating all kinds of devices and uses — for example:
laser discs [remember them], lighting systems, smoke machines, networked MPEG video players, hydraulic equipment, etc.. See list at end of this post
or
Check out the original 1991 spec
MIDI SHOW CONTROL (MSC) 1.0
http://www.richmondsounddesign.com/txt/mscspec.txt
THE SHOW CONTROL MAILING LIST
There is a very intelligent and informative mailing list where dedicated professional Show Control developers discuss everything from precise programming issues, bug sleuthing, general brainstorming, security, project research and Show Control business. The list also addresses life beyond current MSC spec - next generation MIDI Show control if you will, Show Control over Ethernet, Wifi and so forth.
http://groups.yahoo.com/group/show-control/
Let’s be clear MIDI is only a subset of ShowControl. And most show control developers would not could not take Flash seriously for many applications and formany reasons - at least until now perhaps/ Among the obvious that there was NO midi, no hardware I/o, and not even an attempt to syncchronize sound with video. Flash sucks compared to other systems. Likewise Public security of big instsallations could be a nightmare in a theme park based on Flash. But the name of the game is to use each tool for what it doews BEST. And as all here know, Flash has some special virtues.
So for designing and developing end user interfaces, especially the many stages of design mockups, aided by the growing portability of Flash makes it very attractive. For Show Control applications, Flash 8 offers convivial, easily customizable options. Thus the need for MIDI.
In reality even if Flash had MIDI in the player, [which it should], full professional use would be limited due to likely timing unreliability. There are better and well known tools for handling that. But Show Control developers are famously ingenious and route around most problems by combining technologies in new ways, optimizing the benefits of each.
For small scale applications or where timing and public safety are not an issues, MIDI in the Flash Player opens up a fascinating new world of using FLASH to drive an exciting range of external media hardware. Flash will probably never be able to compete with the likes of MAX, PD [PureData], KeyKit and others. But given the skill of programmers on this list I would not be surprised at what emerges, especially when used WITH those other MIDI-friendly hardware and software.
KeyKit [Tim Thompson] FREE
http://nosuch.com/keykit/
PD [PureData] FREE
http://puredata.info/
http://www-crca.ucsd.edu/~msp/software.html
GEM for PD
“GEM” stands for Graphics Environment for Multimedia and is an external (plugin) for the computer-music software PD.
http://gem.iem.at/
MAX/MSP [commercial sold by Cycling74]
http://www.cycling74.com/products/index.html
Be sure to look at ‘Jitter’
- a set of over 150 brilliant video, matrix, and 3D graphics objects for the Max programming environment.
http://www.google.com/search?q=max+msp
Today we can begin to freely experiment interfacing Flash to the above using Alexis Isaac’s promissing Flash MIDI and the plugin he has developed
http://www.alexisisaac.net/products/flashMidi/
The goodies emerging rapidly in the world of “Processing” further suggest why Flash Player should include MIDI.
http://processing.org/
enjoy,
Jason
*** excerpt from MSC 1.0 spec ***
Hex command_format
——————————————————
00 reserved for extensions
01 Lighting (General Category)
02 Moving Lights
03 Colour Changers
04 Strobes
05 Lasers
06 Chasers
10 Sound (General Category)
11 Music
12 CD Players
13 EPROM Playback
14 Audio Tape Machines
15 Intercoms
16 Amplifiers
17 Audio Effects Devices
18 Equalisers
20 Machinery (General Category)
21 Rigging
22 Flys
23 Lifts
24 Turntables
25 Trusses
26 Robots
27 Animation
28 Floats
29 Breakaways
2A Barges
30 Video (General Category)
31 Video Tape Machines
32 Video Cassette Machines
33 Video Disc Players
34 Video Switchers
35 Video Effects
36 Video Character Generators
37 Video Still Stores
38 Video Monitors
40 Projection (General Category)
41 Film Projectors
42 Slide Projectors
43 Video Projectors
44 Dissolvers
45 Shutter Controls
50 Process Control (General Category)
51 Hydraulic Oil
52 H20
53 CO2
54 Compressed Air
55 Natural Gas
56 Fog
57 Smoke
58 Cracked Haze
60 Pyro (General Category)
61 Fireworks
62 Explosions
63 Flame
64 Smoke pots
7F All-types
_______________________________________________
Jason Cunliffe
oops, I forgot to include links to the I-CUBEX set of MIDI sensors
http://infusionsystems.com/catalog/all_categories.php?osCsid=540ddd8aa0ac63e6fb8054c988afefd4
and to Alcorn McBride Show Control equipment
http://www.alcorn.com/products/showcontrol/index.html
Jason
_______________________________________________
Weyert de Boer
Yes, MAXMSP and PureData are nice applications, do you know how to trigger a AppleScript with MaxMSP? I would like to change a playlist in iTunes when a specific value is set. If you know something may I contact you off list?
_______________________________________________
Weyert de Boer
> In addition to keyboards, drum boxes, synths, FX devices and PCs,
> there is a valuable world of MIDI control surfaces which include
> Mixers. The control surfaces are rally a subset of the “industrial”
> uses of MIDI which fall under the rubrik “MIDI SHOW CONTROL” used by
> major
Keyboards are very nice for re-using in physical interfaces, earlier today I mentioned a music painting device which was based on the electronics of an old keyboard.The keyboard only needs to have a circuit board instead of those plastic ones.
_______________________________________________
Jason Cunliffe
> Yes, MAXMSP and PureData are nice applications, do you know how to
> trigger a AppleScript with MaxMSP? I would like to change a playlist in
> iTunes when a specific value is set. If you know something may I contact
> you off list?
Sorry I am not up on Applescript at all…
Try contracting some of these people:
http://www.synthesisters.com/hypermail/max-msp/Sep05/30402.html
good luck,
Jason
_______________________________________________
Weyert de Boer
In combination of a MIDI converter thingy you can control stuff. At a seminar a group used a gypsy suite as the interface of a game. If you did specific gestures with your arms and/or legs. A specific MIDI command got send out to MaxMSP from MotionBuilder. This would then control create the appropriate sounds such as rain. Really fun.
_______________________________________________
Martin Wood
This is another side which would be great to open up.
It would be amazing to integrate flash with MIDI controllers and audio software. I’ve thought of many interfaces to control audio which would be a lot easier to create in flash.
Also i can imagine some amazing interactive work combining flash and the jazz mutant lemur.
mmm, the lemur…
http://www.jazzmutant.com/lemur_overview.php
_______________________________________________
hank williams
Of course, in thinking about this, with the new binary socket functionality, midi control could all be done by just writing a simple localhost to midi gateway. This would be pretty simple to do, and is probably what is needed anyway to deal with different drivers etc. And it makes total sense that if you want to control some local hardware that you need to download a piece of software.
problem solved!!
Regards
Hank
_______________________________________________
Martin Wood
Sure, it can already be done without binary sockets using a simple xml socket based server which translates whichever protocol to and from XML, for instance theres already flosc, a flash / java solution for OSC communication, and the same idea can be done for MIDI…but it would just be nice to minimise the number of pieces in the puzzle.
still, my main interest is still in getting better sound in flash itself.
I agree with you in that i see no reason not to allow access to the sound buffer of flash itself. Its no different from writing bitmap data.
thanks,
Martin
_______________________________________________
hank williams
Yeah, I guess you really dont need binary for this although it might be a tad more efficient. But the data is really pretty low bandwidth so a XML socket server would be fine. And you are right it would be a nice to have in flash. Its just that I recently setup a small studio and everything seems to need a driver. I just wonder how comprehensive/seamless a system built into flash could be. I dont know much about how midi works on the pc and whether the OS already has a kind of internal midi bus so that app developers dont need to worry about compatibility. If all that is already there then it could be done in flash wihout any compatibility issues.
Regards
Hank
_______________________________________________
Jason Cunliffe
MIDI bandwidth demands can be very misleading. So routing via XML sockets, Jabber whatever is _great_ for some applications and hopeless real-time performance-wise for others. Just depends..
Consider simple example of even few sliders built in Flash app from which you want to send/receive continuous control messages to some midi device.
XIFF library might come in handy for passing non-real-time Flash MIDI in collaborative apps.
For example remote Flash driven Audio/Video jukebox type instruments remixing hardware playback from Flash-TiVos. This could mean an end to Flash Video horrors and much more like interactive video apps that Director suggested way back when..
Re: interface idea, I encourage everyone to play with KeyKit. It has a brilliant unique modular interface GUI design including a lovely idea of savable ‘pages’. Stuff like that can now finally be done in Flash. without too much grief. [Based on Plan9 I think.]. If anyone has a heads-up on actionscript to do that, I’d love to hear from them.
Keykit has so much to inspire and offer FlashMIDI. Cross platform, programmable free, fun, very different.
KeyKit could make a powerful background midi processor for FlashMIDI uses. For example already includes a neato telnet interface.Nasty timing MIDI timing and routing issues could be handled by KeyKit, Flash could do cool parts of the interface.
oh And let’s not forget GeoMaestro KeyKit extension:
http://www.zogotounga.net/GM/eGM0.html
I believe adding MIDI to the Flash Player could have similar catalytic creative effect to when Atari first built a MIDI port into their computers — opening the floodgates to some of the best original sequencer software like Dr Ts. Commodore Amiga did the same thing when they included Video in gamut. Revolutionary in its day and still some great ideas from that machine have not reappeared .. like multiple multi-tasked ’screens’ you can switch between. Avoided the hideous visual clutter we have since advent of rectangular windows forever. KeyKit “Pages” is actually a pickup on that same interface idea.
Now that PCs and now [again] merging deeper with consumer video appliances in the form of new DVD recorders all with Tivo functions, it seems to me that Flash _should_ be an ideal user interface environment.
- Jason
_______________________________________________
Jason Cunliffe
Hank
Can you expand on that idea please. Sounds interesting…
thanks, Jason
_______________________________________________
Tyler Wright
We built a MIDI sequencer for the PC that talks to Flash through an XMLSocket (though we just send smaller strings, not XML). On both Mac and Windows versions the code to make a sequencer capable of changing instruments, transposing, changing tempo, and indivudual volume of instruments, etc. was less than 70K. There are API’s for each OS that abstract the different hardware and allow for such small implementation.
The great thing about Alexis Isaac’s MIDI Sequencer is that you can build music on the fly based on user interaction. We could have easily streamed MP3’s from the server with a better sound if we wanted to be static. We even looked into building MP3’s on the server dynamically, but the smallest process we could find would be at least 3% processor per song. That sounds small, but it means 33 users geting sound simultaneously is 99% processor. It absolutely has to be on the client. It would be a great solution to avoid requiring an additional download from a less-known source than Macromedia/Adobe and immediately integrate dynamic sound into your applications.
Tyler
_______________________________________________
hank williams
Well, its basically what tyler describes but using binary sockets instead of xml sockets. The binary socket is just more efficient. I am not sure what the issues are in implementing any of this since I am not a midi guru, so I cant provide to much detail. Sounds like tyler knows what he is talking about.
Regards
Hank
_______________________________________________
Tyler Wright
I know people who know what they’re talking about.
I’ve been fortunate enough to work on a project with some amazing developers and experience the difficulty it is to get Flash to communicate with other applications.
Thank you everyone for the great response. I will put together a summary of this conversation and respond or post it to my blog for your approval. I want to make sure it’s a fair and unbias (as much as possible) representation of the developer community before I send it off.
Thanks again!
Tyler
_______________________________________________
Jason Cunliffe
Excellent - Yes do please write up a blogsume.
Thanks Tyler !
Jason
_______________________________________________
Martin Wood
One thing that i think could be added to the Sound class now which would help in creating audio applications in flash is to have a ‘polyphony’ setting.
if i make a new sound object and want to control how many voices it uses it would be great to just say :
var snd:Sound = new Sound();
snd.polyphony = 1;
and know that every time i call play it will automatically cut the last playing sample, rather than having to call stop.
Similarly i could use a higher polyphony to control how many channels that sound object can use before it starts to re-use old ones when play is called.
The problem is that when too many samples are playing the whole audio system just shuts down.
Also what would be handy is to get a report on how many channels are in use and how many are spare, maybe a couple of static functions
Sound.channelsInUse and Sound.availableChannels
so at any time i can find out :
var spare:Number = Sound.availableChannels - Sound.channelsInUse;
and limit the creation of sound channels to have certain polyphony so i dont overload the system.
just some more ideas that would help in making audio applications
thanks,
Martin
_______________________________________________
Conrad Engelbrecht
hi tyler
not enough time now to read all the posts on midi in flash. but i know i would like this. i will also try to spread the word. am i right in thinking you were going to start a petition? if so i would appreciate it if you could send the link. as flash 8.5 player is still in alpha, there is an outside chance they might listen…
best
conrad
_______________________________________________
Andy Weisner
I dunno if this helps anyone but I figured that with all the many responses to the MIDI thread this might be interesting for some..and no one mentioned it yet. I do use flash and actionscript with MIDI and sound input control on a regular basis nowadays and the combination rocks indeed.
OK its not for online use and its not meant strictly for music but there exists a software tool which enables MIDI and sound levels detection in flash movies now , while wwe wait for MIDI in flashplayer.
Beside being a flash developer Im also a VJ (visual jockey-live visual projections at music events) and I found out a while ago that one of the visuals .avi loop mix VJ tools I use namely RESOLUME ( can download trial at http://www.resolume.com) has not only swf support, like many other VJ software I use which support flash through quicktime which means only flash 5 actionscripts or timeline movies, but Resolume seems to support flash in a different way (???) and renders up to 3 fullscreen 30fps flash movies (or avi Movies of course) in realtime with additional live FX with directX at the same time,mixed as layers It lets me use all swf movies with full actionscript and flash features support up to flash 8. As Resolume also sends continously MIDI controllers and keys as variables to flash , as well as about 40 variables for sound band levels and a stream of strings , all which updates some named flash variables and can be used with actionscript to synchronize and control a flash movie with MIDI controllers and/or music.
With one line of code one can synchronize for example the size,position or blur filter of a movieclip to the bass level or to a midi knob.
Its like a mixer and remote control for flash and can be used great for live performaces and advanced presentations.
definetly opens some new possibilities for using flash and actionscript in a creative way.
I find it to be a different but very rewarding use of my flash and actionscript skills outside of the web stuff I do in the agency, and esp. with flash8 with all the filters and blendmodes its really fun now . ..and finally a great use for all this experimental math and physics flash actionscripts which I do love.
Andy Weisner
_______________________________________________
John Dowdell
Tyler Wright wrote:
> This conversation has now been posted to http://www.xtyler.com/code/2
> I’ve posted a summary along with a direct quote of the entire conversation
> which I hope will continue to receive contributions. If everyone approves,
> I’ll send it off to MM, though I know they also keep up in the Flashcoders
> list and have perhaps already most of this.
Thanks. In the recent rush I haven’t been able to read “Who wants MIDI?” much less abstract it.
Were you able to get a sense from the whole list of (a) which specific MIDI (or MIDI-like) features & implementations are most desired; and (b) how much they’d be willing to pay for this in player size?
Do you know what size of MIDI engine would work best for you, for which MIDI abilities, for instance?
> Anyone also know the link the the MM wish list?
Searching “macromedia wish list” pulls up
http://www.macromedia.com/go/wish
I know that MIDI or Beatnik or other audio-via-instructions has been on the wishlist for awhile, although specific wishes vary in the amount of the General MIDI spec they need, and wishes vary in how much of a player download effect they’d accept.
> I’d like to take a poll. Do you think MIDI should be included
> in the Flash Player? Why or why not? I want both votes and
> opinions as I’ll organize the results and send them off to
> “Adobe, formerly known as Macromedia”. Please respond with
> some sort of opinion whether it’s pro or con.
Hmm, that sounds sort of mom-and-apple-pie… who would *not* want a new ability? The big questions are precisely which abilities under that general “midi” label most people actually want most, and what they’d be willing to pay for it, in terms of slower audience adoption or greater disparity across devices or whatever….?
jd
–
John Dowdell . Adobe Developer Support . San Francisco CA USA
Weblog: http://weblogs.macromedia.com/jd
Aggregator: http://weblogs.macromedia.com/mxna
Technotes: http://www.macromedia.com/support/
_______________________________________________
Stan Vassilev
I’ll give a hint
- Don’t include default General Midi Bank (that’d be at least 1-2 MB to sound reasonable, so it’s out of question, that’s like bundling fonts in the player, pointless), just provide the engine and some sort of instrument/patch format we can load
- This is similar to the MOD/XM “mod players” from the old days, they are basically light music synths that you load with patches and they have some realtime processing of the patch, like layering multiple samples, LFO (low frequency oscilation), ADSR envelope (attack, decay, sustain, release), vibrato/tremolo, velocity, panning.
- A simple MOD/XM engine (or subset) can fit in something like 20kb or less (honestly, I’m talking real implementations here), and with lots of the audio stuff abstracted in these modern OS days you can squeeze even more from it maybe.
Regards, Stan Vassilev
_______________________________________________
Martin Wood
I urge you to get someone to have a good read through the discussion. There was a lot of good input and i think improving the audio capabilities of flash would be a huge step forward for what i consider to be the most neglected aspect of flash.
Or, if all else fails then persuade adobe to buy these guys
http://www.fmod.org/
–
Martin Wood
_______________________________________________
fla coder
xnice!
_______________________________________________
Tyler Wright
Thank you.
I understand that “Who wants (feature)?” is a very general request. In this instance I was probing for feedback on the exact same questions you’re asking, which the developers of Flashcoders gave.
To sum up directly, there were a few areas covered in these conversations:
Developers would like to build MIDI dynamically through low level commands and then play it. If it would bloat the filesize of the Flash Player, then simply add the ability to play a flat MIDI file loaded from a server. Then to be dynamic, allow that same loadMidi method to load from an internal binaryArray as the AS 3 Loader class allows. Then individual applications take on the filesize of sequencing MIDI instead of the player itself. The last wish which I know very little about is to offer an interface to see the MIDI devices on the users OS and to connect and communicate with them (with the users permission, like a webcam). The Flash Player shouldn’t care what the device is, as long the user is notified.
Developers would also love to have some sort of buffer in which they could manipulate sound waves directly (binary data in FP8.5) and have it streamed out to the OS as Flash does with sound files. Once again, just a Loader concept of playing files from a binaryArray seems like it would add very little to the file size. I don’t know the costs of these solutions and I am just one opinion on this list. I didn’t hear anything on the list about a simple api or make it easy to handle these powerful abilities. I personally would rather see the Player size kept to a minimum, even it it meant I had to do a lot of custom development in ActionScript. AS Libraries would be built to make it available for the common man I’m sure. But the abilities have to be there first so users aren’t forced to download/install 3rd party plugins.
Thats my say, for what it’s worth.
Tyler
_______________________________________________
Tyler Wright
Also, am I incorrect in understanding that many hand-held devices with Flash Lite already have MIDI available in some form?
I don’t know what the implemenations are or what they would mean to the common OS. Does anyone have experience with this?
_______________________________________________
John Dowdell
Tyler Wright wrote:
> Also, am I incorrect in understanding that many hand-held devices with Flash > Lite already have MIDI available in some form?
I haven’t checked various pocket devices on MIDI recently, but the problem on desktops has been when you get varying capabilities on various platforms, so that the same MIDI file can sound different on different machines.
jd
_______________________________________________
Weyert de Boer
Hi John,
> I haven’t checked various pocket devices on MIDI recently, but the
> problem on desktops has been when you get varying capabilities on
> various platforms, so that the same MIDI file can sound different on
> different machines.
Yes, but that’s probably because every platform and/or videocard has different standard soundback. Of course this can be a issue, but not when you use MIDI for controlling external devices Anyway it’s a good idea have a look at this, I can’t believe this is a big change within the Flash player. Oh well, I will check it out for yah
Yours,
Weyert de Boer
_______________________________________________
Tyler Wright
I understand that there are various complications, especially when building a cross platform browser plugin like Flash. What else do you need from us to decide if this is a worthy investment? Some of the developers on this list could perhaps build a viable solution for you but in the end it has to be the Flash Player team that makes any sort of commitment. Maybe in a seperate forum some of the more experienced developers (I’m not one of them) could help advise/consult on more in-depth technicalities.
Tyler
_______________________________________________
Jason Cunliffe
John
> Were you able to get a sense from the whole list of (a) which specific
> MIDI (or MIDI-like) features & implementations are most desired; and (b)
> how much they’d be willing to pay for this in player size?
————————————————————————-
*** MIDI in Flash Player needs to implement Midi Show Control.****
————————————————————————-
As follow-up to Tyler’s blogsume and your question, here are the online key resources to implement the above
Even implementing the old spec MSC 1.0 would be a great start for FlashMIDI. That has been superseded by v1.1 and discussion is now underway [October 2005] to design the next generation known as MD-MIDI..
MIDI Industry Investigates Major Update to MIDI Protocol
http://www.midi.org/newsviews/hdmidipr.shtml
Some references
http://en.wikipedia.org/wiki/MIDI_Show_Control#External_links
MSC 1.0
http://www.richmondsounddesign.com/txt/mscspec.txt
MID Manufacturers Association
http://www.midi.org/
see page on complete 1.0 spec
http://www.midi.org/about-midi/specinfo.shtml
Partial list of Sound & Show Control Projects that use MIDI Show Control
http://www.midi.org/about-midi/msc/mscchart.shtml
Finally, I highly recommend Adobe-Macromedia to run any plans for implementing MIDI in the Flash Player past the excellent people who populate the Show Control email forum. It’s low volume very high signal to noise professional discussion [averages around 100 messages per month until you hit a hot topic :-)]
http://groups.yahoo.com/group/show-control/
Closing comment:
I’d argue for implementing MSC and not getting yoo hung up on MIDI sound as much. Focus on how easily MIDI can enable interoperabilty among the growing catalog of so many interestng devices and software. FlashMIDI would provide a valuable missing part of that environment by virtue of easily customizable user interfaces - superb scalable graphics and typogpraphy etc. The world of touch screesn, kiosks handhelds..They all need to interface with eaach other but also to sophisticated external hardware - videowalls, projectors etc.
The data demands of these applicatosin are often quite small. You get a big bang for your buck and less perforamnce issues. That also makes it MSC an excellent candidate for MIDI in the FLash Player v1.
I hope this helps
Thank You.
Best wishes
Jason Cunliffe
_______________________________________________
Tyler Wright
> - Don’t include default General Midi Bank (that’d be at least 1-2 MB to
> sound reasonable, so it’s out of question, that’s like bundling fonts in the
> player, pointless)
What is “default General Midi Bank”? Are you referring to some sort of additional sound set? I would definately agree do not bloat the player with a sound font. However, most soundcards will play MIDI sound straight off of a MIDI file without the need for any additional software. If a “simple MOD/XM engine” were a viable option I’m all for that!
> ————————————————————————-
> *** MIDI in Flash Player needs to implement Midi Show Control.****
> ————————————————————————-
I’d never heard of Show Control before this conversation. Is it something that could be light-weight in implementation? Also, is it something the common user would use? If not or if it’s specific to a vocation then it seems it shouldn’t be included in the Flash Player, but that an independant application be built to interface with/through Flash.
> I’d argue for implementing MSC and not getting yoo hung up on MIDI sound as
> much.
> Focus on how easily MIDI can enable interoperabilty among the growing
> catalog of so many interestng devices and software.
I feel the opposite. Though I think interfacing with other various devices would be very cool, it could add a lot of size to the player for functionality specific to a particular machine/device setup. The ubiquity of the player is in it’s universal behavior. I could go to any computer anywhere in the world and use an application that plays sheet music that you build, or that customizes a soundtrack to a game as it progresses. It would be great to build for devices and I think it should eventually go that route, but it’s the audio that benefits everyone.
Tyler
_______________________________________________