Error 15 MIDI Input Buffer Overflow

Bome Products
1

I'm done developing my MT project, at least, all features work. I still have a problem which I've mentioned before. I thought I would leave it for last since it has been so hard to pin down. It makes my project unusable for performance, which is pretty devastating to me since I've spent quite a few years getting it to this point.

I'm wondering if there is any way for me to "trap" the offending inputs as they are received and execute code in MT. The log window (if it is turned on) shows the message that it's truncated, so I can't tell from that, what has happened. Maybe some software is available to help with that?

In any case, I would be glad to send you my code so that you could look it over. There are more than 400 presets and over 60,000 lines of code and comments when exported as text, but I've disabled all presets that aren't involved in the problem, leaving less than 20 presets in order to make inspecting the code easier.

I can describe the problem generally this way: It's intermittent, in the sense that I can do the same things successfully with my controllers many times over, but then out of the blue, I get the error. 15 message The offending controller then neither sends or receives any MIDI to/from MT, but turning it off and then on restores the connection (with the MT message "MIDI Buffer OK".

Until I do this, I'm not even able to exit MT completely. If I exit, and try to launch MT again, I get messages telling me that the ports for my controllers are already being used. I notice then that though MT doesn't appear in the Apps section of Task Manager, it does appear in the Background processes section and I can't end the task. I have to restart my computer. I can usually force the error by grabbing 8 or 9 faders on my AKAI APC Mini and scrubbing them back and forth as rapidly as I can. Sometimes the error appears fairly quickly - after maybe 2 or 3 repetitions, but sometimes I can keep scrubbing for 10, 15 or more repetitions before I get the error. This is especially true if I've deactivated some of my presets. I've tried to isolate the problem by disabling presets that I suspect could be causing the problem, but even with only quite essential presets still active, I'm still having the error.

I think the problem may be that for most of the inputs that are receiving faders, any one message is being received by many presets and many translators within them. That is because fader movement controls several aspects of my program.

At any rate, I'm stumped and frustrated that I can't really use my program to play for people. any help would be much appreciated.

Thanks,

Gabriel

2

Hi Gabriel,

 

From what you said, I assume you have pinned this down to the APC mini. Is this the device you need to power cycle to correct the problem? If for some reason, you are not seeing it in the log file, and you have a Windows machine (which I’m sure you do). Maybe you can try and capture the point of error in MIDI OX?

I’m happy to take a quick look at the project file if you either post it here or email it to me. I have an APC MINI so maybe I can duplicate the error. I suspect you have a lot of MIDI traffic with such a complex file so may not be able to duplicate your situation however without artificially generating some more MIDI traffic. So a sense of what MIDI traffic is occurring when you are having the problem would be helpful. Is there also any or much MIDIBuddy traffic when this occurs?

Regards,

Steve

 

3

Hi Steve,

Yes, the APC Mini’s cause the error, but I think any controller with several faders or pots mapped to my project would cause the error as well. And, yes, there’s a lot of traffic, but you’ll be able to see it for yourself since you have an APC Mini. Only one is necessary 🙁

I’ll send you a copy of my program which has all but a few presets disabled. This serves to highlight the offending sections of the code. I’ll also send a copy of the Live set which is similarly simplified to make it easier to analyse.

When I last posted about this problem, Florian had mentioned the possibility of increasing the size of the input buffer. I believe I wouldn’t have a problem with increased response time given the way I trigger samples, so I guess that might be a final option to consider if there’s not actually a fixable bug in my code.

Thanks,

Gabriel

4

OK, Gabriel. I will watch for your email.

Steve

5

 

 

 

Hi Steve,

Here are the simplified files for Live and MT Pro. I’ve deleted all MT Pro code that doesn’t pertain to the problem I’m having- so now you only need to regard a handful of Presets instead of 400.

In Live, I’ve also deleted all but 3 tracks and 8 scenes, all returns, and all effects so that you can load the file.

I’ll check to see if I send Fader info to APC’s anywhere. I think I only send back LED on/off messages, but sometimes there are a lot of them. For two reasons….

1) I sometimes need to turn on only one of the 8 LEDs for the round buttons. Only one should be lit at a time to tell me which row has an active clip. I don’t keep track of which LED is on, so I turn them all off and then turn on the one I want lit. Of course I could have used the LEDs under the rectangular buttons to tell me which clip is playing, but I use them to tell me if I’m using Reverb or Delay or Low End Boost or EQ in that track, or one of several other effects.
2) I think I may be sending the same message repeatedly in response to the fader movement. Now, I can’t remember if I limit that. I used to do it with a timer that suppressed sending of the messages – limiting them to once in 500ms, I think. Then I tried another method, but ultimately, I’m not sure if I do that anymore.
Looks like the Live file is still to big to attach. I’ll send it to you directly.
Thanks,
Gabriel


Attachments:
1524680953426_Simplified-Test-for-MIDI-Input-Buffer-Overflow.als
6

OK, I was able to do some testing. Here is what I did.

  1. Moved the faders of APC MINI really fast like you said and yes it hung after a few moments.
  2. Used Bome SendSX to send button data really fast to APC MINI. No problem at full speed with over 64K of data.
  3. Did the same above with SendSX execept used fader data instead of button data. No problem
  4. Opened MIDI OX. Set up loopback from APC MINI to APC MINI with monitoring on. Moved faders fast for about 30 seconds. It hung. Was not able to close MIDI OX (just like MT Pro) until after unplugging and re-plugging in the APC MINI.

So my deduction is that there is a problem in the APC MINI or Windows 10 driver when quickly both sending and receiving same fader data over a period of time over a period of time. It appears to crash the APC40 and hang the Windows driver. Of course when you unplug the device, the Windows driver deletes it which clears the problem and you can close the application. Then you plug it back in and everything is back to normal.

My recommendation, ensure that fader data is never sent to the APC MINI. I don’t think button data would be a problem but then again, I cannot push buttons on the APC MINI fast enough to determine if a similar problem would happen with buttons if I could.

I hope this helps!

 

Steve

 

7

Thanks for this analysis. I think you’re telling me this:
You set up Midiox to receive data from the APC and then to send it back into the APC…. a loop.

And that it’s not that MT Pro is receiving data too fast from the faders, but, instead, that MT Pro is sending too much data for the APC to digest so quickly. Or maybe you’re saying that it takes both sending and receiving of that data to crash the APC.

I didn’t think I was ever sending raw fader messages back to the APC. Are you seeing CC messages going to the APC when running my project, or just Note On/Note Off messages intended to turn the LEDs on or off? If there are CC messages being sent to the APC, I wouldn’t have intended that and should be able to fix it.

If it’s note on/off messages, I should be able to limit the flow one way or another.
Thanks,
Gabriel

8

Steve,

I disabled all outputs to the APC. Now I’m sending MIDI only to Live and to instance 2 of MT Pro (which isn’t running) I’m still sending to instance 2 to see if that causes problems.

Here’s a potential problem. In order to throttle sends to instance 2 (I only need to send one MIDI message to instance 2 for a fader movement) I use a timer that blocks transmission until the timer expires. What I notice is that when I move a volume fader through its entire range, the timer gets started about 125 times. Is that potentially gumming up the works? I’ll eliminate it and also the sends to Instance 2 to see what happens, tho I think I’ve done that test once already.

Thanks,

Gabriel

9

Hi fingerlight, chiming in… at first let me tell you how much I understand how annoying this must be for you. Steve’s isolated test might indeed indicate that the APC driver has a problem with too much data being sent to the APC when it is also receiving a lot. You could enable ONLY the MIDI IN and MIDI OUT checkboxes in the Log Window and see what’s being logged there (that way, the log shouldn’t get truncated so soon).
Are you using version 1.8.2 or 1.8.3 beta?

10

The reason I used MIDI OX is to try and determine if it a MT Pro problem or not. Duplicating the problem in MIDI OX sort of confirms the problem must lie somewhere else.

For the test with Ableton Live, the last logged data was note data, however maybe the issue is that when APC MINI is sending fader data fast while it is also receiving any data, it would hang. I can’t actually send anything but fader data that fast with the APC MINI. So maybe that is it. I don’t know if it is Windows driver problem or APC MINI problem or combination of both. You say it only hangs with the APC MINI so I have to lean in that direction.

11

Hi Florian,
Thanks for your input, and for the commiseration.
I’m using 1.82
I believe that the problem is not generated by sending data to the APC. I simplified the code so that no MIDI is ever sent to the APC and still got the error message.

I thought I had posted a simplified version of my project, but now see that I sent the Live file instead, so I’ll send the simplified project file now. It has only the pertinent sections of code and no MIDI sent to the APC.
Also, I disabled the timer which throttles messages sent to a second instance of MT Pro. I think I just discarded the post which describes how I’m using the timer, so I’ll explain again.
First – I’m only running one instance of MT Pro right now, so that’s not a consideration. When I do use the second instance, I need to send information from instance 1 to instance 2 whenever a fader is moved, I just need to send it once, but the message would get sent many many times as the fader continues to move, so I instituted a timer that throttles the send until the timer expires. The problem is that the timer gets launched about 125 times as the fader moves through its entire range. Can launching a timer so often create a problem. The timer has a period of 500ms I think.


Attachments:
1524693210260_Simplified-Test-for-MIDI-Input-Buffer-Overflow-WITH-NO-midi-to-APC.bmtp
12

Hi Steve,
The version of my project without MIDI data being sent to the APC Mini still has the problem. I will say that removing the preset which starts a timer whenever a volume fader is moved does seem to make it harder to produce the error, though I can still do it.
So, is it a bad idea to repeatedly start a 500 ms timer… 125 times in maybe 1/2 second?
And again, if I’m never sending data to the APC, , it seems that’s not the root cause anyway.
Thanks,
Gabriel

13

I’ll try to test using the Novation Zero SL to see if it has the same problem.
BTW, just letting you know that I sent a simplified MT Pro file to you just now. It might have gotten lost in the flurry of messages.

14

I don’t think changing the timer will do anything. It is strange you are still having the problem. What is the last MIDI message the log shows when it fails and to what device was it sent?

15

I just sent 19205 bytes of button data at full speed several times using Bome SendSX with no problem. Then I tried it while moving the faders at the same time and the port hung. I could not close SendSX until I unplugged and re-plugged in the APC MINI.
I think I test it with a different controller and see if the other one has the same problem. I will test with APC40 MK2.
That might narrow it down to the driver.

16

The problem does not duplicate when sending massive button data to APC40 MK2 or my Worlde Easycontrol.9.
Did you say you tried turning all output off going to the APC MINI to see if the problem still occurs?

17

Yes… I did turn off all output going to the APC Mini. The latest “Simplified” bmtp file I sent disables all sends to the Mini. I still get the problem. You’ll find that file attached to my answer to Florian’s post. It’s called “Simplified Test for MIDI Input Buffer Overflow WITH NO midi to APC.bmtp”

I’m wondering what output you ever see going back to the APC. Do you see anything other than 90 xx yy (LED on/off)? So it seems that sending massive amounts of button data to the MINI hangs it if the faders are moving at the same time, but that this doesn’t hang the APC40.

I also did a test with another controller… the Novation Zero SL. My heart sank when I saw that I couldn’t make it hang. That’s good news as far as that controller goes, but not so good in terms of my setup which is largely built around the Mini’s They’re the only controllers with the particular configuration of buttons and faders I needed. So I have 8 of them at this point. I’m inextricably bound to using them.

Still another test I did was to simply to hang the Mini by moving its faders (a lot) while sending the output to Midiox (but not returning any MIDI to the Mini. It doesn’t hang, as you also pointed out with a similar test.)

Getting AKAI to do anything about the controllers is probably very unlikely. I tried to reach them about another issue with the Mini’s quite a while ago and could never get anything like real attention from them.

It seems strange, though that without any MIDI going to the MINI, it hangs under some conditions (lots of fader movement). I’m going to recheck this to see that there’s absolutely nothing going to the MINI with the latest “Simplified” bmtp.

If the problem really seems to be related to simultaneous input and output from the Mini, perhaps you’ll be willing to help me revamp my code so that simultaneous i/o is somehow prevented or at least throttled to the point that the error is eliminated… but let’s see what further testing comes up with.
Thanks,
Gabriel

18

Tested with the following other controllers (while moving faders or knobs)
Novation Launchkey MINI – OK
Samson Conspiracy – Hang when moving faders but not when leaving static
Novation Launch Control – OK
Novation Launch Control XL – OK

That is all I have to test with that has faders or knobs.

Yes, I’m only sending notes

19

Hi Steve,

You’ve helped a lot by pinning down the cause of the error condition to the APC Mini driver (or perhaps the Windows driver). I think you’re exactly right.

What has been confusing is that sometimes the first swipe of a fader or two can cause the error, but other times I can furiously “scrub” 8 faders for many repetitions before the error occurs. I think I now understand the problem better. I think it’s purely by chance that MT sends a MIDI message to the APC at the exact same moment as the APC is outputting a message to MT Pro. When an APC volume fader is swept through its entire range:

  • The APC sends 128 MIDI CC values (0->127) for a full sweep of the fader.
  • MT responds, among other messages, with the following MIDI string every time it receives one of these values, so, for a full sweep it sends the string 128 times!!!:

90 52 00 90 53 00 90 54 00 90 55 00 90 56 00 90 57 00 90 58 00 90 59 00 90 ss 01

Where ss = a number between 52 and 59 (hex)

You might ask why I’m turning off all LEDs and then turning on just one of them. It’s clumsy, but explaining why I do that would be a distraction here. It has to do with the fact that I don’t have anymore global variables to store the LED states. But I won’t discuss this for now.

There’s a pretty good chance that given this amount of traffic, eventually there will be the type of collision that freezes the APC driver, disabling any communication with MT. I’m not sure just what nuances of movement cause the timing that produces a collision. Maybe it’s just luck combined with some slight variations in the speed at which messages are flying back and forth….something like that.

Solution 1: A solution might be to somehow block MT from sending MIDI to the APC while it’s receiving MIDI from the APC. I think that would be tricky but may be possible. Do you have an idea how this could be coded?

Solution 2: Another solution might be to reserve one particular button for each track (I have 24 buttons per track that launch clips – I could spare one of them and still have 23). Instead of launching a clip, it would instruct MT pro to send the stream described above (8 LEDs off and then one of them on) to the APC. I’ve tried doing this with one track and it seems to prevent the error. I’ll need to test more before I commit to changing my code for all tracks. I think I could also figure out a way to reduce the MIDI stream sent to the APC by turning off only the LED that’s on before turning on the new one, but I need more globals to do that, and just don’t have them.

Can you think of another way to prevent simultaneous in/out MIDI to and from the APC? Or some other unrelated solution?

In any case, I’m feeling much better now since it feels as if a solution might be at hand.

Thanks,

Gabriel

20

Hi Gabriel,

Solution 1 is probably your best bet but you would need another global variable. You simply set the variable to 1 and put a rule in each translator bound for the APC MINI as follows:

if ga==1 the exit rules, skip outgoing action
When the APC MINI is ready for more rules you set the global variable (ga in this example) back to 0 and then
the translator will fire on the next input.

As far as global variables are concerned. How many states are you tracking with each button. Every global variable is a 32 bit signed integer so there may be a bit mapping technique you can use to take more advantage or the 32 to bits available for a single global variable. I can track 64 button on/off states with just 2 global variables, each button representing a single bit of the global variable. If you want to track 3-4 states it would take 2 bits. For 5-8 states (colors) it would take 3 bits. So with a single variable with 8 color states, you could track 10 buttons with a single variable. If interested, I can send you an example of how this is done,

The other way you could disable input without using a global variable is to put all APC MINI incoming messages under a separate preset, and the a translator to disable/enable that preset as needed. If you need to enable/disable multiple presets, you would need to use a single timer event that fire multiple translator, each disabling/enabling a different preset.

We may also need to look at the APC MINI manual. I know on the Novation products you can turn on/off LED’s with Sysex messages instead of note messages. If APC MINI supports this, maybe it would be more reliable, than sending multiple note messages. I’m not sure if APC MINI supports this however.

Steve