User Limits and Axes' Commanded Positions: Runaway

[scott]

No, I can’t imagine what you could do to produce this behavior, which is why I’m trying to reproduce it as well. I’ve run thousands and thousands of loops. I’m using different threads to load motion and to trigger the TRIGGERED_MODIFY action. I’ll switch to a UserLimit for the TRIGGERED_MODIFY now.

What are you using for your PT motion empty_count? Can you check FramesToExecute() for Axis 0? Your most recent plot looks like it was ~60ms without calling MovePT(). Is it possible the motion supervisor runs out of frames to execute, leaving you in an undefined state?

[todd_mm]

We have emptyCount hardcoded to 10.

I can’t elucidate or justify this decision. As far as I know, we copied it out of an example and haven’t changed it since.

[scott]

Ok that’s good, I just wanted to be sure it wasn’t set to -1 (disabled). At the default sample rate of 1000 Hz, that’s 10ms.

[scott]

Updated my test code to having a UserLimit cause a TRIGGERED_MODIFY here on Axis 1, while 1ms points are streamed to an XYZ MultiAxis (keeping X at 0, Y incrementing, and Z fixed at a large value). So I think I have most of what you described. I have never seen a failure, my X position is always at 0.

[todd_mm]

I’m deliberately keeping all axes away from zero and each other so that anything resembling an assignment to anything that exists will produce a noticeable change.

[scott]

Ok I’m now forcing X to a non-zero value and very different from Z.

[todd_mm]

Here are some more pictures.

The second plot is the new stuff. It breaks apart the status words and tracks them, bit by bit.

Here’s the second one, zoomed in to the instant the command position jumped.

The command position jump happened at sample #2766977.
I don’t see really see anything new. What I do notice is that the Triggered Modify bit went high on all three axes and the multiaxis within the same sample. Also, the estop/error state went high 2ms later on all three axes and the multiaxis.

I was half hoping to see one of those bits come on first, but since everytihng’s running way faster inside the RTOS than 1 KHz, the scope data just can’t illustrate it.

[todd_mm]

@scott, Andrea and I were speculating about the order in which things ought (?) to happen. Should I expect otherwise simultaneous events (like three user limits being triggered by an input) to be processed in:

• axis (ID) order (e.g. 0, 1, 2, …)
• user limit number order
• some other prioritized ordering?

[scott]

yes all the firmware objects are processed in order, starting from 0

[scott]

any idea why you have such a long gap without calling MovePT before the TRIGGERED_MODIFY occurs?

also it appears that you are getting AT_TARGET on Axis 1 before the TRIGGERED_MODIFY which means its commanded motion is complete?

[todd_mm]

One reason is that I added some checks before invoking MovePT() so as not to call it if any of the stopping action bits were set. (This hasn’t always been the case.)

When a probe strikes, we attempt to transition the MultiAxis motion state to STOPPED/IDLE while we clean up. I suspect this accounts for the long gaps around the probe strikes.
So, the AT_TARGET probably indicates that we’ve sent a stop point and nothing’s MOVING any more. We normally send motion every 10ms.

Here’s the same data for one of the other thousands of strikes.

It looks like there’s still a gap preceding the TriggeredModify of maybe 35ms (good scenario) and 66ms (bad scenario).
Nothing I’ve said previously would account for that.

[scott]

On the good strikes, do you ever get the AT_TARGET bit going high before the TRIGGERED_MODIFY?

[todd_mm]

No. Now, I don’t have a lot of data to work with. The scope only captures 10 seconds of data by default (I only just now turned it up to 500 seconds).

Here is the latest test, filtering for AT_TARGET and TRIGGERED_MODIFY

Here’s a closer view of the anomaly.

I’ll check some of my more recent data files to see if the answer to this question is always the same.

UPDATE: I only have one other data file with all the axis status words in it, but it looks just like the one illustrated here. This certainly could be a reliable pattern, @scott

[scott]

Ok, in my testing, I never reach AT_TARGET because I’m always asynchronously triggering the TRIGGERED_MODIFY while the PT motion is running.

So in your case, you must be calling a MovePT(final=true) and the command trajectory completes before the TRIGGERED_MODIFY?

TRIGGERED_MODIFY should not do anything if you trigger it an the Axis/MultiAxis is not in motion.

[todd_mm]

This particular type of test that we’re running is attempting to shake out timing bugs when the probe triggers right at (or very close to) the end of the move.

The commanded stop is something we do before “disarming” the probe (a form of cleanup). We put the multi axis in a STOPPED/IDLE state (and wait for it before proceeding) so that we can be certain nothing is moving while we’re changing internal states and sending commands to the drives. That procedure is also inhibiting further calls to MovePT() until cleanup is finished.

Here’s the anomaly again, but only looking at the Axes’ STATUS.AT_TARGET, MultiAxis.MOTION_STATUS.TriggeredModify, and Probe input, armed/success bits.

It looks like what you’ve described (slightly augmented by me) is what’s happening here.

Supposing that the probe triggers just after the end of the commanded decel, but before we’ve disabled the user limit.
What would you expect to happen then?

[scott]

Well I don’t expect the behavior you’re seeing. I will say TRIGGERED_MODIFY is the most unusual of actions because it does not induce any new states (STOPPING, STOPPING_ERROR, ERROR, etc.). It is also the newest of all the actions, so it doesn’t have as much mileage.

I’ll modify my tests to do a short PT move (instead of continuous), to be more consistent with what you’re seeing.

In the meantime, if you are needing to get the feature release, I would suggest considering an action other than TRIGGERED_MODIFY for this case.

[todd_mm]

I’ll test using RSIActionE_STOP_MODIFY as the user limit action.

[scott]

Hey, I’ve reworked my testing app and started calling Y.TriggeredModify() every ~50ms in one thread while another thread is doing PT motion and I’ve been able to reproduce an unexpected command position jump on X and sometimes Y. No such jumps with Y.EStopModify(). I’ll dig in to see how I can make it more reproducible to be able to see what’s going on. More later.

[scott]

Calling MultiAxis.TriggeredModify in the same way does not reproduce the issue, so it appears to be related to the propagation of a TRIGGERED_MODIFY action from an Axis to a MultiAxis.

[todd_mm]

I ran a couple of tests with RSIActionE_STOP_MODIFY as the action of the one user limit. I’m seeing the same basic behavior when it fails.

I’m fairly confident that I’m configuring the user limit properly. Here’s a screen shot of RapidSetup after the last failure. You may see some problem I don’t.

Anyway, I half expected different bits to be turned on in the motion status words of the axes/multiaxis, but it looks very much the same.

The commanded decel/stop finishes just after the user limit input triggers (goes low). In this case, axis 0 was commanded to the axis 2 position @2757560, 1 ms after the error bits went high.