Stop Actions and Multiple MultiAxes

[todd_mm]

[RMP 10.4.3] (We’ll be upgrading soon, though this behavior is probably the same.)

Multiple Coordinated Motion Groups

I am attempting to run multiple motion coordinated motion groups at once. I have, say, two MultiAxis objects, each containing a mutually exclusive list of axes. I am attempting to command motion on each of them via MovePT(...) (just like I do when there’s only one coordinated motion group).

Probing, using User Limits with Output Capture and Actions

Now, I’m attempting to probe on each of these motion groups simultaneously and independently. This means that I have a user limit for each axis that captures the position when a particular input goes high. For each MultiAxis, one of the user limits also has an E-Stop action.

The Problem

When I probe with only one of these motion groups at a time, things work fine.

However, when I probe with both of these two motion groups simultaneously, things work not-so-fine. I get a lot of halting motion that shouldn’t be there. I captured the positions and velocities for some of the axes involved, and it looks as if there are infrequent occurrences of adjacent positions that are identical (i.e. velocity is zero) in the middle of an ordinary (nonzero velocity) move.

Identifying the Culprit

It is difficult to prove exactly who is at fault here because I don’t currently have a means of saving every scrap of data that runs through the process. Here are some likely culprits.

• the motion/trajectory planner
  • If this were the culprit, I would expect some of these errors to occur when I only probed with one motion group.
  • The trajectory planners for each motion group are isolated (mostly, as far as I can prove) and do not care what the other is doing.
• the interlocutor between the trajectory planner and RMP
  • There’s room for error here because I maintain this code.
  • It seems unlikely that I would re-send a single point (as opposed to re-sending a batch of points) because I usually send a burst of points at a time.
    • If I were doing something like this, I would expect it to be as a result of MovePT() failing or being in a state when it would fail if I called it.
• RMP
  • This is the component I have the least knowledge of.
  • There could very well be some config/setup that I’m doing incorrectly.
  • The only way to prove (to me) that it’s RMP is to prove that it isn’t the other two.

Patterns of Behavior

I captured a bunch of data in the firmware (using recorders) hoping to find patterns in the circumstances when I see these Single-Instance-Zero-Velocity commands. Axis0 is in one motion group. Axis3 is in another.

These two graphs illustrate what seems like a strong pattern. Look for these features.

• The commanded velocity (“Vel/Accel”) will have occasional blips to zero (solid, light red/blue lines).
  • The blue lines are instances when Axis3 halts. The solid red lines are when Axis0 halts.
• When the user limit fires and the axis is commanded to a stop by RMP, the axis firmware (“MultiAxis.FW”) status bits seem to indicate it.
  • The dashed lines indicate when the input (that triggers the user limit) activates.

Now, notice that when there’s motion blip for one axis, if you look at the corresponding red/blue line in the axis firmware status for the other axis, the other axis is almost always reacting to a user limit (e-stop) action. (I’m sorry; the plots are very busy, but I’ve attempted to distill the information down to what we might need for further discussion.)

Is the Gun responsible or the person holding the gun?

My first question is: Am I supposed to do anything special to prevent this kind of behavior?

I mean, is this behavior what you’d expect, given certain configuration/setup? Is there something I could be doing that would cause a user limit action targeting axes in one motion group to affect axes in a different motion group? (BTW, I also captured the MultiAxis firmware status, and it looks nearly identical to the Axis firmware status.)

You guys sometimes inform me of setup steps that I didn’t do when I want non-default behavior, so I guess I’m asking if there is any such setting that would affect this that already is in my power to control?

[todd_mm]

Here’s a closer look at some of these blips.

Axis0

Axis3

Note: the blue lines are when Axis3 blips. The red lines are when Axis0 blips.

[jacob]

Hi @todd_mm ,

Can you show me some scopes of the same with just a single probe for comparison?

I can’t think of a reason why an RMP process that works with 1 probe would stop working when a 2nd hits. We run these things sequentially and if they are MultiAxis groups are isolated they shouldn’t affect one another. I would focus my attention on the interrupt code that is looking for the probe. What sort of process do you kick off on probe/userlimit interrupt? How sure you that the whole process is isolated from other events that may take before the first is fully resolved.

[todd_mm]

Here is the same (kind) of data for when I probe with one motion group at a time.

[todd_mm]

@jacob

How sure [are] you that the whole process is isolated from other events that may take before the first is fully resolved[?]

This is a fair question. Things can get complicated (at least, too complicated to post the code and hope it’s comprehensible), so I’ll try to provide an honest simplification. (-8

When it’s time to clean up from a probe operation, I attempt to transition the multi axis to IDLE/STOPPED/ERROR. The goal of this is to ensure that nothing is happening while we do some recalculation and ensure that the commanded position becomes the actual position (since some of the planned motion may not have been executed—due to the probe strike). I can confirm that many/most of the 0-velocity blips I see happen during this operation.

During this time, one thread signals the motion FSM thread (the one that invokes MovePT(...)) to stop streaming points (nicely), then waits for the MultiAxis state to change to one of the target states.

When the motion FSM thread receives the signal, it sends a stop point and waits for the MultiAxis state to change. (This is greatly simplified, but describes what I’m doing WRT the RapidCode API.)

There is interplay between the various threads that are controlling the respective MultiAxis objects, but most of it is about timing. I’m trying to operate these objects in parallel and simultaneously. There are some operations that happen before and after the motion FSM operation loop that are not parallelized (e.g. synchronizing I/O points between my application and RMP). I do report current motor positions back to my application—and this could conceivably have some bearing on the motion data sent to RMP in the next iteration. I don’t currently have that data, but I’ll look into getting it.

[todd_mm]

@todd_mm

I don’t currently have that data, but I’ll look into getting it.

So, I did another run, and now I have some evidence to suggest that I’m commanding this problem. I’ll need to dig in further. Thanks for chatting with me about it.