MultiAxis: Timeout in ClearFaults()

What conditions will cause MultiAxis::ClearFaults() to raise a timeout exception?

Timeout :: {motion.c, line 8915}  (Error 10)
    (RSI::RapidCode::Impl::MultiAxis::ClearFaults) (Object 4)
    (File ..\..\source\multiaxis.cpp) (Line 1560)
    (Version 8.1.6 for 04.04.02.RMP)

RapidSetup reports that the MultiAxis state is “ERROR” and each of the mapped axes has out-of-frames as the error source.

It takes more than 512 Samples for the Motion Supervisor to hit IDLE.

So, I can only clear faults when the object is idle?

Don’t call ClearFaults while Moving, but you can in other states. This timeout was an example where it failed to transition from some other state to IDLE.

This raises an interesting question (for me). I’m pretty sure we’re just calling some/all of the reset-style API functions, hoping they magically work, without really understanding when or why to call them.

For example, when do I actually want to ClearFaults()? We have been doing it as a way to put the object back into a known, good state. This doesn’t seem to be its purpose.

When do want to I Abort() (or one of its cousins, e.g. EStopAbort())? I see a description of what the stopping actions do, but that doesn’t help me understand when I’d want to do it.

In the end, I need to know how to go from whatever state the Axis/MultiAxis is in to a good state. If it’s more complicated than Abort();ClearFaults(); then I can do that, but I need to know how to accomplish it. Is there some documentation to this effect? For example, how do I go from a STOPPED state to STREAMING? Or ERROR to STREAMING? Is there any sort of state diagram or anything that would help me implement this intelligently?

ClearFaults() is designed to remove errors and restore to an idle ready state. Its intended use case is after motion/events have completed. It resets the Motion Supervisor to prepare for new motion. It clears any errors on the associated object. It sets the ClearFaults bit on the Controlword of the associated node for several cycles. It is pretty effective, but can fail if something is very wrong (such as the timeout above) or simply not be effective if an error source is re-triggering.

— We’ll see about getting the following into a topic. —

There are 5 different common ways to handle the need to end motion. Stop(), EStop(), TriggeredModify(), EStopModify(), and Abort(). Deciding which to use should be based on your understanding of your system.

  • How much mass are you moving and at what speeds?
  • Do have external forces to consider such as gravity (a vertical axis)?
  • Is there a possibility of someone or something being pinned by moving parts of the system?

Lets consider our tools:

  • Stop() - Time based. Defaults to half a second. Doesn’t necessarily indicate an error. Can be resumed. It is unique in that you are allowed to continue the motion which was interrupted. You can think of it like a pause. It is internally using Feedrate to change the processing of the motion profile to zero.
  • EStop() - Time based. Defaults to 50ms. Indicates an error. This is our default action for most limits.
  • TriggeredModify() - Deceleration based. Good for stopping as quickly as possible for variable speeds. You will want to change the default Deceleration. It should be set based on how much mass you are moving and how quickly you can safely bring it to rest.
  • EStopModify() - Deceleration based. Same as above with an implied error and likely faster deceleration. Change the default deceleration which is likely completely inappropriate for your system.
  • Abort() - Unlike the rest of the actions, this disables the Axis. It creates an uncontrolled stop based on mass/inertia. If there is a possibility of pinning someone with machine, you likely want to use this. If outside forces such as gravity would cause a crash, you likely don’t want to use this.
  • EStopAbort() & EStopModifyAbort() - This is simply a combination of the above actions. It does a controlled stop as described above, then Aborts to leave the system disabled.

Which you want to use will depend on an understanding of your application. Consider your largest loads at your fastest speeds. Are you initiating the action through code or a limit on the system? Remember you can customize the speeds and decelerations of these events. You could even set them to different values for different loads if significantly different.

You could also build a new motion profile for whatever custom behavior you want to see and override existing motion.

Thanks for this detailed info. I assume that I would only use one of these stopping actions if I wanted to transition from STREAMING to STOPPED. Is that correct?
The next thing I need to know is how to transition between the other Axis/MultiAxis states.

These are the states I know about:

  • RSIStateIDLE
  • RSIStateMOVING
  • RSIStateSTOPPING
  • RSIStateSTOPPED
  • RSIStateSTOPPING_ERROR
  • RSIStateERROR

Are there preconditions or postconditions for these states that are always true? Axis::StateGet() mentions them but doesn’t give more detail. The Stopping Actions overview page mentions some states as targets for various stopping actions, but doesn’t completely answer my question about how to properly change states. How do I transition between them?

In the immediate case I’m working on, I need to go from STOPPED (or ERROR) to MOVING. I had been clearing faults as a routine step to prepare the MultiAxis for motion, but that’s causing the timeout. Do I need to first get it into IDLE?

Is there anything like a state transition diagram that would help me understand how to manage the motion supervisor states better?

Which samples are you referring to here? EtherCAT samples (from the logical network ring) or motion samples (via Move*())? (I would guess the former, but I’d like to make sure I understand.) If I wanted to get into IDLE, would I just wait (approximately) 512 milliseconds?

We don’t have any diagrams created.

3 states are temporary states.

  • MOVING will only remain so during a motion profile and will end after the motion profile is complete and settling criteria has been met. (Note Steaming motion and Velocity moves will keep you in MOVING indefinitely.)
  • STOPPING should only last StopTime. (Triggered Modify leaves you in the MOVING state.)
  • STOPPING_ERROR should only last EStopTime or for the EStopModify Deceleration period.
    I recommend letting these complete before doing other actions.

Stable States - IDLE, STOPPED, ERROR

  • Idle is just the default/neutral/ready state.
  • STOPPED will allow you to RESUME() the motion or start new moves. I recommend ClearFaults if you don’t intend to resume. You can’t do some things like PositionSets while stopped.
  • ERROR indicates you need to ClearFaults() or better yet figure out why you are in an error state with SourceGet() and respond intelligently to error.

To get from STOPPED/ERROR to MOVING, consider the following helper.

public static void ReadyAxis(Axis axis)
{
    //Ensure Axis isn't moving.
    switch (axis.StateGet())
    {
        case RSIState.RSIStateSTOPPING:
        case RSIState.RSIStateSTOPPING_ERROR:
            axis.MotionDoneWait(Convert.ToInt32(axis.StopTimeGet() * 1000));                            // Wait Stop Time and check again.
            RSIState state = axis.StateGet();
            if ((state == RSIState.RSIStateSTOPPING_ERROR) || (state == RSIState.RSIStateSTOPPING))
            {
                axis.Abort();
            }
            break;
        case RSIState.RSIStateMOVING:
            axis.Stop();
            axis.MotionDoneWait(Convert.ToInt32(axis.StopTimeGet() * 1000));
            break;
        case RSIState.RSIStateSTOPPED:
        case RSIState.RSIStateERROR:
        case RSIState.RSIStateIDLE:
        default:
            break;
    }

    ClearFaults(axis);                  // Clear Axis Faults.

    if (!axis.AmpEnableGet())           // Enable Axis if needed.
        axis.AmpEnableSet(true);
}

This function doesn’t care what is going on. It will interrupt motion, wait for actions to complete, Abort, and Enable as appropriate. You may want to instead report states/sources to user, intelligently decide if a motion should be aborted, handle multiaxis objects, etc.

1 Like

We are always going to match the Motion Controller Sample Rate and the EtherCATNetwork cycle time. In our documentation, samples should always refer to MotionController samples. The 512 samples is a TIMEOUT value indicating there was an error. It should normally takes much much fewer samples to successfully transition to IDLE as far as states are concerned. Ideally, I’d like to see it in 1 sample but it can often take a few samples.

If you see ClearFaults taking a long time, something is wrong. I just don’t have enough information to determine what might be wrong in the initial posting here.

Jacob already gave you a pretty in depth response but if you are still interested in a state diagram we recently added one to our topics page.

1 Like

That document is very informative. Thank you.

1 Like

New(?) location: Stopping Actions - Docs