Differences between RapidSetup and C++ Application Using RSI::RapidCode API

RapidCode API
1

RSI 8.0.7have a customer machine that is behaving strangely. I do not have a theory about what’s wrong. Let me describe what I observe.

If my application starts RMP.RTA, then tries to start the network, it fails.

Timed out waiting for network to start. (Error 1000000022) (RSI::RapidCode::Impl::MotionController::NetworkStart) (Object 0) (File …..\source\motioncontroller.cpp) (Line 2992) (Version 8.0.7 for 04.04.02.RMP)

The motion controller logs have this to say:

Starting EtherCAT network…
[StartAttemptBegin] Network State: (262): SHUTDOWN
RMP motion controller network log messages:
Network Log Message: /!\ 00:03:31.457 EtherCAT EthCatDr.cpp:3032 Last cyclic frame was 1093 us
Network Log Message: (i) 00:28:22.544 EtherCAT RMPNetworkStarter.cpp:161 Main Network Loop finished
Network Log Message: (i) 00:28:22.544 EtherCAT RMPNetworkFirmware.cpp:1465 Node #(0) : AL Status (0x0), Code (0x0)
Network Log Message: (i) 00:28:22.544 EtherCAT RMPNetworkFirmware.cpp:1465 Node #(1) : AL Status (0x0), Code (0x0)
Network Log Message: (i) 00:28:22.544 EtherCAT RMPNetworkFirmware.cpp:1465 Node #(2) : AL Status (0x0), Code (0x0)
Network Log Message: (i) 00:28:22.544 EtherCAT RMPNetworkFirmware.cpp:1465 Node #(3) : AL Status (0x0), Code (0x0)
Network Log Message: (i) 00:28:23.794 EtherCAT RMPNetworkFirmware.cpp:599 Exiting ServiceChannel Thread
Network Log Message: (i) 00:28:23.794 EtherCAT EthCatDr.cpp:981 MKEtherCATDriver::Stopping
Network Log Message: (X) 00:28:24.004 EtherCAT EthCatDr.cpp:679 EtherCAT communication broken during system operation
Network Log Message: (X) 00:28:24.005 Controller YMsgDisplayClient.cpp:58 UserInfo : Error : E30
Network Log Message: /!\ 00:28:24.005 EtherCAT EthCatDr.cpp:745 State changed from Running to StoppingOnError
Network Log Message: (i) 00:28:24.317 EtherCAT EthCatDr.cpp:981 MKEtherCATDriver::Stopping
Network Log Message: (i) 00:28:25.201 EtherCAT EthCatDr.cpp:1119 MKEtherCATDriver::Clear
Network Log Message: /!\ 00:28:25.661 EtherCAT EthCatDr.cpp:745 State changed from StoppingOnError to Error
Network Log Message: (X) 00:28:28.326 EtherCAT EthCatDr.cpp:1163 Timeout waiting for motion engines to stop
Network Log Message: /!\ 00:28:29.576 EtherCAT EthCatDr.cpp:3865 → Close driver
Network State: (262): SHUTDOWN

I don’t know what to make of the error messages. “EtherCAT communication broken during system operation” and “UserInfo : Error : E30”

Curiously, if I start RapidSetup (while my application is still running), it reports the network state as “OPERATIONAL” when this happens. The drives show up as network nodes, but do not have any axis objects. Ordinarily, my application would attempt to re-start the network, but it checks the network state before attempting that, and I suspect that the API is also reporting that the network is operational.

If RapidSetup starts the RTA, then, when my application starts the network, it gets the timeout exception and no drive nodes have axes. If I close my application (which calls MotionController::Delete()) and restart it, the network reports that it is operational and all the nodes now have axes. (In fact, if I start RapidSetup after shutting down my application, it shows all the drive nodes as having axes.)

With my application not running, if I then shutdown the network in RapidSetup, then start my application, starting the network times out, but the nodes now have axes (and the network state reports operational).

Huh?

Can you help me make sense of the error messages logged during the timeout? I didn’t see any unusual events in the Windows event log (including the INtime log). We have a number of other customers running this version of the software, and I don’t see this problem on those machines. In fact, I didn’t see it on this machine until recently.

I’m not inclined to suspect the wiring/network, just because if I do everything in RapidSetup, things seem to work well. What meaningful differences are there between what I’m doing with the C++ library and RapidSetup? (I’m calling MotionController::CreateFromSoftware(char*) to create the host object, and MotionController::Delete() when I shut down.)

I checked envvars. They’re all pointing to the correct RSI directory.

Any ideas as to where to look for problems?

2

Sounds like it could simply be timing out. Is this a slow PC?

We added configurable timeouts to NetworkStart(…) in 8.1.4, which are needed on slower PCs and/or systems with a high number of EtherCAT nodes.

The network log messages you’re seeing seem like they could be from a previous run - unrelated to your attempted NetworkStart().

Have you tried the rsiconfig utility? I recommend it for configuring / validating / starting your network and configuring all your Axis objects: rsiconfig - Docs As of 8.1.6, it will also generate an ENI file if it’s missing.

1 Like
3

I clear the network log before attempting to start the network. Start the network, then read the log messages. I saw these same messages every time I tried to start the network in my application.

The machine is (or can be) fairly busy. That could explain a behavioral difference between my application and RapidSetup. Updating the RSI software isn’t the best option, just because this is a production system, and we’d also need to build our application against a new version (and probably do at least a little testing) before turning them loose (if it fixed the problem).

How would rsiconfig help me? Just as another tool to verify that the network start succeeds/fails?

4

Which parameters are you passing to NetworkStart(…)? RapidSetup uses different parameters depending on what is required (probing, generating ENI file, etc.). After RapidSetup starts the network, it will refresh itself (MotionController delete, then create). The re-creation is required for it to detect and configure the discovered Axis objects, etc.

Agreed, let’s not change software versions. rsiconfig is a nice way of dealing with configurations and network without requiring any code compilation. (all human-editable XML)

5

We’re calling NetworkStart() with no parameters.

6

@todd_mm

We use EtherCATNodeInfo.xml to determine axis count and IO count. Without it, you would just see the Node Name with no sub items. NetworkStart() without parameters indicates that all the files (EtherCATNodeInfo.xml and others) will be in the working directory such as C:\RSI\X.X.XX\ for RapidSetup. If not, you can use the overload to specify where you can find the important RapidCode files. I’m guessing that you’ve got a more complex folder structure than all together. You may be running into an issue where files were moved and you may have more than one set of Rapidcode on the system. RapidSetup finds everything it needs, but the application doesn’t.

Do a comparison of the folders between working/failing systems. Remove extra versions and check path to ensure we aren’t looking in multiple places.

7

I did check for this before posting, but it’s good to be reminded of its importance.

I cannot find any other RSI directories (dirs containing the RSI files) on this machine.
We have (in the past) made copies of RSIQVC.dll to “prevent” DLL loading problems (on non-RMP systems), but I couldn’t find any stray RSIQVC.dll’s in the path or anywhere that Windows would look to automatically load it.

I’ll consider refreshing the files in the RSI directory.

8

@scott, can you give me some more detail about how high CPU use would cause this kind of timeout in the host software?

9

INtime grabs a core for its use. The only affect of high use from the Windows side should be on Cache/access to the resources. This is a problem that represents itself with as Jitter. You can run the Graphical Jitter Tool to see if you are getting poor results during normal or peek load. It would normally represent as Network errors rather than timeouts.

Network Start takes variable time based on the processing power of the PC and the number of nodes. We choose a timeout that will not work for all cases, but have an overload that lets you specify your own timeouts as of 8.1.4.

Discovering the Network (The first part of NetworkStart) should have populated Axes in the Motion Controller/RapidSetup.

I think you are generally in a state where you took long enough that RapidCode signaled a Timeout, but the EtherCAT Master ultimately succeeded in starting the network (as seen in RapidSetup). I don’t know why RapidSetup would show a lack of Axes/IO under nodes. However, RapidSetup doesn’t automatically refresh its list if Discover/Start happens from another source. This can be demonstrated by using two RapidSetups.

If you use View->Refresh does RapidSetup populate Axes/IO after having initiated a Start from your application and giving it time to complete?

2 Likes