Could not delete ServiceChannelSemaphore 0x0000CDCD

RapidCode API
1

HI,

i am facing issue where customer application keep disconnected by it own self.
upon checking found this issue:

RMPNetwork: (i) 00:00:19.714 EtherCAT RMPNetworkStarter.cpp:92 Starting RMPNetwork firmware
RMPNetwork: (X) 00:00:20.974 EtherCAT RMPNetworkFirmware.cpp:2071 Problem finding RMP process handle
RMPNetwork: (X) 00:00:20.974 EtherCAT RMPNetworkStarter.cpp:265 Unable to copy network log messages due to lack of RMP memory mapping.
RMPNetwork: (i) 00:00:22.236 EtherCAT RMPNetworkFirmware.cpp:2022 Could not delete ServiceChannelSemaphore 0x0000CDCD GetLastRtError: (0x00000006).

Please advice what is this error?
customer intime kernel memory is setup to 1024 MB.

2

Hi Stephen,

From the log you gave, it seems like the network is failing to start, but it is hard to say the cause without more information. Could you answer the following questions to help narrow it down:

  1. Is there anything in the log before this snippet, or any logs from the run that shut down?
  2. How is RMP and RMPNetwork being started, through RapidCode, rsiconfig, or RapidSetup/RapidSetupX?
  3. When the network shut down, was there anything different happening on the machine?

The INtime kernel memory setting shouldn’t be related to this issue.

3

HI Lucas,

Below is the log from rapid setup X before the restart:

ERROR, 06:55:08.379 PM, , , , Network, , Network state changed from Operational to Error.
SUGGESTION, 06:55:08.387 PM, , , , Network, , Verify that all cables are connected correctly.
WARN, 06:55:08.557 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 8, Timed out trying to read Service Channel (SDO)
WARN, 06:55:08.557 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 7, Timed out trying to read Service Channel (SDO)
ERROR, 06:55:08.564 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 15, Timed out trying to read Service Channel (SDO)
ERROR, 06:55:08.584 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 4, Timed out trying to read Service Channel (SDO)
ERROR, 06:55:08.584 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 0, Timed out trying to read Service Channel (SDO)
ERROR, 06:55:08.586 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 5, Timed out trying to read Service Channel (SDO)
ERROR, 06:55:08.588 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 10, Timed out trying to read Service Channel (SDO)
ERROR, 06:55:08.588 PM, rapidcodenetworknode.cpp, 420, RapidCodeNetworkNode::ServiceChannelReadCore, RSIMessageTIMEOUT, 9,

currently we are still monitoring and trying to identify what cause the shutdwon.

4

“Network state changed from Operational to Error” means the EtherCAT master lost communication with one or more drives and shut the network down. The most common causes are a cable or connector, a drive losing power or dropping out of Operational, or electrical noise corrupting frames.

RapidSetupX doesn’t retrieve the RMPNetwork log after a runtime drop, but the messages stay stored in the controller until the network is started again. If you have a way to run RapidCode against the live controller, then next time it happens, before restarting anything, call NetworkLogMessageCountGet() and then NetworkLogMessageGet(i) for each message, and post the output.

A few more questions:

  • What version of the RMP are you running, and what language is the application written in?
  • Did any drive fault or lose power around when the shutdown occured?
  • How are RMP and RMPNetwork started on that machine, from your application, rsiconfig, or RapidSetup/RapidSetupX?
5
  1. version of the RMP is 10.6.6, written in C#
  2. didn’t observe power lost or driver fault. After shutdown happen, customer able to restart network without issue. only in this case while we collect the logging with RMPNetwork.RTA we saw the semaphore error before network start and rapid setup.
  3. start in customer application.
6

Hi Stephen,

Thanks. A couple of things on the first log: “Problem finding RMP process handle” means RMPNetwork.rta started while RMP.rta was not running, so it had nothing to attach to and exited. The semaphore message is cleanup noise on the way out. How did you run RMPNetwork.rta when you collected that log? If you started it yourself to watch the output, that explains it and we can set it aside. If your application started it, that’s a separate problem worth knowing about.

On the disconnect, the network restarting cleanly and no drive fault being observed is consistent with a brief cable or connector drop, or a drive briefly dropping off the network. Those look identical from the software side.

Two things you can do now:

First, RapidSetup has a Packet Errors view under Network Diagnostics that shows communication errors per port for every node. Those counters reset each time the network starts, so on your system they show how clean the connections have been since the last restart. If you can leave it running for a while, a count that keeps climbing on one port points at a connection that is getting errors but hasn’t failed yet.

Second, every EtherCAT device keeps a Lost Link counter per port in its own hardware. Those are not cleared by a network restart, only by powering the device down, so they may still hold the answer from the drops you have already had. With the network Operational:

for (int n = 0; n < controller.NetworkNodeCountGet(); n++)
{
    var node = controller.NetworkNodeGet(n);
    for (int port = 0; port < 4; port++)
    {
        int lostLinks = node.ServiceChannelRead(0x310 + port, 0, 1);
        Console.WriteLine($"Node {n} port {port}: {lostLinks}");
    }
}

Note: power cycling a drive will reset its Lost Link counter.

After the next drop, restart the controller but do not start the network. Then wait a few seconds for RMPNetwork to finish shutting down and read the log messages with NetworkLogMessageCountGet() and NetworkLogMessageGet(i) . Starting the network clears them, so this has to happen before the recovery restart.