RapidSetupX crashes on reconnecting to server

RapidSetup Tool
1

Hello!

We are noticing that RapidSetupX is crashing when attempting to reconnect to a motion controller/server after a certain sequence of events. What we observe is:

  1. Connect to server and launch motion controller via RapidSetupX
  2. Connect to server + Motion Controller via internal software using RMP C++
  3. Issue sequence of commands from internal software
  4. At some point during 2-3 (exact timing is unknown), RapidSetupX will show it’s not connected to the server.
  5. Attempts to reconnect to the same server causes RapidSetupX to exit.
  6. Reopening RapidSetupX to connect to the server does not resolve the issue.
  7. Restarting RapidSetupX AND shutting down the still running active rmp process allows RapidSetupX to function

I have a log file of the crash that I can share (but not sure how to upload it on the forum) but here is a snippet:

[11:45:59.236] [ERR] Axis (index: 0) does not exist. Increase MotionController's AxisCount (0).
[11:48:06.358] [ERR] Could not ping RapidServer (100.74.136.31:50061). Server Threadpool Exhausted Disconnected.
[11:49:11.532] [INF] The user connected to RapidServer 10.6.7 (100.74.136.31:50061) (hitl-1.codlet-coho.ts.net) in 0.489 seconds.
[11:49:12.373] [ERR] Collection was modified; enumeration operation may not execute.
2

Are you currently using RapidSetupX version 10.6.7 or 10.6.7.1?

If you are on 10.6.7, we recommend upgrading to 10.6.7.1, as we encountered issues with gRPC communication in 10.6.7 that have been resolved in the latest version.

image
image1208×367 16.1 KB

3

Confirm, I am on RapidSetupX 10.6.7.1

Screenshot 2025-04-23 093135
Screenshot 2025-04-23 093135464×244 6.1 KB

1 Like
4

Okay, thank you for verifying the version.

Is your rapidserver console showing any errors by any chance? Or is there anything on the rapidserver service logs?

5

Would you be able to expand on what commands are being sent here?

6

Is there a different way I should be getting logs from the rapidserver service?

Screenshot 2025-04-24 105710
Screenshot 2025-04-24 105710967×304 15.9 KB

We issue a number of cmds to the controller

We also configure the axes

  • userunitsset
    -defaultvel/accel/decell
  • errlimit
    -homemethod/vel/accel/decell
    -settlingtime
    -swlimitsset
    -interruptenable

We also issue enable/disable and move motors and set iopoints

One theory that comes to mind is that we are not yet quite handling interrupts in our SW. Not sure if that would affect things.

7

That’s the correct way to get the rapidserver output.

We suspect your C++ program’s changing of objects counts (AxisCountSet(), etc.) is the likely culprit. rapidserver tries to handle object count changes from other processes (your C++ app) but sometimes (especially if a client like RapidSetupX is running/polling) it could fail. You could validate this by disconnecting RapidSetupX temporarily while your code initializes.

Also we would recommend you consider trying rsiconfig, as its purpose is to start the RMP, the EtherCAT network, configure all object counts and configure Axis settings. It does this with XML in a version-independent manner, so you machine’s configurations are all in hand-editable files and don’t require recompilation of your source code.

Either way let us know if either suggestion helps.

1 Like
8

Ah I just noticed if you don’t use sudo, you won’t see the rapidserver ouput. So try:

sudo systemctl status rapidserver

9

Gotcha! Thanks! We’re pursuing a slightly different architecture now where we start RMP and configure it once and have rapidsetupX + other clients join purely as clients.
We discovered another instance in logs where we saw:

5/5/2025 5:07:11 PM    Error    Status(StatusCode="ResourceExhausted", Detail="Server Threadpool Exhausted")    Grpc.Core.RpcException    Line 0       at RSI.Shared.Services.Rmp.RmpGrpcServiceClient.RecorderAsync(RecorderRequest request)
   at RSI.Shared.Services.Rmp.RecorderService.<.ctor>b__0_0(Object request)
   at CallSite.Target(Closure, CallSite, Object)
   at RSI.Shared.Services.Rmp.RapidService.CallRpcAndUpdateAsync(Object request)
   at CallSite.Target(Closure, CallSite, Object)
   at RSI.Shared.Services.Rmp.RapidService.RequestAsync(Object configOrActionMessage, MessageType messageType)
   at RSI.Shared.Services.Rmp.RapidService.ActionSetAsync(Object actionMessage)
   at RapidSetupX.Models.Scope.ScottPlotScopeX.Records_UpdateFromServerAsync()

and so we disabled recorders and noticed different behavior. RapidSetupX seems like it’s no longer crashing.