Ldtra: Runtime Error!

todd_mm · August 31, 2021, 12:34pm

[RMP 8.3.1] [INtime 6.3.18110.7]

I’m getting an error when I attempt to do anything with RMP, including starting RapidSetup.

This just started happening on an outgoing (customer) machine.

The machine’s been running fine for weeks, including RMP and my application.
- I’ve got other machines at the same software level that run fine.
I don’t recall making any INtime-level changes.
RMP hasn’t changed except for config files.
- I don’t see any corrupt files.
My app has changed a lot, but it’s crashing outside the app, too.
I reinstalled INtime with no net change.
I unzipped a pristine RMP distribution, copied in the license file, and it behaved the same way.

Are there common problems that this is a symptom of?

scott · August 31, 2021, 12:43pm

No, this is not a familiar error. Windows Update? Security settings?

You could try the latest INtime we support, 6.4.21125.2. Let us know if you need a link. It’s what we’re using for 10.3, but I’ve just tested it with 7.3.4 and it seems fine.

todd_mm · August 31, 2021, 2:24pm

Additional details:

Even INtime apps (?) fail in this way, like:
- jitter.exe (when you try to “Start” the monitoring)

Conclusion

TL;DR: %TEMP%\ldrta.txt Exceeded 4 GiB

I ran ProcMon and noticed that ldrta.exe was writing to %TEMP%\ldrta.txt. Hoping it was a log file with some potentially useful detail about what was going wrong, I went to look at it, and noticed that it had a very conspicuous size: 4,294,991,241 bytes.

I renamed the file, and INtime/ldrta.exe works.

Pertinent Scenario Information

I had been testing recovering from network failures (loss of power, hot-changes to network topology, etc.), all of which involve repeatedly and endlessly attempting to restart the network and/or restart the RTA.

I suppose all those attempts to restart the RTA over the past 4-6 weeks are what filled this (log) file up.

I counted the number of lines in the file that looked like RMPNetwork was (re)loaded. In my case, it happened 63,456 times.

Fix?

This is probably a niche issue, but I think it’s something TenaSys should do something about. The error message I received was not particularly helpful in finding the problem.

There was nothing in the event log.
The error message suggested that the RTA might be responsible.

I’d tell TenASys, but they (already told me that they) don’t want to talk to me (since I’m not a customer).

I’d suggest that they rotate the log every so many MiB. AFAICT, the file doesn’t contain timestamps, so there seems to be little value in keeping much of the data.

scott · August 31, 2021, 3:47pm

Interesting. The weird thing is that none of us at RSI has the file, ldrta.txt. Our test agent PCs are probably starting/stopping the network even more frequently and they also do not have this file.

Do you know if you have turned on some special logging setting? I see some settings under INtime Configuration -> Miscellaneous -> System logging, but mine are all set to No. Yours?

todd_mm · August 31, 2021, 4:04pm

Just so that I don’t mistype it:

I don’t see any logging settings enabled.

I exported my INtime config to a file. I couldn’t attach it here, so I posted it to pastebin (good until 2021-09-30). Pastebin warned me that my config file contained potentially offensive information (!!) so I had to base64 encode it in order to paste something that you could get to.

todd_mm · August 31, 2021, 4:11pm

FWIW, as a part of my failure recovery process, we will also stop and restart the INtime node.

todd_mm · August 31, 2021, 4:16pm

Oops. I pasted the wrong config.

Here’s the one on the 6.3.18110.7 machine: pastebin (base64 encoded)

todd_mm · August 31, 2021, 4:23pm

The text file contains instances of this behavior:

Pid 0x12cc: Create mutex handle 00000238
Pid 0x12cc: About to WFSO on mutex (load RSL)
Pid 0x12cc: PID 000012cc RslReq 00011ab0 RslRsp 00011ab8
Pid 0x12cc: LoadRSLFile(ie1ge.rsl)
Pid 0x12cc: OpenRslFile: rsl=ie1ge.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\ie1ge.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\ie1ge.rsl
Pid 0x12cc: LoadRSLFile(RtRsl.rsl)
Pid 0x12cc: OpenRslFile: rsl=RtRsl.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\rt\lib\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\bin\RtRsl.rsl
Pid 0x12cc: LoadRSLFile(pcibus2.rsl)
Pid 0x12cc: OpenRslFile: rsl=pcibus2.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\pcibus2.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\pcibus2.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\rt\lib\pcibus2.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\bin\pcibus2.rsl
Pid 0x12cc: LoadRSLFile(RtRsl.rsl)
Pid 0x12cc: OpenRslFile: rsl=RtRsl.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\rt\lib\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\bin\RtRsl.rsl
Pid 0x12cc: LoadRSLFile(clib.rsl)
Pid 0x12cc: OpenRslFile: rsl=clib.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\clib.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\clib.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\rt\lib\clib.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\bin\clib.rsl
Pid 0x12cc: LoadRSLFile(RtRsl.rsl)
Pid 0x12cc: OpenRslFile: rsl=RtRsl.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\rt\lib\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\bin\RtRsl.rsl
Pid 0x12cc: LoadRSLFile(clib.rsl)
Pid 0x12cc: OpenRslFile: rsl=clib.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\clib.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\clib.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\rt\lib\clib.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\bin\clib.rsl
Pid 0x12cc: LoadRSLFile(RtRsl.rsl)
Pid 0x12cc: OpenRslFile: rsl=RtRsl.rsl
Pid 0x12cc: Trying load path /path/to/rsidir\RSI\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\\network7\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\rt\lib\RtRsl.rsl
Pid 0x12cc: C:\Program Files (x86)\INtime\bin\RtRsl.rsl
Pid 0x12cc: release mutex (load RSL)
Pid 0x12cc: closing mutex handle

todd_mm · August 31, 2021, 4:31pm

I did a quick survey of machines around here. I don’t see the file on any other machines. IDK why the file might only sometimes get created. This particular machine spent much more time in error states, where my application was continuously probing the network and less frequently restarting the RTOS with:

RTOS::RTOSGet()->INtimeStop();
RTOS::RTOSGet()->INtimeStart();

todd_mm · August 31, 2021, 5:22pm

Is it possible that enabling tracing in RMP would cause this? My application is doing that for some things for which we haven’t completely ironed out the implementation.

EDIT: (Clarification) “Tracing” is done with RapidCodeObject::Trace*(...) API functions.

scott · August 31, 2021, 5:27pm

If you’re talking about the Trace methods in RapidCode, no, those would not have anything to do with INtime logging/tracing.

stefan · August 31, 2021, 5:27pm

What do you have configured in your node’s SYSLOG settings here:

In particular I’m curious about LOGLEVEL and DESTINATION (I think that 2 is actually the default for Windows, all of mine are set to 2 and I’ve never touched that)

todd_mm · August 31, 2021, 5:33pm

Same as you: dest(2) level(0)

stefan · August 31, 2021, 7:38pm

So I tried a novel method and checked the changelogs! Looks like this was fixed in version 6.3.18220.1
https://www.tenasys.com/resources/knowledge-base/knowledge-base-page/?pageNum=165

(I couldn’t figure out how to make the text smaller, but imagine reading this in a 1pt font)

todd_mm · August 31, 2021, 9:26pm

That might make sense of why this control is different. I’ve upgrade all the others.