Logix chassis lockup and other symptoms, suggestions

harryting

Lifetime Supporting Member
Join Date
May 2002
Location
Puget Sound
Posts
2,594
I'm also planning on a call to Rockwell in the morning. Meanwhile, I'm here to see if anyone seen anything like this but i don't have much hope since this is a weird one.

key facts: system with multiple Logix PLC, L62, L72, etc... running for years. No one was around at the time and no power bumps that we know of.

What happened: not one but two PLC chassis locked up, also, one redundant system lost sync and the backup chassis PLC (2 of them) lost their program.

When I say chassis locked up, I mean I can ping to the network card but can't see the backplane on RSLinx and OPC can't talk to the PLC.

We cycled power on the chassis and got them back up. Getting the redundant PLC back in sync was much harder but eventually we got it with some help with RA phone support.

Now everything is running but we are still getting OPC comm error on our HMI once a while. we didn't used to get these OPC comm error before the whole incident and it's having me on the edge a bit. I also never seen a chassis, not to mention two chassis that locked up like this. All the lights on the PLC seems fine when the chassis locked up too. It seems there are still some issue with the Chassis as we are getting intermitten comm error from the OPC.

add1: I see some task watchdog warning as the task runs over the period (set at 20ms but runs over 20ms sometime in task monitor) but I dont' believe that's a big deal?
add2: we see "battery alarm" on the L61 PLC once we get them back online in RSLogix and "energy storage alarm" on the L72 but locally at the PLC, no fault LED light are lit for those. Both went away later.

I shall update when I remember or find out more.
 
Last edited:
All are 20.xx, don't remember the L62 firmware but the L72 redundant one is on the 20.58 redundancy bundle. Remember it well since that's one of the version that got the Kepware issue.
 
Update:

None of the card match the serial # or manf date of the problem chips. We then update the firmware on the EN2T/C card, followed by replacing one of the processor. None worked. We will try to replace one of the chassis on Monday. At that time, everything including the chassis except the IO card would be new and if that doesn't fix it I'll go back to look at the network even though it doesn't look like the network is the cause (can still ping EN2T, etc) and other PLCs on the network (it's a ring but not DLR) isn't affected.

The network switches are all Hirschmann S30 and it's hard to view the status and configuration since the software, "HiVision" is no longer supported but we did dug out the old laptop and took a peek, nothing seems to be out of place.

I'll also try to wireshark one of the EN2T as well...

Stay tuned. this is about the most perplexing thing I ever dealt with.
 
“It’s a ring but not DLR” isn’t quite a red flag, maybe a yellow one, so what protocol is it based on?
 
Not sure, probably one of the proprietary ring protocol. I mean, every industrial switch vendor got one. It wouldn't explain why the chassis locked up.

Another step I thought of is to disconnect one chassis from everything and hook up a temporary HMI/OPC to it and see if the intermittent comm error still comes in.
 
I have seen bad power cause these kind of issues. A good Line Filter is a must.


Any new loads added to recently that may be creating Harmonic issues?
 
I'll work on a diagram. another piece of new info I got from the tech:

Multiple PLCs on the network, in addition to the 3 that is flaking out had either "Battery Fault" or "Energy Storage Fault" after the initial incident. However, the LED fault light on the PLC itself never lit up.

The power "should" be very stable as it's fed by redundant 120VDC to 24VDC Power supply. 120VDC is constantly fed from set of large battery bank. I will take a look at the battery charger to see if there are any logs available.

No new loads added recently.
 
Well, an update. The TLDR is that we found it's most likely the switches (Hirchmann MS30) that caused the issue but I have never seen any network issue that will lock up Logix chassis before, I guess first time for everything.

After replacing the rack, PLC, and the network card and still getting the same intermittent comm loss. We started to dig into the network more. While we were messing with the switches one of the chassis locked up AGAIN! Same symptom, can see the EN2T on RSLinx but not the chassis. Got connected to the PLC with serial (slow as heck) but found no errors. One weird thing is that a Panelview Plus can still talk to the PLC. During this time we also noticed one of the switch is report a bunch of CRC error out of one of the ring port. After power-cycle the rack, the comm came back but still intermittent losses We ended up disconnecting the ring from the port that got the CRC error and.... BANG! no more comm issue.

I would say Hirschmann is very fast on their technical support reply and there's no cost. They suggested starting with physical media, SFP, etc.. typical stuff but I'm doubting that's the issue but we will try with replacing that fiber patch. There's no SFP on that module as it's built-in transceiver. It also worries me that there's almost no spare part for this line of switch, even on ebay it's very slim picking so I'll be looking at options. Maybe Stratix 5700? Does 5700 also support EtherChannel?
 

Similar Topics

Hi - I'm a newbie to the PLC world but was recently asked if we could store more than the 2G that the CompactLogix 5370 allows by its max size SD...
Replies
13
Views
1,249
I am working for a company that does chrome plating and the line is VERY old lol So they did a controls upgrade(before my time here) and replaced...
Replies
4
Views
1,762
Hi All, Has anybody had any experience with this module - Softing OPC UA Server Controllogix module (setup, operation, etc.)? For this new...
Replies
7
Views
2,722
1) I am currently trying to integrate a Prosoft MVI56E-MCM Module into a 1756-L61 CONTROLLOGIX5561 that is a redundant system (CPU is redundant...
Replies
13
Views
3,354
Is there a way to know which is the primary and secondary chassis through gsv? I have read that "PhysicalChassis" (Chassis A/Chassis B) is an...
Replies
2
Views
2,206
Back
Top Bottom