High Energy particles from the cosmos causing bit flips

[id10t_error]

After watching The Universe is Hostile to Computers from Veritasium, I suddenly remembered a lost weekend years ago trying to reproduce the same unexpected behavior. So, operators 1,200,431, particle 1, if anyone is keeping score.

[Nova5]

Hrm... More... PEBKAC.

[Peter Nachtwey]

That is why our newer controllers use ECC.

[ndzied1]

This reminds me of story an engineer told me when I was just starting out. He was working to analyze the heating of train cars and was doing testing out in the field.

They had a train car decked out with very accurate temperature sensors. They started noticing on clear nights some of the readings were slightly below the ambient temperature. After multiple checks the sensors were deemed correct. After a little more head scratching and calculating the found out that on clear nights there was some radiation heat transfer to space that they had not considered. The car was warmer than space so the direction of heat transfer was out of the car which made it cool. It was a very small amount but measurable in this case with their accurate instruments.

[I_Automation]

The first time I heard of this was an election recount.


In a small town one lady overwhelmingly won, but the odd part was she won by a lot more than the number of all the registered voters that could have voted.


Turns out the recount was off by precisely 0010000000000000 binary, somehow in the original count bit #14 got turned on and it couldn't be replicated. There weren't 8,192 registered voters in that precinct.



If bit #27 got tripped that would give someone running for president an extra 67.1 million votes

[Dayvieboy]

Had a project @ a Blommer Chocolate where I have consulted for 30 years now

In a paste mixing system with 6 PLC's, 10 pumps & ~300 valves
Bits would get flipped that operators had no access to from a GUI

Emergency service call, production down.
Pumps & Valves world go to disabled or test states in the HOA loop
First couple of times I put them, back scratching my head

In a 16 bit control word Hand-Off-Auto, Test, Calling, Map, Output etc.
any of the bits would flip randomly

Then when it happened again I added automatic reset to default automatic mode, an event counter & an alarm with a time stamp to log it & so production would not be affected or be mad again that I do not sleep with my phone

Also added a huge red flashing notice on every SCADA screen in the plant that cleared in 10 seconds "Paste Mixing Error"

Debug counter & alarm kept going up every couple of weeks or so when I was not there

One day it finally went off when I was in my office, I walked out to investigate in a plant as big as s football field, 2 stories

On my way up the stairs I see a welder with his ground cable clipped
to the 6" communication gutter 15 feet up in the air, watched & saw that
as he welded the bits flipped & the alarm incremented counts

Investigated further & found that the only valves & pumps affected were
isolated to one communication message between the master PLC & one of the remotes & the cable was in that gutter

Somehow those bits in the Cat 5 cable would get scrambled but
the communication packet did not error & still pass them through

The alarm code & counters are still there &
have not incremented since construction in that area was completed
over a decade ago

Had I not been lucky enough to catch it
It would have magically fixed itself
& I hate when that happens

[Dayvieboy]

Does it ever make sense to use 192.168.xxx.xxx to put equipment on?

Using the most popular private subnet in the world
for machines seems to invite many problems and make troubleshooting harder

I was just on project having issues going online with a PLC.
Turns out they were thinking the PLC was alive but were actually pinging something that is not even a PLC a 1/4 mile & 2 buildings away.

(IT is dealing with this as private networks should never be seen public)

I have had problems several times over the years when people
plug things into an incorrect network, do not use a VLAN/NAT
or put the PLC on a the public network
& several other reasons.

192.168.1.xxx is the Rockwell option for Ethernet adaptors
when using the hardware thumbwheel.

Curious what other subnets people use for their machine networks.




Some info I found on private networks:

https://sourcedaddy.com/networking/private-subnets.html

https://www.computerweekly.com/news/...e-IP-explained

https://en.wikipedia.org/wiki/Private_network

[ArcReactorKC]

Originally Posted by Dayvieboy

Does it ever make sense to use 192.168.xxx.xxx to put equipment on?

Using the most popular private subnet in the world
for machines seems to invite many problems and make troubleshooting harder

I was just on project having issues going online with a PLC.
Turns out they were thinking the PLC was alive but were actually pinging something that is not even a PLC a 1/4 mile & 2 buildings away.

(IT is dealing with this as private networks should never be seen public)

I have had problems several times over the years when people
plug things into an incorrect network, do not use a VLAN/NAT
or put the PLC on a the public network
& several other reasons.

192.168.1.xxx is the Rockwell option for Ethernet adaptors
when using the hardware thumbwheel.

Curious what other subnets people use for their machine networks.




Some info I found on private networks:

https://sourcedaddy.com/networking/private-subnets.html

https://www.computerweekly.com/news/...e-IP-explained

https://en.wikipedia.org/wiki/Private_network

Depending on the network and customer I may still use 192.168.xxx.xxx

However most often I use the tried and true 123.123.xxx.xxx very very rarely do I have an incident where IT or some other entity is using the 123 subnet. Originally I used 10.10.xxx.xxx but had more than one collision with IT so I switched.

[mad4x4]

Interesting little Vid. We had similar a few years ago and it was described as a "Stuck Bit" we could never get a straight answer from Stratus on what a "Stuck bit " was.

[Nova5]

Originally Posted by ArcReactorKC

Depending on the network and customer I may still use 192.168.xxx.xxx

However most often I use the tried and true 123.123.xxx.xxx very very rarely do I have an incident where IT or some other entity is using the 123 subnet. Originally I used 10.10.xxx.xxx but had more than one collision with IT so I switched.

why are you using a publicly routable 123 address?

[I_Automation]

Originally Posted by ArcReactorKC

Depending on the network and customer I may still use 192.168.xxx.xxx.

When I first set up my home network I, not being normal, didn't want to use the "normal" 192.168. so I picked a unique number all my own.


Then I added something (a camera I think) that could ONLY be set to 192.168.127.(2-255) and I had to reset and reprogram my router and every other device I had set an IP address on. That included other cameras, a SLC5/05 and 3 NET-ENI's for SLC5/03's. Also RSLinx ETH drivers on multiple computers and every page of 2 AdvancedHMI setups for the home system.


That was a pain (resetting an installed camera means pulling it down, reprogramming and reinstalling it) so 192.168.127.xxx it is. Also with my IP DVR for the camera's it has to be set to a specific IP for each camera, if they were not permanent IP's it would lose the camera every time it reconnected.

[Nova5]

Well... fixed to 192.168.127.x... you were kinda screwed anyway when you left the entire 192.169 range. Otherwise just open the subnet mask a bit to encompass the 127 and it coulda worked with less trouble. But I'd wager that camera got on the Never Buy Again list.

[I_Automation]

RSLinx (not $$$ Gateway edition) can be setup to use a TeamViewer VPN connection.


Others here have done it and I checked it out, but have not done it myself yet.



This thread is one.