>The FAAs safety risk management is built around a rigorous process, but that is also a massive slowdown to reasonable progress.
If this is the case, why was an airport knocked offline due to a cut fiber optic? A safety focused system would have a backup system say satellite or mobile 4G/5G. But they didn't have such.
This isn't about safety. Its about bureaucracy wrapped up as safety.
The primary was being inspected and the secondary failed.
There were two independent systems.
Which is why many systems on most airliners are redundant in threes (though under double fault may operate in a degraded mode).
There was a third system, the fallback they used to prevent loss of life, legacy systems and a system shutdown.
You can argue thats not a sufficient third redundancy, I don't have enough context to throw an opinion out there on the safety of this particular infrastructure project.
I am confident that the fallback process was considered and kept in place as the tertiary backup.
Excellent point though. It took me some thinking to put myself back in SRM mindset and what they'd have considered here.
> Which is why many systems on most airliners are redundant in threes
Not really. Triplex redundancy is based on a failure model where failures can produce arbitrary output. Three systems allows majority voting (often, median select) to provide correct output in the presence of a single arbitrary failure. It allows detection of double-fail-silent failures and uncorrelated double failures, but that's not really the point, since continued operation under these conditions is at risk, and failing nonoperational isn't often acceptable for these systems. (You'll also see quadruplex systems that use the same voting mechanism to detect and disable failed channels, allowing handling two arbitrary non-simultaneous failures, for systems that need a longer window of continued operation after failure.)
If you change the failure modes, you change the architecture. These days you often see dual-dual systems where each pair of systems cross-checks and disables itself on mismatch, converting a single arbitrary failure to a single silent failure; which then allows downstream systems to accept inputs from two such fail-silent systems and trust either that is providing output, removing the need for a median select.
Very interesting, I’d love to learn more. Is there anything you’d recommend in terms of a general introduction to different kinds of fault tolerance? Literature, video material, could be whatever.
Unfortunately this has been on-the-job learning for me over the last few decades so I don't have great sources to point you to, but this is covered in systems engineering courses for aerospace and automotive applications.
Same thing with control-theory - there’s so much established knowledge out there that one really needs to know but which isn’t taught to kids in uni or even postgrad level. Learning these things on-the-job means you don’t know what you don’t know…
What’s a better way? We still don’t have state-regulation/certification for software in safety critical systems, which terrifies me when I think about it.
We have DO-178C for aviation, which (a) is a pretty good standard and (b) is a heck of a lot better than ISO26262. It's "voluntary", but it's the only game in town, so…
The 747 had 4 independent hydraulic systems (one for each engine). There were still failures that took out all 4. What comes to mind is the one where the rear pressure bulkhead failed (due to a faulty repair), and the decompression took out the rudder and all 4 hydraulics.
The pilot no longer had flight controls. He desperately tried to control the airplane using variable thrust, but it just wasn't good enough, and crashed into a mountainside.
The fix was to add automatic valves in the lines so loss of pressure wouldn't bleed out the whole system.
JAL 123
“Two is one and one is none” as the saying goes.
> The primary was being inspected
So in reality you didn't have two systems.
otherwise you would have shut down operations until you had another connection live, so you had 2 online connections.
Ok but downtime isn't the same as failure. Downtime means planes can't take off. Failure means people die.
An more important question is what do they plan to replace it with?
>If this is the case, why was an airport knocked offline due to a cut fiber optic? A safety focused system would have a backup system say satellite or mobile 4G/5G. But they didn't have such.
There is a strong difference between an airport going down and ATC causing and accident.
Uptime isn't safety, they have very orthogonal characteristics.
Ridiculous false dichotomy. The bureaucracy can get in the way sometimes, but ultimately it is about safety, and the FAA has done an amazing job over the past several decades regulating the skies and played a massive role in making aviation as safe as it is. Standards are rigid and very well defined and change slowly because aviation is an industry where mistakes create massive disasters. There is some collateral inefficiency from things moving slow but thats way preferable to the collateral damage that would be caused by "move fast break things".
Although you were generally right, you can't say that after the parties between Boeing and FAA agents, nor FAA using Boeing consultants to inspect Boeing, fail inspections, still pass them, crash 4 planes and be humiliated by having Boeing Max flight permissions only revoked by the European authority. And the Macdonnell history with FAA says it is not new.
I Googled this exact phrase: "Macdonnell history with FAA". Woah, that is quite a complex history. It is a good read for those unaware.
Macdonnell was bought by Boeing, and the legend says it's precisely this _expertise_ that Boeing bought. The Boeing self-burning batteries and wrongly-handcrafted airframes of 2013 are signature of Macdonnell's way of managing issues.
Really going to ignore what happened between FAA and Boeing with the MAX?
The assumption that moving slower and more bureaucratically increases safety and reliability is one of the most harmful myths out there.
The problem is that in the FAA's domain, there are legitimately situations where bureaucracy is indispensable. For example, they just issued an AD to require inspections of a few specific bolts on early production 787s because Boeing thinks they MAY have bolted the wings on with incorrectly sized shims, which over a decade MAY contribute to accelerated fatigue on these parts. Not a single plane has crashed but this kind of thing has caused other planes to crash in the past.
These airplanes are flying all over the world, the inspections need ultrasonic data, and the FAA has to validate that all of them are done. Even identifying this issue required lots of bureaucracy.
Multiply that by the number of airplanes and airplane models and it's easy to see why they move so slowly: when you are putting millions of people in the air each day, enforcing absurdly specific rules with an iron fist requires a monster of a bureaucracy.
But moving faster and without accountability is generally worse.
In a perfect world, we would move at a steady pace, as to keep up with the world but without rushing, and people would be accountable without a heavy bureaucratic processes.
Unfortunately, that's not how it works in practice, at least not for long, so between the two, we usually favor the slow and bureaucratic process when safety and reliability matter.
In fact, it's created a bunch of risks. Like the fact that nobody manufacturers floppy disks or floppy disk readers anymore.
I can understand not changing a system for change sake, but the floppy disk has been obsolete for nearly 20 years now. It's been hard to find a floppy disk reader for the last 10 years. Going so slow that you can't respond to that change is far FAR too slow.
I worked for a salesforce partner and we spent the two years I was there working with a large ag equipment manufacturer. They had two legacy systems that were still (as I was told) some combination of Fortran and early Unix systems. They had been reluctant to upgrade them because one of the their main manufacturing machines was still running on it and they were terrified if they upgraded and something went wrong, it would cost them millions in production time.
As you pointed out, you can't find floppy discs or disc manufacturers any more. I do remember one of the VP's of this company saying as long as they keep teaching Fortran to college engineers, they have no reason to upgrade yet. That got a few raised eyebrows from our team. We relented and upgraded what they were comfortable with - but man oh man it was quite the eye opener for me as a young dev how the thought processes of the people running these companies really are.
Fortran is still used in scientific computing.
As long as those manufacturing systems were air-gapped and working, I can understand being conservative with changes. Of course at some point you're at risk of hardware failures too, for things that are likely NLA.
> Of course at some point you're at risk of hardware failures too, for things that are likely NLA.
Yeah, that's really the biggest issue with these systems. Were I one of these manufacturers I'd invest in getting the current software running in a VM specifically to make it easier to transition to new hardware. The last thing they'll want to deal with is that their software was written for Sun Spark and that no longer exists.
That said, hard to go wrong porting over the x86 at this point.
> ...getting the current software running in a VM specifically to make it easier to transition to new hardware.
Some of the signal control infrastructure for the Melbourne rail network was ported over to an emulated PDP-11 System for this reason. This was previously discussed on HN here: https://news.ycombinator.com/item?id=42514553
“NLA”?
no longer available
Fortran is still the choice for Nuclear Engineering too.
My understanding, from a long retired uncle (after a 40 year career at the National Institues of Health writing nothing but Fortran), is that Claude writes perfectly idiomatic, performant Fortran. Which I suppose is not a surprise given how it was trained, but I still found amusing to hear.
Have the idioms of Fortran changed much over the decades? Because idiomatic C++ today is unrecognizable to how people had to write for VC++ Win32 until well-until the 2010s.
Yes, infamously so. It was fixed-width at one point.
My humbling Claude experience was pointing at a codebase it took me weeks to get to a good 2003 standard, and watch it do the next fifteen years, cleanup, refactoring, feature additions that needed the tech debt solved I never got round to first, in minutes.
Linux 7.4 got fixes for floppy disk handling, as recently as yesterday!
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
A bit odd that they didn't at least update to using USB thumb drives. Those are readily available and at least as reliable as magnetic floppy disks.
Yeah, that's what I'd have expected especially because I believe the USB mass media standard was part of the original standard. You'd need no special drivers to work with a modern USB thumb drive even if you are dealing with windows 98 (Win95 had usb support, but I believe you still had to install the drivers for it).
Not only drivers. The initial release of Windows 95 predates USB itself, so the entire USB stack didn't appear until later OEM service releases that AFAIK were never released as updates for existing installs or even as retail products (unless you count MSDN or OEM System Builder products with their associated license restrictions).
You can get floppy disks and USB-C floppy drives on Amazon. Drives are under $20.
It was bad enough in 1986 when F-16 pilots had to bring their own cassette players on missions.
https://www.youtube.com/watch?v=GVHEFiLQ3GU
The old man in me says this is a reference to the movie "Iron Eagle", without clicking on the link. :)
Creating a rigorous process to ensure safety will create somewhat of a bureaucracy.
But creating a bureaucracy doesn't mean you've created a rigorous process to ensure safety.
Yup, it can create a situation where you have a huge incentive towards workarounds and hacky fixes because doing it properly incurs huge costs. This is especially bad when a fundamentally quite bad design makes it through certification (most processes don't really prevent this).
Presumably such a confident assertion is based on sufficient actual experience with slower safety protocols such that you can state, positively, that faster-moving processes produce safer, human-life critical, systems... ?
You really, really need to work in industrials controls before your opinion carries weight on this - there are plenty of 40 year old systems humming along keeping the world turning that are bulletproof reliable.
'Move fast and break things' is fine when the result of failure is someone failing to make a billion dollars at their IPO. It's not acceptable in the world where failure means dams break, reactors go supercritical, planes collide, or other similarly apocalyptic scenarios.
Yes - but those examples don't all have the same risk modalities. With dams and reactors, most failure variables can be permanently contained by initial system design/engineering and construction quality, continually inspected surroundings and internals, and physics certainty. Human inputs are limited and daily inspection matters, but any momentary error or lapse of operator attention won't result in instant destruction. You have hours and days with Dams, minutes and hours with reactors. In aviation it only takes a split second.
Aviation involves constant unpredictable human inputs and physical states. Hundreds of vehicles, not installations designed to have as few moving parts as possible. It is not yet solved, the status quo merely has a societally tolerable death rate. There can be new sensor hardware, more safety redundancies, and info synthesis/notification interfaces that are currently blocked by old hardware bottlenecks. Controller and pilot time and attention is limited, and it can be wasted less.
> Controller and pilot time and attention is limited, and it can be wasted less.
And inversely... when pilots leave too much to automation, their skills may degrade to the point that by the time you actually need a capable human crew because something has gone terribly wrong, the capability of that crew has degraded too much, and the FAA has warned about this in the past [1].
[1] https://medium.com/faa/the-dangers-of-overreliance-on-automa...
I feel a bit weird that I'm fairly certain that I read this discussion a while ago, in another post...
The FAA doesn't run airports.
The fiber cut was to the Philadelphia TRACON facility, which is run by the FAA. Also, the FAA runs the towers at airports, unless it's a contract facility. No idea how the infra is handled at contract facilities, but I'd be shocked if it didn't follow FAA guidelines, such that the FAA is dictating the levels of redundancy available.
Also, there's a big difference between a small towered airport in a more rural area, and a major international airport with 10x or more the commercial traffic. So there's probably some difference of buildout depending on the facility usage.
Quick edit: looks like there were two fiber cut incidents recently, one at an airport last month, and the TRACON issue that happened 1.5 months ago!
You may recall we went on an orgy of firing federal workers and cutting “waste fraud and abuse”. That impacted engineering, operations and boring jobs like finance and contracts. After the helicopter / airliner collision in DC they accelerated a bunch of stuff to show fire and motion.
Backhoes can disrupt connections, but so can failures to pay bills.
Systems like the radar in question often cannot work with broadband and satellite links.
Engineering will not save you from incompetent governance.
The presence of bureaucracy doesn't mean safety isn't the main consideration, it also isn't a given that bureaucracy is a negative in this instance. The safety record speaks for itself: you are safer in a commercial airplane than just about anywhere else you can go.
And yet with the 747MAX we saw that bureaucrats at the FAA were heavily influenced by Boeing.
The adage is that “you can’t test quality into a product.” The FAA stands as a final test that you attempted to rigorously engineer for safety, but that is no substitute for a company culture that prioritizes safety. It’s like when you release a bug to production and you turn around and blame QA. Yes they missed it but so did you.
And "bureaucracy" really means "grift".