Crowdstrike I.T. Security And Q.A. To blame.

I worked in computer education classrooms and we often pushed out 100’s of New installs weekly
So I can see how this could happen

I also performed tens of thousands of remote network installs on Unix, Linux, and PCs.
Batches of a 30 to 100’s of computers at a time.
Nation wide installs, updates, and upgrades.

I.T. Is responsible for the Security of the corporate computers.
But Q.A. is responsible for New releases being thoroughly tested with the operating systems.

Back in 1990’s I worked at a Famous Anti-virus company working in Q.A. And we rolled out an update that also modified a system config boot file.
But if customers had already edited and customized the system boot file with some code it might not load the system correctly and cause a conflict. And the computer will Not Boot.

So we had a BIG problem Many of our customer computers servers did not boot upon update install and mandatory system reboot.

So I can see how this problem could happen with Crowdstrike.

Software Distribution group who pushes out the updates is Not to blame their job is just to rollout the update and work with Health check
Group to make sure all the computers are working fine.

Q.A. Is the group that let this Bug get thru testing prior to release.

The fix was to boot into DOS and edit the System boot configuration file manually.

But our Managers were PISSED.

But hey, What is Customer support for ?

Q.A. Is a tough job and you take allot of heat and presure to meet release dates.
Q.A. Never gets any reward.
It’s hard to find good Q.A. Peoples.
You have to be a Programmer, Software engineer, I.T. Professional, Network engineer, and Hacker all in one.

It’s a hard position to fill.

So this problem has happened before back in 1990’s
Just not at the scale of this Crowdstrike customer base.

Apparently southwest Airlines avoided it by still using Windows 3.1 and 95 https://www.forbes.com/sites/tedreed/2024/07/ … n-time-all-day/

Obviously it wasn’t the OS that saved them but the fact they don’t use cloudstrike, but still thought you guys might get a laugh out of that. Now to get them to see the light and upgrade to win98SE 😂

This is what happens when people stop using their brains and fail to realize their security software is just as harmful/counter-productive than the bad software it is designed to keep out. There's a reason the term "snake oil" is used.

TheMobRules wrote on 2024-07-20, 18:17:

Their development process must be completely f****ed up. Obviously a lot of Crowdstrike higher ups/executives should be fired and never allowed to work in the field again, but what will probably happen is that a few lowly developer/QAs (that were probably under a lot of pressure to begin with) will be fired as scapegoats while the managers will be promoted for whatever reason. And executives will somehow get their bonus no matter how much this affects the company.

And that's exactly what will happen.
-Hey [middle manager], when do you think we can have a proper QA team back, because since they all resigned last year the secretary is currently doing the testing?
-Management is handling this [senior developer]. In the meantime [senior manager] wants x deployed by the end of the week
-Untested?
-You ensure it's fine.
The world collapses, [senior developer] is fired and managers nag how they cannot find anyone competent and that makes their bonuses smaller.

Yoghoo wrote on 2024-07-21, 00:49:

Of course he has not. But some people on this forum read somewhere in a topic the word "Windows" and then the Microsoft bashing begins. Seems to be a popular thing to do for the last 20 years or so. Especially by the Linux converts. 😉

I too have had a salary based on supporting proprietary software, but it is still a pile of shit, overall. At work you can't say that, the decision makers get all incensed that you are flagging their inability to make technical decisions, but here isn't work.

Also, after ILOVEYOU, SQL Slammer, Nimda, code red/blue, stuxnet, print nightmare.... going over the same old detractions is the thing that can help enable the next vast and wide-spread problem.

Proprietary software exists for the publishers primarily, and whilst they say it is about the users, enshitification illustrates it is not. FOSS has a lot of problems, but if a FOSS projects starts to become shit it can be forked and the users kept as the priority. Proprietary gets worse, and once into monopoly (though that is where it starts) it is guaranteed.

But yeah, this is Micros~1 bashing, it is FUD, blah fucking blah 😀

progman.exe wrote on 2024-07-20, 15:18:

Windows loaded a driver file into the kernel, where the contents of the file were all nulls. […]
Show full quote

Windows loaded a driver file into the kernel, where the contents of the file were all nulls.

For whatever reason Crowdstrike seemed to send that out, though maybe something mangled it upon installation. In theory any publisher that has stuff at kernel level could have had this occur: Windows, it seems, will load driver files that should fail even basic validity tests.

MS has a monopoly, and monopoly markets have bad and expensive products. Yes, the schadenfreude is strong here, but from a dry economic point of view, this massive outage is symptom of the Microsoft monopoly.

Oh yeah, IP law mandates effective monopolies, or at least very partitioned up markets. IP is enforced by international treaty and overrules nation state law. To get rid of the contemporary tech monopolies will be more difficult than Standard Oil.....

They've been doing this for many years, long before the IE vs. Netscape incident, despite the fact that Windows machines have a much higher market share in the consumer market than Linux or macOS. They continue to misuse their power. A month ago, the EU fined Microsoft $2.4 billion USD for violating EU competition rules. I feel they have settled, but there is more to come.

Microsoft says only 1% of total Windows computers impacted.

It’s NOT always Fun and Parties in corporate I.T.
Sometime you actually have to roll up your sleeves and type on the keyboard.
It’s not always Windows and mouse based remote administration.

Boot off a DOS floppy and edit the System boot config files manually. Or you could even write a batch file to do this.
You could make a USB thumb drive to Automatically edit the know bad system files.

Back in 1990’s we had 1.44 floppy diskettes that could do all kinds of automated tasks.
Including Trouble shooting, System diagnosis, network installs, etc.

This is what set the 1993 IBM Ps/2 computer apart from everyone else.
It was loaded with system administration scripts and batch files to backup, diagnose, trouble shoot, restore, and recover the PC incase of failure.

Intel486dx33 wrote on 2024-07-21, 16:15:

It’s NOT always Fun and Parties in corporate I.T.

It's NEVER fun and parties in corporate I.T. Chronic outsourcing has killed productivity and any sense of community. Incompetence is up, innovation is down.

There is no joy.

badmojo wrote on 2024-07-21, 23:15:

Intel486dx33 wrote on 2024-07-21, 16:15:

It’s NOT always Fun and Parties in corporate I.T.

It's NEVER fun and parties in corporate I.T. Chronic outsourcing has killed productivity and any sense of community. Incompetence is up, innovation is down.

There is no joy.

Thats because it’s the contractors doing all the work.
All employees do is complain about stock options and take credit for other peoples work.

Intel486dx33 wrote on 2024-07-21, 16:15:

Boot off a DOS floppy

What year do you think it is? How many of the airport check-in terminals, supermarket cash registers, and other affected machines are designed to boot from arbitrary media? Let alone have a standard display, input device and filesystem as required by DOS?

Intel486dx33 wrote on 2024-07-22, 00:55:

Thats because it’s the contractors doing all the work.
All employees do is complain about stock options and take credit for other peoples work.

Wrong as usual.

Should have stuck with DOS.

javispedro1 wrote on 2024-07-20, 18:02:

I remember experiencing myself an issue in the late 90s where after virus definitions update suddenly McAfee decided to delete a 9x system file, and this wasn't an isolated occurrence (even MS defender is guilty of this...).

I found it amusing that George Kurtz co-founder of CrowdStrike was CTO of Mcafee when it's update caused XP machines to bluescreen back in 2010.
https://www.zdnet.com/article/defective-mcafe … tdown-of-xp-pcs

I.T. is a mess at the moment as its still moving from traditional servers to cloud based systems and everyone is still trying to work out what works best.
But thats not going to change, majority of systems are at least some what back online less then 2 days later and everyone will carry on like nothing happened because proper planning and fixing is expensive and boring.

I missed the disconnected days, when the only possible threat comes from the floppy disks shared among friends.

https://uk.finance.yahoo.com/news/crowdstrike … -093000219.html

the whole world isn't vulnerable to tech's fragile webs of interdependence, but a large portion of the world has allowed itself to become so

Norton Commander wrote on 2024-07-20, 17:40:

My thoughts? I'm glad I no longer work in IT hell. Long hours, on call 24/7 and weekend plans cancelled because of incidents like this. As DosFreak pointed out incompetent leadership runs rampant in this field. My last day in IT I worked 28 hours straight. That was the last straw. Friday afternoons I like to have a cold beer in my hand, not a cup of strong coffee because it's going to be a long weekend at work.

Those working to fix this mess have my sympathy.

What did you choose after IT? I am also thinking about dropping the IT circus.

My thinking is this is a case of bad QA process.

Regardless of the OS, a decision was made to trust a certain software to be the guardian of the system. At that trust power and level of operation, if the software malfunctions it can bring down the entire system. Too bad the developer had a hole in their QA process through which they let the malformed bits get into the system.

So whether it is Win11/Win10/XP/Linux or even DOS... you gave access to the kernel.. that's a big decision no OS can protect against.

That is not to say that I'm ok with current SW update practices, just that this incident isn't exactly a case of a consumer-level "MS shoved an update that broke my system". Consumers nowadays are not even given the option to disable or opt-out of autoupdates and telemetry crap.
Business-level OS setup is different, and IT will vet what goes into the system and nothing would be shoved in without approval.

https://www.youtube.com/watch?v=wAzEJxOo1ts

A good technical explanation about what happened and who may be to blame...

wbahnassi wrote on 2024-07-23, 13:20:

My thinking is this is a case of bad QA process. […]
Show full quote

My thinking is this is a case of bad QA process.

Regardless of the OS, a decision was made to trust a certain software to be the guardian of the system. At that trust power and level of operation, if the software malfunctions it can bring down the entire system. Too bad the developer had a hole in their QA process through which they let the malformed bits get into the system.

So whether it is Win11/Win10/XP/Linux or even DOS... you gave access to the kernel.. that's a big decision no OS can protect against.

That is not to say that I'm ok with current SW update practices, just that this incident isn't exactly a case of a consumer-level "MS shoved an update that broke my system". Consumers nowadays are not even given the option to disable or opt-out of autoupdates and telemetry crap.
Business-level OS setup is different, and IT will vet what goes into the system and nothing would be shoved in without approval.

Software developers

Since this gas happened many times before. These Antivirus companies should not have such control over your computer servers.

wbahnassi wrote on 2024-07-23, 13:20:

So whether it is Win11/Win10/XP/Linux or even DOS... you gave access to the kernel.. that's a big decision no OS can protect against.

Sure it can. Apple doesn't let anyone/anything touch the macOS kernel (for now).

Microsoft on the other hand doesn't have that choice due to certain EU antitrust case outcomes... It's funny that this wouldn't have happened if MS didn't allow any alternative to using Windows Defender.