Hey from the GitHub team. Outages like this are incredibly painful and we'll share a post-mortem once our investigation is complete.
It stings to have this happen as we're putting a lot of effort specifically into the core product, growing teams like Actions and increasing performance-focused initiatives on key areas like pull requests where we're already making solid progress[1]. Would love if you would reach out to me in DM around the perf issues you mentioned with diffs.
There's a lot of architecture, scaling, and performance work that we're prioritizing as we work to meet the growing code demand.
We're still investigating today's outage and we'll share a write up on our status page, and in our February Availability Report, with details on root cause and steps we're taking to mitigate moving forward.
Literally everyone who has used Github to look at a pull request in say the last year has experienced the ridiculous performance issues. It's a constant laughing point on HN at this point. There is no way you don't know this. Inviting to take this to a private channel, along with the rest of your comment really, is simply standard corporate PR.
Yes agreed it's been a huge problem, and we shipped changes last week to address some of the gnarly p99 interactions. It doesn't fix everything and large PRs have a lot of room to be faster. It's still good to know where some worst performance issues are to see if there's anything particularly problematic or if a future change will help.
Hopefully the published postmortem will announce that all features will be frozen for the foreseeable future and every last employee will be focused on reliability and uptime?
I don’t think GitHub cares about reliability if it does anything less than that.
I know people have other problems with Google, but they do actually have incredibly high uptime. This policy was frequently applied to entire orgs or divisions of the company if they had one outage too many.
For what it's worth, I doubt that people think it's the engineering teams that are the problem; it feels as though leadership just doesn't give a crap about it, because, after all, if you have a captive audience you can do whatever you want.
(See also: Windows, Internet Explorer, ActiveX, etc. for how that turned out)
It's great that you're working on improving the product, but the (maybe cynical) view that I've heard more than anything is that when faced with the choice of improving the core product that everyone wants and needs or adding functionality to the core product that no one wants or needs and which is actively making the product worse (e.g. PR slop), management is too focused on the latter.
What GitHub needs is a leader who is willing and able to say no to the forces enshittifying the product with crap like Copilot, but GitHub has become a subsidiary of Copilot instead and that doesn't bode well.
> people think it's the engineering teams that are the problem;
It could be, some people are just terrible at their job. Lots of teams have low quality standards for their work.
Maybe that still comes down to leaders but for different reasons. You can ship useless features without downtime.