> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
It was "multiple human errors". But yeah no matter what, in any case or exceptional circumstances, this thing was not supposed to be screwed up.
On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps.
source:
What's the eMudhra security scare that has rattled the internet?
https://theprint.in/tech/whats-the-emudhra-security-scare-th...
These "wrapping" keys were not "weaker". They were completely unfit for purpose. They have not explained why they used a 3-bit key for this purpose.
If they go to the trouble of wrapping them, use real keys. Or don't do it.
Also, it's a big "human error" to not notice this for a year and a half.
I have to imagine that the 3 bit numbers were slots/indexes on the HSM, and rather than actually retrieving the key(or maybe it did), the slot number got used as the key. I can't otherwise imagine how they'd generate such low number keys...
I have not read up on all of this, but I would have guessed they were artifacts created and validated by a testcase. "testcase 1-8".
The key '4' was just the smallest key, there was a set of keys all below 300
AI ceremony.
Having worked around these types of devices and their interfaces, I doubt they intended to create weak keys.
It is more likely they had a script which placed a parameter in the wrong field, due to something like using positional arguments. Or they misread the documentation for some interface.
Still incompetence. And I would love to see my theory confirmed or refuted.
The previous discussion advanced the theory that they fucked up the string handling. Eg in Python, output = key[0] would result in one-byte keys like "4" if you expect "key" to be an array, but it's actually a string.
> worked around these types of devices and their interfaces..
This is not a problem of devices and interfaces. You are supposed to generate the wrapping key on the HSM itself. And the HSM ensures that the keys are generated with a strong enough security parameter, not some silly 1 byte string. Then the HSM wraps the key inside of itself and exports it in the encrypted form. You take this ciphertext, and the wrapping key and send it over to the other party.
So if they generate the wrapping key with a funny value, that means they didn't use HSM to generate it but their system openssl. But then, HSMs typically don't allow you to export private keys without wrapping them first unless you first tweak around with security flags (not recommended at all), or simply generate the private key using your system openssl too (LMFAO).
Source: I work for one of the major HSM vendors.
I hear you in terms of how these kinds of mistakes typically happen. But why did the script ever produce two sets of keys? The initial root cause was somebody picked the weak key not the strong key. But what was the reason for ever producing a weak key in the first place? They say the script produced two outputs - what on earth for??
Stepping back a little to look at the point and gravity of the exercise makes me wonder why they didn’t just fly it over.
I mean, why have the ability to generate weak keys to begin with? Serious question, I've never done this sort of work. What sort of utility does that bring to the procedures and workflows one does in that line of work?
If you didn't know better, you'd think this had been done on purpose by some three-letter agency.
Testing could be a reason.
To be clearer, I’m not sure these keys were generated, but being able to use weak keys is something you may want to do for testing.
Yeah. I can't think of any development process where you wouldn't simply use a production grade key in development. In fact, that's exactly what we do in development with snake oil keys.
Generating a key with "4" was probably done as a joke, but also is insane to have ever lived outside some devs computer.
> These "wrapping" keys were not "weaker". They were completely unfit for purpose.
Oh come on. They clearly didn't intend to use 4 as a private key or to wrap it. They had just purchased millions of dollars of HSM's to store them. It was something more subtle than that.
Have a little imagination. The transfer procedure sounds bespoke - possibly just a shell script using openssl to encrypt things at one end, and decrypt them at the other. Let the man who has written 100 shell scripts without a bug in one cast the first stone.
But not so subtle that good software engineering procedures wouldn't have picked it up. In fact today just an LLM review of the shell script probably would have caught it. The Internet Death Sentence was warranted - because poor software engineering procedures are unforgivable in a CA. But those procedures and policies aren't easy or cheap to come by. If they were just "don't chose 4 as a primary key" my grandkids could do it.
Unwrapped extraction is the kind of thing security by checkbox covers and process review might catch so it is never clear in a case like this that some party with the capability to influence the process didn't want keys intentionally weak to emulate unwrap.. Then someone else may have introduced the swap of keys that would never have been seen with keys that would (be forever) in a public scrutiny DB.
I don't think it's implied the wrapping keys were the weak ones. More like, the script that ran the wrapping still had the fast "debug" generation of a pair of keys to test the wrapper on enabled, for example.
Well said. I choose not to believe this was an error.
vibe coded?
and imagine being responsible for halving your corp's stock
wait, a 3-bit key??
One of the 384-bit keys was literally 4. All of them fit in a byte.
Their complaint about social media reeks of "and we would have gotten away with it, if it wasn't for those meddling kids". [1]
[1] https://www.youtube.com/watch?v=hXUqwuzcGeU
We deeply and sincerely apologize for getting caught and not being able to cover it up.
This is pretty normal in India unfortunately. Crony capitalism is the dominant business engine in the country, so pretty much any significant business, esp in regulated sectors like govt mandated services(which is the bulk of eMudhra's operations) is impossible to set up without significant bribes being paid.
As a consequence, these businesses are very well protected from normal market forces and are pretty much above the law. Had this incident not been an international one, they had a very good chance of getting away with it, and the people reporting this would have landed in jail.
This is 100% accurate. State run media would have squashed this article if it was not for an international incident response. They have federal laws that prevent such disclosures because it affects the national image and could cause social unrest.
Since you seem knowledgeable about this topic.
Would you say the system forces people to be like that or people are the reason why the system is like that?
Would you do business (i.e. trust) with someone that comes from that system?
You're asking a chicken and egg question, I believe it becomes a self-reinforcing loop
In regards to doing business - someone that comes from a corrupt society with a culture of taking advantage of others should be approached more carefully than someone that grew up in the same town as you, of course
This is a lesson learned the hard way, where I moved to a different country and treated everyone with openness and generosity (of time, resources, willingness to help), just to realize people from certain cultures mistake this behavior for weakness and try to take advantage as much as possible
Which country is this? So I can avoid it.
> Would you say the system forces people to be like that or people are the reason why the system is like that?
It's the courts. Courts are meant to bring in an error correction mechanism. When courts fail, the country goes into open-loop control. No errors are corrected, and eventually, people stop even detecting errors. https://x.com/captgouda24/status/2110089855380369511
> Would you do business (i.e. trust) with someone that comes from that system?
Vet rigorously and continuously! Start with small business, and grow it based on interpersonal trust. Since the system lacks accountability, you must ensure that yourself.
Wild to me that our healthcare data lives there.
Depends on the contract your healthcare providers/payers/etc. have with those who do their IT.
Source: work on healthcare IT projects and can't let anyone outside of the US look at prod data no matter what due to the contract we have with our clients.
[dead]
>capitalism is the dominant business engine in the country,
The government _mandates_ that certain services be used. In which world does this rhyme with "capitalism"?
I said _crony_ capitalism is the dominant business engine in the country.
The govt mandated services are owned by private capital(hence the capitalism part) and govt picks winners by granting licenses to a select few(hence the crony prefix)
Any world where sufficiently wealthy capitalists can bend the govt to their will, i.e., most of them.
It does, but I wonder how much of that phrasing is cultural.
When working with different businesses across the globe, I’ve noticed Indian correspondence to incidents tends to be more a show of strength even if it’s transparently obvious that there’s no merit to the content.
Whereas western European correspondence is often more apologetic and empathetic. Even if it might sound insincere to those affected.
But it’s all just words at the end of the day. It never means much unless the business is willing to back real change to mitigate similar issues in the future and resolve undisclosed existing problems. However there’s no way for us to know if this is happening or not.
I don't know if it's really specific to one culture. Look at US companies after they go through a lengthy legal saga, it's obvious they screwed up, they're paying fines or having to adhere to court-imposed consequences. Their press releases:
"We admit no wrongdoing. We are only settling to move past this. Our business is strong/secure/competent."
It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
>It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness.
It's the first rule you learn after joining basically any organisation larger than 5 people.
People on HN who are surprised are seemingly self-employed prodigies who dropped out of middle school.
I’ve worked for a lot of enterprises across the globe and I can tell you that you’re flat out wrong. That’s simply not how all organisations operate.
Just take a look at any of the thousands of postmortems published over the last decade and you’ll see that the tone of incident response communication can vary massively.
The truth would have come out eventually- after a bunch of security cluster fuck incidents.
Their OTP solution is bruteforceable as well: https://mastodon.catgirl.cloud/@49016/117475921584308861
This becomes an inverse Russian roulette: only one of six chambers is not loaded.
Waiting for a release like "... the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP 1FA place in our business. In view of this, after detailed deliberations our board has concluded that withdrawing from global OTP 1FA is the right decision for eMudhra and its shareholders"
> This becomes an inverse Russian roulette: only one of six chambers is not loaded.
At what point does it become Russian roulette with a Glock?
One chamber is enough to play Russian roulette... once. However, the probability to get to play a second time is rather slim.
Link is broken, gives some randomly generated text when using Brave on Android. Their AI bot detection is not very good.
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
That Web site thinks that my browser is an AI scraper for some reason.
It's not clear what you have linked there.
Paste from mastodon:
> fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise:
"However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP's place in our business"
Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.
Then indirectly the security and trust of the entire Internet rests in the hands of a few government babus in India... that's scary.
Their Internet registry people also weaponized AS0 (zero) as a method to take networks off air via BGP RPKI for the ranges they're responsible for. There's been a number of situations where multinational companies who operate in India who are forced to use IPs from the India RIR have been fat fingered by some staffer who makes changes then goes home for the day: meanwhile a multinational bank is down for 24 hours within the country because said agency doesn't believe in 24/7 support.
Typically they do this for non pay but they've dropped the hammer multiple other times incorrectly (and good luck getting a RFO out of anyone there).
For good or bad, thats by design. These is a significant portion of physical infrastructure associated with internet data flow. The only challenge governments face as of now is being able to segment the traffic to their wishes in its entirety.
Companies already control all the traffic flow on their infrastructure using end-point security solutions, the same can be extended at ISP level, and then Country level
It’s called regulatory capture.
It’s get worse when these companies work with regulators to create more regulation that they can sell more of their services.
https://www.amazon.co.uk/Compliance-Industrial-Complex-Opera...
Regulation like this makes sense, but then the government should also srutinze these companies
Well Indian regulator must have visited office canteen to see no unapproved food was served there.
Only when fully satisfied they'd have given license to operate this security business.
"bias"
"out of proportion to SSL/TLS’s place in our business" is an insane comparison. They are saying they care so little about SSL/TLS to begin with that it's not fair.
It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care.
"We are bad at this job, but we don't do it often so don't think we are bad at other jobs".
Yeah, the proportion if anything is too small. This is the sort of company that should not be doing anything related to security.
This is what I found regarding how it happened, but still not a full explanation - but there was some name confusion and obvious incompetence.
>According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla).
https://www.fairssl.se/en/articles/emudhra-ca-noegler-knaekk...
Fantastic key ceremony. Great time had by all. Can't remember a damn thing, we were all so plastered.
I wonder. Say I wanted to prevent something stupid like this from happening in the future. What would be a sensible check? The higher level authority doesn't see the private key itself, right? Just the certificate request. And I wouldn't want someone to manually inspect the keys to see that it looks random-ish. Perhaps there could be a small check script added to the generation of certificate requests that says "hey, your key a.) is mostly zeroes, b.) appears to be ASCII text, or c.) seems otherwise extremely non-random, please use -f to continue anyway".
I don't know if it will work on something as small as a private key, but you can compress the key. If the compressed key is significantly smaller then the original, it lacks entropy. Pure random data is mostly incompressible.
Note that 32 0x00 is a possible output from a pure random generator while still being quite compressible, you would probably still want to throw that one out though. After buying a lottery ticket.
Do not allow devs to modify ceremonial hardware.
I was about to reply with "they didn't have a choice" but on second thought perhaps CAs are important enough that they should never be allowed to export keys from an HSM and should instead be required to physically deliver and install the HSM wherever it needs to be in the world. Inconvenient and expensive but likely for the best.
Right? I don’t know why anybody would trust this company for anything.
Someone higher up in the hierarchy telling you to trust this. You have no option to challenge it.
Russian Roulette is pretty accurate if not an understatement.
If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place.
As a trust provider you get one chance. Squander it and you're out. That is how it should be.
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
In any legal dispute involving them, Exhibit A will be "look at how they ran their SSL/TLS business". That won't go well.
They got their image hurt, and pulling the culturally appropriate response here which is to never apologize and to never accept fault. Instead quit/resign and say that other stuff is just fine. It's really not fine, and you can bet that the same corporate philosophy that made the CA error will have serious security errors in their other products.
I think it doesn't matter what they say because they'll just get away with it. It's as simple as that. They'll get away with it.
[flagged]
There is no such honor code and stop spreading this BS propaganda. Izzat is not even an Indian word used in any Indian language. It is Arabic which got mixed in with Persian which in turn got mixed in with Urdu.
Well reposted.
West has something similar. Politicians never apologize, and avoiding Indian certificate authorities (and recuriters) is racisist!!!