Hi, I’m helping a hospital in Buenos Aires that has lost their IPv4 due to their ISP moving to CGNAT, putting their on prem imaging servers offline for 7 days. (it’s ok for us though, we have a backup failover isp)
Might have to do something horrible like:
- setup an ipv4 tunnel, which will require hacking a VPN to allow TCP listening, and routing connections to some external proxy with a dedicated static ipv4 address
- or do some modem router level nat port forwarding, and use a ddns to avoid ip changes breaking stuff. but that won’t work probably since public ip is shared
will investigate the isp to see if they are truly short in ipv4 or if they are just moving clients off them in order to sell them to higher bidders and plans, which is likely since I’ve been upsold into optic fiber, static ip surcharge, sd-wan, dedicated connection so far
if you have any advice or RIR contacts (or an ipv4 block to donate, invest or sell), I’d be thankful, my email is in my profile
The first thing you need to ask is: Why do you need IPv4?
I don't know much about your imaging servers: Do they accept incoming https connections, or is it some other kind of TCP or UDP protocol? Assuming it's https, why do you need to stick with IPv4?
> Hi, I’m helping a hospital in Buenos Aires that has lost their IPv4 due to their ISP moving to CGNAT, putting their on prem imaging servers offline for 7 days. (it’s ok for us though, we have a backup failover isp)
Two more things:
1: If you require IPv4, that needs to be part of the contract with your ISP. If they just arbitrarily turned you off, it means that you didn't contract for the services you thought you had.
2: IPv6 is supposed to solve a lot of these problems... So again, why can't you move to IPv6?
> I don't know much about your imaging servers: Do they accept incoming https connections, or is it some other kind of TCP or UDP protocol?
Yes
>why do you need to stick with IPv4?
To support devices and routes that do not have ipv6 support?
>1: If you require IPv4, that needs to be part of the contract with your ISP. If they just arbitrarily turned you off, it means that you didn't contract for the services you thought you had.
That is correct in principle. Although, in the absence of a contract, if a service has been provided for a long time in one way, I'd say there's an argument for breach of contract when that way changes for the worst.
Another argument for breach of contract, which can work jointly with the previous, are industry standards, as far as they define the terms that are implied by common words like "internet connection".
Finally, there's the specifications themselves, since the service is billed as "internet", one can refer to the Internet Protocol Specification, although if they do offer IPv6, that might be reasonably considered a substitute, it is a material change that was not agreed to originally (this being a pre IPv6 contract). That said, an IPv4 with a local address might technically still be defined as internet, what is broken in this case is TCP, not the IP, although by lacking a public IP address, TCP is broken via an IP protocol failure, so there's an argument that something is being sold as Internet but it isn't really internet, there's no written contract necessary, the term used to describe the product creates the contract in the same way that selling a carton labeled Milk would create a contract based on milk coming from mammaries of a domestic mammal, and not pressed Soy Juice with artificial coloring.
>2: IPv6 is supposed to solve a lot of these problems... So again, why can't you move to IPv6?
I'm open to proposals to moving IPv6, I'm not content with having that be forced and removing it from people and institutions, if there's a shortage, then that means no new IP addresses can be issued, why are we normalizing the removal of IPv4 addresses? Are they being removed from lower priority users in order to serve high priority usecases? If so is a hospital a low priority user?
The main issue with IPv6 seems to be IPv6 proponents that try to kill IPv4, no one will migrate if you force them to, if you propose, we will consider.
> The main issue with IPv6 seems to be IPv6 proponents that try to kill IPv4, no one will migrate if you force them to, if you propose, we will consider.
My experience is more the opposite. I keep seeing people say things like "I don't do v6 at home because I don't need it". These people are outright telling us that they're not going to start using v6 unless we make them need it.
Note that I didn't say "migrate to" v6. They're not even willing to do the first step of that ("start using v6"), even with no pressure to do the last step ("stop using v4").
I read an anecdote from Facebook that they had huge trouble getting their developers to stop writing v4-only code. Eventually they ended up removing v4 from their developer machines, who all of a sudden were able to write v6-capable code -- and it didn't take long for them to start asking when they could drop support for v4.
People will moan and moan and moan about v6 until you make them use it, at which point they discover that it was never as bad as they imagined it being.
So you admit that IPv6 people routinely create the problem if it doesn't exist. Maybe there's some nuance where you need to go backwards in order to go forwards?
In any case it's not relevant to the case from the root comment, since the ISP doesn't offer IPv6. IPv6 folk are siding with the ISP here or getting distracted with IPv6.
Essentially IPv6 people are delusional about the state of support of the tech and the extent of issues it brings, there's regions where there's no IPv6 support, so when a network with IPv4 has an IP related problem, IPv6 folk will blame it on IPv4 even if it's entirely unrelated, even if the infrastructure isn't even at dual-stack phase.
One of the thing the ISP did was try to upgrade the connection from copper to fiber optics, like instead of understanding the issue, "let's just try upgrading from your OLD technology which is OUTDATED, no wonder you were having issues? Also have you tried turning it off an on again that helps? Talk to you tomorrow.". IPv6 guys so far have been a distraction, get walmart to run their website on IPv6 before you push it to a hospital in southamerica who is having downtime.
The professional services that I work with give plenty of notice (years) before turning off services. Were you using an internet connection that's intended for hosting servers? (As in you were paying for a dedicated static IP.) If so, then you have a case.
Otherwise, your arguments about what "internet" is are flawed. IPv6 has been a thing for about 30 years, with a well-publicized problem with IPv4 address space and migration plan. I suspect that it's (IPv6) older than your hospital's contract. Even in the IPv4 world, your IP can change at any time and you can be put behind a NAT. That's why ISPs will sell you a static IP if you need one.
>> why do you need to stick with IPv4?
> To support devices and routes that do not have ipv6 support?
> Are they being removed from lower priority users in order to serve high priority usecases? If so is a hospital a low priority user?
After reading your response, I suspect the real problem is attitude and experience. The hospital should have IT staff that know to purchase a static IPv4 IP. This should be part of the contract with the ISP. If it wasn't, ultimately it's up to the IT staff to approach the ISP and arrange to purchase one.
> The hospital should have IT staff that know to purchase a static IPv4 IP
I purchased a static IPv4 IP, and they made the CGNAT IP static.
I thought about making a blog post about it, but I realize there's so much nuance and details. I can't tell if it's just venting about work, or a genuinely interesting clusterfuck in some forlorn corner of the new world.
> I purchased a static IPv4 IP, and they made the CGNAT IP static.
Thank you, I needed a good laugh today, that is amazing!
Have you noted that in the mainline Linux build config file, the option to enable IPv4 functionality is CONFIG_LEGACY_IP= ?
I can tell you that in cicd for production Linux systems today, we have
which in a Linux build config file means "do not include this in the build"._
What might work well for your organization is ZeroTier which will make you immune to what any ISPs do. The downside is your organization would be dependent on ZeroTier, but ZeroTier is much less likely to screw you over than any ISP.
_I appreciate your speculation on the principles of contract law. Like you, I believe those principles should be upheld and I believe in morality. If you lived under a monarchy and I were of the monarch, I would cross-examine your ISP in public and solve your problem in minutes. I would also on occasion take someone to the side of the road and chop off their head in front of the entire village, and everyone would know on a visceral level they are all much better off with someone taking out the trash now and then.
But, as many technical workers know, the contract stuff takes months or years to resolve, and technical issues involving IP addresses are best worked around with technical approaches.
_
What I would suggest your network team (you and whoever) do in your shoes is:
- set up NAT64 and DNS64 at the edges of subnets that contain devices that don't do IPv6 properly, because if these devices do not have updates to do IPv6 properly then they are broken in a fundamental way, and a workaround is necessary;
- dictate that starting on December 1st, all new devices and VMs have IPv6 addresses with all userspace services listening on IPv6 addresses or some nftables rule to kludge IPv6:$port -> 127.0.0.1:$port if necessary, with firewall configured to block that IPv6 traffic, then at a later date firewalls not blocking those IPv6 addresses, then at a later date prohibit any new IPv4 addresses being assigned anywhere with special exception to devices tagged "legacy" or "end of life" requiring documentation of such;
- use a combination of IPv6 GUA addresses assigned by whatever ISP and IPv6 ULA addresses chosen by your team -- in IPv6, multiple local gateways and multiple addresses on a network interface issued by those local gateways via RA is relatively a norm rather than an exception (requires userspace tooling in IPv4, DHCP doesn't have this functionality), so leverage this advantage;
- use IPv6 AH (authenticated headers) wherever possible;
- because in an ideal world, ISPs would behave consistently and IPv6 GUA addresses would always be stable, but we aren't living in an ideal world, so be ready to update your internal DNS to drop AAAA records containing IPv6 GUA that an upstream ISP fumbled;
- or, if you have the internal resources, register your own AS and command your own official block of IPv6 addresses;
- use a combination of OpenTOFU (Terraform) and Salt or Ansible, all versioned with Git, be very strict with your team about versioning with Git, to manage everything, and use
and Salt or Ansible equivalents for continuous monitoring;- move userspace services to Podman where possible.
Personally I like Salt. I know Ansible is more popular out there with more inertia.
_
>> why are we normalizing the removal of IPv4 addresses?
This was normalized a long time ago actually. "The future is already here, it's just not evenly distributed".
You and I are total strangers on different corners of the globe. You may not like what you're about to read, but it's better that you get a hint where the wind is blowing sooner than not.
If my team were consulting with your organization's execs and we caught a hint that the existing internal resources were not already proactively staying current with internet protocol (which is IPv6), we would point out there is urgency to start right away with emphasis that the existing internal team is over two decades behind staying current.
Or, if a detected hint the internal team were at all resistant to IPv6, we would advise the executive team to start developing a plan to flush out the replacements for the internal team ASAP.
Many network engineers at many ISPs would gladly drop all IPv4 traffic if they thought they could get away with it.
IPv6 only seems more complicated if you still need to unlearn IPv4. IPv6 does have a steeper learning curve, but once that learning curve is overcome, IPv6 is actually much simpler than IPv4. More people than not don't like reading RFCs, but professionals do slowly read RFCs on their own time.
Thanks, I'll take a look into the recommended protocols.
One clarification I'd like to make is that IPv6 is not offered, I can only get IPv4 addresses through the ISP DHCP, there is no ipv6 being offered with command 'dhclient -6' for example on linux, neither does window pick any ipv6 up so it doesn't seem to be an us problem.
It's often that Argentinians miss nuances when copying ethical gray areas from the Americans, and without those slight nuances the debate often gets mixed up with a much more subtle debate.
To make a parallel, Trump issued a memecoin days before entering office, it wasn't technically while being president, it was a memecoin so there was no underlying asset, a gray area, see? America.
Now Milei saw and culted the cargo, he made a coin during his office mandate, and he mentioned it would be used to fund Argentina tech projects, so it's an unregulated security. The nuances are lost, and it's just plain bad.
EDIT1:
>Or, if a detected hint the internal team were at all resistant to IPv6, we would advise the executive team to start developing a plan to flush out the replacements for the internal team ASAP.
>we would point out there is urgency to start right away with emphasis that the existing internal team is over two decades behind staying current.
I appreciate the candor
>You and I are total strangers on different corners of the globe.
This is true and it might impact the nuances and extra-technical semantics of the protocol. For example, in here dropping IPv4 support might mean dropping support for older devices, which might be less common in America, ISPs might not support it, so it's not just about the device but the network, to give you an idea I just got off the phone with the ISP which is THE biggest ISP in Argentina, they don't offer IPv6, period.
>If my team were consulting with your organization's execs
Here might be a big disconnect, my organization execs would probably be way out of budget for whatever your org may charge, that would never happen.
If you want to point the salesmen of your org to try and stir some shit up like that with the ISP, then that may make a bit more sense, the budgets may align for importing American consulting, and berating a big org chart that includes engineers might make more sense than trying to replace a 'team' of 1 or 2 fractionalist generalist IT guys on <500$ month contracts.
The ISP brand is Personal, local corp is Telecom Argentina S.A, Asynchronous System Number AS7303, trades as TEO in NYSE I think, owned by Grupo Clarin which comes is a traditional Media and news company. IPv6 is not offered to consumers right now.
>we would advise the executive team to start developing a plan to flush out the replacements for the internal team ASAP.
go nuts
Everything I am suggesting here is what an internal team with full ownership of the network should do. An external team could chisel little by little, and progress the network in phases.
What is shared here is a general direction. I am sure some people I know where all they do is networking could tell me harshly some of these ways could be done better, and often these people are right ; but sometimes these same people are actually wrong and don't fully consider the positions of people in IT roles at companies that aren't ISPs.
All suggested below is not necessarily the ultimate way but a general direction.
_
>> can only get IPv4 addresses through the ISP DHCP
ISP-owned IP addresses, either of legacy IP or IPv6, are capricious and can suddenly change, as you experienced recently.
I suggest to adopt a mindset of "our network" and "their network", rather than functionally have your network be a passive extension of their network by using their IP addresses.
The router announcements (IPv6 RA) or DHCP (IPv4) for any IP addresses within your network should always be issued by devices controlled solely by our organization.
NAT64 and tunnels exist, any upstream ISP is not a blocker.
Modern Linux has eBPF and eXpress Data Path. Both Linux and FreeBSD (use HardenedBSD if you can) have DPDK. They can make for surprisingly high performance edge routers on x86_64 server boxes over a decade old, exceeding what most campuses or even datacenter footprints actually need and at low cost. Even without eBPF & XDP or DPDK, these on x86_64 are often "good enough" for router and firewall.
DNS running on these or adjacent boxes internal to the organization is essential.
The addresses at the gateways, either of IPv4 or IPv6, can be made highly available with VRRPv3.
IPv6 has an elegant way to handle multiple upstream ISPs without VRRP or CARP at the gateways, but even Linux is just recently implementing what may only be a piece of them (the BSDs will later), some mention of them here https://news.ycombinator.com/item?id=50143420
_
What I would do, which is opinionated so I'm not necessarily suggesting others do it, assuming tunneling will be involved: if the organizational wherewithal is there to register an AS and do BGP, is pay the AUD 500 fee to register an AS and have a chunk of IPv6 space for to use IPv6 GUA addresses that officially belong to the organization.
If not, which is often enough the case because we know how some organizations can be slow (not making any assumptions about your's), the remaining avenue is to use IPv6 ULA addresses which are the spiritual counterpart to IPv4 RFC1918 addresses.
You can embed MAC addresses in IPv6 ULA addresses, like the MAC 82c6.7943.acb2 put in an IPv6 like
and use DHCPv6 to always assign a specific IPv6 ULA address to a specific MAC address, where you have thought through the potential information-disclosure implications and you are certain you want to do that.Then I would use IPv4-only ISP paths for what they're good for: dumb paths to tunnel IPv6 packets to where there is proper internet.
For the option with an AS, BGP announcements on the other side of the tunnel. Personally I would do SCION on the gateways and out the other side of the tunnel, but you don't want to do SCION unless you are also crazy (like a fox).
For the IPv6 ULA option, PAT which is the spiritual counterpart to IPv4's NAT but is also technologically simpler.
With IPv6 ULA and no tunneling, NAT64 and DNS64 can work for outbound.
Site to site tunnels will be tricky no matter what. Tailscale, ZeroTier, Netbird, and similar may be able to do much of the work for you ; these should run in a DMZ on another side of a router/firewall that filters the appropriate IPv6 ULA prefixes.
I would try to see if I could make vanilla Wireguard work with NAT traversal techniques using some cloud-hosted VMs that help with the "hole punching", but not everyone would do that.
There aren't many options that will all work well but almost always more than one option that can work well, all subject to the specific site and inter-organizational resources.
_
>> dropping IPv4 support might mean dropping support for older devices
For as long as it makes organizational sense to continue using these devices, put them behind an OpenBSD box that performs NAT46 and DNS46, with IPv4 addresses statically mapped to IPv6 addresses, maybe ones that contain the MAC addresses of the devices.
With Salt or Ansible, such a scheme with NAT, DNS, DHCP configuration may all be templatized and in perfect alignment, and is scalable.
Authoritative inventory of canonical identifiers/names versioned with Git, what Ansible calls "inventory", is imperative.
Sometimes this looks like a couple of old x86_64 boxes doing CARP (similar to VRRP) on the IPv4 side for a few dozen or couple hundred devices downstream.
Sometimes this looks like, in a handful of distant corners of the larger network, a cheap single-board computer between a single device and the rest of the network ; without a wifi chip or the wifi chip removed by heating (board-repair style), unless downstream of a wifi AP.
_
>> the ISP which is THE biggest ISP in Argentina, they don't offer IPv6 >> The ISP brand is Personal, local corp is Telecom Argentina S.A, Asynchronous System Number AS7303, trades as TEO in NYSE I think, owned by Grupo Clarin which comes is a traditional Media and news company.
I can think of some service provider network engineers that would gladly drop all of Argentina's IPv4 off the entire internet and "IPv6 prefix announcements welcome", if the business sides would let them do it.
Something like this will probably start happening sooner than people think.
>I can think of some service provider network engineers that would gladly drop all of Argentina's IPv4 off the entire internet and "IPv6 prefix announcements welcome", if the business sides would let them do it.
Yes, there's no shortage of employees that would vote to reduce their workload by removing their service offering, if you ask a physician here if the hospital should stop taking patients of X type, they'd be onboard, less work for them.
It's a dream though, try to make an AAAA request for any website, it's not there, even government websites.
>Something like this will probably start happening sooner than people think.
I'm even trying american companies, like citi.com, walmart.com, there's no AAAA records there.
>For as long as it makes organizational sense to continue using these devices, put them behind an OpenBSD box that performs NAT46 and DNS46, with IPv4 addresses statically mapped to IPv6 addresses, maybe ones that contain the MAC addresses of the devices.
This is an external facing service, so we don't control the devices or routes that patients and external physicians will be accessing the website.
>Tunneling
Yeah, I'm going to go this route, IPv4 tunnel, it's a horrible hack, but this is hacker news after all.
If the www. is prefixed or not, results for domains of Argentina differ. The results aren't almost-perfect like US, but IPv6 is apparently already somewhat prevalent in Argentina.
The list from ChatGPT:
"This 2026 list ranks organizations by LinkedIn employee count, not revenue, and includes public bodies and universities as well as companies. I’ve kept the listed order; entries without a domain in the source are omitted."
Almost all large companies in the US are on IPv6 and more of the smaller companies are on IPv6 than not.
If your device isn't able to reach 2606:4700:4700::1111, try using the DNS server 1.1.1.1
and run this bash command:
and this one too Then maybe call up your ISP and rant about them allowing your country to devolve into some 3rd world shit hole stuck on legacy IP and over two decades behind the Western countries.Argentina deserves better!
>If your device isn't able to reach 2606:4700:4700::1111, try using the DNS server 1.1.1.1
We run a public facing server, we aren't going to ask patients and external physicians to configure a DNS or run a (linux) script
>Then maybe call up your ISP and rant
You wouldn't be surprised how unhelpful and unfun that is
You're a hospital. You have an actual contract with the ISP, right? Does it say you get an IPv4 address? If not, ask them for one, and pay up.
You don't need an address block from an RIR. You just need your ISP to give you an address.
You could also update to IPv6.
>You're a hospital.
I work for the hospital, occasionally, they are my client.
> You have an actual contract with the ISP, right?
They do not, it's an implied consumer contract, like when someone buys a hot dog from a stand.
I'd like a written contract too, but when trying to negotiate one, they want a lot more money for them, it depends of the nuances of the terms, but I've talked to sales reps, and it comes down to having a 1Gbps line that works right now vs a 100Mbps line that's more expensive, with the only difference being that the contract is now written and there's a symbolic SLA with some 10% penalty, and a single professional beration receiver instead of multiple anonymous on call rotation.
It's not a clear choice, especially considering that the dedicated route with SLA means introducing change risk to the operational and SLA benefit of the ISP, fiber, new holes, new lines, etc..
>You could also update to IPv6.
I would, they don't offer IPv6
>You don't need an address block from an RIR. You just need your ISP to give you an address.
Yeah, maybe. If I get an IPv4 block though what I'd do is a regional entity whether for profit or NFP to offer emergency regional tunneling, potentially while finding alternative revenue streams like collecting lists of wronged ISP customers and notifying them of class action lawsuits they may participate in. A 25K check, or a 5K check with an ipv4 block and I'm golden.
Well if your hospital is running on the cheapest possible internet made for houses, then you have no rights. You should GET a contract that includes what you actually need.
Email sent!