> what Windows actually expects the vtable to look like
I'm not a Windows-knower, are vtable layouts part of the user<->kernel ABI?
> what Windows actually expects the vtable to look like
I'm not a Windows-knower, are vtable layouts part of the user<->kernel ABI?
All of the major kernel ABIs are using C function interfaces as their stable ABI (except Linux, which uses an assembly syscall instruction as the stable interface, although you still rely on a lot of the ancillary C ABI for things like struct layout or stack layout).
For C++ ABIs, there are really only 2.5 major ABIs: the MSVC ABI, used by MSVC and clang wanting to be compatible with MSVC, and the Itanium ABI, used for everything else. There are some slight variants on the Itanium ABI which makes the ".5": ARM uses a different layout for exception handling tables, and there are some flags you can set to use a more compressed vtable layout (32-bit offsets instead of 64-bit pointers). (There are other older ABIs, but either companies stopped making a C++ compiler or they switched to Itanium.)
Windows COM APIs (which isn't part of the kernel, they're still userspace libraries) rely on an IDL which is meant to be directly compatible with the C++ vtable. That said, they also use a restricted subset of C++ that all of the ABIs are going to agree on for vtable layout--if you don't overload any functions, and you don't have any virtual inheritance, there's pretty much only one possible sane vtable layout, and everyone does that.
Being pedantic, the COM vtable layout is defined independently of C++. The Windows headers used to (and maybe still do) have macros that could declare COM vtables as explicit structs so you could use Windows COM interfaces from C.
That used to be the case anyway. It's possible they binned the C support in the switch to 64-bit. I haven't looked since.
It’s part of the MSVC x64 ABI, not a part of the kernel ABI. The kernel and user mode (Win32) APIs all use C linkage so vtables are not relevant and the ABI is a convention of the compiler not the OS. Practically speaking though if your software runs on Windows it will probably use the MSVC ABI.
Last time I checked, software compiled with both Cygwin and MSYS2 internally uses SysV calling convention, only switching to WINAPI when calling, well, Win32 API.
It is more complicated than that. MSYS2 isn't just Cygwin. It has different environments.
MSYS environment is just Cygwin and it uses Itanium IIRC for C and C++ non-Win32 calls since it emulates POSIX. If it links with Windows C runtime they use __cdecl. Win32 API calls always use __stdcall convention / ABI.
However, Mingw environments (Mingw64, UCRT, CLANG etc.) are intended for native Windows development, hence, they use normal MSVC __cdecl ABI for C library calls and internal function calls too. Win32 calls are again __stdcall.
Unlike C, for C++ all MSYS environments use Itanium ABI since they link with GNU libstdc++ or Clang libc++. I think it is possible to use MSVC C++ ABI with Clang but it could be complicated since it requires recompilation of at least libc++.
Yes, you either follow Itanium ABI (non Win) or MSVC ABI. Technically nothing stops _your_ compiler from implementing totally own conventions but it will only be compatible with itself
GHC uses its own calling convention yet it's compatible with Win API just fine. The secret, of course, is to implement not only your bespoke calling convention, but also the other ones that you care to interoperate with.