If you don't trust your devs, CI is rarely going to protect you (at least in most setups)
Your dev could just replace all the tests with a "return true" to bypass traditional CI, too,
This makes an explicit step where you are vouching that you have run tests. If it turns out you didn't, it is going to be found out when someone actually DOES run tests and they fail... at that point you can discipline the developer for lying.
It isn't a matter of trust, its a matter of developers are humans. Humans make mistakes, and continuous integration helps catch mistakes.
I've been coding for 25+ years, and I sometimes I forget to remove debug output, or I forget to update an explicit exception I threw somewhere, or I forget to make sure dependencies build across all the platforms we need.
Running your build a second machine helps to catch a lot of those mistakes. It also helps enforce automation and good configuration practices and lint/formatting standards and all that, but fundamentally its helps us because humans make mistakes.
CI doesn't have to mean setting up a 12 step pipeline in GH actions with 3k lines of yaml and crazy triggers and workflows. It can just be an old dev machine (https://www.jamesshore.com/v2/blog/2006/continuous-integrati...) that pulls the latest code, runs `script/build`, and POSTs results somewhere.
Maybe local signoff works for 37signals, if so, thats great. But it isn't continuous integration.
If you don't run anything different on the CI machine than locally, then how is your post an argument against signoffs? Why is it not continuous integration if the step is run on a different computer?
Humans are easily confused though. I've been 100% sure I've run the tests and haven't many times. It's quite easy to do: make a change, run the tests, fix it, run again, everything works. Make some trivial refactor that "shouldn't change anything". Your brain says "it's fine, I ran the tests, remember?" and you push. But you didn't run the tests.
"First you must not fool yourself, and you are the easiest to fool"
Dev replacing all tests with return true would should up in the PR review right? I mean a dev that malignant needs to be fired yesterday and someone approving that PR can be shown the door right alongside them.
The most common mistake you'll get here is people running tests, then making a small change later, then having tests fail unexpectedly. Even accidentally. This has happened to me before with regular CI, and often hit me during an automerge I fully expected to pass.
Requiring devs to certify that they ran the tests might be useful for compliance, if that's a thing your org has to worry about and you're really dead set against using CI.
Not to sound like a shill but Nix solves this very elegantly: tests are just a special kind of build (usually with a synthetic output like "echo ok > $out") and can therefore be "cached" like any other build. But because build recipes are deterministic, and Nix uses sandboxed, pure build environments, "caching" is just a special case of "uploading build artifacts". Meaning your cache = build artifact server ("binary store"), and it's shared between everyone (CI, build farm, dev laptops, ..), again because build recipes are deterministic. Meaning: if your laptop (or VM) is the same architecture as your CI you can just run your CI steps locally and push that to the shared store, and your CI will now automatically see the tests have already been run and not run them again. And vice-versa. Across all branches and all collaborators who share that same cache.
This is all deterministic and doesn't require any human in the loop: the test result is hashed by all of the inputs, so you never accidentally spuriously pass a test locally which shouldn't have.
It requires almost no setup at all. There are SaaS which make this basically turnkey (cachix, nixbuild.net, garnix, ..). Getting your app to build in Nix in the first place, though? Years of tears.
Nix lives in this group of tools where "easy things are hard, hard things are easy", and this is a classic example.
Lots of confusion between two things in this thread: 1) is this a good idea? 2) is this a good implementation of the idea?
Whether this is a good idea has been discussed to death already, but assuming you want this (which most people won't, the readme says as much), is this a good implementation of the idea? Yeah, it is.
Requiring a CI pass to merge, and a tool to add the CI pass after some unspecified process, seems like a neat, minimal implementation that still prompts the author enough to prevent most accidental misses. Is it complete? Of course not, but checklists don't need to be complete to be useful. Atul Gawande's book "The Checklist Manifesto" talks about this a bit, just the act of asking is often enough to break us out of habits and get us thinking, and will often turn up more issues than are on the checklist itself.
At Google we have a ton of tooling that amounts to light automation around self-certification. Many checks on PRs (CLs) only require you to say "yes I've done the thing", because that's often sufficient.
Surely if this is about creating a step in a checklist, all you need is a box to tick in the PR template, and that would be an even simpler version of this, requiring far fewer moving parts and being easier to use.
I think part of the criticism of (1) here comes from the complexity of the solution, which makes it feel like it should be competing with a more fully-fledged CI solution. But for a tool where the goal is really just to let the developers assert that they've run some tests, it's surely a lot more complicated than it needs to be, no?
PR templates are optional, many tools bypass them, my previous company tried using them and couldn't make processes stick with them because people used various tools to create PRs.
I'd argue that depending on PR templates, vs depending on devs having the signoff tool installed, is a pretty similar level of moving parts.
However perhaps more important than moving parts is failure modes. The failure mode of this tool is that your PR is blocked from merging until you run the tool. The failure mode of the PR template is that you never realise you missed the checkbox.
> for a tool where the goal is really just to let the developers assert that they've run some tests, it's surely a lot more complicated than it needs to be, no?
I think the point is to have the right amount of friction. Too little, as mentioned above, and you don't realise when you've lost the protection from the process. To me this is a good solution exactly because it gives you a little bit of friction (but not much and cheap).
Solutions with less friction include: just write in the PR description that you ran the tests, or just know that you ran the tests but don't write it down anywhere. I'd suggest that the safety nets provided by these options are less useful because they have less friction.
Is this really more friction? I can very easily imagine creating an alias that runs `git push && gh signoff` for the cases where I know the tests have passed, and I can equally easily imagine running that alias and forgetting that I've just committed a change and forgotten to run the tests. If anything, opening a browser and marking a checkbox there feels like more friction to me.
I could imagine there being more friction if the tool somehow checked that a test suite had been run on the current commit, but that doesn't seem to be the case. It's just a more convenient way of checking a checkbox.
The biggest advantage I can see with this is that the status check gets reset with every push. That's harder to do with a regular checkbox, but I can imagine there's still a way, and I can imagine that way would probably still be simpler than this.
> Remote CI runners are fantastic for repeatable builds, comprehensive test suites, and parallelized execution. But many apps don't need all that. Maybe yours doesn't either.
My reading of that is that if you need repeatable builds, comprehensive test suites, and/or parallelised execution, then this is not the tool for you.
I think a checkbox, suggested in another comment is a worse implementation.
I could certainly see the appeal of this sort of idea. Once your engineering org gets to a certain size, you can end up spending an eye-watering amount of money on CI compute - spinning up runners and executing thousands of tests for every single commit of every single pull request. That cost could decrease by a lot if, for PRs at least, you could use the dev's local machine instead.
The three prerequisites in my mind would be, that the CI workflow runs inside a local cluster of VMs (to match a real CI environment), that the results of the CI run are still uploaded and published somewhere, and that it's only used for pull requests (whereas a real CI is still used for the main branch).
This tool fails the second test; it doesn't publish the test results or in any way associate them with the status check.
Also, a larger organization is one in which human error is more likely, so I would expect a tool that relies on individual engineers not to make mistakes to be less suitable there.
I could _maybe, maybe_ see self-hosted GitHub runners inside of VMs running on developer machines. The VMs would have to be re-created after every request however.
I don't think this is a great idea, however, as now CI is dependent on my flaky laptop's wifi/internet connection/IP address, has the potential to be contaminated by something running on my machine, build logs can be modified, environment shape/architectures are all different and can't be easily controlled, I now have access to all of the CI secrets and can impersonate the CI, etc.
If this is just running tests locally, it seems deeply flawed - e.g. the tests would work even if I forgot to commit new files.
OTOH if starting a new container, pulling my branch and then doing everything, it's definitely as good as running on remote CI because it's basically replicating the same behaviour. And it would likely still be much faster since:
* CI machines are underpowered compared to dev laptops/desktops. e.g. our CI builds run on a container with 4 vCPUs and 16GB RAM. In contrast my laptop has 16 cores and 48 GB RAM
* docker pull itself takes a relatively long time on CI. If I'm running it locally it would just use the cached layers
The tool does actually check whether you have any uncommitted changes in Git, and fails in that case. So you're protected from that particular mistake. You're not protected from mistakes related to running the tests or checking their results, though, because the tool has nothing to do with that.
I wrote and released a similar concept in 2012 in response to co-workers constantly breaking my code and not running tests before committing. https://rubygems.org/gems/git_test/
That was before we had a reliable CI server and let me tell you, fixing the CI server was a much better investment than convincing everyone to run and document their tests locally. The basecamp tool is more polished than what I cobbled together, but I personally won’t be reaching for it any time soon.
It's actually worse than that; this tool doesn't run any tests or do anything with test results. All it does is require each developer to run `gh signoff` before their PR can be merged; the only thing that command checks is that there aren't uncommitted changes in Git. So if your colleagues aren't already locally running your tests the right way, this does nothing at all to help you.
I do like the idea of a variant of this that also includes some signing and logs that share, well, more of the state involved in this.
This sort of work would likely mean that test suites would be built in a way to be run in a more diverse sets of environments. That leads to just less brittle tests overall, and hopefully a better foundation to debug things.
Given how fast dev machines are, doing this might not be a bad idea for small teams _if_ you have some way to ensure reproducible builds/tests. Otherwise I can imagine a lot of "works on my machine" cases
Though if you are an org of 1000+ engineers working in big monorepos better stick to CI
This might sound crazy to a lot of people, but I actually wish I had the same lint/mypy/flake8 CI from work on my personal projects. I don’t need anything fancy but something on github that’s single click that forces discipline.
For example, Run you lint script on pre-commit. You can use `git config core.hooksPath .githooks` in your package.json/cargo.toml/composer.json/etc scripts prepare hook to maintain your scripts in a `.githooks` directory in the repo.
You could also run your code coverage command on pre-commit to help with discipline. Sky is the limit.
Do you use any kind of version control on your personal projects? If so, you should be able to set up a precommit hook. Or is the difficulty in finding a project template that has all the right tools configured in the right way?
I agree with the premise (dev machines are under utilized) but the environment (or lack thereof) of the CI machine is its major selling point. No stupid forgot to add dependency x. Of course it works, you developed it on your machine.
At that rate you might as well force merge, no? That's equivalent to signing off and doesn't make the CI have a false positive signal when it hasn't actually run.
I mean, obviously you want to run the local CI in some isolated way. But it's not such a bad idea for many projects.
At PyTorch, we have the reverse problem, it's basically infeasible to run the CI locally, you really do need the cloud setup to cover all of the operating system / hardware configurations, and the parallelization in cloud also saves you loads of time.
This misses the point of CI. The CI environment is neutral, one of its goals is to avoid the "it works on my machine" effect. If it doesn’t at least run in a container or a VM, it’s not CI.
This simply creates a passing status in the repo so that the pr can be merged.
There is no obligation on the dev to actually run the tests locally, so you could just save time by disabling status checks.
The point of a ci pipeline is that the build ran in a more controlled environment and here is the build log for everyone to see proof.
If you don't trust your devs, CI is rarely going to protect you (at least in most setups)
Your dev could just replace all the tests with a "return true" to bypass traditional CI, too,
This makes an explicit step where you are vouching that you have run tests. If it turns out you didn't, it is going to be found out when someone actually DOES run tests and they fail... at that point you can discipline the developer for lying.
It isn't a matter of trust, its a matter of developers are humans. Humans make mistakes, and continuous integration helps catch mistakes.
I've been coding for 25+ years, and I sometimes I forget to remove debug output, or I forget to update an explicit exception I threw somewhere, or I forget to make sure dependencies build across all the platforms we need.
Running your build a second machine helps to catch a lot of those mistakes. It also helps enforce automation and good configuration practices and lint/formatting standards and all that, but fundamentally its helps us because humans make mistakes.
CI doesn't have to mean setting up a 12 step pipeline in GH actions with 3k lines of yaml and crazy triggers and workflows. It can just be an old dev machine (https://www.jamesshore.com/v2/blog/2006/continuous-integrati...) that pulls the latest code, runs `script/build`, and POSTs results somewhere.
Maybe local signoff works for 37signals, if so, thats great. But it isn't continuous integration.
If you don't run anything different on the CI machine than locally, then how is your post an argument against signoffs? Why is it not continuous integration if the step is run on a different computer?
Humans are easily confused though. I've been 100% sure I've run the tests and haven't many times. It's quite easy to do: make a change, run the tests, fix it, run again, everything works. Make some trivial refactor that "shouldn't change anything". Your brain says "it's fine, I ran the tests, remember?" and you push. But you didn't run the tests.
"First you must not fool yourself, and you are the easiest to fool"
Dev replacing all tests with return true would should up in the PR review right? I mean a dev that malignant needs to be fired yesterday and someone approving that PR can be shown the door right alongside them.
The most common mistake you'll get here is people running tests, then making a small change later, then having tests fail unexpectedly. Even accidentally. This has happened to me before with regular CI, and often hit me during an automerge I fully expected to pass.
That and having local changes to either uncommitted files, or something external like a library version that works here but will fail on deployment.
Most of this signoff stuff isn't about actual trust. It's about demonstrating controls for audits.
You do it on a shared server because you can pull it up and show your auditor "see, we follow our SDLC process".
This is a layer on top of CI for people who didn't need CI but use CI anyway to not use CI...
Requiring devs to certify that they ran the tests might be useful for compliance, if that's a thing your org has to worry about and you're really dead set against using CI.
It says sign off on your own work in the title.
I’m unclear why this isn’t just a PR template with a checkbox
Not to sound like a shill but Nix solves this very elegantly: tests are just a special kind of build (usually with a synthetic output like "echo ok > $out") and can therefore be "cached" like any other build. But because build recipes are deterministic, and Nix uses sandboxed, pure build environments, "caching" is just a special case of "uploading build artifacts". Meaning your cache = build artifact server ("binary store"), and it's shared between everyone (CI, build farm, dev laptops, ..), again because build recipes are deterministic. Meaning: if your laptop (or VM) is the same architecture as your CI you can just run your CI steps locally and push that to the shared store, and your CI will now automatically see the tests have already been run and not run them again. And vice-versa. Across all branches and all collaborators who share that same cache.
This is all deterministic and doesn't require any human in the loop: the test result is hashed by all of the inputs, so you never accidentally spuriously pass a test locally which shouldn't have.
It requires almost no setup at all. There are SaaS which make this basically turnkey (cachix, nixbuild.net, garnix, ..). Getting your app to build in Nix in the first place, though? Years of tears.
Nix lives in this group of tools where "easy things are hard, hard things are easy", and this is a classic example.
Lots of confusion between two things in this thread: 1) is this a good idea? 2) is this a good implementation of the idea?
Whether this is a good idea has been discussed to death already, but assuming you want this (which most people won't, the readme says as much), is this a good implementation of the idea? Yeah, it is.
Requiring a CI pass to merge, and a tool to add the CI pass after some unspecified process, seems like a neat, minimal implementation that still prompts the author enough to prevent most accidental misses. Is it complete? Of course not, but checklists don't need to be complete to be useful. Atul Gawande's book "The Checklist Manifesto" talks about this a bit, just the act of asking is often enough to break us out of habits and get us thinking, and will often turn up more issues than are on the checklist itself.
At Google we have a ton of tooling that amounts to light automation around self-certification. Many checks on PRs (CLs) only require you to say "yes I've done the thing", because that's often sufficient.
Surely if this is about creating a step in a checklist, all you need is a box to tick in the PR template, and that would be an even simpler version of this, requiring far fewer moving parts and being easier to use.
I think part of the criticism of (1) here comes from the complexity of the solution, which makes it feel like it should be competing with a more fully-fledged CI solution. But for a tool where the goal is really just to let the developers assert that they've run some tests, it's surely a lot more complicated than it needs to be, no?
PR templates are optional, many tools bypass them, my previous company tried using them and couldn't make processes stick with them because people used various tools to create PRs.
I'd argue that depending on PR templates, vs depending on devs having the signoff tool installed, is a pretty similar level of moving parts.
However perhaps more important than moving parts is failure modes. The failure mode of this tool is that your PR is blocked from merging until you run the tool. The failure mode of the PR template is that you never realise you missed the checkbox.
> for a tool where the goal is really just to let the developers assert that they've run some tests, it's surely a lot more complicated than it needs to be, no?
I think the point is to have the right amount of friction. Too little, as mentioned above, and you don't realise when you've lost the protection from the process. To me this is a good solution exactly because it gives you a little bit of friction (but not much and cheap).
Solutions with less friction include: just write in the PR description that you ran the tests, or just know that you ran the tests but don't write it down anywhere. I'd suggest that the safety nets provided by these options are less useful because they have less friction.
Is this really more friction? I can very easily imagine creating an alias that runs `git push && gh signoff` for the cases where I know the tests have passed, and I can equally easily imagine running that alias and forgetting that I've just committed a change and forgotten to run the tests. If anything, opening a browser and marking a checkbox there feels like more friction to me.
I could imagine there being more friction if the tool somehow checked that a test suite had been run on the current commit, but that doesn't seem to be the case. It's just a more convenient way of checking a checkbox.
The biggest advantage I can see with this is that the status check gets reset with every push. That's harder to do with a regular checkbox, but I can imagine there's still a way, and I can imagine that way would probably still be simpler than this.
What would constitute a bad implementation of the idea?
(Also, I don't see anything in the readme that says most people shouldn't use it.)
> Remote CI runners are fantastic for repeatable builds, comprehensive test suites, and parallelized execution. But many apps don't need all that. Maybe yours doesn't either.
My reading of that is that if you need repeatable builds, comprehensive test suites, and/or parallelised execution, then this is not the tool for you.
I think a checkbox, suggested in another comment is a worse implementation.
I could certainly see the appeal of this sort of idea. Once your engineering org gets to a certain size, you can end up spending an eye-watering amount of money on CI compute - spinning up runners and executing thousands of tests for every single commit of every single pull request. That cost could decrease by a lot if, for PRs at least, you could use the dev's local machine instead.
The three prerequisites in my mind would be, that the CI workflow runs inside a local cluster of VMs (to match a real CI environment), that the results of the CI run are still uploaded and published somewhere, and that it's only used for pull requests (whereas a real CI is still used for the main branch).
This tool fails the second test; it doesn't publish the test results or in any way associate them with the status check.
Also, a larger organization is one in which human error is more likely, so I would expect a tool that relies on individual engineers not to make mistakes to be less suitable there.
Yes, I know the particular tool linked in OP does not pass the prerequisites.
I could _maybe, maybe_ see self-hosted GitHub runners inside of VMs running on developer machines. The VMs would have to be re-created after every request however.
I don't think this is a great idea, however, as now CI is dependent on my flaky laptop's wifi/internet connection/IP address, has the potential to be contaminated by something running on my machine, build logs can be modified, environment shape/architectures are all different and can't be easily controlled, I now have access to all of the CI secrets and can impersonate the CI, etc.
Why not actually running CI on your computer? Most CI system work via an agent, so you can join your laptop as a worker to the CI system?
https://docs.github.com/en/actions/hosting-your-own-runners/...
You could "favor" your own laptop as a target runner for the CI when it's your PR for example
https://docs.github.com/en/actions/writing-workflows/choosin...
If this is just running tests locally, it seems deeply flawed - e.g. the tests would work even if I forgot to commit new files.
OTOH if starting a new container, pulling my branch and then doing everything, it's definitely as good as running on remote CI because it's basically replicating the same behaviour. And it would likely still be much faster since:
* CI machines are underpowered compared to dev laptops/desktops. e.g. our CI builds run on a container with 4 vCPUs and 16GB RAM. In contrast my laptop has 16 cores and 48 GB RAM
* docker pull itself takes a relatively long time on CI. If I'm running it locally it would just use the cached layers
The tool does actually check whether you have any uncommitted changes in Git, and fails in that case. So you're protected from that particular mistake. You're not protected from mistakes related to running the tests or checking their results, though, because the tool has nothing to do with that.
The Docker CLI is supposed to support caching on GitHub Actions (https://docs.docker.com/build/cache/backends/gha/) but I suppose I haven't checked how fast it is in practice.
> CI machines are underpowered compared to dev laptops/desktops.
Ha, I wish. My company thinks 8gb of RAM on a 6 year old machine is plenty of power for the devs.
My browser tabs right now are using that much ram ;_;
I wrote and released a similar concept in 2012 in response to co-workers constantly breaking my code and not running tests before committing. https://rubygems.org/gems/git_test/
That was before we had a reliable CI server and let me tell you, fixing the CI server was a much better investment than convincing everyone to run and document their tests locally. The basecamp tool is more polished than what I cobbled together, but I personally won’t be reaching for it any time soon.
It's actually worse than that; this tool doesn't run any tests or do anything with test results. All it does is require each developer to run `gh signoff` before their PR can be merged; the only thing that command checks is that there aren't uncommitted changes in Git. So if your colleagues aren't already locally running your tests the right way, this does nothing at all to help you.
I do like the idea of a variant of this that also includes some signing and logs that share, well, more of the state involved in this.
This sort of work would likely mean that test suites would be built in a way to be run in a more diverse sets of environments. That leads to just less brittle tests overall, and hopefully a better foundation to debug things.
Given how fast dev machines are, doing this might not be a bad idea for small teams _if_ you have some way to ensure reproducible builds/tests. Otherwise I can imagine a lot of "works on my machine" cases
Though if you are an org of 1000+ engineers working in big monorepos better stick to CI
Docker?
This might sound crazy to a lot of people, but I actually wish I had the same lint/mypy/flake8 CI from work on my personal projects. I don’t need anything fancy but something on github that’s single click that forces discipline.
Use a Git hook: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
For example, Run you lint script on pre-commit. You can use `git config core.hooksPath .githooks` in your package.json/cargo.toml/composer.json/etc scripts prepare hook to maintain your scripts in a `.githooks` directory in the repo.
You could also run your code coverage command on pre-commit to help with discipline. Sky is the limit.
Do you use any kind of version control on your personal projects? If so, you should be able to set up a precommit hook. Or is the difficulty in finding a project template that has all the right tools configured in the right way?
I agree with the premise (dev machines are under utilized) but the environment (or lack thereof) of the CI machine is its major selling point. No stupid forgot to add dependency x. Of course it works, you developed it on your machine.
At that rate you might as well force merge, no? That's equivalent to signing off and doesn't make the CI have a false positive signal when it hasn't actually run.
Passes on every undeclared dependency that is available locally but would render the official artifact unusable.
I’m deeply confused … what does this actually do? Mark your commit as having tests passed?
Yep: https://github.com/basecamp/gh-signoff/blob/0e402078ad1483cf...
I mean, obviously you want to run the local CI in some isolated way. But it's not such a bad idea for many projects.
At PyTorch, we have the reverse problem, it's basically infeasible to run the CI locally, you really do need the cloud setup to cover all of the operating system / hardware configurations, and the parallelization in cloud also saves you loads of time.
This misses the point of CI. The CI environment is neutral, one of its goals is to avoid the "it works on my machine" effect. If it doesn’t at least run in a container or a VM, it’s not CI.
Now we are back to "hey bro, it signed off on my computer" :shrug_emoji: