I was noodling around with personal access tokens today on GitHub and found out that you actually can restrict tokens to specific repositories, orgs, etc. Not sure if actions is a scope that is available or not.