The CI secrets problem nobody talks about with AI fixes

Most of the talk about AI code fixes is about quality. Is the fix right? Does it break something else? Those are fair questions. There is a second one that gets less attention, and it is about what happens after the push.

A tool that pushes a commit to your pull request branch starts a CI run. That run executes the code in the commit. If a model wrote the code, then model-written code is now running in your CI, with whatever your CI can read.

A short refresher on secrets in GitHub Actions

Repository secrets are values you store in GitHub, such as deploy tokens and API keys. A workflow reads them through secrets.NAME. GitHub passes them to a job when the job runs.

GitHub also limits who gets them. Two rules matter here.

• Runs started from a pull request opened from a fork do not get secrets by default.
• Runs triggered by Dependabot do not get your normal Actions secrets either. Dependabot has its own separate secret store. This is deliberate. A dependency bump is opened by a bot, from an external source, and GitHub does not want that run to read your secrets.

Same-repository pull requests are different. A branch in your own repository, pushed by a person with write access, runs with secrets. That includes branches opened by Renovate, since Renovate pushes branches to your repository like any other contributor would.

What changes when a bot pushes a fix

Now picture the sequence. Dependabot opens a bump. CI goes red. A fixing tool pushes a commit to the Dependabot branch. The next run is triggered by the commit, and the actor on that run is the fixing tool's bot account, not Dependabot.

We tested this. On a private test repository we made a throwaway secret with a made-up value. A workflow printed who started the run, and only whether the secret was visible, yes or no, never the value. The results:

Run	Pushed by	Secret visible
A normal PR branch	the repository owner	yes
A Dependabot PR	Dependabot	no
The same Dependabot branch, after Patchlit's push	Patchlit's app	yes
A normal PR branch, after Patchlit's push (control)	Patchlit's app	yes

Read the third row against the second. GitHub gave Dependabot's own run no secrets. A push from our app onto that branch gave the next run all of them.

The test was small. It covered one repository, with Actions secrets only. We did not test repositories with Dependabot secrets configured, workflows that use pull_request_target, environments with required reviewers, or token permissions other than read. We say so because the safe reading of an untested case is not "fine".

Why this matters

On a Renovate PR, or any same-repository PR, the run gets secrets whoever pushed. For those, this is the general risk of running new code in CI. On a Dependabot branch it is worse, because the push turns a no-secrets run into a with-secrets run.

The code that runs is model-written. The model reads CI logs, issue text and repository files, and strangers can write some of that text. A prompt injection through a log or an issue is a real class of attack against any tool like this. Checks before the push reduce the odds. They refuse changes to CI config, lifecycle scripts, lockfiles and new web hosts. They cannot prove that source or test code, which CI will run, is harmless. We cover those checks in a separate post about letting CI judge the fix.

So we treat it as a design problem and not a hope. We will not describe how someone would abuse this, and nothing here needs it. The point is that the exposure is real, it is easy to miss, and it is worth asking any tool that pushes code about.

The default: deny

Patchlit's answer is to decline by default.

Before any push, and before any model call, Patchlit reads the workflow files in the repository. It reads them on the PR head, where a push-triggered run takes its workflow from. It also reads them on the default branch, because pull_request_target workflows run from there, with secrets.

If any workflow uses a secret other than the built-in GITHUB_TOKEN, Patchlit pushes nothing. It marks the dependency PR as declined, counts it apart from the green rate, and posts one comment naming the files that triggered the decision. It says this once per PR, not on every red run.

Some details of the scan.

• It is blunt on purpose. The match is case-insensitive. After the per-run token is set aside, any remaining use of the word secrets counts. That covers secrets.NAME, secrets: inherit, a bare ${{ secrets }}, and a reusable workflow's secrets: declaration. A step that only mentions the word is declined too.
• It fails closed. If a workflow file cannot be read, or is too large, or the default branch cannot be looked up, Patchlit does not push.
• It reads GitHub Actions only. CI on other systems, such as CircleCI or Jenkins, holds its own secrets, and Patchlit cannot see them. A reusable workflow in another repository is caught only when the caller passes secrets to it.

A false decline costs you a comment and a switch. A false allow costs you a secret. We chose the cheaper error.

The opt-in

The owner can turn on Secrets OK for a repository on the Repositories page. It is per repository, and it is off by default. Once it is on, Patchlit works on that repository as usual.

We checked this live on the test repository. With the switch off, Patchlit declined, made no model calls, and left the PR with only Dependabot's commit. With the switch on, it pushed a fix, CI went green on the second attempt, and the audit came back clean.

The switch is one setting for the whole repository. A per-workflow or per-environment allowance would be better, and we have not built it. The rules page lists this setting with the others.

What you can do whatever tool you use

This advice does not depend on Patchlit.

1. Know which workflows use secrets. List them. Many repositories have more than the owner remembers.
2. Keep secrets out of pull request workflows. Run tests without them. Put deploys behind a trigger that only runs on your default branch.
3. Use environments with required reviewers for anything that holds a real credential, so a person approves each use.
4. Set the default token to read-only and grant more per job.
5. Ask what any code-writing tool pushes, and what runs after. The question is the one this post is about.
6. Read the diff. Treat a bot's commit like a pull request from a stranger who passed the tests.

What is still open

We should be clear about the limits.

• One switch covers the whole repository.
• CI on other systems is not seen.
• Repositories with Dependabot secrets, pull_request_target flows, and environments with required reviewers were not tested.
• The scan is a text match. It tells you where secrets are used. It does not tell you whether the code that runs is safe.

If you want the full picture of what Patchlit can access and what it refuses to do, the security page covers it, and how it works shows where the secrets check sits in the loop.