Debugging a PR that doesn't start

[ndw]

(I hope this is the right place to ask this question; I've tried looking through the documentation but I see only a very small troubleshooting section and my problem isn't that.)

I have my own Woodpecker instance running and connected to Codeberg. It's been working fine. I had to update the DNS for the Woodpecker instance recently, but that all seemed to go smoothly. I was able to log back into Codeberg and connect Woodpecker to it.

Today I created a PR. If I look on the "Pull requests" tab for that repository on my Woodpecker instance, I see it listed, but no pipelines ever start for it. I don't see any evidence of errors anywhere I can think to look. I've got three agents running in Docker. They all seem to be running fine. The logs for the server container don't show any errors.

I'm baffled. I've tried pushing to the main branch directly, and I've tried pushing to some other repositories that I have connected. None of them actually start any pipelines, so it appears to be some global issue. I've tried restarting, both the Docker containers and, out of desperation, the host itself. No changes.

I'm running 2.9.0 and I tried doing a pull just to be sure.

[qwerty287]

Can you set the log level to debug or trace and check again if there's something in the logs?

[ndw]

Good idea, but alas, I don't see anything that looks remarkable (to my untrained eye):

[
  {
    "level": "info",
    "time": "2025-09-15T07:33:24Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/cmd/server/server.go:264",
    "message": "starting http server ..."
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 0.02064,
    "method": "GET",
    "path": "/",
    "status": 200,
    "time": "2025-09-15T07:33:42Z",
    "user-agent": "Scrapy/2.12.0 (+https://scrapy.org)",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "::1",
    "latency": 0.047179,
    "method": "GET",
    "path": "/healthz",
    "status": 204,
    "time": "2025-09-15T07:33:54Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "::1",
    "latency": 0.04048,
    "method": "GET",
    "path": "/healthz",
    "status": 204,
    "time": "2025-09-15T07:34:24Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "url": "https://codeberg.org",
    "oauth2-url": "",
    "skip-verify": false,
    "oauth-client-id-set": true,
    "oauth-client-secret-set": true,
    "type": "forgejo",
    "time": "2025-09-15T07:34:53Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/forge/setup/setup.go:106",
    "message": "setting up forge"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 209.106451,
    "method": "GET",
    "path": "/api/user/repos",
    "status": 200,
    "time": "2025-09-15T07:34:53Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "::1",
    "latency": 0.05032,
    "method": "GET",
    "path": "/healthz",
    "status": 204,
    "time": "2025-09-15T07:34:54Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "::1",
    "latency": 0.04199,
    "method": "GET",
    "path": "/healthz",
    "status": 204,
    "time": "2025-09-15T07:35:24Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 1.0627,
    "method": "GET",
    "path": "/api/orgs/1",
    "status": 200,
    "time": "2025-09-15T07:35:49Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 0.492139,
    "method": "GET",
    "path": "/api/orgs/1/permissions",
    "status": 200,
    "time": "2025-09-15T07:35:49Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 1.573879,
    "method": "GET",
    "path": "/api/user/repos",
    "status": 200,
    "time": "2025-09-15T07:35:49Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "::1",
    "latency": 0.027899,
    "method": "GET",
    "path": "/healthz",
    "status": 204,
    "time": "2025-09-15T07:35:54Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 2.623279,
    "method": "GET",
    "path": "/api/user/repos",
    "status": 200,
    "time": "2025-09-15T07:35:58Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "time": "2025-09-15T07:36:00Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:128",
    "message": "synced user permission for ndw xmlcalabash/xmlcalabash3"
  },
  {
    "level": "debug",
    "time": "2025-09-15T07:36:00Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:157",
    "message": "ndw granted &{UserID:1 RepoID:4 Repo:0xc000f771e0 Pull:true Push:true Admin:true Synced:1757921760 Created:0 Updated:1757921760} permission to xmlcalabash/xmlcalabash3"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 61.267879,
    "method": "GET",
    "path": "/api/repos/4/permissions",
    "status": 200,
    "time": "2025-09-15T07:36:00Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "time": "2025-09-15T07:36:00Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:157",
    "message": "ndw granted &{UserID:1 RepoID:4 Repo:<nil> Pull:true Push:true Admin:true Synced:1757921760 Created:1748695277 Updated:1757921760} permission to xmlcalabash/xmlcalabash3"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 0.72992,
    "method": "GET",
    "path": "/api/repos/4",
    "status": 200,
    "time": "2025-09-15T07:36:00Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 0.890549,
    "method": "GET",
    "path": "/api/badges/4/status.svg",
    "status": 200,
    "time": "2025-09-15T07:36:00Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "time": "2025-09-15T07:36:00Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:157",
    "message": "ndw granted &{UserID:1 RepoID:4 Repo:<nil> Pull:true Push:true Admin:true Synced:1757921760 Created:1748695277 Updated:1757921760} permission to xmlcalabash/xmlcalabash3"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 4.886919,
    "method": "GET",
    "path": "/api/repos/4/pipelines",
    "status": 200,
    "time": "2025-09-15T07:36:00Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "time": "2025-09-15T07:36:03Z",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/session/repo.go:157",
    "message": "ndw granted &{UserID:1 RepoID:4 Repo:<nil> Pull:true Push:true Admin:true Synced:1757921760 Created:1748695277 Updated:1757921760} permission to xmlcalabash/xmlcalabash3"
  },
  {
    "level": "debug",
    "ip": "172.18.0.1",
    "latency": 171.671426,
    "method": "GET",
    "path": "/api/repos/4/pull_requests",
    "status": 200,
    "time": "2025-09-15T07:36:04Z",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:142.0) Gecko/20100101 Firefox/142.0",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "::1",
    "latency": 0.031679,
    "method": "GET",
    "path": "/healthz",
    "status": 204,
    "time": "2025-09-15T07:36:24Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  },
  {
    "level": "debug",
    "ip": "::1",
    "latency": 0.05174,
    "method": "GET",
    "path": "/healthz",
    "status": 204,
    "time": "2025-09-15T07:36:54Z",
    "user-agent": "Go-http-client/1.1",
    "caller": "/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/router/middleware/logger.go:59"
  }
]

There's a push of a new PR near the end of the logs. I don't know if Repo: <nil> is significant.

[zc-devs]

I had to update the DNS for the Woodpecker instance recently
Today I created a PR, but no pipelines ever start for it.
I've tried pushing to the main branch directly, and I've tried pushing to some other repositories that I have connected. None of them actually start any pipelines.
The logs for the server container don't show any errors.

Maybe WebHooks then? Repair (all) repo(s).
https://woodpecker-ci.org/search/?q=webhook&version=docs%2F2.8%2F

---

I'm running 2.9.0

There is no 2.9.0, BTW. https://github.com/woodpecker-ci/woodpecker/tags?after=v3.2.0

---

#5038, #4444

[ndw]

Sorry, 3.9.0

[zc-devs]

Also, if my memory serves me well, there still will be old webhooks after repair operation. So, you might want to clean them up in (each, enabled in WP) forge repository.

[ndw]

The only instructions I could find for repairing all the webhooks relied on woodpecker-cli which I cannot get configured. (It tries to xdg-open something but I'm only on an ssh connection.)

However, I did find repair link on the admin pages and I did find some stale webhooks on the forge. Pushing the former and deleting the latter seems to have fixed things.

Thank you!

[zc-devs]

The only instructions I could find for repairing all the webhooks relied on woodpecker-cli which I cannot get configured.

I didn't find either at a time of writing #4444.

Log into Woodpecker under Administrator. Then hit Settings -> Repositories -> Repair all.