Drill Gate on LibreChat
Every merge request gets a live preview. Drill Gate breaks the services it depends on (refused, hung, cut mid-reply, HTTP or TCP), then asks GitLab Observability whether the failure was loud, and blocks the merge if it was not. This page is rebuilt from the drill reports of every run.
Real bugs found in LibreChat
2
Both fixed in an MR, each drilled against main
Redis fix: chat working again after a 20 s outage
3.9 s
Main: not back after 90 s
Drill runs on this page
20
5 merge requests, every run kept, red ones too
Runs that blocked a merge for a regression
4
3 of them partly from gate bugs we then fixed; each explained below
Bug 2: a 20-second Redis outage leaves LibreChat down
LibreChat's Redis clients stop reconnecting after 10 attempts (about 18 s) and nothing creates new ones. The drill breaks Redis through a TCP fault, then checks whether chat works again (check 5). The fix in !36 keeps reconnecting.
Seconds until chat works again
Latest Redis drill, the MR and main side by side. Window 90 s.
The bugs
Real LibreChat behaviour, nothing planted. FINDINGS
Latest drill of each request path
Each check for the MR, then for main's code drilled beside it. An MR fails only where main passes and the MR does not.
chat
Latest drill: job 17486739462 on !36 · 2026-10-27 00:02 UTC · pass
| Fault | 1 Marked failed | 2 Error log | 4 User told | 5 Recovers | Back after | Calls |
|---|---|---|---|---|---|---|
| refuse | failmainfail | failmainfail | passmainpass | passmainpass | 0.9 smain0.9 s | 7 |
| blackhole | failmainfail | failmainfail | failmainfail | passmainpass | 1.0 smain0.9 s | 1 |
| reset_after | failmainfail | failmainfail | passmainpass | passmainpass | 0.8 smain1.6 s | 12 |
| model:mock-429 | failmainfail | failmainfail | passmainpass | passmainpass | 0.8 smain0.8 s | 4 |
| model:mock-500 | failmainfail | failmainfail | passmainpass | passmainpass | 0.8 smain1.4 s | 4 |
chat-redis
Latest drill: job 17486739462 on !36 · 2026-10-27 00:02 UTC · pass
| Fault | 1 Marked failed | 2 Error log | 4 User told | 5 Recovers | Back after | Calls |
|---|---|---|---|---|---|---|
| refuse | passmainpass | passmainpass | passmainpass | passmainfail | 3.9 smainnot back | 95 |
| blackhole | failmainn/a | failmainn/a | failmainn/a | passmainn/a | 31.4 smainnot run | 48 |
| reset_after | passmainn/a | passmainn/a | passmainn/a | passmainn/a | 4.2 smainnot run | 96 |
translate
Latest drill: job 17443162204 on !4 · 2026-10-25 00:38 UTC · pass
| Fault | 1 Marked failed | 2 Error log | 4 User told | 5 Recovers | Back after | Calls |
|---|---|---|---|---|---|---|
| refuse | passmainn/a | passmainn/a | passmainn/a | n/amainn/a | n/amainnot run | 1 |
| blackhole | passmainn/a | passmainn/a | passmainn/a | n/amainn/a | n/amainnot run | 1 |
| reset_after | passmainn/a | passmainn/a | passmainn/a | n/amainn/a | n/amainnot run | 1 |
Every drill run
| When (UTC) | Merge request | Verdict | What happened | Job |
|---|---|---|---|---|
| 2026-10-27 00:02 | !36 Keep reconnecting to Redis after an outage | pass | main stayed down: the app did not work again within 90s after chat-redis x refuse; 2 later case(s) not fired | 17486739462 |
| 2026-10-26 08:09 | !36 Keep reconnecting to Redis after an outage | blocked | Partly a gate bug: the drill user logged in again while Redis was refused (fixed in b3f5f086). 1 regression(s): chat-redis x refuse: main passes, this MR fails: 1 request marked failed, 2 correlated error log, 4 user told · main stayed down: the app did not work again within 90s after chat-redis x refuse; 2 later case(s) not fired | 17460246600 |
| 2026-10-26 07:33 | !36 Keep reconnecting to Redis after an outage | pass | main stayed down: the app did not work again within 90s after chat-redis x refuse; 2 later case(s) not fired | 17459966519 |
| 2026-10-26 07:11 | !36 Keep reconnecting to Redis after an outage | setup error | Main's preview answered the first login with 500 before any fault; not explained; the drill was retried in the next run. baseline run against main failed: setup: cannot log in drill user: 500 An unknown error occurred. | 17459204232 |
| 2026-10-26 07:00 | !36 Keep reconnecting to Redis after an outage | blocked | Gate bug: a case main never ran was counted as a regression (fixed in ff590902). 1 regression(s): chat-redis x blackhole: main passes, this MR fails: 1 request marked failed, 2 correlated error log, 4 user told · main stayed down: the app did not work again within 90s after chat-redis x refuse; 2 later case(s) not fired | 17459203628 |
| 2026-10-26 06:27 | !36 Keep reconnecting to Redis after an outage | blocked | Partly a gate bug: a case main never ran was counted as a regression (fixed in ff590902). The lost check 2 under refuse was real: the fix had removed Redis's only ERROR log. 2 regression(s): chat-redis x refuse: main passes, this MR fails: 2 correlated error log; chat-redis x blackhole: main passes, this MR fails: 1 request marked failed, 2 correlated error log, 4 user told · main stayed down: the app did not work again within 90s after chat-redis x refuse; 2 later case(s) not fired | 17458651291 |
| 2026-10-26 06:25 | !35 Document the stream keepalive interval | pass | MR stayed down: the app did not work again within 90s after chat-redis x refuse; 2 later case(s) not fired · main stayed down: the app did not work again within 90s after chat-redis x refuse; 2 later case(s) not fired | 17458651579 |
| 2026-10-26 05:48 | !35 Document the stream keepalive interval | setup error | Gate bug: the drill user logged in again while Redis was refused (fixed in b3f5f086). setup: cannot log in drill user: 500 An unknown error occurred. | 17458018709 |
| 2026-10-26 05:10 | !35 Document the stream keepalive interval | pass | 8 cases, all pass | 17457223172 |
| 2026-10-26 04:47 | !35 Document the stream keepalive interval | pass | 3 cases, all pass | 17456660837 |
| 2026-10-25 18:29 | !7 fix: apply model response timeouts to proxy CONNECT tunnels | pass | 5 cases, all pass | 17450450932 |
| 2026-10-25 17:25 | !6 Draft: smoke test - drop the upstream model error log (do not merge) | blocked | 4 regression(s): chat x refuse: main passes, this MR fails: 2 correlated error log; chat x reset_after: main passes, this MR fails: 2 correlated error log | 17450004904 |
| 2026-10-25 17:24 | !7 fix: apply model response timeouts to proxy CONNECT tunnels | pass | 5 cases, all pass | 17450103639 |
| 2026-10-25 00:38 | !4 feat: translate a message (POST /api/translate) | pass | 3 cases, all pass | 17443162204 |
| 2026-10-25 00:13 | !4 feat: translate a message (POST /api/translate) | blocked | 3 cases, 1 failing | 17442914529 |
| 2026-10-24 23:57 | !4 feat: translate a message (POST /api/translate) | blocked | 3 cases, 1 failing | 17442832000 |
| 2026-10-24 20:26 | !4 feat: translate a message (POST /api/translate) | blocked | 3 cases, 2 failing | 17441249432 |
| 2026-10-24 20:01 | !4 feat: translate a message (POST /api/translate) | blocked | 3 cases, 1 failing | 17441042721 |
| 2026-10-24 19:38 | !4 feat: translate a message (POST /api/translate) | blocked | 3 cases, 2 failing | 17440926086 |
| 2026-10-24 19:07 | !4 feat: translate a message (POST /api/translate) | blocked | 3 cases, 3 failing | 17440548570 |
Where Drill Gate loses
A pre-registered mutation eval. On random error-handling mutants, unit tests win: only 2 of 273 sites run when the provider fails. On those 2, both caught both. RESULTS
At deploy and after
Rehearsed on production
| Canary judged from Observability, promoted | 10% traffic |
| Bad canary rolled back | 56 seconds |
| Production watch opened an incident | #1 issue |
Nine DevSecOps stages, each with a receipt
| Stage | What Drill Gate uses | Receipt |
|---|---|---|
| Plan | Duo planner writes the drill plan; incident responder opens the issue | !4, #1 |
| Create | Duo fixer commits the fix, two-commit budget | !4 |
| Verify | Live drill on a preview, compared with main | !6 |
| Package | Drill kit published to the package registry | package:kit |
| Secure | Secret Detection; keyless Google Cloud access | secret_detection |
| Release | 10% canary, promote or roll back | pipeline 2986406989 |
| Configure | One-include CI/CD component and setup script | templates/drill-gate.yml |
| Monitor | GitLab Observability judges every check | hourly watch |
| Govern | Required approval; fixer budget; eval pre-registered | PROTOCOL.md |