Meta JiTTests (Catching Just-in-Time Tests) are software tests that a large language model writes for one specific code change, runs before that change lands, and then throws away. A normal test suite is built to pass and live in the codebase for years. A catching JiTTest is built to fail when a change introduces behavior nobody intended. Meta says the system has moved past the research stage into its production infrastructure, and that in the first half of 2026 it helped catch 74 serious privacy bugs before they shipped.
What is the Meta JiTTests?
Meta JiTTests refers to Meta’s research and testing processes for evaluating new technologies, features, and AI systems before wider deployment. These tests help assess performance, accuracy, reliability, and potential improvements, allowing Meta to develop better digital experiences across its platforms.
| Fact | Detail |
| Full name | Catching Just-in-Time Tests (Catching JiTTests) |
| Built by | Meta’s engineering and research teams; Mark Harman is the corresponding author |
| First defined | “Harden and Catch” keynote paper, FSE 2025 (Harman, O’Hearn, Sengupta) |
| Main paper | “Just-in-Time Catching Test Generation at Meta”, arXiv, 30 January 2026; presented at FSE 2026 |
| Scale | Backend systems with hundreds of millions of lines of code |
| Core idea | Generate a test for each risky diff that fails only if the change breaks something |
| Status | Part of Meta’s internal infrastructure (October 2026) |
What exactly is a Meta JiTTest?
To understand JiTTests, you need the two terms Meta’s researchers use for automated tests.
- Hardening tests pass when they are written. They protect today’s behavior from future regressions. Almost all automated test generation over the past few decades has produced this kind of test.
- Catching tests are meant to fail. They expose a bug in the change being reviewed right now.
A JiTTest is a catching test generated “just in time”: when a pull request (Meta calls it a diff) is submitted, and before it lands. Meta targets tests that fail on the proposed revision but pass on its parent. That pattern points to a behavior change the diff introduced.
Here’s the detail most explainers skip: a catching test cannot be merged with the code it tests, because it fails on that code. So it has only two possible outcomes. Either an engineer agrees it found a real problem and fixes the code, or the test is judged unhelpful and discarded. Either way, it never becomes part of the permanent test suite. That’s why JiTTests need no test maintenance and no test code review.
Why Meta built them: test debt meets AI-written code
Traditional tests carry a hidden assumption: whoever writes them must anticipate every future change to the code. Nobody can do that well, so many tests never catch anything, and the ones that fail often fail for the wrong reasons (false positives).
In February 2026, Meta’s engineering blog argued that agentic development makes this worse. When AI coding agents produce changes faster than humans can write and maintain matching tests, test maintenance and false positives become unmanageable. JiTTests flip the model: instead of guessing about all future changes, they only ask one question, “what could this particular change break?”
How Meta JiTTests work, step by step
Meta’s October 2026 write-up describes a five-stage pipeline:
- Targeting. The system runs overnight on spare compute capacity and picks the previous day’s “risky” diffs, using a scoring approach similar to Meta’s Diff Risk Score.
- Test generation. Two workflows run in parallel (see below).
- LLM-as-Judge. An ensemble of several LLMs reviews each failing test and classifies it as a likely true or false positive.
- RubFake. A rule-based assessor (Rule-Based False-Positive Killing Environment) applies deterministic filters.
- Engineer outreach. Based on the combined ranking, the system decides whether a catch is worth an engineer’s attention.
The intent-aware workflow
An LLM reads the diff’s code, title and summary and infers what the engineer was trying to do. It then lists the risks of implementing that intent, creates mutants (copies of the code with deliberately injected faults) for each risk, and generates tests that “kill” those mutants.
The “dodgy diff” workflow
This workflow skips intent entirely. It treats the whole diff as if it were a mutant and assumes the change is buggy, then uses mutation-guided test generation to surface behavioral differences from the parent revision.
Tests from both workflows run against the real diff. Any that fail become weak catches, raw signals that still need filtering before a human sees them.
Traditional tests vs. hardening tests vs. Meta JiTTests
Most comparisons pit “old testing” against “AI testing.” The more useful split has three columns:
| Hand-written tests | AI hardening tests | Meta JiTTests (catching) | |
| Who writes them | Engineers | LLM | LLM |
| When they’re created | As code lands | Any time | When a diff is submitted |
| Expected result at creation | Pass | Pass | Fail if the diff has a bug |
| Stored in codebase | Yes | Yes | No, discarded afterward |
| Maintenance | Ongoing | Ongoing | None |
| Test code review | Needed | Needed | Not needed |
| Human involvement | Constant | At review time | Only when a likely bug is found |
| Main job | Guard existing behavior | Guard existing behavior | Question the new change |
The key takeaway: AI-generated hardening tests still add to your maintenance burden. JiTTests are the first approach on this list that doesn’t.
The numbers, read carefully
Meta has published results from two stages.
The research paper (January 2026) analyzed 22,126 generated tests. Its main findings:
- Change-aware generation produced candidate catches 4x more often than hardening tests and 20x more often than tests that failed by coincidence.
- The rule-based and LLM-based assessors cut human review load by 70%.
- Of 41 candidate catches sent to engineers, 8 were confirmed real bugs, and 4 of those would have caused serious failures.
The full-scale deployment (October 2026):
- 589 engineers, 64% of those contacted, engaged with JiTTest reports.
- Engineers reported a 60% true-positive rate.
- In the first half of 2026, JiTTests caught 74 serious (SEV-level) privacy bugs at the diff stage, and Meta reports privacy SEV occurrence fell 20% in the same period.
Three common misreadings to avoid
- “4x more bugs than traditional testing.” Several secondary articles say this. The paper’s 4x compares how often two generation methods produce candidate catches. It is not a claim that JiTTests find four times as many bugs as a team’s existing test suite.
- “20% accuracy” vs “60% accuracy.” The 8-of-41 figure comes from the early study. The 60% figure is engineer-reported at full scale, after more filtering. They measure different stages, so they don’t contradict each other, but they also shouldn’t be averaged or swapped.
- “JiTTests caused a 20% drop in privacy incidents.” Meta reports the two happened in the same period. That’s a strong signal, not a controlled experiment.
Human-on-the-loop, not human-in-the-loop
Meta frames JiTTests as human-on-the-loop. Instead of reviewing every generated test, engineers set success criteria, watch aggregate results, and step in at the policy level. Day to day, an engineer rarely reads test code. They answer one question: was this behavior change intended?
What your team can borrow, without Meta’s infrastructure
Meta has described an internal system, not a downloadable product. But the ideas transfer. Here’s a practical adaptation for smaller teams (our suggestions, not Meta’s published method):
- Split “guard” tests from “question” tests. Keep your permanent suite for stable behavior. Treat AI-generated, per-PR tests as disposable by default.
- Capture intent in every PR. A two-line “what this change should and shouldn’t do” in the description gives an LLM something to test against, much like Meta’s intent-aware workflow.
- Use the parent/child rule. Run each generated test on the PR branch and on its base. Only a test that fails on the branch and passes on the base is a candidate catch.
- Filter with rules before AI. Drop tests that depend on time, randomness or network calls, or that simply assert the change you meant to make. Deterministic filters are cheap and predictable.
- Only test risky diffs. Start with code touching payments, auth, privacy or data deletion rather than every typo fix.
- Measure true positives, not test counts. The question that matters is how many reports engineers agree were real.
AI coding assistants such as GitHub Copilot, Cline, or the editors in our Windsurf vs Cursor comparison can draft candidate tests. The filtering and parent/child check are what turn drafts into catches.
Limitations and open questions
- LLM judges are non-deterministic. Meta pairs them with RubFake partly for this reason.
- Not every catch is real. A 60% true-positive rate still means a meaningful share of reports don’t pan out.
- Infrastructure assumptions. The pipeline relies on mature CI, risk scoring and spare compute. Smaller teams will need lighter versions.
- Limited public detail. Meta’s posts don’t name the programming languages or specific systems involved.
Conclusion
Meta JiTTests make a simple bet: as AI writes more code, the best test is one that exists only long enough to ask a single, sharp question about a single change. The published results suggest the bet is paying off at Meta’s scale. For everyone else, the lesson isn’t to copy Meta’s pipeline. It’s to stop treating every generated test as something you have to keep.
Want to find the right AI coding and testing tools for your team? Browse the Rentaai AI tools library to compare code assistants, agent frameworks and developer tools in one place.
Frequently asked questions
What does JiTTest stand for?
JiTTest stands for Just-in-Time Test. Meta’s full term is Catching Just-in-Time Test, meaning a test generated when a code change is submitted, designed to catch bugs in that change before it lands.
How are JiTTests different from regular unit tests?
Regular unit tests are written to pass and stay in the codebase. JiTTests are generated by an LLM for one change, are expected to fail if that change has a bug, and are discarded afterward, so they need no maintenance.
Are Meta JiTTests open source?
Meta has published research papers and engineering blog posts describing the approach, but the system itself runs inside Meta’s infrastructure. Teams can apply the published ideas with their own tools.
Do JiTTests replace code review?
No. Meta describes them as different from LLM-based code review: reviewers check code as a human would, while JiTTests produce executable tests that show a concrete behavior change.
What is a “weak catch”?
A weak catch is a generated test that fails on a code change. It might signal a real bug or a false alarm, so it goes through LLM and rule-based assessors before an engineer is contacted.


