Testing Moodle plugins when AI agents write the code

Tests are how you know code works without watching it being written. That was always true. It matters more now, because a lot of Moodle plugin code is written by AI agents, quickly, in sessions nobody reads line by line. When an agent writes the code, the tests stop being an afterthought. They are the spec the agent works to and the only evidence you have that it did the job.

The catch: agents are good at making tests pass. That is not the same as making the code correct. So the useful question is not “does it have tests?” but “would these tests fail if the code were wrong?”

This post explains the testing words you will hear, shows how each one maps to Moodle’s own tools, and then covers what changes when an agent is the one writing the code.

The jargon in one table

What is a unit test?

A unit test calls one small piece of code with known input and checks the output. No database, no browser. Moodle’s PHPUnit guide describes basic_testcase as “intended for very simple tests that do not modify the database, file system, or any PHP globals”.¹1 Moodle HQ, Writing PHPUnit tests (5.3). A function that formats a grade, or parses a CSV row, is a good fit. These tests run in milliseconds, so you can run hundreds of them on every change.

What is an integration test?

An integration test checks that your code works with the things around it: the database, the file API, events, capabilities. Most Moodle plugin tests are really this kind. They extend advanced_testcase, which the guide calls “an enhanced testcase class which includes a number of test helpers for easier testing of Moodle code”.¹ It resets the database and data root between tests, as long as you call $this->resetAfterTest() before you change anything.

namespace local_greet;

final class greeter_test extends \advanced_testcase {
    public function test_greeting_uses_first_name(): void {
        $this->resetAfterTest();
        $user = $this->getDataGenerator()->create_user(['firstname' => 'Amina']);

        $this->assertSame('Hello, Amina', greeter::for_user($user));
    }
}

PHPUnit itself is a Composer dependency, not part of a Moodle install. You set a separate phpunit_prefix and phpunit_dataroot in config.php, run the init script, then vendor/bin/phpunit. Moodle 5.0 uses PHPUnit 11.4.²2 Moodle HQ, PHPUnit.

What are fixtures and data generators?

A fixture is the data a test needs before it starts: a course, two users, an enrolment. Writing that by hand with SQL inserts is slow and breaks on every schema change. Moodle’s answer is data generators. $this->getDataGenerator() can create users, categories, courses, cohorts, enrolments, groups, roles and more. A plugin can ship its own generator in tests/generator/lib.php, which “must extend the \component_generator_base class”, and tests reach it through get_plugin_generator().¹

The same generators feed Behat, which is why writing one pays off twice.

What is acceptance testing, and what is Behat?

An acceptance test describes a feature the way a user would, then runs it. Moodle uses Behat, which its docs call “a framework for behavior driven development (BDD)”. Scenarios are written in Gherkin, a “human-readable language used to define features that can be parsed and translated into PHP methods”.³3 Moodle HQ, Behat. Feature files live in your plugin’s tests/behat folder.⁴4 Moodle HQ, Writing acceptance tests.

@local @local_greet
Feature: Greeting on the dashboard
  Scenario: A student sees their own name
    Given the following "users" exist:
      | username | firstname |
      | student1 | Amina     |
    When I log in as "student1"
    Then I should see "Hello, Amina"

That Given the following "users" exist step calls the data generators. You can extend it so your plugin’s own entities can be created the same way.⁴

What is an end-to-end test?

End-to-end means the whole system, front to back: real browser, real JavaScript, real database, real page loads. In Moodle, that is a Behat scenario tagged @javascript, which drives a browser through Selenium. Moodle’s docs say the tag “should only be used, if JavaScript is needed to execute your test”, because those runs are much slower.⁴ Use them for the paths a user actually clicks through: a modal form, a drag and drop, a React component in Moodle 5.2 and later.

What is code coverage, and why does it lie?

Coverage measures which lines of your code ran while the tests ran. 90% coverage means 90% of lines were executed at least once. It does not mean 90% of lines were checked.

Here is a test with full coverage of for_user() that checks almost nothing:

public function test_greeting(): void {
    $this->resetAfterTest();
    $user = $this->getDataGenerator()->create_user();

    $this->assertIsString(greeter::for_user($user));
}

Every line ran, so coverage is happy. Change the greeting to "Goodbye", drop the first name, return an empty string: the test still passes. Swap the assertion for $this->assertTrue(true) and coverage does not move at all.

Moodle’s PHPUnit guide still “strongly” recommends checking coverage,¹ and it is right to: low coverage is a real warning, because untested code is certainly unchecked. High coverage just isn’t proof of anything. moodle-plugin-ci can report it with --coverage-text or --coverage-clover, using pcov or Xdebug.⁵5 Moodle HQ, moodle-plugin-ci PHPUnitCommand.php.

What is mutation testing?

Mutation testing tests your tests. A tool makes one small change to your code (flips >= to >, deletes a line, swaps true for false) and runs the suite. Infection, the main PHP tool, puts it plainly: “If mutated program produces failing tests, this is called a killed mutant.” And: “If tests are green with mutated code, then we have an escaped mutant.”⁶6 Infection, Introduction to mutation testing. Every escaped mutant is a bug your tests would not catch.

The score is the Mutation Score Indicator (MSI), the share of mutants your tests caught. Infection’s own guide walks through an example with 67% code coverage but a 47% MSI, and concludes that “Code Coverage was a terrible quality measurement in this example.”⁶

A Moodle example. Say a plugin marks an activity complete when $grade >= $passgrade. A test with a grade of 90 and a pass mark of 50 passes. So does the mutant with >. Only a test where the grade equals the pass mark kills it. Mutation testing finds that missing boundary for you.

Moodle ships no mutation testing setup, and the moodle-plugin-ci workflow template has no mutation step.⁷7 Moodle HQ, moodle-plugin-ci gha.dist.yml. If you want it, you add Infection yourself. It is slow, because it runs the covering tests again for every mutant, so you can run it on changed files or nightly instead of on every push.

What is static analysis?

Static analysis reads code without running it. PHPStan’s own pitch is “Find Bugs Without Writing Tests”.⁸8 PHPStan, PHPStan: Find Bugs Without Writing Tests. It has rule levels from 0 to 10; level 0 catches unknown classes, unknown functions and wrong argument counts, and level 10 complains about any implicit mixed.⁹9 PHPStan, Rule Levels. Psalm does the same job, and its security analysis tracks user input through the code to see whether it reaches places like HTML output without being escaped.¹⁰10 Psalm, Security Analysis.

Moodle has no official PHPStan setup and moodle-plugin-ci has no PHPStan command. We covered that gap in our post on Moodle’s developer tooling. For agent-written code it is one of the cheapest checks there is: a call to a method that does not exist fails in seconds, before any test runs.

What are coding standards checks?

These check style and known bad patterns. Moodle’s are in moodle-cs, “the Moodle Coding Style configurations, written as PHP CodeSniffer rulesets”, with a moodle standard and a stricter moodle-extra that adds “recommended best practices”.¹¹11 Moodle HQ, moodle-cs. They will not tell you the code works. They will tell you it looks like Moodle code, which matters when a reviewer, or a future agent, has to read it.

What is a security test?

A security test tries to do the wrong thing and checks that it fails. In a Moodle plugin that usually means: a student calls a teacher-only web service and gets required_capability_exception; a form post without a valid sesskey is rejected; user input in a $DB query goes through placeholders; output is escaped. Most of these are ordinary PHPUnit tests with a hostile setup. Taint analysis (Psalm) adds a second check that does not depend on you imagining every attack.

What is a flaky test?

A flaky test passes and fails on the same code. Timing in browser tests is the usual cause. Moodle HQ keeps a list of flaky Behat scenarios and tags them @skip in CI (more on that in the developer tooling post).¹²12 Moodle HQ, moodle-skip-tag-injector. Flaky tests are worse than no tests in one way: once people learn that red sometimes means nothing, they stop reading red.

What does moodle-plugin-ci actually run?

moodle-plugin-ci is Moodle HQ’s tool for testing a plugin in CI. It installs Moodle, installs your plugin into it, and runs the checks. Its docs say “All of these tests and tools are run everytime a change is pushed to a GitHub branch or pull request.”¹³13 Moodle HQ, moodle-plugin-ci documentation. GitHub Actions is the main target; you copy gha.dist.yml into .github/workflows. Travis support is deprecated.¹³

The template’s steps, in order:⁷

1. phplint: syntax errors.
2. phpmd: PHP Mess Detector, for overly complex or unused code.
3. phpcs --max-warnings 0: moodle-cs coding standards.
4. phpdoc --max-warnings 0: Moodle’s PHPDoc rules.
5. validate: the plugin’s required files and structure.
6. savepoints: upgrade savepoints in db/upgrade.php.
7. mustache: template linting.
8. grunt --max-lint-warnings 0: JavaScript and CSS build and lint.
9. phpunit --fail-on-warning: the PHPUnit suite.
10. behat --profile chrome: Behat in Chrome.

The template runs that list across a matrix of PHP versions and databases (PostgreSQL and MariaDB in the example). For a plugin that claims to support Moodle 4.5 through 5.3, the matrix should cover each branch and each PHP version those branches accept. A plugin that is only green on one combination has been tested on one combination. Our Moodle 5 requirements post lists the PHP versions per release.

Note what is not in the list: coverage thresholds, mutation testing, static analysis beyond phpmd, and dedicated security checks. “moodle-plugin-ci passes” is a solid floor. It is not the ceiling.

How does testing change when an agent writes the code?

Three things change.

Tests become the spec. An agent builds what it can check. Anthropic’s Claude Code guidance says that without something to run, “‘looks done’ is all it has to go on. In practice that makes you the test suite.”¹⁴14 Anthropic, Best practices for Claude Code. A failing test that describes the feature is the clearest instruction you can give. It is also the clearest record afterwards of what was asked for.

Agents game weak tests. This is not malice; the agent is told to make the tests pass, and it finds the shortest path there. Kent Beck listed it among his warning signs that an agent was going off track: “Any indication that the genie was cheating, for example by disabling or deleting tests.”¹⁵15 Kent Beck, Augmented Coding: Beyond the Vibes (25 June 2025). Anthropic’s own prompting guide has a section titled “Avoid focusing on passing tests and hardcoding”, and its sample prompt says “Tests are there to verify correctness, not to define the solution.”¹⁶16 Anthropic, Prompting best practices. Birgitta Böckeler, testing TDD prompts with coding agents for Martin Fowler’s site, found agents that “sometimes skipped or faked the red step”, and tests that “checked the implementation’s output against itself”. Writing tests first “doesn’t reliably prevent this.”¹⁷17 Birgitta Böckeler, martinfowler.com, TDD inside the agent loop: theater or actual value? (10 August 2026). Her answer was not more process. It was measurement: “I monitor and improve regression quality with the help of mutation testing.”¹⁷

That is the core argument. An agent can raise coverage to 100% with assertions like assertIsString, or with mocks so thorough that the test checks the mock. Coverage cannot see the difference. Mutation testing can, because a mocked-out test kills no mutants.

Flakiness costs more. A person learns which Behat scenario is flaky and reruns it. An agent sees a red result and starts changing code to fix a problem that is not there. A July 2026 study of over 200,000 test files found agent-written tests had assertion strength close to human ones (85.37% strong assertions against 88.1%) but a higher flakiness risk, and concluded agents lack the “environmental awareness” needed to write stable, hermetic tests.¹⁸18 Jhanglani, Desai, Kansara and AlOmar, arXiv, Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects (July 2026). So the tests an agent writes need the same review as its code, with an eye on timing, shared state and order dependence.

What does a good setup look like?

For agent-written Moodle plugins, this is the order we would add things in:

• moodle-plugin-ci green on the full matrix. Non-negotiable, and cheap to set up.
• Tests written or reviewed by a person before the agent implements. Especially boundary cases. They are the spec.
• A rule that the agent does not edit existing tests to make them pass. If a test is wrong, it says so and a person decides.
• PHPStan at a low level with a baseline. Fast, deterministic, catches the invented-method class of mistake.
• Mutation testing on the code that matters. Grading, completion, access rules, money. Not every file.
• Behat for each user-facing path, @javascript only where needed. And a quarantine for anything flaky, fixed or deleted quickly.
• Evidence, not claims. Anthropic’s guidance again: “‘All tests pass’ on its own tells you very little.”¹⁴ Ask for the command and its output.

This is also why fast delivery is believable. In our post on building a plugin in days, speed comes from agents and a senior review. Trust comes from checks that would fail if the agent were wrong.

Tests as the contract

The list of tests is the most honest description of what you are buying. Our own packages are defined that way. Essentials means unit tests plus moodle-plugin-ci passing. Complete adds Behat, end-to-end tests, coverage, mutation and security tests, plus documentation and a demo video. Whoever builds your plugin, ask for the test list in writing, and ask what would make each test fail.