Complement is a black box integration testing framework for Matrix homeservers. It allows us to write end-to-end tests that interact with real Synapse homeservers to ensure everything works at a holistic level.

Nothing beyond a normal Complement setup (just Go and Docker).

Run tests from the Complement repo:

# Run the tests
./scripts-dev/complement.sh

# To run a whole group of tests, you can specify part of the test path:
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate
# To run a specific test, you can specify the whole name structure:
scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate/Parallel/POST_/createRoom_makes_a_public_room
# Generally though, the `-run` parameter accepts regex patterns, so you can match however you like:
scripts-dev/complement.sh ./tests/... -run 'TestRoomCreate/Parallel/POST_/createRoom_makes_a_(.*)'

It's often nice to develop on Synapse and write Complement tests at the same time. Here is how to run your local Synapse checkout against your local Complement checkout.

# To run a specific test
COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh ./tests/csapi/... -run TestRoomCreate

The above will run a monolithic (single-process) Synapse with SQLite as the database. For other configurations, try:

• Passing POSTGRES=1 as an environment variable to use the Postgres database instead.
• Passing WORKERS=1 as an environment variable to use a workerised setup instead. This option implies the use of Postgres.
  • If setting WORKERS=1, optionally set WORKER_TYPES= to declare which worker types you wish to test. A simple comma-delimited string containing the worker types defined from the WORKERS_CONFIG template in here. A safe example would be WORKER_TYPES="federation_inbound, federation_sender, synchrotron". See the worker documentation for additional information on workers.
• Passing ASYNCIO_REACTOR=1 as an environment variable to use the asyncio-backed reactor with Twisted instead of the default one.
• Passing PODMAN=1 will use the podman container runtime, instead of docker.
• Passing UNIX_SOCKETS=1 will utilise Unix socket functionality for Synapse, Redis, and Postgres(when applicable).

To increase the log level for the tests, set SYNAPSE_TEST_LOG_LEVEL, e.g:

SYNAPSE_TEST_LOG_LEVEL=DEBUG COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestRoomCreate

In-repo Complement tests are tests that are vendored into this project. We use the in-repo test suite to test Synapse specific behaviors like the admin API.

To run the in-repo Complement tests, use the --in-repo command line argument.

# Run only a specific test package.
# Note: test packages are relative to the `./complement` directory in this project
./scripts-dev/complement.sh --in-repo ./tests/...

# Similarly, you can also use `-run` to specify all or part of a specific test path to run
scripts-dev/complement.sh --in-repo ./tests/... -run TestIntraShardFederation

If you're curious what the database looks like after you run some tests, here are some steps to get you going in Synapse:

1. In your Complement test comment out defer deployment.Destroy(t) and replace with defer time.Sleep(2 * time.Hour) to keep the homeserver running after the tests complete
2. Start the Complement tests
3. Find the name of the container, docker ps -f name=complement_ (this will filter for just the Complement related Docker containers)
4. Access the container replacing the name with what you found in the previous step: docker exec -it complement_1_hs_with_application_service.hs1_2 /bin/bash
5. Install sqlite (database driver), apt-get update && apt-get install -y sqlite3
6. Then run sqlite3 and open the database .open /conf/homeserver.db (this db path comes from the Synapse homeserver.yaml)