Connectors Testing Framework
Create and run automated tests with the Connectors Testing Framework using the Rover CLI
Requires ≥ Rover 0.36, Federation 2.12.1
The Connectors Testing Framework is a testing automation tool for Connectors to ensure that API changes don't break your current implementation.
Test suites are YAML files in the ./tests/ directory with the .connector.yml extension. Each test suite can have multiple test cases, and each test case can target a different Connector in your schema.
Text
1.
2└── tests/
3 ├── test-a.connectors.yaml
4 ├── test-b.connectors.yaml
5 └── ...Usage
shell
1rover connector testSee all available CLI options.
Test suite example
YAML
tests/my_test_suite.connector.yml
1config:
2 schema: path/to/schema.graphql
3
4tests:
5 - name: Get Slideshow
6 connector: Query.slideshow
7 apiResponseBody: |
8 {"data": {"author":"Yours Truly","title":"Sample Slide Show","slides":["Wake up to WonderWidgets!","Overview"]}}
9 expect:
10 connectorRequest:
11 method: GET
12 url: http://my-url.com/path/segments
13 connectorResponse: |
14 {"author":"Yours Truly","title":"Sample Slide Show","slides":["Wake up to WonderWidgets!","Overview"]} Test suites
A test suite contains two main blocks:
config: the test suite configuration, where you name the test suite and define the path to the schematests: the test cases, where you define the assertions for Connector requests and responses
YAML
tests/my_test_suite.connector.yml
1# Test suite configuration
2config:
3 # Path to the GraphQL schema. Can also be provided in the CLI using `--schema-file`
4 schema: fixtures/schema.graphql
5 # Test suite name. If omitted, defaults to the YAML file relative path.
6 name: "my_test_suite_name"
7
8# Array of test cases
9tests:
10 - # Some test casenote
If a path is absolute, for example, it starts with
/ on Unix or C:/ on Windows, the tool uses it directly. If a path is relative, it's resolved from the project's root directory.Test cases
Test cases are the building blocks of a test suite and contain the following sections:
name: the name of the test case. Required.connector: the target Connector, defined using the Connector ID or the Connector coordinates. See Connector IDs for more information. Required.skip: a Boolean to indicate if you want to skip this specific test case. Defaults tofalse.variables: the variables the Connector uses to make requests. See Variables documentation for more information.apiResponseBodyorapiResponseBodyFile: the API response body to pass to the Connector. UseapiResponseBodyto include a simple response body inline, orapiResponseBodyFilefor a path to a file for more complex responses.apiResponseHeaders: the API response headers to pass to the Connector.status: the API response status to pass to the Connector.expect: the expectations and assertions for the target Connector.
YAML
1# Array of test cases
2tests:
3 # Generic test name to display. Required.
4 - name: ShouldGreetWithArgName
5
6 # The target Connector, defined using the Connector ID or the Connector coordinates. Required.
7 connector: query_helloWorld
8
9 # Skips test case and all its expectations. Defaults to `false`.
10 skip: true
11
12 # Any variables needed for the Connector
13 variables:
14
15 # variables map under `$args`
16 $args:
17 name: "Name to greet"
18
19 # variables map under `$context`
20 $context:
21 key: value
22
23 # variables map under `$this`
24 $this:
25 key: value
26
27 # variables array of maps under `$batch`
28 $batch:
29 - id: product-id
30
31 # variables map under `$config`
32 $config:
33 key: value
34
35 # variables map under `$request.headers`
36 $requestHeaders:
37 key: value
38
39 ### API Responses ###
40 ## Mock responses expected by the Connector.
41 ## Use either `apiResponseBody` or `apiResponseBodyFile`. If either are missing, `apiResponseBody` will be set to `Json::null`, with no response mapping guarantees.
42
43 # Mock API response body. Use this when the API response body is simple.
44 apiResponseBody: |
45 {
46 "greeting": "Hello Some Name"
47 }
48
49 # Mock API response body file. Use this when the API response body is complex or too large for the YAML file.
50 apiResponseBodyFile: fixtures/mocks/responseFile.json
51
52 # Any necessary response headers from the Mock API. Optional.
53 apiResponseHeaders:
54 content-type: application/json; charset=utf-8
55
56 # Response status, used to check if `is_success` field is correct. Optional. Defaults to `200`.
57 status: 200
58
59 ### Test Expectation ###
60 expect: # Test expectationsTest expectations
Define expectations for Connector request, response, problems, and errors in the expect block of a test case.
Connector request
You can define Connector request expectations in the following ways:
Descriptive form, where you identify each parameter of the request
URL based request
Using the results from analyzed data request
Descriptive form
In this format, you identify each parameter of the Connector request explicitly.
YAML
1 connectorRequest:
2 # Checks if the HTTP::Method called is as expected. Optional. Defaults to `GET`
3 method: GET
4
5 # Expected request body. Use `body` for simple requests
6 body: |
7 <Some request body>
8 # Expected request body file. Use `bodyFile` for complex requests that need to be stored in a file
9 bodyFile: path/to/requestBody.json
10
11 ## The following fields are used to verify if the called URI is as expected.
12
13 # URI scheme. Optional. Defaults to `http`
14 scheme: https
15
16 # URL. Optional. Defaults to `localhost:8080`
17 origin: jsonplaceholder.typicode.com
18
19 # URI Path, Optional. Defaults to `/`
20 path: /greeting
21
22 # URI query params. Optional. Defaults to empty.
23 queryParams:
24 name: "Some Name"
25
26 # Verifies expected request headers. Optional.
27 headers:
28 "content-type": "application/json; charset=utf-8"URL based request
YAML
1 connectorRequest:
2 # Checks if the HTTP::Method called is as expected. Optional. Defaults to `GET`
3 method: GET
4
5 # Expected request body. Use `body` for simple requests
6 body: |
7 <Some request body>
8 # Expected request body file. Use `bodyFile` for complex requests that need to be stored in a file
9 bodyFile: path/to/requestBody.json
10
11 # Specifies the URL of the request
12 url: https://jsonplaceholder.typicode.com/greeting?name=SomeName
13
14 # Verifies expected request headers. Optional.
15 headers:
16 "content-type": "application/json; charset=utf-8"Using the results from rover connector analyze
You need an analysis to use this format. See Analyze Connector requests for more information.
YAML
1 # Uses all the data in the analyze `.http` to create a mock expectation.
2 # Use this for complex requests
3 connectorRequestHttp: "path/to/analyzed/request/<id>_request.http"Connector response
The Connector response is optional. You can define it inline or in a file. You can also define optional response headers.
YAML
1 # Use `connectorResponse` for simple responses
2 connectorResponse: '{"helloWorld":{"greeting":"Name to greet"}}'
3 # Use `connectorResponseFile` for complex/large responses
4 connectorResponseFile: "path/to/analyzed/request/<id>_response.json"
5 # Verifies expected response headers. Optional. Defaults to empty.
6 connectorResponseHeaders:
7 "content-type": "application/json; charset=utf-8"Connectors errors and problems
Errors are standard GraphQL errors output, which can be request errors, field errors, or network errors.
Problems are issues in the Connector that occurred during selection mapping, including runtime issues such as missing fields, or incorrect method arguments.
You can define expectations for errors and problems by matching the full message string using message or matching partially with a substring using contains_message.
Matching the full message string
YAML
1 # Vec of `{message: String, path: String}`. Optional.
2 problems:
3 # Match the full message string
4 - message: "my message"
5 # Path to the field that caused the problem. Optional.
6 path: "path"
7
8 # Vec of `{message: String, extensions: IndexMap<String, JSON>}`. Optional.
9 errors:
10 # Match the full message string
11 - message: Request failed
12 # Emits a warning in Verbose mode if this differs from the asserted value. Optional.
13 extensions:
14 code: CONNECTOR_FETCH
15 service: test_connector
16 connector:
17 coordinate: test_connector:Query.helloWorld[0]Matching a substring of the message string
YAML
1 # Vec of `{message: String, path: String}`. Optional.
2 problems:
3 # Check if message contains a substring (`message` always supersedes `contains_message`)
4 - contains_message: "some substring"
5 # Path to the field that caused the problem. Optional.
6 path: "path"
7
8 # Vec of `{message: String, extensions: IndexMap<String, JSON>}`. Optional.
9 errors:
10 # Check if message contains a substring (`message` always supersedes `contains_message`)
11 - contains_message: "some substring"
12 # Emits a warning in Verbose mode if this differs from the asserted value. Optional.
13 extensions:
14 code: CONNECTOR_FETCH
15 service: test_connector
16 connector:
17 coordinate: test_connector:Query.helloWorld[0]note
If any Connector contains an unexpected
Problem or Error, an assertion fails. For example: GraphQL::Error::UNEXPECTED.Current limitations
The Connector Testing Framework only supports JSON and plaintext requests and responses.
It doesn't support non-JSON values in the
bodyfield.Variables only support string values
Full test suite example
YAML
1# Test suite configuration
2config:
3 # Path to the GraphQL schema. Can also be provided in the CLI using `--schema-file`
4 schema: path/to/schema.graphql
5 # Test suite name. If omitted, defaults to the YAML file relative path.
6 name: "my_test_suite_name"
7
8# Array of test cases
9tests:
10 # The name of the test case. Required.
11 - name: ShouldGreetWithArgName
12
13 # The target Connector, defined using the Connector ID or the Connector coordinatesRequired.
14 connector: Query.helloWorld
15
16 # Skips test case and all its expectations. Defaults to `false`.
17 skip: false
18
19 # Any variables needed for the Connector
20 variables:
21 # variables map under `$args`
22 $args:
23 name: "Name to greet"
24 # variables map under `$context`
25 $context:
26 key: value
27 # variables map under `$this`
28 $this:
29 key: value
30 # variables array of maps under `$batch`
31 $batch:
32 - id: product-id
33 # variables map under `$config`
34 $config:
35 key: value
36 # variables map under `$request.headers`
37 $requestHeaders:
38 key: value
39
40 ### API Responses ###
41 ## Mock responses expected by the Connector.
42 ## Use either `apiResponseBody` or `apiResponseBodyFile`. If either are missing, `apiResponseBody` will be set to `Json::null`, with no response mapping guarantees.
43
44 # Mock API response body. Use this when the API response body is simple.
45 apiResponseBody: |
46 {
47 "greeting": "Hello Name to greet"
48 }
49
50 ### OR ###
51
52 # Mock API response body file. Use this when the API response body is complex or too large for the YAML file.
53 apiResponseBodyFile: fixtures/mocks/responseFile.json
54
55 # Any necessary response headers from the Mock API. Optional.
56 apiResponseHeaders:
57 content-type: application/json; charset=utf-8
58
59 # Response status, used to check if `is_success` field is correct. Optional. Defaults to `200`.
60 status: 200
61
62 ### Test Expectation ###
63 expect:
64 # Uses all the data in the analyze `.http` to create a mock expectation.
65 # Use this for complex requests
66 connectorRequestHttp: "path/to/analyzed/request/00000000-0000-0000-0000-000000000000_request.http"
67
68 ### OR ###
69
70 # Connector request expectations
71 connectorRequest:
72 # Checks if the HTTP method called is as expected. Optional. Defaults to `GET`
73 method: GET
74 # Expected request body. Use `body` for simple requests
75 body: |
76 <Some request body>
77
78 ### OR ###
79 # Expected request body file. Use `bodyFile` for complex requests that need to be stored in a file
80 bodyFile: path/to/requestBody.json
81
82 ### OR ###
83
84 # Specifies the URL of the request
85 url: https://jsonplaceholder.typicode.com/greeting?name=Name to greet
86
87 ### OR ###
88
89 # URI components for building the expected request URL
90 # URI scheme. Optional. Defaults to `http`
91 scheme: https
92 # URL origin. Optional. Defaults to `localhost:8080`
93 origin: jsonplaceholder.typicode.com
94 # URI path. Optional. Defaults to `/`
95 path: /greeting
96 # URI query params. Optional. Defaults to empty.
97 queryParams:
98 name: "Name to greet"
99 # Verifies expected request headers. Optional.
100 headers:
101 "content-type": "application/json; charset=utf-8"
102
103 # Use `connectorResponseFile` for complex/large responses
104 connectorResponseFile: fixtures/expected/some_response_file.json
105
106 ### OR ###
107
108 # Use `connectorResponse` for simple responses
109 connectorResponse: '{"helloWorld":{"greeting":"Name to greet"}}'
110
111 # Verifies expected response headers. Optional. Defaults to empty.
112 connectorResponseHeaders:
113 "content-type": "application/json; charset=utf-8"
114
115 # Asserts problems related to Connectors request and response. Optional.
116 # Vec of `{message: String, path: String}`
117 problems:
118 # Match the full message string
119 - message: "my message"
120 # Check if message contains a substring (`message` always supersedes `contains_message`)
121 contains_message: "some substring"
122 # Optional
123 path: "path"
124
125 # Asserts errors related to Connectors request and response. Optional.
126 # Vec of `{message: String, extensions: IndexMap<String, JSON>}`
127 errors:
128 # Match the full message string
129 - message: Request failed
130 # Check if message contains a substring (`message` always supersedes `contains_message`)
131 contains_message: "some substring"
132 # Optional field and will emit a warning in Verbose mode if they differ from the asserted value or are present.
133 extensions:
134 code: CONNECTOR_FETCH
135 service: test_connector
136 connector:
137 coordinate: test_connector:Query.helloWorld[0]Defining shared configurations across test cases
If your tests have shared configuration values, you can define a config.common property. This helps avoid defining the same values in each test case. All fields in common are optional.
When a common value is present in the test case, it replaces the common value. For example, if you define a common value for variables.$args, and a test case defines a value for variables.$args, the test case value replaces the common value.
For structured data such as headers, the tool appends the common data instead of replacing it.
Example using `config.common`
YAML
1config:
2 schema: path/to/schema.graphql
3 common: # Set of common test case configuration for this test suite
4 connector: helloworld
5 variables:
6 $args: # only used if test case `variables.$args` is empty or not present
7 key: value
8 $context: # only used if test case `variables.$context` is empty or not present
9 key: value
10 $this: # only used if test case `variables.$this` is empty or not present
11 key: value
12 $batch: # only used if test case `variables.$batch` is empty or not present
13 - id: some-id
14 $config: # only used if test case `variables.$config` is empty or not present
15 key: value
16 $requestHeaders: # only used if test case `variables.$requestHeaders` is empty or not present
17 key: value
18 apiResponseBody: | # can be overridden by test case `apiResponseBody`
19 <some body>
20 apiResponseBodyFile: path/to/api/responseBody.json # can be overridden by test case `apiResponseBodyFile`
21 apiResponseHeaders: # can be overridden by test case `apiResponseHeaders`
22 content-type: application/json; charset=utf-8
23 status: 200 # can be overridden by test case `status`
24 expectedResponseFile: path/to/expectedResponse.json # can be overridden by test case `expected.connectorResponseFile`
25 expectedResponse: | # can be overridden by test case `expected.connectorResponse`
26 <some response>
27 expectedRequestMethod: PUT # can be overridden by test case `expected.connectorRequest.method`
28 ### URI ###
29 expectedRequestScheme: https # can be overridden by test case `expected.connectorRequest.scheme`
30 expectedRequestOrigin: localhost:8080 # can be overridden by test case `expected.connectorRequest.origin`
31 expectedRequestPath: /greeting # can be overridden by test case `expected.connectorRequest.path`
32 expectedRequestQueryParams: # can be overridden by test case `expected.connectorRequest.queryParams`
33 # only used if test case `expected.connectorRequest.queryParams` is empty or not present
34 key: "value"
35
36 ### OR ###
37 # If you prefer using a full URI
38 expectedRequestUrl: https://localhost:8080/greeting?key=value
39
40tests:
41 - name: Should map the greeting including the provided name
42 expect:
43 connectorRequest:
44 method: GET # overrides `expectedRequestMethod`
45 origin: someurl.com # overrides `expectedRequestOrigin`
46More examples
Example: All fields explicitly defined
YAML
1config:
2 schema: fixtures/schema.graphql
3
4tests:
5 - name: GreetsWithProperName
6 connector: helloworld
7 variables:
8 $args:
9 name: Some Name
10 $context:
11 key: value
12 apiResponseBody: |
13 {
14 "greeting": "Hello Some Name"
15 }
16 apiResponseHeaders:
17 content-type: application/json; charset=utf-8
18 status: 200
19 expect:
20 connectorRequest:
21 method: GET
22 scheme: https
23 origin: jsonplaceholder.typicode.com
24 path: /greeting
25 queryParams:
26 name: "Some Name"
27 connectorResponse: |
28 {
29 "greeting": "Hello Some Name"
30 }
31 connectorResponseHeaders:
32 "content-type": "application/json; charset=utf-8"GraphQL
1extend schema
2 @link(url: "https://specs.apollo.dev/federation/v2.11", import: ["@key"])
3 @link(url: "https://specs.apollo.dev/connect/v0.2", import: ["@connect", "@source"])
4 @source(name: "myApi", http: { baseURL: "https://jsonplaceholder.typicode.com" } )
5
6type Query {
7 helloWorld(name: String): String
8 @connect(
9 id: "helloworld"
10 source: "myApi"
11 errors: {
12 message: """$("a custom error message")"""
13 }
14 http: { GET: "/greeting?name={$args.name}"}
15 selection: """
16 greeting
17 """
18 )
19}Example: From analyzed request
YAML
1config:
2 schema: tests/schema.graphql
3
4tests:
5 - name: FromCurlGenerated
6 connector: slideshow_by_name
7 variables:
8 $args:
9 name: Your-Name
10 $config:
11 user_key: 123
12 apiResponseBodyFile: "path/to/analyzed/request/00000000-0000-0000-0000-000000000000_body.json"
13 expect:
14 connectorRequestHttp: "path/to/analyzed/request/00000000-0000-0000-0000-000000000000_request.http"
15 connectorResponse: '{"author":"Yours Truly","title":"Sample Slide Show","slides":["Wake up to WonderWidgets!","Overview"]}'GraphQL
1extend schema
2 @link(url: "https://specs.apollo.dev/federation/v2.10", import: ["@key"])
3 @link(url: "https://specs.apollo.dev/connect/v0.2", import: ["@source", "@connect"])
4 @source(
5 name: "httpbin"
6 http: {
7 baseURL: "https://httpbin.org/"
8 }
9 )
10
11type Mutation {
12 json(name: String): Json @connect(
13 source: "httpbin"
14 id: "slideshow_by_name"
15 http: {
16 GET: "json?name={$args.name}"
17 headers: [
18 { name: "accept", value: "{$config.accept}" }
19 { name: "x-user", value: "{$config.user}" }
20 ]
21 }
22 selection: """
23 accessControlAllowOrigin: {$request.headers.'access-control-allow-origin'->first}
24 accessControlAllowCredentials: {$request.headers.'access-control-allow-credentials'->first}
25 slideshow {
26 author
27 date
28 slides
29 title
30 }
31
32 """
33 )
34}
35
36type Slides {
37 title: String
38 type: String
39 items: [String]
40}
41
42type Slideshow {
43 author: String
44 date: String
45 slides: [Slides]
46 title: String
47}
48
49type Json {
50 accessControlAllowOrigin: String
51 accessControlAllowCredentials: String
52 slideshow: Slideshow
53}Example: Expected output
Test Suite:
YAML
1config:
2 schema: schemas/schema.graphql
3 name: mapping_problems
4
5tests:
6 - name: Should Handle Mapping Problems
7 connector: helloworld
8 variables:
9 $args:
10 name: Some Name
11 $context:
12 key: value
13 apiResponseBody: |
14 {
15 "randomField": "Hello Some Name"
16 }
17 apiResponseHeaders:
18 content-type: application/json; charset=utf-8
19 status: 200
20 expect:
21 connectorRequest:
22 method: GET
23 scheme: https
24 origin: jsonplaceholder.typicode.com
25 path: /greeting
26 queryParams:
27 name: "Some Name"
28 connectorResponse: '{}'
29 problems:
30 - message: "my message"
31 path: "path"For tests/cases/mapping_problems.connector.ymlsh
1TEST SUITE: mapping_problems - SCHEMA: schemas/schema.graphql
2TEST CASE: Should Handle Mapping Problems @helloworld
3[SUCCESS]: HTTP::Request::Method Should Handle Mapping Problems@helloworld
4[SUCCESS]: HTTP::Request::URI Should Handle Mapping Problems@helloworld
5[SUCCESS]: HTTP::Response::Status: 200 Should Handle Mapping Problems@helloworld
6[SUCCESS]: HTTP::Response::Body Should Handle Mapping Problems@helloworld
7[FAILURE]: Expected GraphQL::Problem NOT FOUND N°1 Should Handle Mapping Problems@helloworld
8- Message: {"message":"my message","path":"path"}
9[FAILURE]: UNMATCHED GraphQL::Problem Occurred N°1 Should Handle Mapping Problems@helloworld
10- Message: {"message":"Property .greeting not found in object","path":"greeting","count":1,"location":"Selection"}
11
12FAILURES:
13
14TEST SUITE: mapping_problems
15
16TEST CASE Should map the greeting including the provided named
17[FAIL]: Expected GraphQL::Problem NOT FOUND
18Message: {"message":"my message","path":"path"}
19[FAIL]: UNMATCHED GraphQL::Problem Occurred
20Message: {"message":"Property .greeting not found in object","path":"greeting","count":1,"location":"Selection"}
21TEST RESULTS: FAILED 4 passed; 2 failed; 0 skipped