Projects (beta) GraphQL API to list contents of a Kanban column

[cpswan]

I've been trying to use the Projects (beta) GraphQL API to export the contents of a column in order to create .csv files for import into Planning Poker.

For old style projects we have a set of scripts that dump out card details (id, description, story points from labels) from the REST API to .csv files that can be imported into Planning Poker. It’s all pretty straightforward: find the project id, find the column id, iterate across the cards (issues) in the column to get what’s needed from them.

For beta projects the first hurdle seems to be that columns aren’t really a thing. It’s just a view onto the underlying data. Status (and the Group by Status view) is perhaps equivalent though. And I can see stuff about Status using a query like:

gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
  query{
    node(id: "MDEyOlByb2plY3ROZXh0MzQ1Ng==") {
      ... on ProjectNext {
        fields(first: 20) {
          nodes {
            id
            name
            settings
          }
        }
      }
    }
  }'

Which gets me a response that includes:

{
  "data": {
    "node": {
      "fields": {
        "nodes": [
          ...
          {
            "id": "MTIzOlByb2plY3ROZXh0RmllbGQ0NTY3OA==",
            "name": "Status",
            "settings": "{\"width\":265,\"options\":[{\"id\":\"f75ad846\",\"name\":\"To Do - Yet to Start\",\"name_html\":\"To Do - Yet to Start\"},{\"id\":\"47fc9ee4\",\"name\":\"PR19.5 14 SP's | 15 Velocity (1 over)\",\"name_html\":\"PR19.5 14 SP's | 15 Velocity (1 over)\"},{\"id\":\"98236657\",\"name\":\"Done\",\"name_html\":\"Done\"}]}"
          },

and if I run that settings response through jq I see:

{
  "width": 265,
  "options": [
    {
      "id": "f75ad846",
      "name": "To Do - Yet to Start",
      "name_html": "To Do - Yet to Start"
    },
    {
      "id": "47fc9ee4",
      "name": "PR19.5 14 SP's | 15 Velocity (1 over)",
      "name_html": "PR19.5 14 SP's | 15 Velocity (1 over)"
    },
    {
      "id": "98236657",
      "name": "Done",
      "name_html": "Done"
    }
  ]
}

So I can see that I want the stuff with id "47fc9ee4"

A query like this gives me everything I need (too much actually because it isn't filtered):

gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
  query{
    node(id: "MDEyOlByb2plY3ROZXh0MzQ1Ng==") {
      ... on ProjectNext {
        items(first: 100) {
          nodes{
            title
            fieldValues(first: 8) {
                  nodes{
                    value
                  }
            }
            content{
              ...on Issue {
                number
                labels(first: 50) {
                  nodes{
                    name
                  }
                }
              }
            }
          }
        }
      }
    }
  }'

But I should be able to filter it down using something like:

value(filter: {
value: "47fc9ee4"
})

As part of a query like this:

gh api graphql --header 'GraphQL-Features: projects_next_graphql' -f query='
  query{
    node(id: "MDEyOlByb2plY3ROZXh0MzQ1Ng==") {
      ... on ProjectNext {
        items(first: 20) {
          nodes{
            title
            id
            fieldValues(first: 8) {
              nodes{
                value(filter: {
                value: "47fc9ee4"
                })
                projectField{
                  name
                }
              }
            }
            content{
              ...on Issue {
                labels(first: 10) {
                  nodes{
                    name
                  }
                }
              }
            }
          }
        }
      }
    }
  }'

But that just gives me gh: Field 'value' doesn't accept argument 'filter', or in it’s full JSON majesty:

{
  "errors": [
    {
      "path": [
        "query",
        "node",
        "... on ProjectNext",
        "items",
        "nodes",
        "fieldValues",
        "nodes",
        "value",
        "filter"
      ],
      "extensions": {
        "code": "argumentNotAccepted",
        "name": "value",
        "typeName": "Field",
        "argumentName": "filter"
      },
      "locations": [
        {
          "line": 11,
          "column": 23
        }
      ],
      "message": "Field 'value' doesn't accept argument 'filter'"
    }
  ]
}

So I’m starting to think that the reason for no filter examples might be that filtering hasn’t been implemented (yet).

But if I view source for the Kanban board there's lots of tasty segments like this:

{
  "memexProjectColumnId": "Status",
  "value": {
    "id": "47fc9ee4"
  }
},

The web front end must be crafting queries along the same lines as what I want, so what part of the GraphQL API needs to be tickled right to get filtered output?

[cpswan]

I got the filtering working in Python, with some new scripts added to my dump_cards repo.

Most of the action is in the atdumpmemex.py module:

#!/usr/bin/env python3
import base64, json, os, requests

# Color constants
# Reference: https://gist.github.com/chrisopedia/8754917
COLERR="\033[0;31m"
COLINFO="\033[0;35m"
COLRESET="\033[m"

graphqlurl = 'https://api.github.com/graphql'
token = os.environ['GITHUB_API_TOKEN']
headers = {"Content-Type": "application/json", 
    "Accept": "application/json",
    "Authorization": "Bearer " + token,
    "GraphQL-Features": "projects_next_graphql" }



def list_memex_projects(org):
    query = ('query{ organization(login: \\"' + org + '\\") '
        '{ projectsNext(first: 20) { nodes { id title } } } }')
    response = requests.post(graphqlurl, 
        headers=headers, 
        data='{"query": '+'\"' + query + '\"}')
    if response.status_code != 200:
        # An error occured
        print(COLERR + "Error getting project list : "
            + str(response.status_code) + " " + response.text + COLRESET)

    json_projects = json.loads(response.text)
    for node in json_projects["data"]["organization"]["projectsNext"]["nodes"]:
        project_id = base64.b64decode(node["id"]).decode("utf-8")
        print(f'{project_id}  {node["title"]}')



def list_memex_columns(project_id):
    b64id = base64.b64encode(project_id.encode("ascii")).decode("utf-8")
    query = ('query{ node(id: \\"' + b64id + '\\")  '
        '{ ... on ProjectNext { fields(first: 20) '
        '{ nodes { id name settings } } } } }')
    response = requests.post(graphqlurl, 
        headers=headers, 
        data='{"query": '+'\"' + query + '\"}')
    if response.status_code != 200:
        # An error occured
        print(COLERR + "Error getting project columns : "
            + str(response.status_code) + " " + response.text + COLRESET)
    
    json_nodes = json.loads(response.text)
    for node in json_nodes["data"]["node"]["fields"]["nodes"]:
        if node["name"] == "Status":
            json_status = json.loads(node["settings"])
            for options in json_status["options"]:
                print(f'{options["id"]} {options["name"]}')
            


def list_memex_cards(column_id, project_id):
    cards_file = column_id + ".csv"
    b64id = base64.b64encode(project_id.encode("ascii")).decode("utf-8")
    query = ('query{ node(id: \\"' + b64id + '\\") '
        '{ ... on ProjectNext { items(first: 100) '
        '{ nodes{ title fieldValues(first: 8) { nodes{ value } } '
        'content{ ...on Issue { number labels(first: 50) '
        '{ nodes{ name } } } } } } } } }')
    response = requests.post(graphqlurl, 
        headers=headers, 
        data='{"query": '+'\"' + query + '\"}')
    if response.status_code != 200:
        # An error occured
        print(COLERR + "Error getting project column cards : "
            + str(response.status_code) + " " + response.text + COLRESET)

    json_cards = json.loads(response.text)
    f = open(cards_file, 'w')
    f.write("Issue Key,Summary,Description,Acceptance Criteria,Story Points\n")
    for card in json_cards["data"]["node"]["items"]["nodes"]:
        for status in card["fieldValues"]["nodes"]:
            if status["value"] == column_id:
                f.write(f'{card["content"]["number"]},{card["title"]},,,')
                for label in card["content"]["labels"]["nodes"]:
                        if (label["name"][-2:]=="SP"):
                            f.write (f'{label["name"].partition(" ")[0]}')
                            # break loop in case there are multiple SP labels
                            break
                f.write ('\n')
    f.close

This doesn't feel like the right GraphQL way, but it works.

[mcsken]

@cpswan I wish you left this discussion unanswered so that it would have potentially gotten to the attention of the GitHub team, and which would have hopefully implemented a proper filtering option on the query level (the same way you can filter in the browser with a filter query of status:"My column name").

While it’s possible to filter the results in-code as you showed, it's not an ideal solution, it should be done in the query. It also makes pagination necessary even if your column is small, as you later found out.

And yes, 3 years later and this seems to be the only way to do this still.

[cpswan]

@mcsken the parlous state of the GitHub GraphQL API is most of the motivation for me writing The GraphQL Way. I think by then (2.5y ago) I had no realistic hope of GitHub actually fixing this. Things have only gotten worse since, with Copilot delivering a hefty bump to revenues and diverting all management attention to AI, AI, AI. To a first approximation, nothing that's broken is getting fixed.

Your point about the browser filtering is key. The GraphQL API is broken because GitHub (the site) doesn't use it.

If you’re creating a new API, and it’s just for external customers, and not being used internally; then something is wrong. Badly wrong. And you likely won’t be getting the immediate feedback to make things right; just lots of whinging and moaning off in the distance.

As Twitter dissolved I suggested some hires that GitHub might make to my friends there (as Twitter had a pretty decent implementation of GraphQL, which they'd talked about publicly). Those folk have dropped out of touch, as the conversations became repetitive and boring (and they came to realise that they weren't empowered to do anything).

Perhaps the most aggravating aspect of this is that the fix could have been done without any breaking changes, as it's natural for GraphQL APIs to be extended that way.

[cpswan]

A colleague pointed out to me that the next challenge will be the 100 node limit on query responses. This wouldn't be a problem if it was possible to filter down to a column, but will be an issue for big project boards (esp. where there are hefty backlogs).

[cpswan]

and that 100 node limit did bite, forcing the use of pagination

[byteflowx]

Hello! Any update on this issue?
I need to get all cards from specific column.

[mgslone]

Seems this is still not possible, sadly.

[br14]

Just wanted to thank you for posting this question and answer. I was looking for a more involved example of how to query a V2 project, and you've saved me literally hours.