Scalable approach to a code search?

[davepeck]

Select Topic Area

Question

Body

About a month ago I built a tiny website called "Big Time T-Strings".

The website shows a list of public repositories on GitHub that use Python 3.14's new t-strings feature, which just shipped.

To do this, an action is run daily that uses gh search code to identify repositories with a pyproject.toml that declares a minimum dependency on Python 3.14. We then shallow clone each such repository and parse the python files, looking for uses of t-strings (contains ast.TemplateStr or imports from string.templatelib).

This has been working without problem until a few days ago. The gh search code invocation that has been used since I first built the project is no longer returning consistent results, and is most definitely not returning a full result set anymore. (3.14 is new enough that there are roughly 500 such repositories at the moment; something changed behind the scenes and I often get far fewer than 100 returned today)

I know from the documentation that gh search code uses the "legacy search API". But I don't really see another better API available to perform a search that looks like this one:

gh search code --filename=pyproject.toml 'requires-python = ">=3.14"' --json repository,path,sha --limit 1000

So, two questions:

1. Does anyone know if something material changed with the "legacy code search" endpoint in the past week or so?

2. Can anyone suggest a scalable approach that meets GitHub's terms of service and that would allow me to identify all such repositories? Is there one?

Thanks! This is just a fun side project, but I'd love to figure out how to do this in a way that doesn't tax GitHub and scales nicely.

Cheers,
Dave

[ogcae]

Select Topic Area

Question

Body

About a month ago I built a tiny website called "Big Time T-Strings".

The website shows a list of public repositories on GitHub that use Python 3.14's new t-strings feature, which just shipped.

To do this, an action is run daily that uses gh search code to identify repositories with a pyproject.toml that declares a minimum dependency on Python 3.14. We then shallow clone each such repository and parse the python files, looking for uses of t-strings (contains ast.TemplateStr or imports from string.templatelib).

This has been working without problem until a few days ago. The gh search code invocation that has been used since I first built the project is no longer returning consistent results, and is most definitely not returning a full result set anymore. (3.14 is new enough that there are roughly 500 such repositories at the moment; something changed behind the scenes and I often get far fewer than 100 returned today)

I know from the documentation that gh search code uses the "legacy search API". But I don't really see another better API available to perform a search that looks like this one:

gh search code --filename=pyproject.toml 'requires-python = ">=3.14"' --json repository,path,sha --limit 1000

So, two questions:

1. Does anyone know if something material changed with the "legacy code search" endpoint in the past week or so?
2. Can anyone suggest a scalable approach that meets GitHub's terms of service and that would allow me to identify all such repositories? Is there one?

Thanks! This is just a fun side project, but I'd love to figure out how to do this in a way that doesn't tax GitHub and scales nicely.

Cheers, Dave

---

this is not you — GitHub quietly tightened legacy code search recently, so wide queries now just don’t return the full result set anymore (it’s throttling / anti-fanout).

quick fix that actually works right now:

don’t do one huge search, do multiple smaller buckets, like:

gh search code --filename=pyproject.toml 'requires-python = ">=3.14"' repo:a*
gh search code --filename=pyproject.toml 'requires-python = ">=3.14"' repo:b*
# ...

then merge all the JSON outputs.

narrow queries don’t get hammered by the new throttle — you get full coverage again.

long-term: migrate to Code Search v2 when it stabilizes, but for now this trick is the most reliable workaround.

if this saved you time; follow me on GitHub :)

Cheers,
ogcae

[davepeck]

Good to know that things have changed!

Bucketing seems like a good idea, but it doesn't appear that wildcards are accepted in practice in repo: or org: qualifiers:

> gh search code --filename=pyproject.toml 'requires-python = ">=3.14"' "repo:t-strings/tdom" --json repository,path,sha | jq 'length'
1
> gh search code --filename=pyproject.toml 'requires-python = ">=3.14"' 'repo:t*' --json repository,path,sha | jq 'length'
0

[ogcae]

I think it's because they're doing updates or the engine is in beta, but either way, check the official documentation :)

Cheers,
ogcae

[davepeck]

Is there any word on when Code Search v2 is going to be available in API form?

As far as I can tell, there's basically no way to accomplish today what Big Time T-Strings accomplished without trouble just a month ago.