API for code search

[kendallm]

Is there an API for interacting with Code Search?

[look]

@kendallm not yet, but we are working on it. Can you let us know some of the use cases you'd like to see? Thanks!

[mario-guerra]

I would like to +1 the request for an API. We are deprecating a large number of old packages and we'd like to be able to programmatically find repositories that are making use of the deprecated packages so we can notify the repo owners that they should upgrade.

[sgandon]

One main use case that we have and similar to @kendallm is that we want to upgrade a library or compile it with a more recent SDK version or deprecate a given library and we would like to be able to find all the repositories that would reference this library in order to contact the owner and make sur upgrading is possible.
and I believe an API would allow us to get response payload that we can furthermore exploit for filtering like only give us the distinct repositories for this search result.

[jetersen]

There are simply some searches that are impossible in the old elastic search code search. I really wish this had an API. I may have to default to HTML scraping of cs.github.com 😂

[gabrielenosso]

I can share my use case: I am working on a Software Catalog to increase the Reusability Index in my Company.
Using the Code Search API would give us a chance to find them easily with the search of specific strings (For instance, in the package.json files of repos).
Can't do that with the "normal" API, as special characters are not supported.

[jpivarski]

I'm a developer of software tools for physicists and, as a part of that, I need quantitative metrics of software adoption and patterns of use among those physicists. I've used a variety of tools to gather this sort of data (PyPI/conda-forge download statistics, regex searches of our conference talks, GitHub stars, surveys), the most incisive metrics are derived from the GitHub API.

Search results are awesome: the most relevant metric is "What code are the physicists actually writing?" (Assuming they put all their code on GitHub, which is close to true nowadays.) But the GitHub Search API is pretty limited and I've needed to do queries in a hacky way... and slowly pull data over the course of hours to avoid rate limits (and secondary rate limits). In part, that was limited by old GitHub search semantics.

I've used the new code search enough to know that I'll be able to write more directed searches with this. I'll want to get the results as text-based lists, and could scrape the HTML, but it would be much nicer to use a formal API.

While we're at it, I usually don't need the full set of information that the GitHub API returns for its results—a page full of JSON for each repo—most of the results that I'm looking for are lists of repos (JSON array of strings) and lists of users (JSON array of strings).

Unlike other use-cases, I'm interested in doing a one-time pull, maybe followed up several months later, not any kind of continuous pipeline.

[OnkarRuikar]

Many of the repos/projects have their dedicated websites. The CS API will allow them to provide more customized code search into the repo directly on their websites.

---

With the API any website could embed the Code Search. They could have more user friendly querying mechanism. The results could be shown in more creative ways.

---

Just a thought, with a clever repo structure and queries, using the API, the content on GitHub could be used as a read only database.

[gabrielenosso]

I can share my use case: I am working on a Software Catalog to increase the Reusability Index in my Company.
Using the Code Search API would give us a chance to find them easily with the search of specific strings (For instance, in the package.json files of repos).
Can't do that with the "normal" API, as special characters are not supported.

[ng-druid]

Integrating JSON path into the search API would enable package.json searching and much more. Map reduce support using custom JavaScript functions passed as JSON encoded strings would also be welcomed.

[lachilles]

Hi @look , is the code search API available yet? I have access to the code search beta and am able to view all the search results I want from the web UI. Would love to have this available programmatically since we have similar use cases to others. Would like to search spring-app-chart versions org wide and find outdated versions. If there's an open issue somewhere that I can subscribe to and be notified when the feature is ready, please share it. Thank you!!

[look]

Hi @lachilles, not yet, we are still working on scaling the system up before we can tackle the API. I will update this issue when we have news to share. Sorry, but thank you for your patience!