COLLECTED BY

Organization: Archive Team

Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.

History is littered with hundreds of conflicts over the future of a community, group, location or business that were "resolved" when one of the parties stepped ahead and destroyed what was there. With the original point of contention destroyed, the debates would fall to the wayside. Archive Team believes that by duplicated condemned data, the conversation and debate can continue, as well as the richness and insight gained by keeping the materials. Our projects have ranged in size from a single volunteer downloading the data to a small-but-critical site, to over 100 volunteers stepping forward to acquire terabytes of user-created data to save for future generations.

The main site for Archive Team is at archiveteam.org and contains up to the date information on various projects, manifestos, plans and walkthroughs.

This collection contains the output of many Archive Team projects, both ongoing and completed. Thanks to the generous providing of disk space by the Internet Archive, multi-terabyte datasets can be made available, as well as in use by the Wayback Machine, providing a path back to lost websites and work.

Our collection has grown to the point of having sub-collections for the type of data we acquire. If you are seeking to browse the contents of these collections, the Wayback Machine is the best first stop. Otherwise, you are free to dig into the stacks to see what you may find.

The Archive Team Panic Downloads are full pulldowns of currently extant websites, meant to serve as emergency backups for needed sites that are in danger of closing, or which will be missed dearly if suddenly lost due to hard drive crashes or server failures.

Collection: ArchiveBot: The Archive Team Crowdsourced Crawler

ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites).

To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel operator permissions in order to issue archiving jobs. The dashboard shows the sites being downloaded currently.

There is a dashboard running for the archivebot process at http://www.archivebot.com.

ArchiveBot's source code can be found at https://github.com/ArchiveTeam/ArchiveBot.

TIMESTAMPS

Module Counts

	Jun 6	Jun 7	Jun 8	Jun 9	Jun 10	Jun 11	Jun 12	Avg Growth
Clojars (Clojure)	26375	26380	26389	26398	26402	26405	26413	6/day
CPAN	41462	41476	41479	41493	41496	41507	41507	7/day
CPAN (search)								5/day
CRAN (R)	15788	15790	15790	15794	15805	15808	15820	5/day
Crates.io (Rust)	41355	41399	41446	41494	41545	41587	41642	48/day
Crystal Shards	5607	5611	5614	5617	5620	5623	5625	3/day
Drupal (php)	45549	45548	45553	45564	45572	45577	45584	7/day
DUB (dlang)								2/day
Gopm (go)								7/day
Hackage (Haskell)	14923	14924	14926	14928	14928	14928	14929	1/day
Hex.pm (Elixir/Erlang)	10089	10093	10094	10097	10099	10105	10108	3/day
Julia								0/day
LuaRocks (Lua)	2483	2483	2483	2483	2483	2484	2484	0/day
Maven Central (Java)	342498	342554	342685	342837	343062	343211	343361	144/day
MELPA (Emacs)	4585	4588	4597	4597	4597	4597	4597	2/day
Nimble (Nim)	1305	1305	1306	1309	1310	1310	1310	1/day
npm (node.js)	1302182	1302750	1303605	1304555	1305496	1306693	1307837	942/day
nuget (.NET)	208126	208126	208284	208284	208609	208609	208609	80/day
Packagist (PHP)	269360	269463	269573	269734	269871	270000	270163	125/day
Pear (PHP)	603	603	603	603	603	603	603	0/day
Perl 6 Ecosystem (perl 6)	1569	1569	1570	1570	1570	1570	1572	0/day
PyPI								177/day
Rubygems.org	159971	159992	160008	160020	160054	160080	160108	23/day
Vim Scripts	5536	5536	5536	5536	5536	5536	5536	0/day

Data is collected by scraping the relevant websites once a day via a cron job and then stored in a Postgresql database for later retrieval. Growth rates are calculated by averaging data over the last week. I'm gathering counts of separate modules, so multiple versions of the same module/package/gem only count once (foo-1.2, foo-1.3 and bar-1.0 would count as 2 total).

(Jun 23, 2019) Update Crates.io, Julia, and LuaRocks after site changes. Also upgrade LOTS of ruby gems and get off a deprecated Heroku stack.

(Jun 3, 2018) Update Drupal, DUB, and PyPI after their sites updated.

(Mar 4, 2018) Added Crystal Shards, as well as JSON endpoints /repositories and /repositories/:id/counts/:start/:finish as first steps towards revamping JS front end. (e.g. /repositories/1/counts/2018-02-01/2018-02-28)

(Sept 10, 2017) Update Crates.io for their new data URL and add Nimble, the Nim package manager. Many thanks to Alex Libman for pointing me to it.

I'd like to add more repositories. If you have suggestions, please send them to erik@debill.org.

CPAN and CPAN (search) used to be two conflicting sources for data about how many modules are in CPAN. During spring of 2011 CPAN got a site refresh and the numbers came into line with each other. It looks funny on the graph, but it's an interesting bit of history.

GoDoc is weird. It's not a true package repository in the same sense as all the others, but as far as I can tell it's the closest GoLang has. Be aware that it pretty drastically overcounts the number of packages. Don't use the raw numbers to compare with other languages. You can still watch the line to see changes in velocity, though.

If you'd like to check out the data in more detail, you are welcome to download it in a CSV file.

about

May	JUN	Jul
	12
2019	2020	2021