jack reed

Sitemaps that scale

2020-01-10T00:00:00+00:00

A few months back, at a Blacklight-LD Working Meeting a few of us were discussing issues with building sitemaps for catalog websites with millions of records. We want to be able to submit sitemaps to search engines so that our content is more discoverable, but generating sitemaps for millions of webpages can sometimes cause headaches.

A few of these headaches include:

Long running processes that create sitemaps can become stale rather quickly
How do we efficiently manage sitemaps when a record is removed, updated, or changed
Out of date sitemaps could lead to 404 errors causing SEO reduction

After a session brainstorming about this problem, a few of us split off to come up with a solution that works in Blacklight. Thanks to everyone at the meeting who helped discuss the problems and work on a solution together.

The first group of us to work on the problem included myself and

The solution

The solution we devised was originally first implemented within a Blacklight catalog application.

The solution relies on two things:

Partitioning your Solr documents in a semi-evenly distributed way
Using prefix queries in Solr efficiently query calculated parts of your index

Our solution takes advantage of Solr’s ability to automatically create efficient partitions of the data by creating a hexadecimal hash of our unique id field using the SignatureUpdateProcessorFactory

<updateRequestProcessorChain name="add_hashed_id">
  <processor class="solr.processor.SignatureUpdateProcessorFactory">
    <bool name="enabled">true</bool>
    <str name="signatureField">hashed_id_ssi</str>
    <bool name="overwriteDupes">false</bool>
    <str name="fields">id</str>
    <str name="signatureClass">solr.processor.Lookup3Signature</str>
  </processor>

  <processor class="solr.LogUpdateProcessorFactory" />
  <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

<requestHandler name="/update" class="solr.UpdateRequestHandler">
  <lst name="defaults">
    <str name="update.chain">add_hashed_id</str>
  </lst>
</requestHandler>

Next, because we know the number of documents and roughly the number of documents that can be displayed in a sitemap urlset (50,000 max), we can determine how many prefix characters we want to query which will give us our urlset.

numberOfDocuments / documentsPerUrlSet = numberUrlSetsNeeded

numberUrlSetsNeeded = 16^y

16 is used as our base here because we our hashes only use 0-9 a-f (hexadecimal positional system).

We can then calculate our exponent.

numberUrlSetsNeeded = 16^y
y = log16(numberUrlSetsNeeded)
y = ln(numberUrlSetsNeeded) / ln(16)

Our exponent (y) here will give us the number of prefix characters we will need to build to create a urlset with less than our target documentsPerUrlSet per set. We then create a sitemapindex containing these sitemap urlsets. So if our exponent is 3 we will have something like this:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/siteindex.xsd"
  xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/000</loc>
  </sitemap>
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/001</loc>
  </sitemap>
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/002</loc>
  </sitemap>
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/003</loc>
  </sitemap>
  ...
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/af4</loc>
  </sitemap>
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/af5</loc>
  </sitemap>
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/af6</loc>
  </sitemap>
  ...
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/ffd</loc>
  </sitemap>
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/ffe</loc>
  </sitemap>
  <sitemap>
    <loc>http://127.0.0.1:3000/sitemap/fff</loc>
  </sitemap>
</sitemapindex>

Putting this into production

This was all theoretical until Charlie Morris put this into production for Penn State Libraries’ catalog. And Charlie has reported this working fairly well so far without too many issues.

We decided then to pull all of this into gem so that we can use it in multiple applications. Jessie Keck, Camille Villa, and myself did that a few days ago. We now have a gem that others can use in Blacklight and GeoBlacklight applications, blacklight_dynamic_sitemap.

Thanks to everyone who worked on putting this solution together!

Who's on First in SearchWorks

2019-04-29T00:00:00+00:00

Over the past few days, I’ve worked on a basic implementation of Who’s on First (WoF) data into Stanford Libraries catalogue, SearchWorks. This work has been part of an experimental effort to enhance discovery using linked data. While some may not consider WoF “Linked Data™”, it has a rich and unique set of concordances and unique features which enable a rich set of applications. Other gazetteers often only contain center points or bounding box geometries for places, one of the unique characteristics of Who’s on First is the data contains full geometry of locations.

It’s also worth pointing out that I look at this from a narrow lens on linked data as a whole. I recognize that there are experts in this field with more substantial experience with linked data and reconciliation approaches. I’m curious to learn more about these and how they might apply to Who’s on First, if you are interested to chat more about alternative approaches feel free to connect with me on Twitter.

Starting with strings

Our ILS team provides us binary MARC records that are used in our indexing processes. These MARC records only contain strings that represent places using the Library of Congress Name Authority File (LCNAF) known labels.

So how do we get from

Tenderloin (San Francisco, Calif.)

The Who’s on First data already contains the LCNAF identifier:

'wof:concordances': {
  'gp:id': 23512024,
  'loc:id': 'n97044389',
  'qs:id': '953338',
  'qs_pg:id': '953338',
  'wd:id': 'Q7464'
},

So if we can get the LCNAF identifier for the string Tenderloin (San Francisco, Calif.) it should be easy enough to get the WoF id. Luckily, Library of Congress has a “Known-label retrieval” service that can be used to lookup known labels.

The known-label retrieval service will take known-label in the url (http://id.loc.gov/authorities/label) and then redirect to the authority record if it is found.

Example:

curl -I -L  'https://id.loc.gov/authorities/label/Tenderloin (San Francisco, Calif.)'

# Redirects to http://id.loc.gov/authorities/names/n97044389

While the service will work with content negotiation Accept headers, I found it easier to just parse the response header to retrieve the LCNAF URI.

Of the top 10,000 geographic terms in SearchWorks, I was able to retrieve 6,497 known label URI’s. Updating the data to get Who’s on First identifiers After we have the LCNAF URI’s we can then lookup a Who’s on First record where the concordances have been added. There is no formalized Who’s on First search service, so the options are:

Searching a download of the data
Using the GitHub search API to search the whosonfirst-data GitHub organization
Using the WoF Spelunker data explorer

Due to time constraints I chose to use the WoF Spelunker to lookup WoF ids based on matched LCNAF identifiers.

https://spelunker.whosonfirst.org/search/?q=n97044389

I’m assuming here also that the first record returning here is the only match and return that WoF id as a match. Using this process, I matched 1084 records.

Displaying the data in SearchWorks

Once we have the data, we can start to use it in our catalogue. Since our index records still only contain the text label, I stored the results in JSON keyed by the text label for quick lookup by the search application.

{
  "Tenderloin (San Francisco, Calif.)" : {
    "loc_id":"n97044389",
    "wof_id":"85865903"
  }
}

We can now start displaying information from Who’s on First records on SearchWorks show pages.

We did something with Who’s on First in a library catalogue! For fun I added a map of the area, and some metadata from WoF.

Future work

This is just a beginning, but was a fun exploration on how library data might start to integrate with Who’s on First. It also starts to tease at the idea of what could be possible by integrating rich geographic data sources with library metadata. But by no means is this ready for primetime. A few things stuck out as next steps to take here.

More LoC concordances - with only around 10% total of our top 10,000 geographic names matching, it would be great to be able to add more Library of Congress ids to Who’s on First. There are some complexities with the data here, there are many labels that aren’t necessarily a place in our “Region” facet like “European Economic Community countries”. Another issue here is that regions or continents are not represented in the same way as other places within Library of Congress identifiers.

Alternate name search - a nice enhancement to our search would be to index alternate names from WoF data. This would allow for searches in other languages or different terms to return more relevant results.

Spatial search - In addition to alternate name search, spatial search enhancements could also be added using WoF data.

Search service for WoF - a more robust way to search Who’s on First data could help resolve some of these data issues. This has been discussed before by those in the library community it would be nice to have this as a shared service with some enhancements meant for reconciliation over the Spelunker.

The case for serving your IIIF content over HTTPS

2017-05-23T00:00:00+00:00

TLDR: IIIF content hosted over HTTP is not fully usable by HTTPS hosted webpages.

In writing this blog post, I realized that I can’t fully understand what all of the barriers are for IIIF adopters in moving to HTTPS. To that end, I would like to know more about this so we can focus the community to provide more useful resources. Would you mind completing this short (4 questions, 3 are multiple choice) survey about your HTTPS adoption?

https://goo.gl/forms/6pvcGUG67yFzPTDD3

Interoperability is a characteristic of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, in either implementation or access, without any restrictions.

Definition of Interoperability ¹

For several years there have been pushes from organizations to migrate websites to use HTTPS²³⁴⁵. This serves as an informational post for IIIF content users and providers on why serving IIIF content over HTTPS is just as important and how to do it.

There are many reasons why as a IIIF content provider you would want to serve your content only using HTTPS, the best reason first:

By serving your content over HTTPS exclusively, your image resources gain interoperability.

But don’t worry, this is not a problem exclusive to IIIF but a larger issue with content on the Web and the way browsers handle security.

What is HTTPS?

Hyper Text Transfer Protocol Secure (HTTPS) is the secure version of HTTP, a protocol that is used to transfer information on the World Wide Web. HTTPS provides a layer of encryption using SSL/TLS. While originally adopted on secure websites (e.g. financial institutions), it is now the preferred⁶ way to serve content on the Web.

Why and how does your content become more interoperable with HTTPS?

TLDR: IIIF content hosted over HTTP is not fully usable by HTTPS hosted webpages.

Your gain in interoperability is not something to do with how the IIIF specifications are written, but in how web browsers implement security policies. For the purpose of this discussion I will talk primarily about the IIIF Image API and the IIIF Presentation API but other IIIF specifications are also implicated.

As websites move to HTTPS only, content hosted over HTTP starts to become unusable.

The problem boils down to something called mixed content⁷. Mixed content describes a scenario when a user visits a site hosted over HTTPS and that page then requests content hosted over HTTP. Browsers specifically block mixed active content⁸ which causes problems for most browser-based IIIF clients. Browser security models prohibit displaying secure content (a web page hosted on HTTPS) with some types of insecure content (IIIF content hosted over HTTP).

How do browsers use IIIF?

IIIF clients implemented in browsers usually request JSON or JSON-LD as a precursor for requesting images. These JSON responses give information to the client in how to display images.

This request/response cycle becomes problematic when the webpage requesting HTTP resources is hosted over HTTPS. Browser content security specifically blocks mixed active content⁸ which includes the JSON responses needed for IIIF clients usually requested as an XMLHttpRequest. For many browser-based IIIF clients hosted over HTTPS, these security restrictions essentially makes HTTP resources unusable. :(

Additional considerations

At the moment, only mixed active content is blocked by the browser’s security model. Mixed passive/display content is not blocked, and this includes <img> resources. This means that a browser-based IIIF client that displays content using <img> element tags should be ok.

So what should I do?

Host all of your content over HTTPS. No exceptions.

Why else should I host my content over HTTPS?

Not only is it important for interoperability, there are other really good reasons to serve everything over HTTPS by default⁹¹⁰¹¹⁶.

Trust

By serving content over HTTPS you can guarantee to your users that they are receiving the content that they requested and nothing else. This provides proof to the user/browser that you are talking to the server that was requested. Internet Service Providers can inject content into pages¹²¹³, using HTTPS prevents them from being able to do this. Serving content over HTTPS using a trusted certificate, can also prevent man-in-the-middle (MITM) attacks¹⁴.

Privacy

By using HTTPS, all of the traffic between a user and the server is encrypted. This encryption layer gives your users a level of privacy ensuring that traffic between your server and your users is not broadcast to bad actors. This guarantees that only the server and browser can read the data that is transmitted between them.

Search engine optimization

Google started using HTTPS as a “ranking signal” for its search results¹⁵ back in 2014. This “signal” seems to rank HTTPS websites as delivering high-quality content. By serving your content in a secured way you can increase your ranking in search results.

Browsers will start marking HTTP as insecure

Google Chrome has decided that it will eventually start marking HTTP webpages as insecure. ⁴

Chrome has already started to remove functionality like Geolocation-API from HTTP hosted sites¹⁶ and more things will be coming.

A great resource from Google on “Mythbusting HTTPS”.

How do I host my IIIF content using HTTPS?

I hope you are convinced now that all of your IIIF content should be hosted over HTTPS. Often times, the largest hurdle here is organizational buy-in. Yet the technical considerations are not trivial at all. Migrating legacy services from HTTP to HTTPS can take a bit of time and is really specific to the technical infrastructure. Some good news here is that as more and more websites move to HTTPS there are more resources than ever to get started. I won’t try and cover how to

Implementing HTTPS, first things first getting a trusted certificate

The first thing you need to implement HTTPS is a trusted certificate. Traditionally these are purchased through a trusted certificate provider and can vary in cost. Often times large organizations have the ability to purchase these through a central IT department that controls DNS.

Some cheap/free options

I wanted to outline a few cheap/free options for obtaining these certificates. sslmate is an options that provides certificates for $15.95 / year for single hosts, and you can obtain a Wildcard SSL for $149.95 / year¹⁷.

A new option is now available, that allows you to obtain certificates for free!

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. It is a service provided by the Internet Security Research Group (ISRG).

Many hosting providers have integration with the service to make installation easier. If you run your own servers, I would recommend taking a look at Digital Ocean’s technical tutorials on installing Let’s Encrypt certificates¹⁸. There are tutorials for many different popular applications, languages and platforms.

Setting up Let’s Encrypt seems like it could be complicated, but the EFF has made it even easier with a new software project Certbot. Certbot “Automatically enable HTTPS on your website with EFF’s Certbot, deploying Let’s Encrypt certificates”¹⁹. This can take out some of the headache of having to renew your certificate.

I chose to host this blog over HTTPS using netlify and it was straightforward to setup. You can read about my experience in this post.

Moving all services to HTTPS

Because IIIF relies on potentially many different services you may need to be intentional on when and how you move your services. Because of the mixed active content problem, one likely needs to migrate Image API services first, before moving Presentation API services.

IIIF community specific problems

In writing this blog post, I realized that I can’t imagine what all of the barriers are for IIIF adopters in moving to HTTPS. To that end, I would like to know more about this so we can focus the community to provide more useful resources. Would you mind completing this short (4 questions, 3 are multiple choice) survey about your HTTPS adoption?

https://goo.gl/forms/6pvcGUG67yFzPTDD3

Thanks and I look forward to continuing the conversation!

Special thanks to Mark Matienzo for reviewing this post for me before I published and Sheila Rabun for helping with the survey.

Moving this site to HTTPS

2017-05-06T00:00:00+00:00

I recently volunteered to write a blog post on why IIIF resources should be served over HTTPS rather than HTTP. Turns out I should probably be serving that same blog post over HTTPS. This is a quick post on my experience in moving my blog to HTTPS.

What’s wrong with GitHub Pages?

Previously, my blog was served using GitHub Pages from a custom domain. I’ve had a great experience with GitHub pages and it has been really easy to setup and use. Unfortunately, GitHub pages using custom domains do not have an option for HTTPS. GitHub Pages does offer HTTPS for non custom domain sites, but that doesn’t really help me here. I would like the flexibility to move my website from hosting providers without having to rely on their domain names.

Researching HTTPS site hosting offerings

I looked into several options to see what was out there for hosting HTTPS sites. A few of my requirements included:

something easy to deploy
painless maintenance
HTTPS :)
custom domain

With the recent advent of Let’s Encrypt I found there were quite a few more options available. I considered hosting static files on a server from Dreamhost. I purchased some domains from Dreamhost years ago and still have an active account. They have seamless integration with Let’s Encrypt, but it seemed like the continuous deployment integrations with GitHub were slim to none.

Setting up and maintaining a server just to publish a blog seemed like even more overkill. Even though I found some great guides from Digital Ocean on doing this. The Digital Ocean offerings at $5 / month for the small droplets are great, I just didn’t want to have to maintain the server or packages on an on-going basis.

Going with Netlify

I had heard about Netlify from somewhere and it seemed to meet all of my requirements. It offers free hosting for custom domain sites, with simple integration for continuous deployment.

Literally.. it took 5 minutes while on a plane to setup.

I only had to make a quick minor change to my blog’s code by adding a Gemfile and Gemfile.lock, previously assumed dependencies by GitHub Pages.

I then changed my DNS, and voila it was done.

The DNS changes took a bit to switch over, but after they were done I was able to enable HTTPS in the Netlify application.

Did anything break?

Yep, I had just a few things break. First, in one of my blog posts I was including MathJax JavaScript using HTTP. This was a quick change to using an updated version hosted via HTTPS from Cloudfare. The only other thing that broke was an iFrame I was including from a HTTP url from GitHub pages. This was just another quick change since GitHub Pages using the github.io domain are also served over HTTPS.

All in all i was an easy and painless process.

Rounding strategies used in IIIF

2016-10-14T00:00:00+00:00

2017-06-19 - Corrected IIIF API Implementation Notes strategy; floor → ceil. Thanks @zimeon

TL:DR

Make sure you are rounding the same way across your stack (I think).

While developing Leaflet-IIIF I’ve noticed differences in the way that IIIF clients and image servers implement calculating aspect ratios. This post serves as a collection of information gathered in hopes that it can help the community steer in a collective direction.

What’s this rounding about?

The IIIF Image API is a super cool, powerful way to serve out images. A primary use of this API is serving tiled images in a standardized way so that multiple clients can use them.

A tiled image example

Here’s an example of a IIIF image that is being served out from the Stanford University Library.

To create this tiled image view, Leaflet-IIIF requests images from the image server at Stanford and then stitches them all back together. The client figures out which images it needs to request to create an optimal experience for the end user.

Requested image: https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/0,0,5426,3820/679,/0/default.jpg

This request is asking for an image that is the full size and is scaled to 679 pixels wide. That 679 pixel width is calculated in Leaflet-IIIF and used to request a canonical url of an image.

Now this is all good so far, and the image seems to load fine but lets look closer.

There is a black line located at the bottom of the image. This black line exists because the image server is expecting to return an image of a different size than the size requested (or thought it was requested). The server then seems to fill the negative space with black pixels.

The returned image comes back as 679 pixels wide and 479 tall. The original dimensions of the image are 5426 x 3820. Requesting a scaled image of 679 pixels wide could return an image of either 478 or 479 pixels depending on how the server calculates the aspect ratio.

\begin{equation} h_2 = \frac{(w_2 * h_1)}{w_1} \end{equation}

\begin{equation} 478.02801326944342 = \frac{(679 * 3820)}{5426} \end{equation}

The image server is expecting to return an image that is appropriately scaled to the aspect ratio using a specific rounding strategy. Since an image server is only going to return an image with integer pixel dimensions it now must make a decision. Do I discard the remainder? Do I round up to the next integer? Do I round to the nearest integer? In this example the image server always rounds up to the next integer. These strategies can be consolidated to the following and are available in most standard math libraries.

Strategy	Description	Example
ceil	always round up to the next integer	`Math.ceil(10.001) -> 11`
floor	always round down to the previous integer	`Math.floor(10.88) -> 10`
round	round to the closest integer	`Math.round(10.4) -> 10`, `Math.round(10.6) -> 11`

Problems these differences can present

Two separate IIIF server implementations have the potential to return a different sized image with the same request. The underlying image processing tools may also be using a different rounding implementation than the server that is returning the tile. These inconsistencies have the potential to cause unintended artifacts in the image viewing experience.

Comparisons of implementations

A non-exhaustive comparison of IIIF rounding implementations:

Software	Rounding method	Example
Image API Implementation Notes	ceil	see: `ws = (width - xr + s - 1) / s # +s-1 in numerator to round up`
go-iiif	floor	`/0,0,3897,4096/245,/0/default.jpg` returns `245 x 257`
riiif	round	`0,0,4264,3248/333,/0/default.jpg` returns `333 x 254` (see ImageMagick)
kakadu	ceil	proof thanks @jpstroop
openjpeg	ceil	proof thanks @jpstroop
loris	ceil	code reference
leaflet-iiif	ceil	code reference
openseadragon	ceil	code reference
iiif_s3	ceil	code reference

See something wrong here? Submit a PR

What should we do?

I’m not exactly sure what we should do. I would assume that at varying levels of the IIIF rounding methodologies are implemented for good reasons. Hopefully for performance reasons at the low levels. What prompted me to start looking into this was a pull request in Leaflet-IIIF aiming to resolve some of the artifacts. This PR prompted a discussion about the canonical uri syntax and what a client can expect back from a IIIF image service. It is counterintuitive that a canonical uri can return images at different sizes.

The community could work to standardize on a particular rounding method. Though the coordination and software changes/upgrades might not be worth the effort. ¯\(ツ)/¯

Hopefully this serves as a resource for others who run into this problem or want to discuss further. Relevant previous discussions:

Minding the Open Data Gap

2015-08-30T00:00:00+00:00

Over the past several years, it has been so encouraging to see government embrace and deploy open data sites. This shift towards open data is monumental and signals a change in how these governmental organizations connect with those they serve. Sites like Data.gov, Oregon Open Data Portal, and Data Driven Detroit are becoming more and more the norm rather than the exception.

Code for America, local advocacy groups, and innovative companies like Mapbox and Azavea have pushed this movement even further. There is even an industry forming around providing open data solutions for organizations. Newer companies like Socrata have stepped into this space with long standing industry behemoth’s like Esri also offering an open data product.

This is all great, and beyond my wildest dreams from when I first started working in the open data space. But since I’ve started working at Stanford University Libraries, I’ve come to believe we need to do even more to provide long-term access to this data. If an organization removes access to a dataset in an open data portal, does the general public even notice?

The lack of interest, the disdain for history is what makes computing not-quite-a-field.
— Alan Kay

People place inherent value on published or physical items. Much of the data published to open data portals is treated ephemerally. One day you may be able to access a data set and the next you may not. And just because a dataset is listed on Data.gov, that doesn’t necessarily mean you can download or use it. Loss of access to open data can happen for a lot of reasons including:

Web and file services are no longer maintained (due to cost, organizational challenges, loss of institutional knowledge, etc.)
Decisions are made to withdraw data that is not being frequently accessed
Historical data is overwritten to make room for current business needs only
Data is not adequately described or curated

Loss of access to this data may not cause any immediate negative ramifications to a government agency, or even its users who are trying to access it. But this loss of data is perpetuating a growing data gap for digitally created data. To illustrate, someone who comes across a dusty old map in a storage closet might think to themselves “Hey this could be valuable”. They may even go as far as checking with a local library or museum to see if they would like the map rather than throwing it away. But with digital only content thought of preservation is rarely considered. The dusty data represented in bytes is much more frequently created and deleted without a second thought. Many libraries and museums are not equipped to preserve such digital content even if they were contacted.

Data that seems as if it should be accessible in an authoritative way, a lot of times isn’t.

Anyone know an open dataset of 2012 US presidential election results? Values in http://t.co/2CDk32h9r5 seem off. Thanks! #followerpower
— Anita Graser (@underdarkGIS) August 23, 2015

So what is the solution? Government agencies, open data advocacy groups, and libraries all have a role to play and should be working together. If enough thought has gone into publishing the data in an online portal, that same data should be preserved in perpetuity. What we need is a distributed Internet Archive for data.

At Stanford we are already preserving all of the data that we serve out through our spatial data infrastructure and our discovery portal, EarthWorks. But one, or even a handful of universities doing this isn’t enough. A coordinated effort and between organizations is needed to provide near and long term access to this huge amount of content.

Who’s up for this?

Visualizing 10 Million GeoNames with Leaflet and Solr Heatmap Facets

2015-06-29T00:00:00+00:00

I wrote this post in response to some requests I had after some tweets a while ago.

A proof of concept for @ApacheSolr with @LeafletJS server side clustering using the new Solr 5.1 heatmap >100000 pts https://t.co/twydmaag7Q
— Jack Reed (@mejackreed) May 7, 2015

A follow up, @ApacheSolr heatmap with >10,000,000 locations using @LeafletJS MarkerClusterer https://t.co/JrG8vudIk6 still really fast
— Jack Reed (@mejackreed) May 8, 2015

A recent addition in Solr 5.1 is a new type of facet ability, Heatmap Faceting. It looks like this is another great addition added to Solr by David Smiley. I was excited to see this feature in the release notes but was curious about the practicality and performance.

The Heatmap Facet will return a grid of counts for documents over a given area. The return type defaults to a 2D array of values, but can also be returned as a 4-byte PNG. These type of return values can be used to generate a heatmap visualization of result hits. Additionally, the Heatmap Facet will take several parameters that modify how the heatmap is calculated or returned. For my experimentation purposes I have only been using the facet.heatmap.geom parameter. facet.heatmap.geom will limit the region that the heatmap is computed on.

Indexing spatial data

I knew that I was going to want to put this feature through some performance trials, so I opted to start with large corpus of spatial data, the GeoNames.org seemed like a suitable dataset to start with. More on indexing GeoNames data into Solr in this other post. For the rest of this, I assume you followed this post using the default example Solr schema.

Collections and fields used here

Collection name	GeoNames title field	Geometry field
`gettingstarted`	`title_t`	`loc_srpt`

Heatmap requests and returns

For this blogpost, I only dealt with the Solr Heatmap Facet return using the 2D array of hit counts. The basic idea of the feature is that I can request a bounding area, and get return hit counts for items within that area.

An example request

A basic Facet Heatmap request:

http://localhost:8983/solr/gettingstarted/select?q=*:*&facet=true&facet.heatmap=loc_srpt&facet.heatmap.geom=["-180 -90" TO "180 90"]&wt=json

Let’s break down this request:

Request parameter	What does it do?
`q=:`	Select all documents
`facet=true`	enable faceting
`facet.heatmap=loc_srpt`	field name for heatmap faceting (needs to be type RPT)
`facet.heatmap.geom=["-180 -90" TO "180 90"]`	the region where the heatmap is computed
`wt=json`	return it in JSON

An example return

The example request above will return hit counts for the entire world. This will be in the form of a 2D array. The return from Solr will look something like this:

// Normal Solr response
...
"facet_counts":{
    ... // facet response fields
    ...
    "facet_heatmaps":{
      "loc_srpt":[
        "gridLevel",2,
        "columns",32,
        "rows",32,
        "minX",-180.0,
        "maxX",180.0,
        "minY",-90.0,
        "maxY",90.0,
        "counts_ints2D", [null, null, [0, 0, ... ]]
...

The response here gives us a lot of useful information we can use to build a mapping interface to visualize the hits. The return isn’t necessarily a JSON object but is an array, lets just call it an object with keys (even array index values) and values (odd array index values).

Response key	What does it tell us?
`gridLevel`	granularity of each grid cell
`columns`	number of columns in 2D array return
`rows`	number of rows in 2D array return
`minX`, `maxX`, `minY`, `maxY`	region heatmap 2D array was computed for
`counts_ints2D`	2D array of integers that are counts for a given region

Transforming it into a grid

counts_ints2D can be transformed into a grid. Below is an equal degree grid computed from a 32 x 32 2D integer array for the entire world ([“-180 -90” TO “180 90”]).

Visualizing with Leaflet.js

The next step is to turn the 2D integer array into a visualization depicting hit counts on a map. Leaflet.js is my goto mapping library so I created a quick Leaflet plugin, Leaflet-Solr-Heatmap, that creates a GeoJSON grid from the resulting response from Solr.

Using this plugin, with the 10 million plus GeoNames corpus will yield a result like this:

The plugin does a really naive classification of density using color. Future work can implement something a bit more scientific.

You may notice, I’ve included in the example Solr response and Leaflet rendering times. Here is a version using the MarkerClusterer functionality in the Leaflet-Solr-Heatmap plugin.

As you zoom further in, the plugin will send a facet query that limits the response area to the map view. This significantly increases performance of the Solr faceting at lower zoom levels. Solr will also dynamically modify the grid resolution as you zoom further in.

Going from here

This is really just a proof of concept of performance for a large geospatial dataset. Its really exciting to this fast performance with such a large dataset. I might next try and do something similar with a polygon geometry dataset to see how that works, maybe quattroshapes? I’m also interested in future work on a Blacklight plugin that will use this functionality to visualize search results. If your interested in working something collaboratively connect with me on Twitter.

Indexing GeoNames into Solr

2015-06-15T00:00:00+00:00

This post walks through a quick and easy way to index GeoNames.org locations into Solr 5.2.1. It uses the Solr default configuration for the gettingstarted collection.

For more on Solr collections vs cores.

The first part of this post is borrowed from the Solr quickstart

Getting Solr 5.2.1 up and going

Download and unzip Solr 5.2.1

$ ls solr*
solr-5.2.1.zip
$ unzip -q solr-5.2.1.zip
$ cd solr-5.2.1/

Start Solr

$ bin/solr start -e cloud -noprompt

You should now be able to successfully navigate to http://127.0.0.1:8983/solr

Formatting GeoNames.org data for Solr

GeoNames provides several data download types available on their website. This post will focus on indexing allCountries.txt which includes all features from GeoNames. This file unzipped is ~1.2 GB which could be troublesome for some. Beginning users may want to start with a smaller dataset such as cities1000.txt which is a smaller subset of the GeoNames data.

Someone out there probably could do all of this in an awesome one liner. These steps are broken up for better understanding of whats going on. We first need to format the GeoNames data into something that is indexable into Solr.

Download and unzip allCountries.zip

Download available from GeoNames.

$ unzip -q allCountries.zip

allCountries.txt comes in a tab-delimited text file in utf-8 encoding. The following fields are provided:

Field	Description
geonameid	integer id of record in geonames database
name	name of geographical point (utf8) varchar(200)
asciiname	name of geographical point in plain ascii characters, varchar(200)
alternatenames	alternatenames, comma separated, ascii names automatically transliterated, convenience attribute from alternatename table, varchar(10000)
latitude	latitude in decimal degrees (wgs84)
longitude	longitude in decimal degrees (wgs84)
more …	we don’t need the rest of these

We won’t use most of these columns, so let’s get rid of the ones we don’t need.

Get rid of columns we don’t need

We only need the 1st, 2nd, 5th, and 6th columns.

$ cut  -f1-2,5-6 allCountries.txt > allCountries_red.txt

Add a header row

Add in a header row to the tsv text file. Note, whitespace delimiters (between id, title_t, lat, lng) should be tab literals.

$ sed '1s/^/id  title_t lat lng\
/g' allCountries_red.txt > allCountries_head.txt

Add a WKT column

This command requires the csvpys version of csvkit software. Running the command will create a new WKT point column loc_srpt using the existing lat and lng columns. *_srpt is a Spatial Recursive Prefix Tree Field Type dynamic Solr field shipped with the default gettingstarted Solr schema.

$ csvpys --tab -s loc_srpt "'POINT(' + ch['lng'] + ' ' + ch['lat'] + ')'" allCountries_head.txt > allCountries_wkt.txt

Only keep the columns we need

Get rid of the lat and lng columns

$ csvcut -c 1,2,5 allCountries_wkt.txt > allCountries_wkt_cut.txt

Convert the tsv to json

$ csvjson -i 2 allCountries_wkt_cut.txt > allCountries.json

Index into Solr

If you are doing this using the full allCountries.txt file, this command can take a while (at least 5 minutes). This command will index over 10 million records into your Solr index. You can check the status of this command by seeing if the document counts in your Solr collection are increasing. You can see this by using the Solr admin interface.

$ curl 'http://localhost:8983/solr/gettingstarted/update?commit=true' --data-binary @allCountries.json -H 'Content-type:application/json'

You should now have your GeoNames data indexed in Solr!

Checkout a Solr query.

// http://127.0.0.1:8983/solr/gettingstarted/select?q=*:*&wt=json&indent=true
{
  "responseHeader":{
    "status":0,
    "QTime":39,
    "params":{
      "indent":"true",
      "q":"*:*",
      "wt":"json"}},
  "response":{"numFound":144573,"start":0,"maxScore":1.0,"docs":[
      {
        "title_t":["El Tarter"],
        "id":"3039154",
        "loc_srpt":"POINT(1.65362 42.57952)",
        "_version_":1504146876751937536},
      {
        "title_t":["Sant Julià de Lòria"],
        "id":"3039163",
        "loc_srpt":"POINT(1.49129 42.46372)",
        "_version_":1504146876821143552},
      {
        "title_t":["Pas de la Casa"],
        "id":"3039604",
        "loc_srpt":"POINT(1.73361 42.54277)",
        "_version_":1504146876823240704},
      ...
  }
}

You can now do all sorts of fun spatial search things in Solr!