• NumPy Illustrated: The Visual Guide to NumPy
• Half-Precision Floating-Point, Visualized
• Johnny•Decimal

Hello! Content from the old excess.org site will be migrated here over the next few months. Thank you for stopping by.

CKAN’s datastore_search now comes with format options and is up to 17x faster.

This article covers:

• new total row calculation
• new result generation
• new record formats

TL;DR:

1. Use CKAN 2.7 or later and resource views and other code that uses datastore_search is faster with no other changes required
2. Update your datastore_search client code to use one of the new records_format=csv and/or include_total=false options to make it much, much faster

I had the pleasure of co-hosting CKANCon last week and presented a short talk on ckanapi and ckanext-scheming.

For pyrf I needed to take data from a frequency plot, which could be any number of points, and present it as a spectrogram that fills the view size exactly. In the spectrogram I only care about the maximum values that appear in the range of frequencies represented by each pixel.

If I could just divide the number of source bins by an integer factor the solution would be simple:

return np.amax(data.reshape((-1, factor)), axis=1)

But I have to be able to handle any number of source bins and output that to any number of pixels.

Fortunately numpy is awesome.

I’ve been working on two new CKAN extensions that I’m really excited about. Since I helped build the custom metadata schema for open.canada.ca/ I’ve never been completely satisfied with the approach we took. These new extensions are the start of a proper solution to the compromises we made.

The new extensions are:

• ckanext-scheming data-driven custom schemas
• ckanext-fluent multilingual text fields