Repeating Subfields and Multiple Text with ckanext-scheming

ckanext-scheming 2.1 now support Datasets with repeating subfields and repeating text fields. Repeating subfieds support custom snippets and validation just like regular fields.

This work is inspired by the excellent ckanext-composite extension and replaces ckanext-repeating repeating_text fields.

You must be using CKAN 2.8 or later and a custom IPackageController plugin to index datasets with repeating subfields.

Continue reading…

Faster datastore_search in CKAN

CKAN’s datastore_search now comes with format options and is up to 17x faster.

This article covers:

  • new total row calculation
  • new result generation
  • new record formats

datastore_search Performance Improvements

TL;DR:

  1. Use CKAN 2.7 or later and resource views and other code that uses datastore_search is faster with no other changes required
  2. Update your datastore_search client code to use one of the new records_format=csv and/or include_total=false options to make it much, much faster
Continue reading…

Resample NumPy Array without Feature Loss

For pyrf I needed to take data from a frequency plot, which could be any number of points, and present it as a spectrogram that fills the view size exactly. In the spectrogram I only care about the maximum values that appear in the range of frequencies represented by each pixel.

If I could just divide the number of source bins by an integer factor the solution would be simple:

return np.amax(data.reshape((-1, factor)), axis=1)

But I have to be able to handle any number of source bins and output that to any number of pixels.

Fortunately numpy is awesome.

Continue reading…

open.canada.ca

data.gc.ca

A site I’ve been working on for the past few months has just launched. data.gc.ca “2.0” (update: now open.canada.ca) is a completely rewritten version of the Open Data portal for the Government of Canada.

I worked on the CKAN portion of the site responsible for the data catalogue. CKAN is an open source Data Portal application written in Python. I built a fully bilingual schema with many additional fields for our datasets. I also optimized, fixed and improved many parts of CKAN itself.

Our team built templates based on the Web Experience Toolkit (WET). WET provides building blocks for creating accessible, mobile-friendly web sites. These templates include a map widget for our geospatial datasets and an interactive data table for previews of tabular data.

All of our work is available on the open-data github page and is released under a permissive license.

data.gc.ca might just be the largest, most fully bilingual and most accessible CKAN site anywhere. More to come!

Painting with Braille

Braille Python Logo

This is something I’ve been wanting to write for a while.

Unicode page U+2800 has all the combinations of a 2x4 grid of Braille dots. Braille dots that line up neatly with the ones on all sides in most fonts. We can paint with this!

Continue reading…

XTerm 256-Colour Chart 2.0

XTerm 256-Colour Chart (cows)

Announcing XTerm Colour Chart 2.0 “Old and busted: Colour Cubes. New hotness: Colour Cows” release.

The script has been completely rewritten. It now requires python 2.3 or later for the optparse command-line parsing module.

Many new features have been added to this version:

  • Five different charts available in 256 and 88-colour modes
  • Origin and rotation of colour cube is now configurable
  • Size of colour cells is now configurable
  • Colour palette now exactly matches xterm defaults
  • Resetting colour palette is now flicker-free
  • Source code is much nicer (and commented!)

xterm_colour_chart on github

Continue reading…