Ian Ward's Blog - excess.org

datastore_search Performance Improvements

Faster DataStore in CKAN 2.7

Posted on 2017-08-04, last modified 2017-08-10.

The newly released CKAN 2.7 adds new functionality and makes DataStore search up to 17x faster at the same time.

This article covers:

new total row calculation
new result generation
new record formats

TL;DR:

Upgrade to 2.7 then resource views and other code that uses datastore_search is faster with no other changes required
Update your datastore_search client code to use one of the new records_format=csv and/or include_total=false options to make it much, much faster

update This post now appears on the ckan.org blog

Tags: Software Python CKAN

Urwid 1.3.1 Released link

Posted on 2015-11-01.

Urwid 1.3.1 is now available at urwid.org and on pypi. This is a bug fix release that includes a number of small fixes.

Complete changelog

Tags: Urwid Software Python

ckanapi and ckanext-scheming link

Posted on 2015-06-01.

I had the pleasure of co-hosting CKANCon last week and presented a short talk on ckanapi and ckanext-scheming.

Here's a link to my slides.

Tags: Video Python CKAN

Urwid 1.3.0 Released link

Posted on 2014-10-17.

Urwid 1.3.0 is now available at urwid.org and on pypi. This is a major feature release including new asyncio event loop support. Thank you to all the new contributors.

Complete changelog

Happy 10th birthday Urwid!

Tags: Urwid Software Python

Urwid 1.2.2 Released link

Posted on 2014-10-06.

Urwid 1.2.2 is now available at urwid.org and on pypi. This is a bug fix release that includes a fix for a serious performance regression.

Complete changelog

Tags: Urwid Software Python

Resample Numpy Array without Feature Loss

Posted on 2014-10-03.

For pyrf I needed to take data from a frequency plot, which could be any number of points, and present it as a spectrogram that fills the view size exactly. In the spectrogram I only care about the maximum values that appear in the range of frequencies represented by each pixel.

If I could just divide the number of source bins by an integer factor the solution would be simple:

return np.amax(data.reshape((-1, factor)), axis=1)

But I have to be able to handle any number of source bins and output that to any number of pixels.

Fortunately numpy is awesome.

Tags: Software Python

CKAN Schemas and Multilingual Fields

Posted on 2014-09-12.

I've been working on two new CKAN extensions that I'm really excited about. Since I helped build the custom metadata schema for http://data.gc.ca/ I've never been completely satisfied with the approach we took. These new extensions are the start of a proper solution to the compromises we made.

The new extensions are:

ckanext-scheming data-driven custom schemas
ckanext-fluent multilingual text fields

Tags: Software CKAN

Speaking at IASA Open Source E-Summit link

Posted on 2014-05-10.

I will be speaking about contributing to Open Source software by webcast at the Global IT Architect Association's E-Summit on Open Source on May 16 at 2pm ET. It's free to join, listen in and ask questions.

Tags: Software Video

The Light and Dark sides of Python name binding

Posted on 2014-04-04, last modified 2014-04-09.

What does bar.foo do? - An exploration of Python assignment, objects, attributes and descriptors

This is a talk I gave March 27, 2014 at the Ottawa Python Authors Group meetup.

Follow along with the IPython notebook version if you would like to play with the code examples.

Tags: Ottawa Software Python OPAG

Learn Python Videos link

Posted on 2014-03-03.

Mahdi Yusuf, one of the Ottawa Python Authors Group organizers and curator of Pycoder's Weekly, has just posted his complete catalog of Python tutorial videos for free:

Neckbeard Republic

Tags: Video Python OPAG

Ian Ward

News