CKAN 2.11 introduces Table Designer: form builder and enforced validation for your data

This article also appears in the CKAN Blog: Taming Dynamic Data with Table Designer and Table Designer was presented in video form at CKAN Monthly Live 22

Publishing high-quality data is hard.

• Data schemas, custom rules, and enforced validation help

Publishing high-quality data over time is much harder.

• Data collected changes
• Validation rules change
• Publishers and tools producing the data change
• The way we use the data changes
• We want use old data and new data together

In CKAN, time series data and regularly updated data is often published as a new CSV or Excel file every release, either replacing an existing file or being added to a growing collection.

The Ottawa Python meetup group had its first hybrid meeting last week. Our host Spiria recorded the presentation and I’ve posted a slightly edited version above.

Thank you to Ivan at Spiria for hosting, Ian and Bryan from WiseWithData for presenting and feeding us, and to everyone who came out. It was a great evening.

We’re turning a movie-playing circuit into an actual interactive game. This video lays out what we need to do:

• Use the HD44780 scrolling feature for the play field
• Design custom CGRAM characters
• Build a controller for user input
• Build a health tracker to halt the circuit on game over
• Tie it all together with new logic, opcodes and program code

In this article we generate mazes using jupyter notebooks, numpy, matplotlib, and the scikit-image flood_fill function.

Almost since I could first draw I’ve been fascinated with creating and solving mazes. Graph paper and mechanical pencils were some of my favorite gifts. Recently I’ve rediscovered some of that same childlike excitement by designing a homebrew cpu, and the next game I’m building for it needs some custom mazes.

There are lots of existing maze generating algorithms but I wanted something that filled cells (not grid edges) and that is fast and simple to do in Python so it would be easy to tweak and customize.

ckanext-scheming is a CKAN extension for customizing dataset, group and organization metadata forms, validation rules and display templates.

With ckanext-scheming 3.0 CKAN dataset metadata forms may now be split across multiple pages using start_form_page.

Example ckanext-scheming schema:

- start_form_page:
    title: Detailed Info
    description:
      These fields improve search and give users important links

  field_name: tag_string
  label: Tags

Each start_form_page block marks a field at the beginning of a new page of fields. All following fields without a new start_form_page block will appear on the same page. This lets us maintain compatibility with existing schemas and makes it easy to reassign pages over time without affecting how metadata appears in the API.

We crank up the bit rate for a widescreen character LCD video experience using our new 512K flash.

Based on the encoder we developed for our Bad Apple on 32K EEPROM video but with our clock running 25x faster, we test the limits of our LCD character display.

In this video we replace our clock’s 4011 CMOS Quad-NAND gate IC with a TTL equivalent and discuss compatibility between different logic families.

We connect an HD44780-powered LCD character display to the circuit and play a simple text animation from the program ROM.

We build a larger address register and use it to test an SST39SF040 512k flash chip.

This is part two of a two-part series. Read part one here.

In this post we use Python to encode the full Bad Apple video (3m39s) into 32 kilobytes. Our playback hardware has no CPU so our video “compression” is limited to the features of our HD44780-powered LCD display.

We create an illusion of a full bitmap display by carefully juggling 8 CGRAM characters across the 8 x 4 video area and lean on LCD display persistence.

This is part one of a two-part series. Read part two here.

This post covers how we interface a 32K EEPROM, a clock and an address register to a 20 x 4 LCD character display module and use it to play a video. We “decompress” the video by expanding our 8-bit data stream to 9 bits for playback in hardware. There is no CPU or microcontroller involved.

We complete our clock module by adding a halt signal and test it with our address register. Now our (future) programs won’t have to run forever.

We add a reset circuit to our clock module that resets our address register to “0” on power-on and when a button is pressed.

We add three more 74LS163’s to our counter to make a 16-bit address register for our homebrew CPU project.

We connect our clock circuit to a 74LS163 synchronous binary counter and see how to count, reset and load values into the counter.

We add a potentiometer and a single-step button to our clock circuit.

In this video we create a clock circuit with a 555 timer in astable mode, and use it to blink a green LED.

This video introduces the first milestone for our homebrew CPU project.