This is a read only archive of pad.okfn.org. See the shutdown announcement for details.

labs-data-packages ##############################################################
## 30 Dec 2013

Agenda

- dpm vs dpm2
- where we all intersect
- reuse of npm or not etc


Frictionless Data
- https://github.com/okfn/frictionlessdata
- http://data.okfn.org/roadmap
- rsync module

Centralization around protocols and concepts - not a specific library (unix philosophy)

http://blog.standardanalytics.io/post/science-for-machines

How do we collaborate together ...

Naming: #hyperdata, #frictionlessdata, [#datapackages - no only one part], 

We want a term that:
    - generates +ve buzz
    - not too hand wavy

Channel: #hyperdata
URL: 
    
## Different types of data for data packages

- File online
- API
- Streaming API from a DB (different from previous)

## What is Frictionless Data

Frictionless Data has these X components

- Data Packages
- Syncing and Storing

Possibly

- "Core" Datasets

## Roadmap

- dpm and lib - done
- Identify key examples (end to end workflows)
- registry - plan it out  (discuss this here: https://github.com/okfn/dpm/issues/5 ...)
- Integrations (R, dat, CKAN, ...)

dpm as continuously running ... - we keep pulling data down

## Decisions

- Agreed: dpm reuses conceptual components of npm but not actual code
- Agreed: go with datapackage.json
- Brand as #HyperData (for channel etc). 


##############################################################

# 23 Dec 2013

Agenda



Questions

## Discussion

## Plan

We will converge on https://github.com/okfn/dpm
- merge existing dpm2 into here ...

SB: lead on development of dpm

Stage 1:
    
dpm init
dpm install {name or url}   # just from url or git url ...
- support for datapackage.json requirements

Stage 2:

dpm view    
web create / init  online
web view

Stage 3:
    
registry
dpm publish
dpm install supports registry

### Other changes

Semi-deprecate: datapackage.js

Start with: dpm

Copy over everything from dpm2 into dpm

Maybe split out later to:

dpm-init
dpm-install

### Resolving questions

datapackage.json vs package.json

Pros of datapackage.json

- This is different so name should be different
- What we have now
- Different makes it easier to identify e.g. on github

Cons

- Means reuse for node modules ...  (but you could just copy)
- Means we could reuse npm [but it looks like we ultimately won't]


### Pieces we need

Data Package:

- init (create)
- install
- view
- publish [depends on registry]
- validate [validate a data file against schema]

Want both command line and web

Use Data
- install into a DB
- use from R
- use from Google Spreadsheets

Components/library needed to build this:

- Data Package create
- Data Package view
- CSV Validator (Basic)
- CSV Validator against JSON Table Schema

Registry / catalog
- post
- get
- ownership

### Init

As a User I want to create a datapackage.json

- I want sensible defaults and assistance
  - generate defaults - look stuff up on disk etc
  - prompt based on defaults etc
- I want data files automatically analysed
- (?) I want data files automatically discovered and added (scan / globbing https://npmjs.org/package/glob)
- I want to be able to easily update later
- I want good help as to what these fields mean
- I want to be able to generate from data files as urls

How should interaction work? here's a think-through of case for adding a data file from a url

- ask you for a url
- analyse the url
- prompt you to ocnfirm analysis
- get response
- update  datapackage.json
- prompt again ... 

### Instructions for creating
As a User I want instructions on how to create a data package

- In progress at http://data.okfn.org/publish


### Install / Clone etc

As a App creator I want to get a bunch of data from data packages into my project

- into my project could mean:
    - data files on disk at a standard location
    - into a database
- Do I want entire data package or just the data resources (?)
- What about ones that are versioned - should we install the version control?
How do we do this?

- dpm-requirements.txt like python requirements
- .dpm/requirements.txt

Where do we install to? Ans: datapackages/{datapackage-name}/...

How do we install from a URL? Answer: get the datapackage.json and then download the resources?

How do we install from a git url or a hg url etc ...