This is a read only archive of pad.okfn.org. See the shutdown announcement for details.

• Acquisition describes gaining access to data, either through any of the methods mentioned above or by generating fresh data, e.g through a survey or observations.
• In the extraction stage, data is converted from whatever input format has been acquired (e.g. XLS files, PDFs or even plain text documents) into a form that can be used for further processing and analysis. This often involves loading data into a database system, such as MySQL or PostgreSQL.
• Cleaning and transforming the data often involves removing invalid records and translating all the columns to use a sane set of values. You may also combine two different datasets into a single table, remove duplicate entries or apply any number of other normalizations. As you acquire data, you will notice that such data often has many inconsistencies: names are used inconsistently, amounts will be stated in badly formatted numbers, while some data may not be usable at all due to file corruptions. In short: data always needs to be cleaned and processed. In fact, processing, augmenting and cleaning the data is very likely to be the most time- and labour-intensive aspect of your project.
• Analysis of data to answer particular questions we will not describe in detail in the following chapters of this book. We presume that you are already the experts in working with your data and using e.g. economic models to answer your questions. The aspects of analysis which we do hope to cover here are automated and large-scale analysis, showing tips and tricks for getting and using data, and having a machine do a lot of the work, for example: network analysis or natural language processing.
• Presentation of data only has impact when it is packaged in an appropriate way for the audiences it needs to aim at.

• 
  
• Acquisition -- existing classifications, controlled names, 
• Various repositories - identity repositories of data 
• metadata - access to eg. the images on flickr, but not looking at the metadata 
• Sensor Networks
• APIs
• Public / Private 
• Source control system 
• Primary source collection - generating your own, but going to those existing collections, rather than working with the library 
• Mass media 
• Social networks
• Translation and interpretation at the point of acquisition 
• Mathematical models, algorithms, teacher / experts/ mentors 
• 
  
• 
  
• Acquisition - what possible data sources are there 
• 
  

•     - extract text - is there a website?
• - recognize song - shazam is closed - any other
• 
  
• PDF
• - Tabula - tables
• - PDF2 Text - text
• - Nitro PDF
• - evernote
• -quablo
• U
• Webpages
• -import.io - extracting tables
• -kimono
• 
  
• Microtasking
• - CrowData
• 
  
• VIDEO/IMAGES/TEXT - crowd sourcing and microtasking to have individuals sort through massive data sets to make basic observations, flag and categorize
• 
  
• CSV kit - pentaho, for etl extraction, opensource but also a commercial version. 
• 
  
• document cloud - online repository of pdf document files, can upload your own for analzyis. (have to persuade journalists as it is built for them, but could also possiblz set up zour own. It is opensource)
• 
  
• alchemy api - for named entitz extraction, natural language processing.
• 
  
• UPTOM (web scraper) 
• 
  
• analice.me - An investigative journalism web platform for data extraction, semantic analysis and structuring information to show relationships in visual ways
• 
  
•