What problem are you solving?:
Extraction, transformation and loading of data: “Screen scraping” of web pages, gathering of data from other sources, cleansing and transformation of data.
Describe your idea:
Library should serve as a base for other transparency applications that would like to deal with data extraction from other sources, data transformations and data loading (into application database).
Provide a library and set of tools for:
* automating and scheduling data management tasks
* parallel managed screen-scraping
* remote control and configuration through database
* make development of data/information extractions easier
* tools for data transformations and data cleansing (to make further analysis easier)
* parallel downloading (1M pages < 2hours)
Objective is to:
* simplification of extraction development
* easy task management and configuration
* immediate and easy to get feedback about success/failure of tasks
* use modular approach for tasks – each task is independent module
Currently used in Datacamp application – transparency application for sharing government data in Slovakia.
What country will this operate in?: Slovakia
Who are you?:
Analyst with background in bsiness intelligence, data warehouse and knowledge management.
Currently working on Datacamp – Fair-play alliance application to publish government data in Slovakia in “Web 2.0″ way.