Support different data file types

[h-bragg]

We should support different data file types: mainly: csv, json, and yaml.

When importing we should support:

• allow different sets of key/values between each 'row'
  • group rows based on their common sets of keys, to reduce time when inserting

Potential Solutions

1. Convert each into .sql and import using mysql command
2. Read the data into memory and write to the database within code

CSV

"id","country_code","country_code_iso2","country_code_iso3","name","decimal_point","thousands_separator","added","updated","deleted"
1,'AF','AF','AFG','Afghanistan','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
2,'AX','AX','ALA','Aland Islands','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
3,'AL','AL','ALB','Albania','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
4,'DZ','DZ','DZA','Algeria','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL
5,'AS','AS','ASM','American Samoa','.',',','2012-09-20 12:43:09','2015-10-07 11:05:58',NULL

Yaml

schema: the_schema
table: country

data:
  - id: 1
    country_code: AF
    country_code_iso2: AF
    country_code_iso3: AFG
    name: Afghanistan
    decimal_point: '.'
    thousands_separator: ','

  - id: 2
    country_code: AX
    country_code_iso2: AX
    country_code_iso3: ALA
    name: Aland Island
    decimal_point: '.'
    thousands_separator: ','

  - id: 3
    country_code: AL
    country_code_iso2: AL
    country_code_iso3: ALB
    name: Albania
    decimal_point: '.'
    thousands_separator: ','

  - id: 4
    country_code: DZ
    country_code_iso2: DZ
    country_code_iso3: DZA
    name: Algeria
    decimal_point: '.'
    thousands_separator: ','

  - id: 5
    country_code: AS
    country_code_iso2: AS
    country_code_iso3: ASM
    name: American Samoa
    decimal_point: '.'
    thousands_separator: ','

Json

{
  "schema": "the_schema",
  "table": "country",
  "data": [
    {
      "id": 1,
      "country_code": "AF",
      "country_code_iso2": "AF",
      "country_code_iso3": "AFG",
      "name": "Afghanistan",
      "decimal_point": ".",
      "thousands_separator": ","
    },
    {
      "id": 2,
      "country_code": "AX",
      "country_code_iso2": "AX",
      "country_code_iso3": "ALA",
      "name": "Aland Island",
      "decimal_point": ".",
      "thousands_separator": ","
    },
    {
      "id": 3,
      "country_code": "AL",
      "country_code_iso2": "AL",
      "country_code_iso3": "ALB",
      "name": "Albania",
      "decimal_point": ".",
      "thousands_separator": ","
    },
    {
      "id": 4,
      "country_code": "DZ",
      "country_code_iso2": "DZ",
      "country_code_iso3": "DZA",
      "name": "Algeria",
      "decimal_point": ".",
      "thousands_separator": ","
    },
    {
      "id": 5,
      "country_code": "AS",
      "country_code_iso2": "AS",
      "country_code_iso3": "ASM",
      "name": "American Samoa",
      "decimal_point": ".",
      "thousands_separator": ","
    }
  ]
}