Close

## Structured Data¶

### Structured data¶

CSV files can only model data where each record has several fields, and each field is a simple datatype, a string or number.

We often want to store data which is more complicated than this, with nested structures of lists and dictionaries. Structured data formats like JSON, YAML, and XML are designed for this.

### JSON¶

JSON is a very common open-standard data format that is used to store structured data in a human-readable way.

This allows us to represent data which is combinations of lists and dictionaries as a text file which looks a bit like a Javascript (or Python) data literal.

In [1]:
import json


Any nested group of dictionaries and lists can be saved:

In [2]:
mydata = {'key': ['value1', 'value2'],
'key2': {'key4':'value3'}}

In [3]:
json.dumps(mydata)

Out[3]:
'{"key": ["value1", "value2"], "key2": {"key4": "value3"}}'

If you would like a more readable output, you can use the indent argument.

In [4]:
print(json.dumps(mydata, indent=4))

{
"key": [
"value1",
"value2"
],
"key2": {
"key4": "value3"
}
}


In [5]:
%%writefile myfile.json
{
"somekey": ["a list", "with values"]
}

Overwriting myfile.json

In [6]:
with open('myfile.json', 'r') as f:

In [7]:
mydataasstring

Out[7]:
'{\n    "somekey": ["a list", "with values"]\n}\n'
In [8]:
mydata = json.loads(mydataasstring)

In [9]:
mydata['somekey']

Out[9]:
['a list', 'with values']

This is a very nice solution for loading and saving Python data structures.

It's a very common way of transferring data on the internet, and of saving datasets to disk.

There's good support in most languages, so it's a nice inter-language file interchange format.

### YAML¶

YAML is a very similar data format to JSON, with some nice additions:

• You don't need to quote strings if they don't have funny characters in
• You can have comment lines, beginning with a #
• You can write dictionaries without the curly brackets: it just notices the colons.
• You can write lists like this:
In [10]:
%%writefile myfile.yaml
somekey:
- a list # Look, this is a list
- with values

Overwriting myfile.yaml

In [11]:
import yaml  # This may need installed as pyyaml

In [12]:
mydata = yaml.load(open('myfile.yaml'))
print(mydata)

{'somekey': ['a list', 'with values']}


YAML is a popular format for ad-hoc data files, but the library doesn't ship with default Python (though it is part of Anaconda and Canopy), so some people still prefer JSON for its universality.

Because YAML gives the option of serialising a list either as newlines with dashes, or with square brackets, you can control this choice:

In [13]:
print(yaml.safe_dump(mydata))

somekey: [a list, with values]


In [14]:
print(yaml.safe_dump(mydata, default_flow_style=False))

somekey:
- a list
- with values



default_flow_style=False uses a "block style" (rather than an "inline" or "flow style") to delineate data structures. See the YAML docs for more details.

### XML¶

Supplementary material: XML is another popular choice when saving nested data structures. It's very careful, but verbose. If your field uses XML data, you'll need to learn a python XML parser (there are a few), and about how XML works.