Introduction

Part 1 and Part 2 of this series showed that the web is a rich source of open data in formats such as CSV, JSON and XML. But how do we get data from these sources into a program? How do we deal with the formatting and extract what we need from the data? This final part of the series shows how these goals can be achieved using the Python programming language.

Note that Python 3 is used here, rather than the older (and increasingly obsolete) Python 2. Note also that other languages besides Python can do these things, with comparable ease in some cases. Python is used here because it is a good choice for such tasks and because it also happens to be my favourite programming language!

Reading Data From The Web

This can be done using the urlopen function from the urllib.request module in Python’s standard library (see the official module documentation for full details). To make a basic HTTP GET request, which is typically what you’ll need to access data, this function requires just one argument: the URL of the resource being accessed. It returns an HTTPResponse object from which we can read the data.

Calling the read method of the HTTPResponse object returns the data as a Python bytes object - essentially a string of bytes. No assumptions are made about the nature of the data. Thus, if you were expecting a text-based format such as CSV, XML or JSON, you must do the translation of bytes to text yourself by calling the decode method on the string of bytes. The decode method accepts an encoding scheme as an optional argument, defaulting to UTF-8 if one isn’t specified. This default will be suitable in most cases.

All of this leads to code like the following example, which acquires CSV data for earthquakes from the USGS Earthquake Hazards Program website discussed in Part 2:

from urllib.request import urlopen

# Construct feed URL for M4.5+ quakes in the past 7 days

base = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/"
feed_url = base + "4.5_week.csv"

# Open URL and read text from it

source = urlopen(feed_url)
text = source.read().decode()

Handling CSV

You now have code that reads one of the earthquake CSV data feeds into your program as a single string of text. Imagine that you wish to process this dataset in order to find the mean and standard deviation of earthquake depths. An examination of the data (e.g., in a spreadsheet application) will tell you that the fourth column holds the depth values you need.

Screenshot of LibreOffice Calc showing earthquake CSV data

Extraction of the depth values can be done using the csv module from Python’s standard library (see the official module documentation for full details). If you are running Python 3.4 or newer, the standard library also has a module called statistics that makes computation of the mean and standard deviation trivial.

One approach is to create a reader object that can scan the lines in the text string that contains all of the data. Calling the splitlines method on the string will return these lines in a list, which is then used to create the reader object. Iterating over the reader object will give you first the column headings (if present), then each record from the dataset. The record will be a list of string values, since the reader object doesn’t know how the data should be interpreted. Depths will therefore need to be converted into floating-point values before being collected into a list. This list of values can then be passed to functions from the statistics module that compute mean and standard deviation.

Suitable code to do all this is shown below. (The code to read the data has been omitted but you should imagine it to be at the location of the ... characters.)

import csv
import statistics

...

# Create reader for the dataset

reader = csv.reader(text.splitlines())

# Read column headings (which are not used here)

headings = next(reader)

# Fetch each record and collect depths into a list

depths = []
for record in reader:
    # Depth is fourth value, at index 3
    # Value is a string and must be converted to a float
    depth = float(record[3])
    depths.append(depth)

# Compute mean & standard deviation of depths

mean = statistics.mean(depths)
stdev = statistics.stdev(depths, mean)

print("Mean    =", mean)
print("Std dev =", stdev)

An alternative and slightly more user-friendly approach is to use a DictReader object. Unlike the normal reader object provided by the csv module, which gives you each record as a list of strings, a DictReader object will give you each record as a dictionary, in which the keys are the column headings. Here’s an example generated from the earthquake data feed:

{'depth': '11.4', 'dmin': '2.379', 'time': '2014-04-24T03:10:12.880Z', 'updated': '2014-04-24T09:31:17.990Z', 'net': 'us', 'id': 'usb000px6r', 'nst': '', 'rms': '1.18', 'mag': '6.6', 'magType': 'mww', 'place': '94km S of Port Hardy, Canada', 'latitude': '49.8459', 'type': 'earthquake', 'longitude': '-127.444', 'gap': '41'}

If DictReader is used then the code needed to build a list of earthquake depths will change to something like this:

reader = csv.DictReader(text.splitlines())

depths = []
for record in reader:
    depth = float(record["depth"])
    depths.append(depth)

Note how “depth” is used to look up the depth value instead of an integer index. This makes the code a little easier to understand.

The code can be reduced in size a little further by using a list comprehension:

reader = csv.DictReader(text.splitlines())

depths = [ float(record["depth"]) for record in reader ]

Handling JSON

This can be done using the json module from Python’s standard library (see the official module documentation for full details). Let us consider how this module can be used to find artists who have been played more than ten times in the past week on BBC radio stations, using the JSON data feed that was mentioned in Part 2.

The first step is to read data from the feed into a single string of text, as discussed above. This string can then be passed to the loads function from the json module, which deserializes the JSON dataset, returning it as a dictionary. Here is the code that you need:

import json
from urllib.request import urlopen

feed_url = "http://www.bbc.co.uk/programmes/music/artists/charts.json"

# Open URL and read text from it

source = urlopen(feed_url)
text = source.read().decode()

# Deserialize the JSON data contained in the text

data = json.loads(text)

The dictionary will have the following format. (Note: this is real data, but records for only the first three artists are shown here.)

{
  "artists_chart" : {
    "artists" : [
      {
        "plays" : 17,
        "name" : "Drake",
        "previous_plays" : 15,
        "gid" : "9fff2f8a-21e6-47de-a2b8-7f449929d43f"
      },
      {
        "plays" : 16,
        "name" : "Nas",
        "previous_plays" : 4,
        "gid" : "cfbc0924-0035-4d6c-8197-f024653af823"
      },
      {
        "plays" : 15,
        "name" : "David Bowie",
        "previous_plays" : 12,
        "gid" : "5441c29d-3602-4898-b1a1-b77fa23b8e50"
      },
      ...
    ],
    "period" : "Past 7 days",
    "end" : "2014-04-24",
    "start" : "2014-04-17"
  }
}

You can see from this example that keys of “artists_chart” and “artists” are required in order to access the list of artist details. Each element of this list is itself a dictionary in which artist name and play count can be accessed using keys called “name” and “plays”, respectively. This leads us to the following code:

...

artists = data["artists_chart"]["artists"]

for artist in artists:
    if artist["plays"] > 10:
        print(artist["name"], artist["plays"])

Simplifying Things With Requests

If you are willing and able to install third-party Python packages on your system, Kenneth Reitz’s excellent Requests library can be used to simplify things considerably. Requests has a much cleaner API for issuing HTTP GET requests like those used in the preceding examples. It also greatly simplifies POST requests, file uploading and authentication. It even has built-in JSON deserialization capabilities.

Using Requests, the first six lines of code in the JSON example can be replaced with four simpler lines:

import requests

feed_url = "http://www.bbc.co.uk/programmes/music/artists/charts.json"

response = requests.get(feed_url)
data = response.json()

...

That’s All Folks!

The source code for this article’s examples is available in a GitHub repository.

I hope you’ve found this series of articles useful; feel free to get in touch if you have questions or comments!



Comments

comments powered by Disqus