ICS 32 Winter 2022, Notes and Examples: Web APIs

Background

In our previous example, we saw that the Python standard library makes it easy to download the contents of a web page. Given a URL, which specifies the location of the page you want to download, the urllib.request.urlopen() function hides nearly all of the details of the conversation between your Python program and the web server that will send the page back to it. You do have to know what HTTP is, and you do have to understand a few details of how it works (e.g., status codes, URLs, and perhaps headers), but you don't have the implement the HTTP protocol from scratch, you don't have to know the precise format of an HTTP request or response, and the function more or less "just works," as long as the page is available, your Internet connection is active, and you open the right URL.

However, writing a program to download a web page — the same page you might see if you visited a particular URL in a web browser — often isn't the right choice. A web page is suitable for display to a human user within a browser, but isn't necessarily a suitable result for a program to download. For example, you can visit YouTube and search for whatever you'd like — say, lakers clippers, if you like basketball — and you'll get back a result like this one:

https://www.youtube.com/results?search_query=lakers+clippers

But if you wanted to write a Python program that was capable of obtaining just the titles of the videos on that page, you'd have a surprisingly difficult time doing it. The problem is that the web page isn't what it first seems. Its structure looks simple and straightforward when displayed in a browser, but it's actually surprisingly complex. If you've never looked at the source code of a web page before, visit the YouTube link above, then right-click in an empty area of the page and select something like View Page Source — it'll be slightly different depending on what browser you use, but will likely be available in any browser running on a laptop or desktop machine, though perhaps not on a smartphone or other small-sized device. What you'll see is a combination of HTML, JavaScript, CSS, and other code used to make the page look the way it does. When your Python program downloads that page, this source code is what your program will get back; finding the important parts — this is sometimes called web scraping — is a challenge, one made worse by the fact that YouTube will make changes to its pages periodically that will render your scraping algorithm incompatible, and you'll be back to square one again.

Using web APIs instead of web pages

Fortunately, some sites provide an alternative interface, one that's intended to be used by programs instead of people. These are sometimes called APIs (application programming interfaces), because their goal is to provide an interface that application programs can use to access them. (Note that the term "API" is actually a pretty open-ended one, used not only to describe interfaces to web sites, but also any other kind of software library; here, though, when I say "API," I'm referring to a web API.)

YouTube is owned by the tech giant Google. In addition to providing YouTube as a web site that you can visit normally in a browser, Google also provides a set of APIs that can be used to access information about YouTube videos, channels, playlists, and users; to upload new videos to your YouTube account; to view an analysis of things like advertisement sales on your own videos; and so on. Some of what's provided requires payment, while other parts are free. While a fair amount of what they provide is only available if you authenticate (i.e., you've gone through a procedure to log into the API using your Google account, which is beyond the scope of what we're doing here), some of the more innocuous functionality — such as finding videos that match a search query, where the security concerns are lighter — is available to anyone, subject only to reasonable usage limits (i.e., how many times per day you can run a query).

Since our goal here is to display titles and descriptions of videos matching a YouTube search query, we'll need an API called the YouTube Data API. The YouTube Data API is a web API, meaning that a program interacts with it by sending an HTTP request — just like downloading a web page — and gets its answer back as an HTTP response. The URL specifies not only the operation we want to perform (e.g., search), but also the parameters for that operation (e.g., the search query). Meanwhile, the response is formatted in a way that's structured so that it will be easy for a program to parse and understand, in a format that's published (it's part of the API), so you can rest assured that it won't change when YouTube periodically changes the look of their web pages for human users.

Using web APIs effectively requires us to learn a handful of new techniques, though we'll find that most of them (certainly YouTube's) are implemented around standards, in ways that are common among most web APIs. These techniques are so common that Python's standard library provides implementations of all of them, so we don't have to deal with any of the low-level details; we just have to know what parts of the library we need, and we'll be in business. To find what we need in the library, we first have to learn a little bit about the standard techniques that are being used.

Standards for web APIs

While there are certainly differences between web APIs, and there are common techniques that we won't need here, the YouTube Data API uses a few standard techniques that we'll need to be familiar with.

URLs with query parameters

Since the very early days of the World Wide Web, it has been necessary to specify URLs that carry parameters, particularly on web sites whose content is generated dynamically. For many years, there has been a standard for URLs that include these kinds of parameters, which are called query parameters. A hypothetical example of a URL with query parameters follows:

http://www.blah.com/some/page?param1=value1&param2=value2&param3=value3

Before the ? character, this looks just like any other URL we've seen previously. The ? is special; it indicates that what follows it will be a sequence of query parameters. Each parameter is specified as a name and a value, with an = separating them; the parameters themselves are separated by & characters.

For example, on the day I originally wrote these notes, I opened Amazon.com in my browser and searched for u2 the joshua tree and here's the URL that my browser directed me to:

https://www.amazon.com/s/ref=nb_sb_noss_2/183-7159112-3775704?url=search-alias%3Daps&field-keywords=u2+the+joshua+tree

We can see that the parameters here are url (whose value is some kind of partial URL, though it's not clear exactly what it's being used for) and field-keywords (which appears to be my original query, with the spaces mysteriously replaced with + characters). We would have to know more about how Amazon's web site is implemented to know for sure what the query parameters mean, but we can sometimes suss out their meaning just by looking at them.

Still, the important thing to realize is that the URLs aren't necessarily stable over time. When I did that same search on Amazon again today, here's where I was directed instead:

https://www.amazon.com/s?k=u2+the+joshua+tree&ref=nb_sb_noss_2

The basic format — query parameters, in other words — is still intact, but the names of these parameters have changed and their order has been rearranged.

URL encoding

So, in general, the syntax for URLs with query parameters is quite simple. After the ?, parameters are separated by & characters, with the name and value of each parameters separated by = characters. But how do we specify a parameter whose value contains an = character? How about a parameter whose name includes a ? character? Or a parameter whose value includes &? Or a parameter whose value includes spaces (which turn out to be illegal in URLs, no matter where you want to put them)?

The answer lies in a technique called URL encoding, in which any character that's considered "special" (i.e., it has a meaning in the syntax, like = or &) is replaced by something else. There are two main rules:

Spaces are replaced with + characters.
Other special characters — anything that isn't legal in a URL, or has a special meaning (such as ?, &, or =) is replaced by a % followed by a "character code" (a numeric code, expressed in hexadecimal (i.e., base 16) corresponding to the character). For example, a ? is replaced with %3F.

A much more detailed explanation of this kind of encoding is here, if you're curious about it, though there's no need for you to understand it any better than I've explained it here. The basic idea is identical to the way that escape sequences are used in Python string literals, so we can include characters in them — like quotes or newlines — that are otherwise syntactically problematic.

We do need to be able to recognize whether something has been URL encoded — which we can pick up visually just by looking for % characters followed by two hexadecimal digits — but the details will be left to part of Python's standard library to implement for us.

JavaScript Object Notation (JSON)

The YouTube Data API, like many web APIs, returns its results in a common format called JavaScript Object Notation — usually referred to by its acronym, JSON. Because many web APIs are consumed by web pages, and the "guts" of many web pages are written in a programming language called JavaScript, JSON (which has its roots in JavaScript) turns out to be a natural format for web APIs to return. And JSON has become so common in web APIs and across the Internet, the Python standard library provides a module that can parse it, so we won't have to.

JSON is actually quite simple. An example follows:

{ "name": "Boo", "age": 13, "qualities": ["smart", "cute", "playful", "relentless", "perfect", "forever"] }

This is a JSON description of an object, which consists of a collection of attributes (much like Python's objects do). In this example, the attribute name has the string value "Boo", the attribute age has the numeric value 13, and the attribute qualities has a value that is an array (you can think of these like Python lists) containing the strings "smart", "cute", "playful", "relentless", "perfect", and "forever".

The details of JSON are described at json.org, but they're not much more complicated than that; the entire standard for JSON is four printed pages long, largely made up of large-sized diagrams.

Using the YouTube Data API

The YouTube Data API allows us to send a wide variety of different kinds of requests, but we'll focus on just one for this example. Our goal is to issue a search query — like we might do on YouTube's web page — and display the titles and descriptions of videos that match the request.

The appropriate request in the YouTube Data API is called search, which is described in detail here:

https://developers.google.com/youtube/v3/docs/search/list

Boiling down their documentation, we need to know only a few things, though you might want to look through and see what else is available, in case you want to experiment with aspects of it that we're not covering here.

A search request is issued using a URL that starts out like this:
- https://www.googleapis.com/youtube/v3/search
Following the word search in the URL, we add a ? character and then include a list of query parameters. There are lots of parameters we might like to pass, but these are the ones we need in order to accomplish our goal:
- key, which is an API key that uniquely identifies us as a user of Google's web APIs. We're required to set up an API key and associate it with our Google account if we want to use Google's APIs, though I've already done that for us (and I'll send out an API key in an email message). You can feel free to create your own, too, if you'd like, by logging into your Google account and then visiting the Google Developers Console.
- type, which specifies what we want to search for. In our case, we want to search for video, though we could also search for other things (channel or playlist) instead.
- part, which specifies what part of YouTube's information you're interested in obtaining. In our case, we want something called a snippet, which briefly describes a few aspects of each video.
- maxResults, which specifies how many results we want (at most).
- q, which specifies our search query.

So a complete URL might look like this:

https://www.googleapis.com/youtube/v3/search?key=YOUR_API_KEY&type=video&part=snippet&maxResults=10&q=lakers+clippers

Of course, we need to be sure that the parameters are URL encoded in case, for example, the search query includes special characters in it. Note the + sign between lakers and clippers in the example above; that's because the search query contained a space, but spaces are URL encoded to + signs. Note, also, that other special characters like ?, &, and = are not URL encoded, because we want them to have their natural meaning in a URL: separating the portions of the URL from each other. If, instead, we'd wanted a search query that included these characters in it, then we'd need to URL encode them.

We then issue the request — an HTTP request, just like we've seen before, albeit one that includes encryption (which is why the URL starts with https instead of http) — and the result is returned as a JSON object.

The only remaining trick is to understand what parts of the JSON response we're interested in. Construct a URL using the pattern above (replacing YOUR_API_KEY with a valid Google API key, such as the one I will email to you) and visit it in a browser, then take a look through the JSON response to see if you can understand what parts of it you might need, given our goal to display the titles and descriptions of the matching videos.

How the Python standard library can help

There are three basic operations we need here:

To URL encode a set of query parameters, so that it's safe to pass them to a web API.
To issue an HTTP request to the URL we built and get the HTTP response.
To parse the response, which will be a JSON object in the format documented in the YouTube Data API documentation.

URL encoding your query parameters

The module urllib.parse, which sounds like it knows how to parse URLs (i.e., to break them into their component parts), happens also to contain a function that knows how to URL encode query parameters. The easiest way to use it is to pass it a list of two-element tuples, and it will generate URL-encoded query parameters from it. For example:

>>> import urllib.parse
>>> urllib.parse.urlencode([('name', 'Boo'), ('age', 13), ('description', 'pekingese/perfect'), ('search', 'best food in heaven')])
'name=Boo&age=13&description=pekingese%2Fperfect&search=best+food+in+heaven'

Notice that not only did it separate the parameters' names and values with =, as well as the parameters from each other using &, but it also performed "percent encoding" of the values (e.g., the slash in the description parameter became %2F, while the space in the search parameter became +).

Issuing an HTTP request and getting the HTTP response

This is actually no different than what we did in the previous example. We create an object of the urllib.request.Request class with the complete URL (including the URL-encoded parameters), then pass it to the urllib.request.urlopen function, just like we did before.

There are some web APIs that would require some additional adjustments to this technique. For example, it's not uncommon for web APIs to require us to use HTTP POST requests (which we've not seen in this course) instead of HTTP GET requests, or to require us to set the values of certain headers in our request. If we needed to do that, we'd pass additional parameters to the urllib.request.Request constructor, meeting whatever the API's particular requirements are:

>>> import urllib.request
>>> request = urllib.request.Request(url, headers = { 'Content-Type': 'application/x-www-form-urlencoded' }, method = 'POST')

Fortunately for us, in this case, we don't need to go down that road; the right way to ask the YouTube Data API this particular question is to use an HTTP GET request, and it doesn't care about what headers we send, but you might find that Project #3 uses APIs with different requirements than YouTube's.

Parsing the JSON response

Once we've retrieved a bytes object containing the content of our HTTP response, we'll have the response text in JSON format. However, it won't be a string; it'll be a bytes object. We could turn it into a string — we've seen already how to do that — but that doesn't leave us in a much better place, as it turns out. A string in JSON format isn't a very convenient thing to have; to process it, we'll need to start searching for curly braces, double quotes, commas, colons, brackets, etc., in order to build an understanding of what's there and act on it. If, fundamentally, a JSON description of an object boils down to "a set of attributes with unique names, each with its own value", then it might be nice if we could take a string in JSON format and turn it into something we could use more conveniently in a Python program.

That would be a tall task, except for one important bit of good news: The Python standard library includes a module called json that knows how to do this already! It can turn JSON into Python dictionaries (and also Python dictionaries back into JSON again, though we won't need that here). For example, if we have a string variable containing JSON describing some object, the function json.loads — where loads is short for "load from string" — can convert the string to a dictionary for us.

>>> import json
>>> x = '{ "name": "Boo", "age": 13, "qualities": ["intelligent", "cute", "playful", "perfect"], "forever": true }'
>>> obj = json.loads(x)
>>> obj['name']
'Boo'
>>> obj['age']
12
>>> for quality in obj['qualities']:
        print(quality)

intelligent
cute
playful
perfect

We can do the same thing with a file object (or something that you can read from like a file object, such as the HTTP response you get back from urllib.request.urlopen); the only difference is that we call json.load instead of json.loads, passing it the file object. It reads the contents of the file and converts it. So, for example, if you pass the HTTP response to json.load, it will read its contents and convert them to a Python object, provided that the contents are valid JSON.

Now that's handy, isn't it?

Web APIs and security

Previously, we've discussed the importance of information security, and how it's a consideration in everything we do (and everything our programs do) online. Of course, web APIs are, by definition, online, so we might reasonably expect for there to be security implications involved in their use, both from the perspective of the provider of the API and also for its users.

Security considerations for the provider of an API

First of all, the provider of a web API will have some of the same concerns we talked about in the broader context of network protocols.

A web API can't immediately know the identity of who connected to it. This is one reason why Google requires an API key to make requests from the YouTube Data API; that API key is associated with a Google account. By generating the API key fairly randomly, and by making it long and difficult to guess, they've set up a situation where they have a pretty good idea who's connecting to them.
A web API can't know the intentions of whoever connected to it, meaning that it will need to be vigilant about interpreting the information that it's sent. If the HTTP request is written incorrectly in some way, better to refuse service (or even close the connection) than to try to recover from it, because there's at least some chance the request was sent incorrectly for malicious reasons rather than just being ham-fisted.

Additionally, a web API needs to protect sensitive information that shouldn't be visible to those who shouldn't obtain it. This is the other reason why Google requires an API key, and also why some operations require an additional authentication to be performed (wherein a user or program logs into the corresponding Google account with the appropriate credentials). If someone has an API key belonging to a Google account and the credentials to log into that account, there's a pretty good chance they're who they claim to be. Meanwhile, the more stringent the Google account login becomes — for example, if they require you to receive and respond to a text message on your phone, additionally — the more stringent the web API's security becomes, correspondingly.

An API key provides one more weapon that can be used to protect security, as well. The API key can be revoked, which is to say that Google can stop honoring it at any time. So, for example, if it's used in ways that are seen as malicious, if it's used too often or from too many different places, Google can simply consider it invalid going forward.

Security considerations for the client of an API

While the provider of a web API usually has more to protect, since they're the holders of information belonging to a large number of people, it's important to note that there are security considerations for the client of a web API, as well.

As we've discussed previously, the client of a web API can't immediately know for sure who it connected to. For this reason, we should prefer to use HTTPS whenever it's an option; as we've seen, HTTPS isn't just about encryption, but also about establish trust in who we've connected to. If we connect to a web API belonging to Google, we want to trust that it's really Google. HTTPS helps us to have that trust.

This is particularly important because web APIs aren't just about querying information that already exists; they're also used by clients to provide new information. For example, Google provides APIs that can used to manipulate users' Google-based email, their calendars, their online documents, their cloud-based computing resources, and so on. If we ask Google to send an email on our behalf, we'd like to know that it's Google we've asked — and this might be doubly important if we're talking about a web API belonging to a bank, a government agency, or something of that nature.