How to Use Youtube JavaScript API to Search for Videos in Node.js

It goes without saying that nowadays, Youtube is an amazing resource for any kind of information. From educational content to entertainment, from documentaries to live shows, Youtube has it all. But with this kind of scale, it is becoming extremely challenging to discover truly great content. It has some pretty decent filtering options, but this might not be enough for a sophisticated user. With all the algorithmic power of Youtube that is at your disposal, try to find the best programming talk from 2010 to 2012, sorted by most comments. It is not as easy as you would think.

That's why I have built Awesome Talks as a weekend project. It simply is a collection of JavaScript conference talks that can be sorted in a way that would allow you to find the most relevant and interesting talks according to your needs. I don't have any grand ambitions for this project. It was simply a means for me to discover awesome talks that relate to JavaScript. But working on this project was an educational experience in interacting with the Youtube JavaScript API. Here is a beginner-friendly tutorial to share my learnings.

In this tutorial, I will show you how to write a program that will search Youtube for public videos using JavaScript and Node.js and save that information on your computer. Here we go!

Getting a Youtube API Key

Getting an API key is my least favorite step when interacting with any API Service since the steps involved can vary a lot depending on the platform we use.

API Keys are used as authorization credentials in Youtube. Authorization gives us access to the resources on Youtube through the API.

There are two different kinds of credentials that we might need depending on the kind of service we are building on top of the Youtube API.

OAuth 2.0
API keys

If we wanted to build a program that is going to perform actions on behalf of a user we would need an OAuth 2.0 token. We won't be building such a program so API keys will suffice for our purposes.

We can first visit [https://console.developers.google.com/apis/credentials](https://console.developers.google.com/apis/credentials) and create a project by clicking on Create Project.

Google will prompt us to enter a project name. We can enter whatever you want. We don't have to choose a Location either. We then click Create.

Clicking Create Credentials creates the API key. We will see a dialog box that shows the API key that is created for us. We should treat this key just like any other secret, like a password. Please make a note of this key since we will be using it in our program.

Enabling Access to the Youtube Data API

Google has a lot of APIs that are available. The API that we are interested in is the Youtube Data API. We can go to the Library section through the sidebar and search for the Youtube Data API.

Click on the Enable button to activate this API for our project. We will be using the API key that we have created to access this service. This is pretty much it. We can now start coding!

Creating a new Node.js Project

Create a folder for the Node.js project and run npm init -y in there to initialize an empty package.json file. We will be using the package.json file to install libraries for this project.

mkdir project-youtube-api
cd project-youtube-api
npm init -y

We can now create two files in this folder. A main.js where we will write the functions that we will use to interact with the Youtube API and a config.js we will use to store the Youtube API key. We can create these files through the graphical interface of our operating system or the command line.

touch main.js
touch config.js

Inside the config.js, we will just have a variable to hold the API key and export that variable to be used from main.js.

const YOUTUBE_API_KEY = '';

module.exports = {
    YOUTUBE_API_KEY,
}

Make sure to assign your API key to the YOUTUBE_API_KEY variable. Next, we will make use of this key from inside the main.js.

const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

function main() {
  console.log("Ready to get Youtube data!");
}

main();

This file currently loads (imports) the YOUTUBE_API_KEY from the config.js file in the same folder. It then checks to see if the YOUTUBE_API_KEY exists. It throws an error if it doesn't.

If the YOUTUBE_API_KEY exists, we proceed to execute the main function. What we would like to do is to use this API key to authenticate with the Youtube API. We will interact with the Youtube API by using the node-fetch library.

Interacting with the Youtube API using Fetch

We will install the node-fetch package to be able to interact with the Youtube API.

npm install --save node-fetch@^2.6.1

We will make use of this library inside the main.js file.

const fetch = require("node-fetch");
const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

async function main(query) {
  console.log("Ready to get Youtube data!");
  const url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

main("JavaScript");

We are importing the node-fetch library on the first line with the require function.

const fetch = require("node-fetch");

We have changed the main function to be an async function since we would like to use the async-await pattern inside this function. Interacting with an API is an asynchronous process, and async-await will allow us to wait for the request to the API to resolve before proceeding to the next line.

This is the endpoint we are using to make a search for a given query using the Youtube API.

const url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;

There are two parameters in this URL right now. The YOUTUBE_API_KEY and the query to be searched for. When executed, this query returns an object with a field called items that have the results that we want. Let's take a look at the first item object to see what kind of properties it has.

console.log(data.items[0]);

The result would look something like this (the particular item might be different for you!)

{
  kind: 'youtube#searchResult',
  etag: 'HXpfXYuctt1KbbEEnnlYDhEiiVM',
  id: { kind: 'youtube#video', videoId: 'PkZNo7MFNFg' },
  snippet: {
    publishedAt: '2018-12-10T14:13:40Z',
    channelId: 'UC8butISFwT-Wl7EV0hUK0BQ',
    title: 'Learn JavaScript - Full Course for Beginners',
    description: 'This complete 134-part JavaScript tutorial for beginners will teach you everything you need to know to get started with the JavaScript programming language.',
    thumbnails: { default: [Object], medium: [Object], high: [Object] },
    channelTitle: 'freeCodeCamp.org',
    liveBroadcastContent: 'none',
    publishTime: '2018-12-10T14:13:40Z'
  }
}

This object has plenty of data that we can use for various purposes. Our script works, but it has a couple of problems. The items array is only returning 5 results by default. According to the totalResults field inside the pageInfo field of the returned object, we should have a million results instead! Let's address these issues.

Changing the Results Per Page in Youtube API

Solving the first issue is easy. Inside the pageInfo field, you can see a reference to a value called resultsPerPage, which is equivalent to 5 at the moment. This is the reason why we are getting 5 results. We need to add a parameter to the url variable to adjust the number of results per page. The API can't return a million results at once. It is paginated. This means the results are divided into separate sections (pages). We can make use of the maxResults parameter to change the number of results we are getting.

async function main(query, resultsPerPage) {
  console.log("Ready to get Youtube data!");
  let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
  if (resultsPerPage) {
    url = `${url}&maxResults=${resultsPerPage}`;
  }

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

main("JavaScript", 25);

We have added a new parameter to the main function called resultsPerPage. We have also made the url variable use the let keyword so we can update its value. Additionally, we have added a conditional to check to see if a resultsPerPage value exists. If it does, we provide it as the value of the maxResults parameter of the URL.

let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
  if (resultsPerPage) {
    url = `${url}&maxResults=${resultsPerPage}`;
  }

This allows us to increase the number of results we are getting per page. But you will notice that the results are capped at 50 items. The rest of the results would be on the next page. Let's take a look at how to get those other results.

Working with Pagination in Youtube API

Notice how the returned object has a field called nextPageToken.

nextPageToken: 'CDIQAA',

The value of that field is an identifier for where we are at in the pagination. You can think of it as a page number. We can use that value in our API call to get the next page of results.

const fetch = require("node-fetch");
const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

async function getYoutubeResults(query, resultsPerPage, pageToken) {
  console.log("Ready to get Youtube data!");
  let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
  if (resultsPerPage) {
    url = `${url}&maxResults=${resultsPerPage}`;
  }
  if (pageToken) {
    url = `${url}&pageToken=${pageToken}`;
  }

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

async function main() {
  const data = await getYoutubeResults("JavaScript", 25);
  await getYoutubeResults("JavaScript", 25, data.nextPageToken);
}

main();

Here, we have changed the structure of our code a bit. I have renamed the main function to be getYoutubeResults. The sole responsibility of this function is to query the Youtube API using the given arguments. The new main function contains the logic for making specific queries using the getYoutubeResults function.

We have also made the getYoutubeResults function to accept a pageToken parameter as an argument. If this argument is provided, this function will use it when constructing the url variable.

if (pageToken) {
	 url = `${url}&pageToken=${pageToken}`;
 }

Now we can run this function, get the data that it returns, and use the nextPageToken field to rerun the function to get the next page of results!

async function main() {
  const data = await getYoutubeResults("JavaScript", 25);
  await getYoutubeResults("JavaScript", 25, data.nextPageToken);
}

This definitely works but feels a bit suboptimal. Surely there must be other ways to go to the next page than manually calling the getYoutubeResults over and over again.

We should have a way to collect these search results before we look at the pagination. Let's implement that first.

Collecting the Youtube Search Results

We will collect the items that are returned from the getYoutubeResults inside an array called videoData.

async function main() {
  const videoData = [];
  const data = await getYoutubeResults("JavaScript", 25);
  videoData.push(...data.items);

  console.log(videoData);
  console.log(`There are ${videoData.length} items in videoData`);

We are using the spread (...) operator to unpack the data.items array, and then we are pushing all those items inside the videoData array. We are then logging the videoData and the number of items in that array to the screen.

Now we have a mechanism to collect the results, let's build a loop that will collect data from all the pages.

async function main() {
  const videoData = [];

  let totalPages = 10;
  let nextPageToken = undefined;

  for (let counter = 0; counter < totalPages; counter = counter + 1) {
    const data = await getYoutubeResults("JavaScript", 50, nextPageToken);
    videoData.push(...data.items);
    nextPageToken = data.nextPageToken;
  }

  console.log(videoData);
  console.log(`There are ${videoData.length} items in videoData`);
}

We introduced two new variables called totalPages and nextPageToken. totalPages is the number of pages that we would like to collect data from. nextPageToken is a variable to store the nextPageToken that is returned from the current page. We are using a for loop to loop through the pages.

for (let counter = 0; counter < totalPages; counter = counter + 1) {
  const data = await getYoutubeResults("JavaScript", 50, nextPageToken);
  videoData.push(...data.items);
  nextPageToken = data.nextPageToken;
}

The loop currently collects data from 10 pages. We would see that we have 500 items stored in the videoData variable at the end of the loop. The last step would be to save this data on our computer.

Storing the Collected Data in a JSON file

We can save the collected data in a file. We generally use JSON data format to save JavaScript data. This format is human-readable, so you can just open the file and see how it looks like. JSON is also a file format that is natively supported in JavaScript. It is straightforward to read and write JSON files from JavaScript files. To be able to write data to the file system, we will first import the fs module.

const fs = require("fs");

The we will convert the videoData into JSON format using the JSON.stringify function.

const videoDataJSON = JSON.stringify(videoData);

We can then save this data into a file. Here, we are saving the file into a file called data.json in the same directory.

fs.writeFileSync("./data.json", videoDataJSON);

If we are to look at the saved data, we will notice it is actually not very readable. We can format this file a bit better by passing additional arguments into the JSON.stringify function.

const videoDataJSON = JSON.stringify(videoData, null, 2);

This should result in a much more readable code. Here is what the final result looks like.

const fs = require("fs");
const fetch = require("node-fetch");
const { YOUTUBE_API_KEY } = require("./config");

if (!YOUTUBE_API_KEY) {
  throw new Error("No API key is provided");
}

async function getYoutubeResults(query, resultsPerPage, pageToken) {
  console.log("Ready to get Youtube data!");
  let url = `https://www.googleapis.com/youtube/v3/search?key=${YOUTUBE_API_KEY}&type=video&part=snippet&q=${query}`;
  if (resultsPerPage) {
    url = `${url}&maxResults=${resultsPerPage}`;
  }
  if (pageToken) {
    url = `${url}&pageToken=${pageToken}`;
  }

  const response = await fetch(url);
  const data = await response.json();
  console.log(data);

  return data;
}

async function main() {
  const videoData = [];

  let totalPages = 10;
  let nextPageToken = undefined;

  for (let counter = 0; counter < totalPages; counter = counter + 1) {
    const data = await getYoutubeResults("JavaScript", 50, nextPageToken);
    videoData.push(...data.items);
    nextPageToken = data.nextPageToken;
  }

  const videoDataJSON = JSON.stringify(videoData, null, 2);
  fs.writeFileSync("./data.json", videoDataJSON);
}

main();

Conclusion

This is it for now! There is a bit more to cover about the Youtube Data API, such as getting videos from the channels, and I will be covering that in another post. As always, be mindful of the usage quotas and limitations of the API that you are using! Feel free to connect with me at https://twitter.com/inspiratory and ask any questions you might have.

Resources

Here are some resources if you wanted to get more information about things that we have covered here:

Documentation for the Youtube Data API: https://developers.google.com/youtube/v3
Github Page for the Node-Fetch library: https://www.npmjs.com/package/node-fetch
How fetch works in general: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API