How to Use the Keywords Everywhere API

Updated on

This tutorial shows how to get keyword data from the Keywords Everywhere API using Python. This script is a bit more robust than the example given in the Keywords Everywhere documentation in that it lets you load an unlimited number of keywords from a text file and then saves the output to JSON files. Once the keyword data is saved on your computer you can analyze it with other tools.

Set Up a Python Virtual Environment

A virtual environment is a set of installed Python packages that is isolated from other virtual environments and your computer’s main Python installation. A virtual environment (or venv) lets you install different versions of dependencies for each project that you’re working on without them conflicting with each other.

It’s best practice to create a separate virtual environment for every Python project you create.

First, create a new directory for your keyword data fetcher and run this terminal command inside of it:

python -m venv .venv

That will create a directory for your virtual environment named .venv. The leading period means that it’s a hidden directory, but you can view it when listing files in the terminal by using the -a flag with ls:

ls -a

After you create the venv, the next step is to activate it. Run this command:

source .venv/bin/activate

After using the source command, you should see the text (.venv) prefixed to your terminal prompt. That means that the venv is active, and you’re working with a set of Python packages that is isolated from your system’s Python.

Install the Required Packages

Run these two commands to install the required dependencies:

pip install requests
pip install python-dotenv

The requests package is for making the HTTP request to Keywords Everywhere’s API, and the python-dotenv package is for loading your API key in a secure way.

Create a .env File

Create a file name .env and put the following content in it, replacing 12345 with the API key that you got from your Keywords Everywhere settings:

KEYWORDS_EVERYWHERE_API_KEY=12345

Also create a file named .gitignore and add this line to it:

.env

If you already have a .gitignore file, just append that line to the file, if it doesn’t already exist. That will ensure that you don’t accidentally publish your secret API key on Github or some other site.

Writing the Python Code

We’ll start by writing a bunch of functions and then put everything together at the end.

Create a new file named script.py and add this code to it:

from datetime import datetime
import json
import logging
import os
import sys
from time import sleep
from typing import Any

from dotenv import load_dotenv
import requests

load_dotenv()

The first seven lines import things from the Python standard library that we’ll use momentarily.

The next two lines import things from the packages that were downloaded with pip.

The last line calls a function that loads the variable from the .env file that was created in the last step.

Creating a Logging Feature

If fetching a lot of keywords, it’s a good idea to log what the script is doing. If something goes wrong, you can inspect the log file to see what happened.

Here’s the code for the logger:

def create_logger(filename: str = "logs.log"):
    """
    This creates a logger that logs to both stdout and a file.

    If something goes wrong with the program, you can inspect the log file to
    see what keywords were missed.
    """
    logger = logging.getLogger(__name__)
    logger.setLevel(logging.DEBUG)
    formatter = logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")

    stdout_handler = logging.StreamHandler(sys.stdout)
    stdout_handler.setLevel(logging.DEBUG)
    stdout_handler.setFormatter(formatter)

    file_handler = logging.FileHandler(filename)
    file_handler.setLevel(logging.DEBUG)
    file_handler.setFormatter(formatter)

    logger.addHandler(file_handler)
    logger.addHandler(stdout_handler)
    return logger


log = create_logger()
log.debug("Logger created")

The details are beyond the scope of this tutorial, but the function will create a log file at logs.log.

The log.debug method is working something like a print function, but it’s writing to a file and has some other features. You can use them in almost the same way.

Loading the API Key

The next step loads the API key. The load_dotenv function in a previous step loaded the API key from the .env file and made it available in the environment. This line loads it into the program:

API_KEY = os.getenv("KEYWORDS_EVERYWHERE_API_KEY")

Loading Keywords from a File

Instead of putting the list of keywords directly in the script, it’s better to load it from a separate file.

Here’s a function that will do that:

def load_keywords(filename: str) -> list[str]:
    """
    Load keywords from a text file.

    Put one search term on each line.
    """
    with open(filename, "r") as f:
        kws = [line.strip() for line in f.readlines() if line.strip()]
        log.debug(f"Loaded {len(kws)} keywords from {filename}")
        return kws

If you haven’t used mypy before, it’s a way to specify types, which can help reduce errors.

The function could have been written like this:

def load_keywords(filename):

but it’s a little safer to specify that filename is a string and that the function returns a list of strings (the keywords):

def load_keywords(filename: str) -> list[str]:

It might seem like a small difference when looking at one function, but as soon as you have to refactor something, the types make it much easier to find bugs.

Splitting Keywords into Chunks

The Keywords Everywhere API only accepts 100 keywords at a time, so we need a function to split the list of keywords that were loaded from the file into chunks of 100:

def split_keywords(keywords: list[str]) -> list[list[str]]:
    """
    Split a list of keywords into chunks of 100 keywords each.

    The API only accepts 100 keywords at a time.
    """
    return [keywords[i : i + 100] for i in range(0, len(keywords), 100)]

Saving the Keywords

Here are two functions that help with saving the keywords.

The create_filename function creates a unique filename based on the date and time. That way each filename is unique and won’t be overwritten.

def create_filename() -> str:
    """
    Create a unique filename based on the current date and time.

    This prevents files from overwriting each other.

    Example filename: `2023-05-14T00:12:45.736981.json`
    """
    current_date = datetime.now()
    return current_date.strftime("%Y-%m-%dT%H:%M:%S.%f%z") + ".json"

The write_to_file function saves the data from the API in the file as JSON:

def write_to_file(data: dict[str, Any]) -> None:
    """
    Write a dictionary of data to a JSON file with a unique filename.
    """
    output_path = os.path.join("output", create_filename())
    with open(output_path, "w") as f:
        json.dump(data, f, indent=4)
        log.debug(f"Wrote data to file {f.name}")

The function could have been written like this:

def write_to_file(data):

but I specified that data has to be a dict where the keys are strings and the values can be Any type, and that the function doesn’t return anything:

def write_to_file(data: dict[str, Any]) -> None:

Fetching the Data

The last helper function fetches the data from the API.

The only required argument is a list of keywords. You can optionally pass in country, currency, and data_source if you want, but it defaults to US values.

def get_keywords_data(
    keywords: list[str],
    country: str = "us",
    currency: str = "USD",
    data_source: str = "gkp",
) -> dict[str, Any]:
    """
    Make a request to KeywordsEverywhere's API.
    """
    num_keywords = len(keywords)
    log.debug(f"Preparing to fetch data for {num_keywords} keywords")
    if num_keywords > 100:
        log.error(
            f"You can only search for 100 keywords at a time. Received {num_keywords}"
        )
        raise ValueError(
            f"You can only search for 100 keywords at a time. Received {num_keywords}"
        )

    payload = {
        "country": country,
        "currency": currency,
        "dataSource": data_source,
        "kw[]": keywords,
    }
    headers = {
        "Accept": "application/json",
        "Authorization": f"Bearer {API_KEY}",
    }

    log.debug(f"Requesting data for {num_keywords} keywords")

    response = requests.post(
        "https://api.keywordseverywhere.com/v1/get_keyword_data",
        data=payload,
        headers=headers,
    )

    if response.status_code == 200:
        log.debug(response.content.decode("utf-8"))
        return response.json()
    else:
        log.error(response.content.decode("utf-8"))
        raise Exception("An error occurred. Check the log file for more details.")

Putting It All Together

The main function wires up all the other functions. It fetches 100 keywords at a time, pausing for 1 second between requests.

def main():
    keywords = load_keywords("keywords.txt")
    batches = split_keywords(keywords)
    num_batches = len(batches)
    delay = 1  # second

    for idx, batch in enumerate(batches):
        log.info(f"Processing batch {idx + 1} of {num_batches}")

        result = get_keywords_data(batch)
        write_to_file(result)

        log.info(f"sleeping for {delay} seconds")
        sleep(delay)

    log.info("Done. Check the output directory for the JSON file(s).")


# Run the function.
main()

All the Code

Here’s the full script in one place:

from datetime import datetime
import json
import logging
import os
import sys
from time import sleep
from typing import Any

from dotenv import load_dotenv
import requests

load_dotenv()


def create_logger(filename: str = "logs.log"):
    """
    This creates a logger that logs to both stdout and a file.

    If something goes wrong with the program, you can inspect the log file to
    see what keywords were missed.
    """
    logger = logging.getLogger(__name__)
    logger.setLevel(logging.DEBUG)
    formatter = logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")

    stdout_handler = logging.StreamHandler(sys.stdout)
    stdout_handler.setLevel(logging.DEBUG)
    stdout_handler.setFormatter(formatter)

    file_handler = logging.FileHandler(filename)
    file_handler.setLevel(logging.DEBUG)
    file_handler.setFormatter(formatter)

    logger.addHandler(file_handler)
    logger.addHandler(stdout_handler)
    return logger


log = create_logger()
log.debug("Logger created")

API_KEY = os.getenv("KEYWORDS_EVERYWHERE_API_KEY")


def load_keywords(filename: str) -> list[str]:
    """
    Load keywords from a text file.

    Put one search term on each line.
    """
    with open(filename, "r") as f:
        kws = [line.strip() for line in f.readlines() if line.strip()]
        log.debug(f"Loaded {len(kws)} keywords from {filename}")
        return kws


def split_keywords(keywords: list[str]) -> list[list[str]]:
    """
    Split a list of keywords into chunks of 100 keywords each.

    The API only accepts 100 keywords at a time.
    """
    return [keywords[i : i + 100] for i in range(0, len(keywords), 100)]


def create_filename() -> str:
    """
    Create a unique filename based on the current date and time.

    This prevents files from overwriting each other.

    Example filename: `2023-05-14T00:12:45.736981.json`
    """
    current_date = datetime.now()
    return current_date.strftime("%Y-%m-%dT%H:%M:%S.%f%z") + ".json"


def write_to_file(data: dict[str, Any]) -> None:
    """
    Write a dictionary of data to a JSON file with a unique filename.
    """
    output_path = os.path.join("output", create_filename())
    with open(output_path, "w") as f:
        json.dump(data, f, indent=4)
        log.debug(f"Wrote data to file {f.name}")


def get_keywords_data(
    keywords: list[str],
    country: str = "us",
    currency: str = "USD",
    data_source: str = "gkp",
) -> dict[str, Any]:
    """
    Make a request to KeywordsEverywhere's API.
    """
    num_keywords = len(keywords)
    log.debug(f"Preparing to fetch data for {num_keywords} keywords")
    if num_keywords > 100:
        log.error(
            f"You can only search for 100 keywords at a time. Received {num_keywords}"
        )
        raise ValueError(
            f"You can only search for 100 keywords at a time. Received {num_keywords}"
        )

    payload = {
        "country": country,
        "currency": currency,
        "dataSource": data_source,
        "kw[]": keywords,
    }
    headers = {
        "Accept": "application/json",
        "Authorization": f"Bearer {API_KEY}",
    }

    log.debug(f"Requesting data for {num_keywords} keywords")

    response = requests.post(
        "https://api.keywordseverywhere.com/v1/get_keyword_data",
        data=payload,
        headers=headers,
    )

    if response.status_code == 200:
        log.debug(response.content.decode("utf-8"))
        return response.json()
    else:
        log.error(response.content.decode("utf-8"))
        raise Exception("An error occurred. Check the log file for more details.")


def main():
    keywords = load_keywords("keywords.txt")
    batches = split_keywords(keywords)
    num_batches = len(batches)
    delay = 1  # second

    for idx, batch in enumerate(batches):
        log.info(f"Processing batch {idx + 1} of {num_batches}")

        result = get_keywords_data(batch)
        write_to_file(result)

        log.info(f"sleeping for {delay} seconds")
        sleep(delay)

    log.info("Done. Check the output directory for the JSON file(s).")


main()

Running the Code

To run the code, create a file named keywords.txt and put one keyword query on each line.

Also create an empty directory named output, which is where the downloaded keyword data will be saved.

Be sure that your venv is activate. If you don’t see that the terminal prompt has the prefix, (.venv), run this command again:

source .venv/bin/activate

When first running the script, test it with just a few keywords to make sure that it works.

When you’re ready to run it, type this command:

python script.py

If everything went okay, you should see the data in the output directory.

If something went wrong, check the logs.log file. If you get stuck, feel free to send a message with the comment form below. Be sure to include your email address if you would like a reply.

If you followed the tutorial exactly, the files in your project directory will look like this, with your saved keyword data files in the output directory:

.
├── .env
├── .gitignore
├── keywords.txt
├── logs.log
├── output/
└── script.py

For more details on the Keywords Everywhere API, also check out their docs.

Tagged with: SEO Keyword ResearchPython

Feedback and Comments

What did you think about this page? Do you have any questions, or is there anything that could be improved? You can leave a comment after clicking on an icon below.