Caching OpenAI API Responses with Upstash Redis

If you used the OpenAI API, you might have noticed that it’s pretty slow and sometimes doesn’t even respond. Especially the GPT-4 model is prone to high latency responses. Also, you pay for every answer you get. Those are all reasons to avoid serving responses directly.

You can store your responses in Upstash Redis to get around these issues. If you’re serving the same response to many clients, this can save you quite some money, and with global deployments, you can be sure users get the info as quickly as possible.

This article will guide you through caching OpenAI API responses to an Upstash Redis database.

Features

We will build a web app showing visitors a list of history jokes.

1. Query the OpenAI API daily to get a history joke related to the current date
2. Cache the joke for future delivery
3. Display all jokes

Technology

We will use the following technologies to build the app:

• Node.js as HTTP server
• The OpenAI API to generate history jokes
• Upstash QStash to trigger a new joke request from the HTTP server to the OpenAI API
• Upstash Redis to store the joke for later use

Prerequisites

• An OpenAI account with API access to get access to an AI
• An Upstash account to send the request to the OpenAI API and cache the AI responses in Upstash Redis
• A Node.js installation to call the OpenAI and Upstash APIs

Implementation

Let’s start by creating the HTTP server to fetch the jokes, store them in Upstash Redis, and display them to visitors.

Setting Up the Project

First, we create a new Node.js project and install the dependencies.

    $ mkdir history-jokes & cd history-jokes
    $ npm init -y
    $ npm i dotenv express axios @upstash/redis @upstash/qstash

Implementing the Server

To implement the server, create an index.js file and copy the following code into it:

require("dotenv").config();
 
const axios = require("axios");
const express = require("express");
const { Redis } = require("@upstash/redis");
const { Receiver } = require("@upstash/qstash");
 
const redisClient = new Redis({
  url: process.env.UPSTASH_REDIS_URL,
  token: process.env.UPSTASH_REDIS_TOKEN,
});
 
const qstashReceiver = new Receiver({
  currentSigningKey: process.env.UPSTASH_QSTASH_CURRENT_SIGNING_KEY,
  nextSigningKey: process.env.UPSTASH_QSTASH_CURRENT_NEXT_KEY,
});
 
const openaiApiClient = axios.create({
  baseURL: "https://api.openai.com/v1",
  headers: {
    Authorization: "Bearer " + process.env.OPENAI_TOKEN,
    "Content-Type": "application/json",
  },
});
 
const server = express();
 
server.use("/generate", async (request, response, next) => {
  // return next()
  let validRequest = false;
  try {
    validRequest = await qstashReceiver.verify({
      signature: request.headers["upstash-signature"],
      body: "",
    });
  } catch (e) {}
  if (!validRequest) return response.status(403).end("Forbidden");
 
  return next();
});
 
server.post("/generate", async (_request, response) => {
  const today = new Date();
  const month = today.toLocaleString("default", { month: "long" });
  let day = today.toLocaleString("default", { day: "numeric" });
  day = day == 1 ? "1st" : day == 2 ? "2nd" : day == "3" ? "3rd" : day + "th";
 
  const { data } = await openaiApiClient.post("/chat/completions", {
    model: "gpt-3.5-turbo",
    messages: [
      {
        role: "system",
        content: "You are a comedian that tells short history jokes.",
      },
      {
        role: "user",
        content: `Please tell me a joke for ${month} the ${day}.`,
      },
    ],
  });
 
  const joke = JSON.stringify({
    date: month + " the " + day,
    text: data.choices[0].message.content,
  });
 
  await redisClient.lpush("jokes", joke);
 
  response.end();
});
 
server.get("/", async (_request, response) => {
  let html = "<h1>History Jokes for Every Day</h1>";
 
  const jokes = await redisClient.lrange("jokes", 0, -1);
 
  html +=
    "<ul>" +
    jokes
      .map(({ date, text }) => `<li><b>${date}</b><br><pre>${text}</pre></li>`)
      .join("") +
    "</ul>";
 
  response.setHeader("Content-Type", "text/html");
  response.end(html);
});
 
server.listen(3000);

Let’s go through the important parts of this code.

We use the redisClient to save our responses into Upstash Redis and load them for display.

The qstashReceiver is responsible for verifying the daily calls from Upstash QStash; this way, we ensure that only QStash calls our /generate endpoint.

We use Axios to create a client for the OpenAI API, so we don’t have to pass the baseUrl and Authentication header whenever we call the API.

Next, we create an Express middleware that uses the qstashReceiver to verify all requests to the /generate endpoint. Since we use QStash only to trigger the endpoint and don’t pass any values, we can use an empty string as the body.

Note: The middleware allows all requests if you un-comment the line with "// return next()". You can use this to test the /generate endpoint on localhost.

The /generate endpoint listens for POST requests and creates and stores the actual jokes. It calculates the current month and day and uses it to create two prompts for the AI. One tells the AI to behave like a comedian, and the second asks to tell a history joke about the current month and day.

We call the OpenAI API with the prompts and save the result into a LIST in Upstash Redis.

The / endpoint listens for GET requests and displays the saved jokes. It loads the LIST from Upstash Redis, formats it as HTML list elements, and sends it to the client.

Deployment

Now that the server is set up, we must create the cloud resources and fill in the API credentials to access them.

Creating a Credential File

To store the credentials, create a .env file with the following content:

OPENAI_TOKEN=""
UPSTASH_REDIS_REST_URL=""
UPSTASH_REDIS_REST_TOKEN=""
UPSTASH_QSTASH_CURRENT_SIGNING_KEY=""
UPSTASH_QSTASH_NEXT_SIGNING_KEY=""

In the next steps, we will fill each of these empty strings.

Creating an OpenAI API Token

Let’s start with the OpenAI API because we only need a key for the existing API; we don’t have to deploy anything.

Go to the OpenAI web console and click the “create new secret key” button. Give the key a name, click “Create secret key”, copy the new key, and paste it into the .env file as the value of OPENAI_TOKEN.

Deploying a Recurring Request with QStash

To tell QStash that it should send a request to the /generate endpoint once a day, you use the “Request Builder”. You find it in the Upstash console. Figure 1 shows the configuration details.

Figure 1: QStash Request Builder

Replace the <HOSTNAME> with the domain that hosts your server. You need a publicly accessible hostname for QStash to work.

You find the singing keys QStash uses in the Upstash console in the “Request Builder” section. Click on the grey “Singing Keys” drop-down, copy each key and paste them in the right places of the .env file.

Deploying an Upstash Redis Database

To create an Upstash Redis database, go to the Upstash console and click the “Create Database” button.

This will prompt you with a dialog to fill in the database configuration. Figure 2 shows the values you should use.

Figure 2: Upstash Redis Configuration

After the creation is completed, you can scroll down to the “REST API” section, which has two buttons, “UPSTASH_REDIS_REST_URL” and “UPSTASH_REDIS_REST_TOKEN”. Click each of these buttons to copy the corresponding credentials and paste them into the correct place in the .env file.

Testing the Website

To test the website on your local machine, deactivate the QStash validation middleware by un-commenting the “// return next()” line in the index.js file.

Run the server with the following command:

    $ node .

If you open the / route in the browser, you should only see “History Jokes for Every Day” since no jokes have been generated.

You must send a POST request to the /generate endpoint to generate a joke. You can do so with this command:

    $ curl -X POST http://localhost/generate

If you refresh the browser page, you should see a joke like in Figure 3.

Figure 3: Joke website

This joke is now cached in Upstash Redis, so no OpenAI API requests will happen when someone visits the website. This gives you sub-second response times while saving you quite some money.

Summary

AI APIs are versatile tools that will transform the Internet as we know it, but right now, they’re slow and expensive. Caching is vital when using these APIs, and Upstash Redis is the easiest way to add it to your stack with just a few clicks.