A key-value storage system designed to handle massive amounts of data with minimal latency. Whether you’re building a URL shortener, a content delivery system, or any application requiring quick data retrieval, Cloudflare KV Storage offers a powerful and flexible solution.

What is Cloudflare KV Storage?

Cloudflare KV (Key-Value) Storage is a globally distributed, eventually consistent key-value store that allows developers to store and retrieve data across Cloudflare's edge network.

Key Features of Cloudflare KV:

• Global Distribution: Data stored in KV is automatically replicated across Cloudflare's global network, ensuring that it is close to users wherever they are.

• Eventual Consistency: While KV is globally distributed, it offers eventual consistency, meaning that updates to a key might take a little time to propagate across all data centers. This is a trade-off for the performance and scalability it provides.

• Low Latency: KV is optimized for read-heavy workloads, delivering fast read access to data.

• Large Storage Capacity: It is designed to handle large volumes of data with millions of keys.

• Integration with Workers: KV is tightly integrated with Cloudflare Workers, making it easy to use KV in serverless functions.

KV Storage vs. SQL/NoSQL Databases

You might wonder how KV Storage compares to more traditional storage solutions like SQL or NoSQL databases. Here’s a quick overview:

SQL Databases

• Structure: Relational, with defined schemas.

• Use Case: Best for applications requiring complex queries, transactions, and strong consistency.

• Performance: Higher latency for global distribution.

NoSQL Databases

• Structure: Flexible, schema-less.

• Use Case: Suitable for applications requiring horizontal scaling and handling large volumes of unstructured data.

• Performance: Better for distributed systems but still may require more management than KV Storage.

KV Storage

• Structure: Simple key-value pairs.

• Use Case: Ideal for caching, configuration, and scenarios with high read frequency and low write frequency.

• Performance: Extremely low latency due to global distribution and eventual consistency.

  Source code

  If you'd like to see the full source code for this project, it's available on GitHub.

Setting Up KV Storage

1. Install wrangler:

wrangler is the CLI tool used to manage Cloudflare Workers projects.

npm install -g wrangler

If you don't have Node.js installed, you'll need to install it first, as wrangler is a Node.js package.

2. Login to Cloudflare:

To link wrangler to your Cloudflare account, run:

wrangler login

This will open a browser window where you can authorize wrangler to access your Cloudflare account.

3. Create a New Workers Project:

Create a new Cloudflare Workers project using wrangler:

wrangler init my-workers-project

This will create a new directory named my-workers-project with the necessary files and structure.

4. Set Up KV Namespace:

1. Log in to the Cloudflare dashboard.

2. Go to Workers and then KV.

3. Create a new KV namespace.

4. Note the namespace's ID.

5. Configure wrangler.toml:

Open the wrangler.toml file in your project directory and add the KV namespace configuration:

name = "my-workers-project"
type = "javascript"

kv_namespaces = [
  { binding = "MY_KV", id = "your-kv-namespace-id" }
]

compatibility_date = "2023-08-01" # Adjust to current date
node_compat = true
nodejs_compat = true
workers_dev = true

Replace "your-kv-namespace-id" with the ID of the KV namespace you created.

6. Create the Worker Script:

Replace the contents of the index.js or src/index.js file with the API script provided earlier (which supports GET, POST, PUT, and DELETE requests).

7. Run Your Worker Locally:

Cloudflare Workers don't run locally in the traditional sense, but you can use wrangler dev to simulate it. This command creates a local development server that mimics the behavior of the Workers platform:

wrangler dev

This will start the worker on a local server, usually available at http://localhost:8787.

8. Test Your API Locally:

You can now use tools like curl, Postman, or a browser to test your API endpoints locally.

For example:

• POST Request:

    curl -X POST http://localhost:8787/post \
      -H 'Content-Type: application/json' \
      -d '{"key": "someKey", "value": "someValue"}'
  

• GET Request:

    bashCopy codecurl http://localhost:8787/get?key=someKey
  

• PUT Request:

    curl -X PUT http://localhost:8787/put \
      -H 'Content-Type: application/json' \
      -d '{"key": "someKey", "value": "newValue"}'
  

• DELETE Request:

    curl -X DELETE http://localhost:8787/delete \
      -H 'Content-Type: application/json' \
      -d '{"key": "someKey"}'
  

9. Deploy to Cloudflare:

Once you're satisfied with your local testing, you can deploy your worker to Cloudflare's network:

wrangler publish

This will deploy your Worker to the Cloudflare network and make it accessible globally.

Here's an example of a URL shortener with authentication using Cloudflare Workers

1.1 Shorten URL

import { generateUniqueShortcode } from '../../utils/generateUniqueShortcode'

export async function handleCreateShortUrl(request, env) {
    const { originalUrl, customShortcode } = await request.json()

    let shortcode

    if (customShortcode) {
        const existingUrl = await env.MY_KV.get(`url:${customShortcode}`)
        if (existingUrl) {
            return new Response(
                'Shortcode already in use. Please choose another one.',
                { status: 409 }
            )
        }
        shortcode = customShortcode
    } else {
        shortcode = await generateUniqueShortcode(env)
    }

    await env.MY_KV.put(
        `url:${shortcode}`,
        JSON.stringify({
            originalUrl,
            createdAt: new Date().toISOString(),
            clicks: 0,
        })
    )
    return new Response(
        JSON.stringify({
            shortUrl: `${new URL(request.url).origin}/go/${shortcode}`,
        }),
        {
            headers: { 'Content-Type': 'application/json' },
        }
    )
}

1.2 Get Original URL

export async function getOriginalUrl(shortcode, env) {
    const urlData = await env.MY_KV.get(`url:${shortcode}`, { type: 'json' })

    if (!urlData) {
        return new Response('Shortcode not found', { status: 404 })
    }
    urlData.clicks += 1
    await env.MY_KV.put(`url:${shortcode}`, JSON.stringify(urlData))
    if (urlData) {
        return new Response(urlData.originalUrl, {
            status: 302,
            headers: { 'Content-Type': 'application/json' },
        })
    } else {
        return new Response(JSON.stringify({ error: 'URL not found' }), {
            status: 404,
            headers: { 'Content-Type': 'application/json' },
        })
    }
}

import { handleCreateShortUrl } from './routes/urlShortener/createShortUrl';

export default {
  async fetch(request, env, ctx) {
    const url = new URL(request.url);

    // Route requests based on the path and method
    if (url.pathname === '/shorten' && request.method === 'POST') {
      // Call the function to handle creating a short URL
      return await handleCreateShortUrl(request, env, ctx);
    }

    // Default response for other routes
    return new Response('Not Found', { status: 404 });
  },
};

Approach: Maintain a User Index

Whenever you create or add a new user, you also update a special key (e.g., userIndex) that stores a list of all user IDs. This way, you can retrieve the list of all users by simply fetching this index key.

Why Use Indexes?

Since Cloudflare KV Storage is a simple key-value store without the ability to list or query keys directly, maintaining an index of keys (like user IDs or URL shortcodes) allows you to keep track of all the items of a certain type. This way, you can simulate querying or listing all users or URLs.

Step-by-Step Implementation

1. Store User Data with Indexing

When you add a new user to KV Storage, you also update the userIndex key with the user's ID.

• Adding a New User:

async function addUser(userId, userData) {
  // Store the user data
  await KV.put(`user:${userId}`, JSON.stringify(userData));

  // Update the user index
  let userIndex = JSON.parse(await KV.get("userIndex")) || [];
  userIndex.push(userId);
  await KV.put("userIndex", JSON.stringify(userIndex));
}

2. Retrieve All Users

To get the list of all users, fetch the userIndex key and then retrieve each user's data.

• Fetching All Users:

async function getAllUsers() {
  // Fetch the user index
  let userIndex = JSON.parse(await KV.get("userIndex")) || [];

  // Retrieve user data for each userId
  let users = [];
  for (let userId of userIndex) {
    let userData = JSON.parse(await KV.get(`user:${userId}`));
    users.push(userData);
  }
  return users;
}

Source code

If you'd like to see the full source code for this project, it's available on GitHub.

Conclusion

Cloudflare KV Storage is a robust tool for developers looking to build scalable, globally distributed applications with minimal complexity. Its ease of use, combined with the performance benefits of Cloudflare’s edge network, makes it an excellent choice for a wide range of use cases, from simple URL shorteners to complex data caching systems.