A key-value storage system designed to handle massive amounts of data with minimal latency. Whether you’re building a URL shortener, a content delivery system, or any application requiring quick data retrieval, Cloudflare KV Storage offers a powerful and flexible solution.
What is Cloudflare KV Storage?
Cloudflare KV (Key-Value) Storage is a globally distributed, eventually consistent key-value store that allows developers to store and retrieve data across Cloudflare's edge network.
Key Features of Cloudflare KV:
Global Distribution: Data stored in KV is automatically replicated across Cloudflare's global network, ensuring that it is close to users wherever they are.
Eventual Consistency: While KV is globally distributed, it offers eventual consistency, meaning that updates to a key might take a little time to propagate across all data centers. This is a trade-off for the performance and scalability it provides.
Low Latency: KV is optimized for read-heavy workloads, delivering fast read access to data.
Large Storage Capacity: It is designed to handle large volumes of data with millions of keys.
Integration with Workers: KV is tightly integrated with Cloudflare Workers, making it easy to use KV in serverless functions.
KV Storage vs. SQL/NoSQL Databases
You might wonder how KV Storage compares to more traditional storage solutions like SQL or NoSQL databases. Here’s a quick overview:
SQL Databases
Structure: Relational, with defined schemas.
Use Case: Best for applications requiring complex queries, transactions, and strong consistency.
Performance: Higher latency for global distribution.
NoSQL Databases
Structure: Flexible, schema-less.
Use Case: Suitable for applications requiring horizontal scaling and handling large volumes of unstructured data.
Performance: Better for distributed systems but still may require more management than KV Storage.
KV Storage
Structure: Simple key-value pairs.
Use Case: Ideal for caching, configuration, and scenarios with high read frequency and low write frequency.
Performance: Extremely low latency due to global distribution and eventual consistency.
Source code
If you'd like to see the full source code for this project, it's available on GitHub.
Setting Up KV Storage
1. Install wrangler:
wrangler is the CLI tool used to manage Cloudflare Workers projects.
npm install -g wrangler
If you don't have Node.js installed, you'll need to install it first, as wrangler is a Node.js package.
2. Login to Cloudflare:
To link wrangler to your Cloudflare account, run:
wrangler login
This will open a browser window where you can authorize wrangler to access your Cloudflare account.
3. Create a New Workers Project:
Create a new Cloudflare Workers project using wrangler:
wrangler init my-workers-project
This will create a new directory named my-workers-project with the necessary files and structure.
4. Set Up KV Namespace:
Log in to the Cloudflare dashboard.
Go to Workers and then KV.
Create a new KV namespace.
Note the namespace's ID.
Open the wrangler.toml file in your project directory and add the KV namespace configuration:
name = "my-workers-project"
type = "javascript"
kv_namespaces = [
{ binding = "MY_KV", id = "your-kv-namespace-id" }
]
compatibility_date = "2023-08-01" # Adjust to current date
node_compat = true
nodejs_compat = true
workers_dev = true
Replace "your-kv-namespace-id" with the ID of the KV namespace you created.
6. Create the Worker Script:
Replace the contents of the index.js or src/index.js file with the API script provided earlier (which supports GET, POST, PUT, and DELETE requests).
7. Run Your Worker Locally:
Cloudflare Workers don't run locally in the traditional sense, but you can use wrangler dev to simulate it. This command creates a local development server that mimics the behavior of the Workers platform:
wrangler dev
This will start the worker on a local server, usually available at http://localhost:8787.
8. Test Your API Locally:
You can now use tools like curl, Postman, or a browser to test your API endpoints locally.
For example:
POST Request:
curl -X POST http://localhost:8787/post \
-H 'Content-Type: application/json' \
-d '{"key": "someKey", "value": "someValue"}'
GET Request:
bashCopy codecurl http://localhost:8787/get?key=someKey
PUT Request:
curl -X PUT http://localhost:8787/put \
-H 'Content-Type: application/json' \
-d '{"key": "someKey", "value": "newValue"}'
DELETE Request:
curl -X DELETE http://localhost:8787/delete \
-H 'Content-Type: application/json' \
-d '{"key": "someKey"}'
9. Deploy to Cloudflare:
Once you're satisfied with your local testing, you can deploy your worker to Cloudflare's network:
wrangler publish
This will deploy your Worker to the Cloudflare network and make it accessible globally.
Here's an example of a URL shortener with authentication using Cloudflare Workers
1.1 Shorten URL
import { generateUniqueShortcode } from '../../utils/generateUniqueShortcode'
export async function handleCreateShortUrl(request, env) {
const { originalUrl, customShortcode } = await request.json()
let shortcode
if (customShortcode) {
const existingUrl = await env.MY_KV.get(`url:${customShortcode}`)
if (existingUrl) {
return new Response(
'Shortcode already in use. Please choose another one.',
{ status: 409 }
)
}
shortcode = customShortcode
} else {
shortcode = await generateUniqueShortcode(env)
}
await env.MY_KV.put(
`url:${shortcode}`,
JSON.stringify({
originalUrl,
createdAt: new Date().toISOString(),
clicks: 0,
})
)
return new Response(
JSON.stringify({
shortUrl: `${new URL(request.url).origin}/go/${shortcode}`,
}),
{
headers: { 'Content-Type': 'application/json' },
}
)
}
1.2 Get Original URL
export async function getOriginalUrl(shortcode, env) {
const urlData = await env.MY_KV.get(`url:${shortcode}`, { type: 'json' })
if (!urlData) {
return new Response('Shortcode not found', { status: 404 })
}
urlData.clicks += 1
await env.MY_KV.put(`url:${shortcode}`, JSON.stringify(urlData))
if (urlData) {
return new Response(urlData.originalUrl, {
status: 302,
headers: { 'Content-Type': 'application/json' },
})
} else {
return new Response(JSON.stringify({ error: 'URL not found' }), {
status: 404,
headers: { 'Content-Type': 'application/json' },
})
}
}
import { handleCreateShortUrl } from './routes/urlShortener/createShortUrl';
export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);
if (url.pathname === '/shorten' && request.method === 'POST') {
return await handleCreateShortUrl(request, env, ctx);
}
return new Response('Not Found', { status: 404 });
},
};
Approach: Maintain a User Index
Whenever you create or add a new user, you also update a special key (e.g., userIndex) that stores a list of all user IDs. This way, you can retrieve the list of all users by simply fetching this index key.
Why Use Indexes?
Since Cloudflare KV Storage is a simple key-value store without the ability to list or query keys directly, maintaining an index of keys (like user IDs or URL shortcodes) allows you to keep track of all the items of a certain type. This way, you can simulate querying or listing all users or URLs.
Step-by-Step Implementation
1. Store User Data with Indexing
When you add a new user to KV Storage, you also update the userIndex key with the user's ID.
async function addUser(userId, userData) {
await KV.put(`user:${userId}`, JSON.stringify(userData));
let userIndex = JSON.parse(await KV.get("userIndex")) || [];
userIndex.push(userId);
await KV.put("userIndex", JSON.stringify(userIndex));
}
2. Retrieve All Users
To get the list of all users, fetch the userIndex key and then retrieve each user's data.
async function getAllUsers() {
let userIndex = JSON.parse(await KV.get("userIndex")) || [];
let users = [];
for (let userId of userIndex) {
let userData = JSON.parse(await KV.get(`user:${userId}`));
users.push(userData);
}
return users;
}
Source code
If you'd like to see the full source code for this project, it's available on GitHub.
Conclusion
Cloudflare KV Storage is a robust tool for developers looking to build scalable, globally distributed applications with minimal complexity. Its ease of use, combined with the performance benefits of Cloudflare’s edge network, makes it an excellent choice for a wide range of use cases, from simple URL shorteners to complex data caching systems.
