How we made 30 GB PSD uploads feel instant without expensive backend conversions.

When we first built our media upload system, the architecture looked completely normal. A user uploaded a file, the backend converted the image, resized variants were generated, and finally the UI displayed previews. At small scale this worked perfectly. But once real production teams started using the platform, everything changed.

Our users regularly upload massive assets including layered PSDs, TIFFs, RAW camera files, and multi-terabyte photography projects. Some uploads are 10–30 GB for a single file. The biggest issue was not the upload speed itself — it was the waiting experience around it.

Imagine a photography studio uploading thousands of RAW files after a shoot. The upload might continue for hours because of multipart transfers, network conditions, and extremely large source files. During that time the web application looked broken. Grid previews were empty, collections appeared incomplete, and teams could not review or organize assets because thumbnails did not exist yet.

The backend made things even worse. Once the upload finally completed, the system started expensive image conversion jobs just to generate previews. PSD flattening, TIFF decoding, resizing, and re-encoding consumed huge CPU and memory resources. The frustrating part was that users already had perfectly good preview images locally on their desktop.

That became the key insight behind the redesign.

Instead of asking how to make 30 GB uploads faster, we started asking a completely different question:

How do we make the system usable BEFORE the upload finishes?

The answer was surprisingly simple.

Every large source file already had a lightweight visual representation available locally.

main.psd  →  thumb.jpg

The thumb image might only be 200 KB while the source PSD could be 30 GB. So instead of waiting for the master upload to complete, we decided to upload the thumb immediately and use it as a temporary display proxy.

That became the foundation of what we now call:

Progressive Proxy Upload Architecture

The desktop application now creates a paired structure where the source file and lightweight thumb live together.

project/
├── main.psd
└── thumb/
    └── thumb.jpg

Both uploads share the same upload namespace in S3.

aws_key/
└── _1747828822/
    ├── main.psd
    └── thumb/
        └── thumb.jpg

This shared namespace turned out to be extremely important because both files now share the same lifecycle, retry behavior, cleanup scope, and asset identity.

Here is the actual workflow.

┌──────────────────────────┐
│ Desktop detects files    │
│                          │
│ main.psd                 │
│ thumb/thumb.jpg          │
└─────────────┬────────────┘
              │
              ▼
┌──────────────────────────┐
│ Upload thumb immediately │
│                          │
│ thumb/thumb.jpg          │
│ uploads in seconds       │
└─────────────┬────────────┘
              │
              ▼
┌──────────────────────────┐
│ Backend fans out thumb   │
│ into final delivery      │
│ paths                    │
└─────────────┬────────────┘
              │
              ▼
┌──────────────────────────┐
│ Web UI instantly shows   │
│ previews                 │
└─────────────┬────────────┘
              │
              ▼
┌──────────────────────────┐
│ Multipart PSD upload     │
│ continues in background  │
│ for minutes or hours     │
└─────────────┬────────────┘
              │
              ▼
┌──────────────────────────┐
│ Upload completes         │
│ Backend checks if        │
│ display assets exist     │
└─────────────┬────────────┘
              │
              ▼
┌──────────────────────────┐
│ YES → Skip expensive     │
│ conversion pipeline      │
└──────────────────────────┘

The most important architectural decision was this:

The thumb is copied directly into the FINAL production delivery paths.

That means the frontend does not need a special preview system. Existing CDN behavior, image URLs, caching, authorization, and rendering pipelines continue working exactly the same way.

The backend does not even treat the thumb as a real asset. No database row is created for it. The thumb only exists as a temporary display proxy that is copied into the same paths the real conversion pipeline would normally generate.

This architecture is very similar to proxy workflows used in video editing and VFX systems. Editors do not immediately work with huge 8K RAW footage. Instead they use lightweight proxy media first and later swap to the high-resolution master during final rendering.

We applied the same philosophy to large image uploads.

Once the thumb is uploaded, users can immediately browse collections, organize folders, search previews, review assets, and continue their workflow while the original file quietly uploads in the background.

The infrastructure improvements were massive.

Previously the backend had to decode giant PSDs, flatten layers, resize variants, encode outputs, and upload converted assets for almost every upload. That consumed huge CPU and memory resources.

With the proxy architecture, most conversions are skipped entirely because the display assets already exist.

Display asset exists?
       ↓
YES → Skip conversion

This dramatically reduced queue congestion, memory spikes, and processing overhead.

Even more importantly, the platform now feels fast.

The actual upload time did not magically disappear. A 30 GB PSD still takes time to upload. But because previews appear instantly, users no longer feel blocked by infrastructure operations.

That completely changed the user experience.

We also added multiple protection layers to ensure thumbs never accidentally become real assets. The desktop watcher ignores thumb directories, handlers filter thumb files again as defense-in-depth, and the backend short-circuits normal asset processing whenever a thumb upload is detected.

One particularly tricky bug appeared during multipart uploads. Small uploads worked perfectly, but large uploads occasionally triggered unnecessary conversions. The issue was caused by multipart uploads requesting a second presigned URL, which generated a different timestamp namespace from the thumb upload.

thumb → timestamp A
source → timestamp B

The thumb populated paths for timestamp A, while the conversion pipeline checked timestamp B.

The fix was ensuring both uploads always shared the exact same namespace.

Once solved, the architecture became extremely stable.

Here is the final architecture in one diagram.

        Lightweight Thumb Upload
                    │
                    ▼
      Fan-out into delivery paths
                    │
                    ▼
       Frontend becomes usable
                    │
                    ▼
     Master upload continues async
                    │
                    ▼
   Conversion pipeline checks paths
                    │
                    ▼
         Existing assets found
                    │
                    ▼
        Skip expensive conversion

This architecture dramatically improved scalability, reduced infrastructure costs, stabilized processing queues, and made multi-hour uploads feel nearly instant from the user’s perspective.

For large-scale media systems, proxy-first upload pipelines become incredibly powerful once asset sizes start growing into gigabytes and terabytes.

And honestly, the biggest lesson from this entire project was simple:

Users do not need the full-resolution master immediately. They only need the system to feel responsive.