Skip to content
algo150Open the app
Frontend system design ยท about 40 minutes

Large media upload

Design the upload feature of a media product. Users upload images and videos, and one video can be several gigabytes. The user selects or drops several files, sees progress for each one, and can cancel. An upload must survive a short network loss. A large upload should continue after a page reload and not start again from zero. Many users are on mobile networks. After the upload, the server processes the file, for example to make thumbnails.

Outline it in the app

Ask first

  • What are the limits: file size, file types, and number of files at one time?
  • Do files go through our server, or directly to object storage with signed URLs?
  • Must an upload resume after a page reload, or only after a network drop?
  • Do we change files on the client, for example resize images or make a preview?
  • What happens after the upload: processing, a virus scan? How long does it take?
  • May the user move to another page of the app while uploads run?

The outline

Write your own outline for each part first, then open it to compare with the points a strong answer covers.

Requirements
  • Functional: select or drop files, validate them, upload several, show progress per file and in total, cancel, retry, show the final state
  • A file of several gigabytes uploads without being read into memory as a whole
  • After a network drop the upload continues from the last confirmed part, not from zero
  • Progress is honest: it shows bytes that were really sent, and 'done' only after the server confirms
  • The user gets a warning before closing the page during an upload
  • Keyboard and screen reader users can add files and can follow progress and errors
Architecture
  • An upload manager outside the components owns a queue of files and runs a limited number of uploads at a time
  • A large file is cut into parts with Blob.slice, for example 8 MB each, and sent part by part, a few in parallel
  • The browser sends parts directly to object storage with signed URLs from our API; our server only starts, signs and completes
  • Each part is retried alone with backoff; when all parts are confirmed, the server joins them
  • Views subscribe to the manager: a drop zone, a list with one row per file, and a total progress bar
  • After completion the client asks for the processing status until the file is ready
Data model
  • Upload item: local id, the File, name, size, type, status (queued, uploading, paused, processing, done, failed, cancelled), bytes sent, error
  • Server session: uploadId, part size, and the list of confirmed parts with their numbers and checksums
  • Part: number, byte range, status, attempts; progress is the confirmed bytes plus the bytes now in flight
  • Resume record in IndexedDB or localStorage: a fingerprint of the file (name, size, last modified), the uploadId and the confirmed parts
  • After a reload the page usually has no access to the file, so the user selects it again and the fingerprint finds the saved session
  • Status changes follow a small state machine, so a cancelled upload cannot become done
Interface
  • POST /uploads { name, size, type } returns { uploadId, partSize } after the server checks the limits
  • POST /uploads/:id/parts/sign { partNumbers } returns a signed URL for each part; the URLs are short-lived
  • PUT <signed URL> with the bytes of one part; the response carries a checksum or ETag for that part
  • GET /uploads/:id returns the confirmed parts, so the client knows where to continue
  • POST /uploads/:id/complete { parts } joins the parts and returns the media id; DELETE /uploads/:id cancels and removes the parts
  • GET /media/:id returns the processing status, and the URLs when the file is ready
  • Uploader props: accept, maxSize, maxFiles, onComplete(media), onError; row props: item, onCancel, onRetry
Optimizations
  • Validate type, size and count on the client for fast feedback, and again on the server, because a client check can be skipped
  • Progress events: XMLHttpRequest reports upload progress and fetch does not; with parts, finished parts can also be counted
  • A few parts in parallel, for example 3 or 4; more parts only share the same bandwidth and use more connections and memory
  • A failed part is retried with exponential backoff; the manager pauses on the offline event and continues on online
  • Previews use URL.createObjectURL and revoke the URL after use; large images can be made smaller on the client before upload
  • Accessibility: a real file input behind the drop zone, a progress element per file, and status announced at steps, not on every percent
  • Edge cases: the same file twice, an empty file, an expired signed URL, and a file that changed on disk during the upload

Pitfalls

  • One request for a file of several gigabytes, so a network drop means starting from zero
  • Reading the whole file into memory, for example as a base64 string
  • Sending every file through our application server when it can go directly to storage
  • Trusting only the file extension or the check on the client
  • Showing 100 percent when the bytes are sent but the server has not confirmed the file

Follow-ups an interviewer asks

  • The network drops at 80 percent of a 4 GB file. What happens next?
  • The user reloads the page during the upload. How does it resume?
  • How do you show progress that is accurate?
  • Why upload directly to storage, and how do you keep that secure?
  • How many parts do you send in parallel, and why not 50?
  • A signed URL expires in the middle of the upload. What do you do?