The problem#

In my previous post, I gave some tips for making a web app save and load its data as a file to give the user control over their data. But for many applications, it's useful to think of the user's data as multiple files, possibly organized into directories. OPFS lets a web app store local data with a filesystem-like API, but, due to security concerns, there's no direct access to the user's real filesystem, so there's no straightforward way for the user to view or manipulate that data.

The solution#

The common way to deal with this kind of issue is to stuff all of the files into one file, reducing it to a solved problem. We'll use the ZIP archive file format as it's pretty universally supported, so the user can likely use such files. In these examples, I use the zip.js library, so you'll have to import zip-fs.min.js (or equivalent) to use them.

In these functions, dir is an OPFS directory: either the root directory from navigator.storage.getDirectory() or a subdirectory's FileSystemDirectoryHandle. They input/output the ZIP files as Blobs; use the helpers from my previous post to actually connect to the user's filesystem.

Downloading a directory as a ZIP is simple:

async function zipDirectory(dir) {
  const zipFs = new zip.fs.FS();
  await zipFs.root.addFileSystemHandle(dir);
  return await zipFs.exportBlob();
}

Reading a ZIP file into OPFS is more complicated and must be done inside a Web Worker (due to using createSyncAccessHandle()):

async function unzipToDirectory(zipfile, dir) {
  const z = new zip.fs.FS();
  await z.importBlob(zipfile);

  async function extract(z, dir) {
    if (z.directory) {
      const childDir = z.name
        ? await dir.getDirectoryHandle(z.name,
                    { create: true })
        : dir;
      for (const child of z.children) {
        await extract(child, childDir);
      }
    } else {
      await writeFile(
        await dir.getFileHandle(z.name, { create: true }),
        await (await z.getBlob()).arrayBuffer());
    }
  }

  await extract(z.root, dir);
}

async function writeFile(file, contents) {
  const handle = await file.createSyncAccessHandle();
  handle.truncate(0);
  if (contents.arrayBuffer) contents = await contents.arrayBuffer();
  handle.write(contents);
  handle.flush();
  handle.close();
}

The details#

Web Workers#

Safari API limitation#

The above code has to be run inside a Web Worker due to Safari only supporting createSyncAccessHandle(), which is only supported in a Web Worker context on all browsers due to doing synchronous IO, and not createWritable(), the other API for writing to OPFS files, which does not have that restriction. Here's the createWritable() variant of the writeFile() function:

async function writeFile(file, contents) {
  const f = await file.createWritable();
  await f.write(contents);
  await f.close();
}

Wrapping function in a Web Worker#

If you just want some code ready to drop in without setting up Web Workers, here's a wrapper based on this StackOverflow answer:

Function.prototype.callAsWorkerAsync = function (...args) {
  return new Promise((resolve, reject) => {
    const code = `self.onmessage = async e =>
        self.postMessage(await (${this.toString()})
                                .call(...e.data));`,
      blob = new Blob([code], { type: "text/javascript" }),
      url = window.URL.createObjectURL(blob),
      worker = new Worker(url);
    worker.onmessage = e => (resolve(e.data),
                         worker.terminate(),
                         window.URL.revokeObjectURL(url));
    worker.onerror = e => (reject(e.message),
                         worker.terminate());
    worker.postMessage(args);
  });
}

// To do "await writeFile(file, contents)" in a Web Worker:
await writeFile.callAsWorkerAsync(null, file, contents);

That will start up a Web Worker, run the function, and then terminate the worker. Note the function is communicated as text and the arguments are serialized to be sent to the worker, so it can't call any functions defined outside of itself. If we replace the direct call in unzipToDirectory() with that call instead, then unzipToDirectory() won't need to run in a Web Worker.

By spinning up a worker for every writeFile() call, this is completely eliminating any possible performance gains from using Web Workers and probably adding some overhead over using createWritable(), but it will work in all browsers. If performance is an issue, then writing a worker that runs the entire unzipToDirectory() function would probably be better, although more complicated as it would need access to the ZIP library.

Other ZIP libraries#

I originally started with using JSZip since it appeared to be more popular than zip.js. I ended up going with the latter because its FS API let me write the above functions more concisely. The JSZip versions are a little longer, but may still be useful if you already have a dependency on JSZip:

async function zipDirectory(dir) {
  async function recurseZip(zip, dir) {
    for await (const handle of dir.values()) {
      if (handle.kind === "directory") {
        await recurseZip(zip.folder(handle.name), handle);
      } else /* handle.kind === "file" */ {
        zip.file(handle.name, await handle.getFile());
      }
    }
    return zip;
  }

  const zip = await recurseZip(new JSZip(), dir);
  return await zip.generateAsync({type: 'blob'});
}

async function unzipToDirectory(zipfile, dir) {
  async function extract(zip, dir) {
    // forEach doesn't return, so collect the Promises.
    const tasks = [];
    zip.forEach((path, handle) => {
      // forEach returns all descendants, filter to children.
      const isNested = path.slice(0, -1).includes('/');
      if (isNested) return;
      async function process() {
        if (handle.dir) {
          const name = path.endsWith('/')
                       ? path.slice(0, -1) : path;
          const subdir = name
            ? await dir.getDirectoryHandle(name,
                                           { create: true })
            : dir;
          await extract(zip.folder(path), subdir);
        } else {
          await writeFile(
            await dir.getFileHandle(path, { create: true }),
            await handle.async('arraybuffer'));
        }
      }
      tasks.push(process());
    });
    await Promise.all(tasks);
  }

  await extract(await JSZip.loadAsync(zipfile), dir);
}

A Weird Imagination

ZIP web app local data