A Weird Imagination

Deleting deeply nested OPFS directories

Posted in

The problem#

The straightforward OPFS API for deleting a directory

await parentDir.removeEntry(name, {recursive: true});

doesn't work if the directory contains too many (several hundred) levels of nested directories. The sensible workaround is never create such a directory structure and wipe the OPFS storage entirely if you ever do by accident, but as discussed previously, I did get in that situation due to a bug and wrote some helpers to deal with it.

The solution#

For any reasonable real-world case, what you actually want is probably removeEntry():

await parentDir.removeEntry(name, {recursive: true});

or to delete everything from the root directory:

const root = await navigator.storage.getDirectory();
for await (const handle of root.values()) {
  await root.removeEntry(handle.name, {recursive: true});
}

or possibly to simply use your browser's settings to delete all site data, which should include OPFS data.

In case you do still want to delete a directory in OPFS without worrying about how deeply nested the directory structure is, you can use

async function removeDirectoryFast(dir) {
  const toDelete = [];
  let i = 0;
  let maxDepth = 0;
  for await (const fileHandle
             of getFilesNonRecursively(dir)) {
    maxDepth = Math.max(maxDepth, fileHandle.depth);
    toDelete.push(fileHandle);
  }
  async function deleteAtDepth(depth) {
    for (const f of toDelete) {
      if (f.depth === depth) {
        await f.parentDir.removeEntry(f.name,
                            {recursive: true});
      }
    }
  }
  const increment = 500; // Works empirically in Firefox.
  for (let depth = maxDepth; depth > 1; depth -= increment) {
    await deleteAtDepth(depth);
  }
  await deleteAtDepth(1);
}

This depends on the getFilesNonRecursively() helper from my previous blog post.

The details#

Read more…

Debugging OPFS

The problem#

While the web developer tools in Firefox and Chrome provide a Storage/Application tab for inspecting the local data stored by a web app, neither shows OPFS files there, making it difficult to tell what's going wrong when you have a bug (which was a problem when writting my recent blog posts about OPFS). There's open Firefox and Chromium bugs about the missing feature, so if it's been a while since this was posted when you're reading this, hopefully this is no longer a problem.

Additionally, the tools I did find all use recursion, resulting in them failing to work on the deeply nested directory tree I created by accident.

The solution#

If you don't have several hundred levels deep of nested directories, you can just use this Chrome extension or this script (or probably this web component, although I couldn't get it to install), all named "opfs-explorer".

The following AsyncIterator returns all of the files in OPFS without using recursion and adds properties to include their full path and parent directory:

async function* getFilesNonRecursively(dir) {
  const stack = [[dir, "", undefined, 0]];
  while (stack.length) {
    const [current, prefix, parentDir] = stack.pop();
    current.relativePath = prefix + current.name;
    current.parentDir = parentDir;
    current.depth = depth;
    yield current;

    if (current.kind === "directory") {
      for await (const handle of current.values()) {
        stack.push([handle,
                    prefix + current.name + "/",
                    current,
                    depth + 1]);
      }
    }
  }
}

And here's the simple HTML display function I've been using that calls that (you will likely want to modify this to your preferences):

async function displayOPFSFileList() {
  const existing = document.getElementById("opfs-file-list");
  const l = document.createElement('ol');
  l.id = "opfs-file-list";
  if (existing) existing.replaceWith(l);
  else document.body.appendChild(l);

  const root = await navigator.storage.getDirectory();
  for await (const fileHandle
             of getFilesNonRecursively(root)) {
    const i = document.createElement("li");
    i.innerText = fileHandle.kind + ": "
                  + (fileHandle.relativePath ?? "(root)");
    if (fileHandle.kind === "file") {
      const content = await fileHandle.getFile();
      const contentStr = content.type.length === 0
                      || content.type.startsWith("text/")
        ? ("\"" + (await content.slice(0, 100).text()).trim()
          + "\"")
        : content.type;
      i.innerText += ": (" + content.size + " bytes) "
                     + contentStr;
    }
    l.appendChild(i);
  }
}

The details#

Read more…

Loading multiple files without ZIP

The problem#

Last time, I showed how you can let a user have control over their data stored in a web app's OPFS by transferring directories in or out of the browser as ZIP files. But it would be more convenient if the user could just transfer folders instead without needing the extra step of going through an archive manager. There's no shortcut for getting data out of the browser: we can only save one file at a time (unless we use the Chrome-only File System Access API). But there is a cross-browser way to load multiple files, or even nested directories, into the browser.

The solution#

The HTML Drag and Drop API supports transferring multiple files and directories, although the details are a bit messy:

const target = document.getElementById("dropTarget");
// Required to make drop work.
target.addEventListener("dragover", (e) => e.preventDefault());
target.addEventListener("drop", async (e) => {
  e.preventDefault();
  await Promise.allSettled([...e.dataTransfer.items]
    .map(async (item) => {
      if (item.getAsFileSystemHandle) {
        await processFileSystemHandle(dir,
          await item.getAsFileSystemHandle());
      } else {
        await processFileSystemEntry(dir,
          item.webkitGetAsEntry());
      }
    })
  );
});

As you can see, for cross-browser support, we need to handle both getAsFileSystemHandle() and webkitGetAsEntry(). And, unfortunately, they return different types, so those two process*() functions really are pretty different:

async function processFileSystemHandle(dir, handle) {
  if (handle.kind === "directory") {
    const subdir = await dir.getDirectoryHandle(handle.name,
                             {create: true});
    for await (const entry of handle.values()) {
      await processFileSystemHandle(subdir, entry);
    }
  } else /* handle.kind === "file" */ {
    await writeFile(await dir.getFileHandle(handle.name,
                              {create: true}),
                    await handle.getFile());
  }
}
async function processFileSystemEntry(dir, entry) {
  async function readDirectory(directory) {
    let dirReader = directory.createReader();
    let getEntries = async () => {
      const results = await (new Promise((resolve, reject) =>
        dirReader.readEntries(resolve, reject)));
      if (results.length) {
        return [...results, ...await getEntries()];
      }
      return [];
    };

    return await getEntries();
  }

  if (entry.isDirectory) {
    const subdir = await dir.getDirectoryHandle(entry.name,
                             {create: true});
    for (const el of await readDirectory(entry)) {
      await processFileSystemEntry(subdir, el);
    }
  } else /* entry.isFile */ {
    const file = new Promise((resolve, reject) =>
      entry.file(resolve, reject));
    await writeFile(await dir.getFileHandle(entry.name,
                              {create: true}),
                    await file);
  }
}

(These assume the writeFile() helper from last week's post to handle writing inside a Web Worker as necessary.)

The details#

Read more…

ZIP web app local data

The problem#

In my previous post, I gave some tips for making a web app save and load its data as a file to give the user control over their data. But for many applications, it's useful to think of the user's data as multiple files, possibly organized into directories. OPFS lets a web app store local data with a filesystem-like API, but, due to security concerns, there's no direct access to the user's real filesystem, so there's no straightforward way for the user to view or manipulate that data.

The solution#

The common way to deal with this kind of issue is to stuff all of the files into one file, reducing it to a solved problem. We'll use the ZIP archive file format as it's pretty universally supported, so the user can likely use such files. In these examples, I use the zip.js library, so you'll have to import zip-fs.min.js (or equivalent) to use them.

In these functions, dir is an OPFS directory: either the root directory from navigator.storage.getDirectory() or a subdirectory's FileSystemDirectoryHandle. They input/output the ZIP files as Blobs; use the helpers from my previous post to actually connect to the user's filesystem.

Downloading a directory as a ZIP is simple:

async function zipDirectory(dir) {
  const zipFs = new zip.fs.FS();
  await zipFs.root.addFileSystemHandle(dir);
  return await zipFs.exportBlob();
}

Reading a ZIP file into OPFS is more complicated and must be done inside a Web Worker (due to using createSyncAccessHandle()):

async function unzipToDirectory(zipfile, dir) {
  const z = new zip.fs.FS();
  await z.importBlob(zipfile);

  async function extract(z, dir) {
    if (z.directory) {
      const childDir = z.name
        ? await dir.getDirectoryHandle(z.name,
                    { create: true })
        : dir;
      for (const child of z.children) {
        await extract(child, childDir);
      }
    } else {
      await writeFile(
        await dir.getFileHandle(z.name, { create: true }),
        await (await z.getBlob()).arrayBuffer());
    }
  }

  await extract(z.root, dir);
}
async function writeFile(file, contents) {
  const handle = await file.createSyncAccessHandle();
  handle.truncate(0);
  if (contents.arrayBuffer) contents = await contents.arrayBuffer();
  handle.write(contents);
  handle.flush();
  handle.close();
}

The details#

Read more…

Keeping web app data local

The problem#

Users don't tend to have a lot of control over their data in web apps. Most often, the data is stored on a server the user does not control—or, if they do control it, we're talking about self-hosting which is much more involved then just navigating to a web app in a browser. Alternatively, the data may be stored locally, but using various browser-specific mechanisms which make it difficult for the user to share, backup, or otherwise reason about the data the web app manipulates.

While desktop apps can replicate these problems, usually they store data in files either explicitly chosen by the user or in well-known locations.

The solution#

Files are a flexible interface to let users do whatever they want with their data, so let's use them for web apps, too.

To save a file to the user's computer, modified from this example:

function saveFile(filename, data, mimeType) {
  const element = document.createElement("a");
  const url = URL.createObjectURL(new Blob([data],
                                  { type: mimeType }));
  element.setAttribute("href", url);
  element.setAttribute("download", filename);
  element.click();
  URL.revokeObjectURL(url);
}
// Save a JSON file:
saveFile("hello.json",
  JSON.stringify({"Hello": "World!"}, null, 2),
  "application/json");

(Consider using beforeunload if the user has unsaved changes to make sure they really do have their data in the file, and not just in the browser.)

To load a file from the user's computer:

function loadFile() {
  const element = document.createElement("input");
  element.type = "file";
  return new Promise((resolve, reject) => {
    element.click();
    element.addEventListener("change",
      () => resolve(element.files[0]));
    element.addEventListener("cancel",
      () => reject("User canceled."));
  });
}
// loadFile() must be called from a real user click.
myButton.addEventListener('click',
  async (e) => myLoadFunc(await loadFile()));

The details#

Read more…

Serverless WebRTC

The problem#

While in my last post, I said serverless WebRTC was too cumbersome, I wanted to try to see how streamlined I could make the process. While researching, I did find a variant of serverless-webrtc called serverless-webrtc-qrcode as well as a similar demo, webrtc-qr that both use QR codes as an easy way to transmit the offer strings. But both require that both sides have a camera to scan QR codes, while my use case is a WebRTC connection between my desktop without a camera and my smartphone.

The solution#

minimal-webrtc now has a checkbox to enable serverless mode. In that mode, the QR code shown by the host is a much longer URL that includes the initial WebRTC offer. Opening that URL on another device (or in another browser window) will show another QR code along with a "Copy" button. With the first device, either press the "Scan QR code" button and point it at the QR code or use some other mechanism to copy the text and paste it into the text area labeled "Paste offer here:".

To run it locally, download the source code and run a web server to serve the wwwroot/ directory. If both devices can run a web server, then you can just access it via localhost on each, but, as before, because WebRTC requires HTTPS, to run it on your local network, you may need to set up a self-signed certificate.

The details#

Read more…

Minimal WebRTC

The problem#

I wanted to stream video of myself and my screen at the same time. My plan was to put the video of myself on my screen and stream the entire screen, but I do not have a camera on my desktop. On the other hand, I do have a smartphone with a camera, so I needed a way to show the video from my phone's camera on my desktop's screen.

There are a few Android apps that promise to do so, but none of the ones I tried worked very well. But I know that video chat works just fine on my phone, including in a web browser using WebRTC which supports peer-to-peer video chat between two web browsers, so it should be easy to show the video from my phone's camera in a browser window on my desktop. Unfortunately, I couldn't find any straightforward solution for setting up just a peer-to-peer video link.

The solution#

Open minimal-webrtc on the computer you want the video streamed to. A QR code will appear; use your smartphone to read it and after approving access to the camera, the video should appear in the first browser window. This is intended to be used for local connections, so it may not work if the two devices are not on the same network. Only the signaling to set up the connection goes to the minimal-webrtc server, the actual video will be sent peer-to-peer over the local network.

To get just the video as a bare window with no decorations, use chromium --app=uri to get rid of the address bar, etc., and this script to remove the rest.

To host it yourself, download the source code1 and use the included run_daphne.sh script (which assumes daphne is installed) and nginx configuration. As WebRTC requires HTTPS, to run it on your local network, you may need to set up a self-signed certificate.

The details#

Read more…

Change title based on visible section

The problem#

In the computer game Keep Talking and Nobody Explodes, the "bomb expert" players are looking at a fictional "bomb manual", often frantically searching for the right page. While the intention is for this document to be printed out—and physical paper makes it relatively easy to keep the headings at the top of each page visible—there is also a web version if you prefer to view it on a screen (or don't have access to a printer). Scrolling through the web version feels a lot more awkward than flipping through the paper version; one workaround I found was to open different pages in different browser tabs or windows, but then identifying which page is in which window is still awkward.

The solution#

I created a userscript, "BombManual.com Tab Title and TOC" that automatically updates the tab title to match the title of the currently visible page. It additionally adds a table of contents to make it easy to quickly open all of the pages in separate tabs or windows.

If you do not already have one, you will need to install a userscript manager extension for your browser to use it. Alternatively, you could bookmark this bookmarklet, but as you would have to click that bookmark on every instance of the page you opened, that's less convenient.

The details#

Read more…

Long Polling in Django Channels

The problem#

In a web app where the display should be constantly up-to-date, the client needs some way to get up-to-date information from the server. One of the simplest ways to do so is to regularly (every few seconds) query the server asking if there is new information. This involves making a lot of requests and is wasteful of bandwidth and processor time on both the client and server (the latter can be improved with caching).

If updates are rare, it makes much more sense for the server to notify the client when they occur, but HTTP is designed around the client making requests to the server, not the other way around. And, furthermore, the Django web framework (like many web frameworks) is built around that model.

The solution#

Of course, this is a well-understood problem and there are a wide variety of APIs and libraries to solve it discussed on the Wikipedia page for Comet. The main workarounds are WebSockets which is a very flexible technology for two-way communication in a web browser and long polling which is a simpler technique which involves merely having the server not answer a request immediately and instead wait until it actually has an update to reply with.

In the rest of this blog post, I discuss the changes I made to convert a Django-based web app that I originally wrote to use a basic polling pattern and hosted using uWSGI to instead use long polling and be hosted using Gunicorn/Uvicorn. I also cover nginx configuration including hosting the app in a subdirectory.

The details#

Read more…

User scripts on iPad

The problem#

Google Forms is a tool that allows for easily setting up simple structured data entry. But it's designed to make it easy to analyze a lot of data that has been entered, not to view a single entry. There is a view to show individual entries, but it's very cluttered due to including all of the options that were not selected as well as those that were selected. A display that showed only the entries that were selected could be used as a quick and dirty way to make a form letter-like website.

To make this problem harder, the solution has to run on iPad, a platform not exactly known for its user programmability.

The solution#

Bookmark this link: hide unselected items in Google Form. Then select that bookmark when on the appropriate Google Forms page. Note that in addition to hiding unselected entries, if the entry that is not select has a value of "Yes", then its entire section will be hidden. If you don't want that behavior, bookmark this variant of the script instead.

The details#

Read more…