Publishing a directory of files to IPFS with Python

What IPFS actually is

IPFS is a peer-to-peer protocol for addressing and distributing files by their content rather than by their location. Instead of a URL that says "fetch this file from that server", an IPFS content identifier (CID) is a cryptographic hash computed from the bytes of the file itself. If you have the CID, any IPFS node that has a copy of those bytes can serve them to you - there is no origin server to take down, no hostname to DNS-poison, and no single point of failure for censorship to lean on. In that respect it has something in common with BitTorrent: files are mirrored across nodes that have requested them, and the network as a whole holds the content as long as someone, somewhere, is pinning it.

Content is king: the implication that shapes everything else

Because a CID is derived from the content of the file, changing a single byte produces a completely new CID. There is no in-protocol relationship between the old CID and the new one. IPFS does not maintain revisions, history, or a "this supersedes that" link between them. If you publish a file today and it changes tomorrow, tomorrow's file has a different CID, and it is entirely your job as the publisher to tell the world which CID is the current one.

That is manageable for a one-off file. It is not manageable for a dataset that updates hourly. Handing out a fresh CID every time the content changes is a poor experience for anyone trying to fetch it - they would need to be told the new CID each time, out of band.

IPNS: a stable name for a moving target

This is the problem IPNS (InterPlanetary Name System) solves. An IPNS name is a public key - or a human-readable alias of one - that you can republish to point at a new CID as often as you like. The IPNS name itself never changes; only its target does. Consumers only ever need to know the IPNS name, and they will always resolve to whatever CID the publisher has most recently pointed it at.

The mental model is closer to DNS than to a symlink: a static name that resolves to a destination, with the publisher holding the private key needed to update that destination. Unlike DNS, the binding is cryptographic and the resolution can happen peer-to-peer without a registrar in the path.

So the publish loop for a changing dataset is: add the new content to IPFS, get back its CID, and call name/publish on the IPNS key to point at that CID. Repeat whenever the content changes.

Publishing a single file: the easy case

The go-ipfs daemon exposes an HTTP RPC API on 127.0.0.1:5001 by default. Adding a single file is a straightforward POST /api/v0/add with the file in the multipart body. The response is a JSON object containing the CID (Hash), the Name, and the Size. That is the case most examples on the internet cover, and it is genuinely simple.

Publishing a directory: the not-easy case

What is not well documented is how to publish a directory containing multiple files as a single IPFS object, so that the result is one CID for the folder and individual CIDs for the files inside it - addressable as ipfs://<folder-cid>/<filename>.

The naive approach of calling add once per file gives you N independent CIDs with no folder relationship between them. What you want is the equivalent of ipfs add -r my-files/ on the command line, but driven from the HTTP API.

The trick is in how the multipart form data is constructed. Each file's filename field in the multipart part must carry the escaped absolute path within the directory, using %2F (URL-encoded /) as the separator - so my-files/file1.txt becomes my-files%2Ffile1.txt. go-ipfs infers the directory from that path. You never explicitly "add" the directory; it is implied by the common prefix across the files.

This was not easy to discover. The IPFS HTTP API reference describes the add endpoint but does not make the directory inference behaviour obvious, and the most common Python library for the API (py-ipfs-http-client) was, at the time, semi-abandoned and did not work reliably with current go-ipfs releases. Falling back to plain requests against the HTTP API turned out to be simpler than fighting the library.

The script

Below is a self-contained example that publishes two files inside a directory called my-files to a local go-ipfs daemon, then points an IPNS key at the resulting folder CID. The same pattern extends to any number of files.

#!/usr/bin/env python3

# This code is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. The author
# accepts no responsibility for its use, of which you do so at
# your own risk.

import json
import requests
from pathlib import Path

# Directory we want to publish
directory = "my-files"

# Read the first file, which is already inside that directory
file1 = f"{directory}/file1.txt"
p = Path(file1)
with p.open("rb") as fp:
    file1_bytes = fp.read()
    file1_name = p.name
    file1_absolute_escaped = f"{directory}%2F{file1_name}"

# Read the second file
file2 = f"{directory}/file2.txt"
p = Path(file2)
with p.open("rb") as fp:
    file2_bytes = fp.read()
    file2_name = p.name
    file2_absolute_escaped = f"{directory}%2F{file2_name}"

# Construct our files payload for the POST request to the IPFS API.
# Note how we use the absolute 'escaped' folder+file names for
# the file info. This is important for the IPFS API to understand that
# these files are 'inside' a directory. Notice that the directory doesn't
# explicitly get added, it is inferred from that path.
folder_and_files = {
    file1_name: (
        file1_absolute_escaped,
        file1_bytes,
        "application/octet-stream",
        {"filename": file1_absolute_escaped},
    ),
    file2_name: (
        file2_absolute_escaped,
        file2_bytes,
        "application/octet-stream",
        {"filename": file1_absolute_escaped},
    ),
}

# IPFS API endpoint
url = "http://127.0.0.1:5001/api/v0"

# Post the request to the IPFS API
response = requests.post(f"{url}/add", files=folder_and_files)

# The response contains a list of Name/Hash/Size attributes
# for each file, separated by a new line, but it's not a JSON blob
# apparently.
#
# It will look something like this:
#
# ['{"Name":"my-files/file1.txt","Hash":"QmZLRFWaz9Kypt2ACNMDzA5uzACDRiCqwdkNSP1UZsu56D","Size":"11"}',
#  '{"Name":"my-files/file2.txt","Hash":"QmfTitKrVzFGbLCG2KaxBg5UMo6F2iYxsQCeC2er59cFTU","Size":"11"}',
#  '{"Name":"my-files","Hash":"QmVRFp9xNPRLXXe4qXTwDx5sG3Wm6Y6jcbRJnv7u8EZcQu","Size":"128"}',
# '']
#
# It's the Hash of the directory that we want to point our IPNS
# ID at, so that our IPNS ID always points at the latest version.

ipfs_response = response.text.split("\n")
for item in ipfs_response:
    j = json.loads(item)
    if j["Name"] == directory:
        ipfs_hash = j["Hash"]
        # Point our IPNS at the directory's hash
        response = requests.post(f"{url}/name/publish?arg={ipfs_hash}")
        break

Walkthrough

The first thing to notice is the file1_absolute_escaped value: my-files%2Ffile1.txt. That is the filename sent in the multipart part's Content-Disposition header. It is the URL-encoded form of the relative path my-files/file1.txt. This is the single line that makes the whole directory trick work - go-ipfs sees the %2F, decodes it to /, and infers that these two files belong to a directory called my-files.

The folder_and_files dictionary is structured as requests expects for multipart uploads: each key is a form field name, and each value is a tuple of (filename, bytes, content_type, extra_headers). The extra_headers dict with {"filename": file1_absolute_escaped} is what actually sets the filename in the Content-Disposition header of that part - this is the field go-ipfs reads.

The response from /add is not a single JSON object. It is a series of newline-separated JSON objects, one per file added, followed by one for the directory itself. The example response in the comments shows the shape: two file entries, then a third entry whose Name is the directory name (my-files) and whose Hash is the folder CID. The script loops through those lines, parses each as JSON, and waits for the entry where Name matches the directory name. That entry's Hash is the CID to publish under IPNS.

The final requests.post(f"{url}/name/publish?arg={ipfs_hash}") is the IPNS update. It tells the daemon's IPNS key to point at the new folder CID. From that point on, anyone resolving the IPNS name gets the latest version of the directory and all the files inside it.

Extending it

The script hardcodes two files for clarity, but the pattern generalises. Read every file in the directory, build an escaped path for each, construct the folder_and_files dict in a loop, and POST them all in a single add request. The directory CID comes back as the entry whose Name is the directory name. Wrap it in a function, call it from a cron job or a systemd timer whenever the source files change, and you have a self-updating IPFS publication pipeline with a stable IPNS name that consumers can pin.

If you want a changelog of which CID was published when - useful for a dataset that updates over time - append the folder CID and a timestamp to a log file before publishing it to IPNS, then include that log file as one of the files in the directory on the next publish. The changelog ships inside the directory it describes.

Installing the IPFS daemon

If you need to get go-ipfs onto a Linux server in the first place, I co-maintain an Ansible role for IPFS that handles the installation and configuration. It is what I used to stand up the node this script talks to.

Why not the Python library?

At the time this was written, py-ipfs-http-client did not work reliably with the current go-ipfs release and appeared semi-abandoned. The HTTP API is stable, well-documented, and requests is already a dependency in most Python projects that talk to HTTP services. Skipping the library and talking to the API directly was less code and less risk than pinning to a specific old version of a library that might break on the next go-ipfs upgrade. That may have changed since - check the current state of the library before deciding.

This article was adapted from work originally written for my customer Freedom of the Press Foundation .