Skip to main content

Tutorial 6: Off-Chain Storage

In Tutorial 5: Common Types and Functions, you learned how to use Merkle trees to refer to large amounts of data stored off-chain.

This tutorial presents a library and pattern to store Merkle trees off-chain and store only the tree's root hash on-chain.

This approach is a step towards unlocking a larger set of applications that require off-chain storage. Future solutions can provide other decentralized options for zkApps that require more trustless solutions.

experimental

This proposed solution to off-chain storage is experimental and is used only for education purposes.

This tutorial provides a single-server solution to data storage for prototyping zkApps and building zkApps where some trust guarantees are reasonable. The solution proposed in this learning tutorial is appropriate for development, but is not recommended for zkApps that require trustlessness.

The single-server solution for prototyping is intended as one of several options for data availability on Mina. Mina doesn't offer an out-of-the-box solution for off-chain storage.

Why Off-Chain Storage?​

When you build an application for testing and local use, you can build and store a Merkle root locally.

However, when you build a production-ready, distributed zkApp, you need more than this. All users that interact with your zkApp must be able to retrieve and modify the latest state.

Any data that modifies a zkApp must be available somewhere for others users to access.

Off-Chain Storage and Decentralization​

Solutions to storage span a large spectrum from inexpensive and more centralized to more expensive and more decentralized. The decentralized solutions are more expensive due to replicating and proving stored data.

Your off-chain storage needs depend on the zkApp you are building and the guarantees you want that zkApp to have.

Solutions under exploration:

  1. A single-server storage solution, presented here.
  2. A multi-server storage solution that can be run by multiple parties for stronger trust guarantees.
  3. A solution that leverages storage on modular blockchains.
  4. A future hard fork to add purchasable on-chain data storage to Mina.
  5. A future hard fork to add data-storage committees to Mina for horizontally scalable storage.

Single-Server Off-Chain Storage​

This tutorial implementation is the single-server off-chain storage solution. The library provides a REST server that anyone can run to store data for one or multiple zkApps and a zkApp library to check on-chain if changes have been backed by the server.

This implementation requires a trust assumption for zkApps that use it: both developers and users must trust whoever is running the server.

This trust assumption makes it useful for prototyping applications that need off-chain storage and for putting applications into production where these trust assumptions are reasonable. This implementation is not appropriate for zkApps where a trustless solution is needed.

To learn more about this implementation and see how it works, see the experimental-zkapp-offchain-storage library.

Grant Opportunity to Develop Single-Server Off-Chain Storage​

Mina Foundation values contributions. You can improve this library to make it more decentralized and useful in production. This development opportunity seeks a developer to start with the library presented here, improve it, make it more decentralized, and run instances for the community. If you are interested in this grant opportunity, send an email to [email protected].

Suggested improvements:

  • Eliminate DDOS vulnerability by adding a token that limits storage requests.
  • Do not store trees for a smart contract if that contract is misconfigured in a way to prevent cleaning up old data.
  • Add support to the client library for connecting to multiple storage servers, enabling correctness under a majority-honest assumption.
  • Switch the project to a more scalable database implementation (for example, Redis).
  • Write an implementation for an automatically scalable service (for example, Cloudflare).

Implement a Project Using Off-Chain Storage​

The sample code for this project is provided at examples/zkapps/06-offchain-storage/contracts with a focus on:

This project implements a Merkle tree where:

  • Each leaf is either empty or stores a number (an o1js field), which is the data.
  • Updates to the tree can update a leaf if the new number in the leaf is greater than the old number.
  • The root of the tree is stored on-chain.
  • The tree itself is stored on an off-chain storage server.

Prerequisites​

  • Make sure you have the latest version of the zkApp CLI installed:

    $ npm install -g zkapp-cli

  • Ensure your environment meets the Prerequisites for zkApp Developer Tutorials.

This tutorial has been tested with:

Create the project​

  1. Create or change to a directory where you have write privileges.

  2. Create a project by using the zk project command:

    $ zk project 06-off-chain-storage

    The zk project command has the ability to scaffold the UI for your project. For this tutorial, select none:

    ? Create an accompanying UI project too? …
      next
      svelte
      nuxt
      empty
    > none

Project structure​

The zk project command creates the 06-off-chain-storage directory that contains the scaffolding for your project.

The files in the src directory files contain the TypeScript code for the smart contract.

Each time you make updates, then build or deploy, the TypeScript code is compiled into JavaScript in the build directory.

For all projects, you run zk commands from the root of your project directory.

Prepare the project​

  1. Change to the project directory, delete the existing files, and create a new src/NumberTreeContract smart contract, and a main.ts file:

    $ cd 06-off-chain-storage
    $ rm src/Add.ts
    $ rm src/Add.test.ts
    $ rm src/interact.ts
    $ zk file src/NumberTreeContract
    $ touch src/main.ts

  2. Edit index.ts to import and export your new smart contract:

    import { NumberTreeContract } from './NumberTreeContract.js';

    export { NumberTreeContract };

  3. Now, add the experimental library for the off-chain storage server:

    $ npm install experimental-zkapp-offchain-storage --save

  4. Install the xmlhttprequest-ts TypeScript wrapper for the built-in HttpClient to emulate the browser XMLHttpRequest object:

    $ npm install --save xmlhttprequest-ts

    This project uses this for network requests when running from Node.js where the browser's XMLHttpRequest is not available by default.

Run your storage server​

When you start this experimental local storage server, a database.json file is created in the current directory to store data for this tutorial.

In a new terminal window, run this command from the root directory of your project:

$ node node_modules/experimental-zkapp-offchain-storage/build/src/storageServer.js

Implement the smart contract​

A full copy of the NumberTreeContract.ts example file is provided.

  1. Open NumberTreeContract.ts in your editor.

  2. Start by adding the imports:

import {
  SmartContract,
  Field,
  MerkleTree,
  state,
  State,
  method,
  DeployArgs,
  Signature,
  PublicKey,
  Permissions,
  Bool,
} from 'o1js';

import {
  OffChainStorage,
  MerkleWitness8,
} from 'experimental-zkapp-offchain-storage';
...

Notice that you import some items from experimental-zkapp-offchain-storage:

The OffChainStorage object contains the functions for interacting with off-chain storage:

  • getPublicKey: A function to get the storage server's public key. This key is also stored in smart contracts to identify what storage server is storing the smart contract's off-chain data.
  • get: A function for fetching the data for a Merkle tree from the storage server, given the root of the tree.
  • requestStore: A function to request storing a tree on the storage server. Returns a proof that the storage server has stored this tree.
  • assertRootUpdateValid: A function used in smart contracts to prove updates to the smart contract's currently stored tree root result in a tree root that is being stored by the storage server.
  • mapToTree: A storage function to convert maps to trees. Internally the storage server is using maps from tree indices to leafs.

The MerkleWitness8 is a type of Merkle tree witness required for o1js to use the same instances of the witness cross-library. Other types of Merkle trees are available for input, such as MerkleWitness32 and MerkleWitness256.

  1. Now, set up your smart contract:
...
export class NumberTreeContract extends SmartContract {
  @state(PublicKey) storageServerPublicKey = State<PublicKey>();
  @state(Field) storageNumber = State<Field>();
  @state(Field) storageTreeRoot = State<Field>();

  deploy(args: DeployArgs) {
    super.deploy(args);
    this.account.permissions.set({
      ...Permissions.default(),
      editState: Permissions.proofOrSignature(),
    });
  }

  @method async initState(storageServerPublicKey: PublicKey) {
    this.storageServerPublicKey.set(storageServerPublicKey);
    this.storageNumber.set(Field(0));

    const emptyTreeRoot = new MerkleTree(8).getRoot();
    this.storageTreeRoot.set(emptyTreeRoot);
  }
...

This code adds three pieces of state to the contract:

  • The public key of the storage server
  • The storageNumber used to ensure the storage server is actively storing states
  • The root of the Merkle tree

Initialize the zkApp state for these three values by setting:

  • The public key of the storage server
  • The storage number to 0
  • Storing the root of an empty tree
  1. Continuing, add the code for the update function on the smart contract:
...
  @method async update(
    leafIsEmpty: Bool,
    oldNum: Field,
    num: Field,
    path: MerkleWitness8,
    storedNewRootNumber: Field,
    storedNewRootSignature: Signature
  ) {
    const storedRoot = this.storageTreeRoot.get();
    this.storageTreeRoot.assertEquals(storedRoot);

    let storedNumber = this.storageNumber.get();
    this.storageNumber.assertEquals(storedNumber);

    let storageServerPublicKey = this.storageServerPublicKey.get();
    this.storageServerPublicKey.assertEquals(storageServerPublicKey);

    let leaf = [oldNum];
    let newLeaf = [num];

    // newLeaf can be a function of the existing leaf
    newLeaf[0].assertGreaterThan(leaf[0]);

    const updates = [
      {
        leaf,
        leafIsEmpty,
        newLeaf,
        newLeafIsEmpty: Bool(false),
        leafWitness: path,
      },
    ];

    const storedNewRoot = OffChainStorage.assertRootUpdateValid(
      storageServerPublicKey,
      storedNumber,
      storedRoot,
      updates,
      storedNewRootNumber,
      storedNewRootSignature
    );

    this.storageTreeRoot.set(storedNewRoot);
    this.storageNumber.set(storedNewRootNumber);
  }
}

This code gets and asserts the current state of the contract, and then performs the update.

First, check that the new leaf is greater than the old leaf.

Then, check the update itself. In this example, perform a single update to the tree. However, you can chain updates together with multiple witnesses to change the tree more than once in a single call to the storage server.

To assert the update is valid, use a assertRootUpdateValid call from the OffChainStorage library. This checks that when the update is applied to the tree represented by the existing on-chain tree root, the data for the new tree is being stored by the storage server.

That completes the smart contract!

Implementing main.ts​

Since much of the logic in the main.ts file is repeated from earlier tutorials, this tutorial reviews just the relevant parts.

  1. Download the main.ts example file.

  2. Move it to the local /06-off-chain-storage/src folder.

  3. Open it in your editor.

The main.ts file contains logic for running the contract locally and for deploying and interacting with it on Devnet.

This is a useful pattern when developing a new contract.

Connect to the off-chain storage server​

To try your contract on Devnet, deploy the contract as usual with zk deploy.

In main.ts, set useLocal to false.

Add the name of your config to the end of your call to node main.js.

This code connects to the storage server on port 3001 and get its public key:

...
const storageServerAddress = 'http://localhost:3001';
const serverPublicKey = await OffChainStorage.getPublicKey(
  storageServerAddress,
  NodeXMLHttpRequest
);
...

In a real application, you would run the storage server on an externally exposed machine, and change this address from localhost to match the storage server.

updateTree function​

Now, review the updateTree function in main.ts.

The goal in this function is to:

  1. Get the currently stored tree from the storage server
  2. Select a random leaf
  3. Change the value at that leaf to a bigger number
  4. Create a transaction that performs this update.

To start, get the existing tree:

...
async function updateTree() {
  const index = BigInt(Math.floor(Math.random() * 4));

  // get the existing tree
  const treeRoot = await zkapp.storageTreeRoot.get();
  const idx2fields = await OffChainStorage.get(
    storageServerAddress,
    zkappPublicKey,
    treeHeight,
    treeRoot,
    NodeXMLHttpRequest
  );

  const tree = OffChainStorage.mapToTree(treeHeight, idx2fields);
  const leafWitness = new MerkleWitness8(tree.getWitness(BigInt(index)));
...

Next, get the current root stored in the contract and request the data for that root from the storage server.

Then, convert that data from a map to a Merkle tree and get a witness for a random index of the Merkle tree.

Continuing:

...
  // get the prior leaf
  const priorLeafIsEmpty = !idx2fields.has(index);
  let priorLeafNumber: Field;
  let newLeafNumber: Field;
  if (!priorLeafIsEmpty) {
    priorLeafNumber = idx2fields.get(index)![0];
    newLeafNumber = priorLeafNumber.add(3);
  } else {
    priorLeafNumber = Field(0);
    newLeafNumber = Field(1);
  }
...

This code checks if the leaf is empty and shapes the update accordingly. If the leaf was empty, set it to one. Otherwise, set the leaf to whatever used to be there, plus 3.

...
  const [storedNewStorageNumber, storedNewStorageSignature] =
    await OffChainStorage.requestStore(
      storageServerAddress,
      zkappPublicKey,
      treeHeight,
      idx2fields,
      NodeXMLHttpRequest
    );
...

Finally, request that the storage server stores the data. If successful, a new storage number and a signature is returned and can be used for updating the smart contract.

Call the smart contract​

To call the smart contract:

...
  const doUpdate = () => {
    zkapp.update(
      Bool(priorLeafIsEmpty),
      priorLeafNumber,
      newLeafNumber,
      leafWitness,
      storedNewStorageNumber,
      storedNewStorageSignature
    );
  };

  if (useLocal) {
    const updateTransaction = await Mina.transaction(
      { sender: feePayerKey.toPublicKey(), fee: transactionFee },
      () => {
        doUpdate();
      }
    );
    updateTransaction.sign([zkappPrivateKey, feePayerKey]);
    await updateTransaction.prove();
    await updateTransaction.send();

That completes the review of the code to interact with the experimental off-chain storage server.

Conclusion​

This tutorial introduced an experimental solution that builds a smart contract that leverages off-chain storage.

Next, check out Tutorial 7: Oracles to learn how to use Oracles to pull in data from the outside world into your zkApp.