Distributing digital content metadata by a distributed file system

Last time we talked about why we are going to set up an International Standard Content Number (ISCN) And now, we define the first version of the ISCN specification and implement it on top of the LikeCoin chain. LikeCoin chain is a Cosmos-based blockchain, and it is one of the ISCN registry who is responsible for storing and distributing the well-defined digital content metadata.

In the previous article, we also discussed the benefit of storing digital content metadata by a blockchain. It records who and when is registering the digital content metadata. At the same time, it can also guarantee integrity once the metadata is registered. Although it is great for storing digital content metadata by a blockchain, there is not good enough for distributing the digital content metadata. Since the users can only interact with LikeCoin chain through an API gateway, they need to find an existing public API gateway or build up an API gateway to access the LikeCoin chain by themself. For the first method, the users do need to know an existing public API gateway, and it needs to be online or not banned by any authorities. And for the second method, it will be too tricky for a user who wants to query the ISCN metadata just once. Therefore, we need to use another technical stack to help the distribution of the ISCN metadata becomes more practicable and more reliable availability. After some research, we lock on InterPlanetary File Systems (IPFS).

IPFS

The IPFS s a protocol and peer-to-peer network for storing and sharing data in a distributed file system, and it is a composition from a set of stacks :

IPLD

What we are focusing on is the InterPlanetary Linked Data (IPLD) layer which is a Merkle Distributed Acyclic Graphs (Merkle DAG) to define the data structure on the IPFS network, and its natures perfectly match to the ISCN metadata:

1. A JSON like data structure which is quite human-readable.
2. A linked data structure by design which is a requirement from the ISCN specification.
3. Using Content IDentifier (CID) as a self-describing content-addressed identifier which uses the hash of the data itself to form an addressable ID that means once the user gets a piece of data from a provided CID, they will know the integrity of the data.

What an ISCN IPLD looks like:

{
  "context": "https://iscn.io/schema/iscn-v1",
  "id": "1/25G6rDEXF2SVLaqMHjX5jur4nnNFJgGoZvms8Hx8LP2C",
  "timestamp": "2020-01-01T12:34:56Z",
  "version": 1,
  "rights": {
    "/": "/ipfs/z4gc5ex1gES8s5vDfVAFXfpvNssKaQ4YKf2agiZN5UDsXDx9QXo"
  },
  "stakeholders": {
    "/": "/ipfs/z4h3dBp1iw52EY8KKUX7kWK6sP7GKfw8AJEnJoqoNtSo6vKJ9wW"
  },
  "content": {
    "/": "/ipfs/z4hviFYTSJE27xnwc8f6XttU6tEbgJqqp7KLvxNfKoPzNvyZJxF"
  }
}

The CID of this ISCN IPLD is z4gAY85sHVv5P1CnxGPPYvL3yFxdZXrJ1j4j8PJLntyPfnbJpwQ.
The id is what an ISCN-ID looks like, and the rights, stakeholders and content are the Merkle links that link to another type of ISCN IPLD objects.

Fortunately, IPFS provides us with a high degree of scalability, allowing us to make an IPLD plugin to handle our specific data structure. Since the LikeCoin chain is implemented in GO, we only need to import github.com/ipfs/go-ipfs/pluginand implement the PluginIPLD interface.

One of the most critical functions is the RegisterBlockDecoders (format.BlockDecoder) error, which tells the IPFS core how to handle different types of ISCN objects:

func RegisterBlockDecoders(decoder ipld.BlockDecoder) error {
  decoder.Register(0x0264, decodeIscnBlock)
  decoder.Register(0x0265, decodeRightsBlock)
  decoder.Register(0x0266, decodeStakeholdersBlock)
  decoder.Register(0x0267, decodeEntityBLock)
  decoder.Register(0x0268, decodeContentBlock)
}

The codes 0x0264, 0x0265, etc. are the codecs for IPLD objects. These codecs are encoded into the CID so that the IPFS core knows which logic should use to process the binary data. Everyone can occupy a codec on the Multicodec Github repo, which is based on “first come first assign” policy. You can see that we only need to map the codec to the corresponding handler for decoding the binary data. In our use case, we do not care about the data encoding, so we leave the function RegisterInputEncParsers (coredag.InputEncParsers) error with no operation.

For binary data encoding, we use canonical CBOR provided by IPFS, which works like Concise Binary Object Representation (CBOR), and it will sort the key-value pairs right before encoding to ensure the encoded binary data is canonical even if the input order of two identical contents is different. The IPLD-plugin only need to deserialize the binary data in the decoder through the same CBOR library to obtain the key-value object so that we can construct it back to a human-readable IPLD JSON string.

After creating this IPFS IPLD-plugin, anyone can start the IPFS client with this plugin installed. They can retrieve ISCN metadata through CID, just like other IPFS content.

LikeCoin chain Datastore

We discussed how we to distribute the ISCN metadata through IPFS network, but where is the data stored per se? Considering the raw data is stored actually on the LikeCoin chain, what if IPFS can directly retrieve the data from the LikeCoin chain too? It sounds perfect, and it actually can be accomplished by adding a Datastore plugin to IPFS.

First of all, the LikeCoin chain validator program runs an IPFS daemon programmatically, that means all of the validators also host an IPFS server at the same time, and of course, those IPFS servers install the IPLD-plugin which can handle the ISCN metadata with IPLD.

The IPFS Datastore plugin is used to handle where and how to retrieve the binary data by a given CID. Therefore, we can create a Datastore plugin, such that whenever someone is requesting the ISCN objects through the IPFS network, it can tell the IPFS core to get the binary data from LikeCoin chain.

For a Datastore plugin, we need to implement at least three functions: get, getSize, has, which get the data, the data size, and the existence of data for a specific CID respectively. The LikeCoin chain Datastore plugin utilizes the codec to identify whether the CID is querying a chain data or not, and if it is a chain data, the request will be forwarded to the chain through RPC endpoints exposed by the chain.

func isISCN(key ds.Key) (bool, *cid.Cid) {
  // convert the key to CID
  c, err := dshelp.DsKeyToCid(ds.NewKey(key.BaseNamespace()))
  if err != nil {
    return false, nil
  }  // check if the CID type is one of the ISCN types
  return iscn.IsIscnObject(c.Type()), &c
}func (a *accessor) Get(key ds.Key) (value []byte, err error) {
  if ok, cid := isISCN(key); ok {
    // call Tendermint RPC endpoint to get chain data for ISCN types
    return a.tmGet("/custom/iscn/cid_get", *cid)
  }
 
  // get non-chain data through original LevelDB flow, skipping
}// `GetSize` and `Has` are similar

This method is versatile enough that any blockchain software can utilize the plugin to connect to the IPFS network, and only need to implement those three functions through RPC endpoints and specifying the CID codec that the blockchain need to respond. In the future, we will modify the plugin so it will be configurable with different backends, RPC endpoints and CID types to let it becomes more flexible.

Retrieving ISCN metadata from IPFS network

By combining the above two plugins, we finally can distribute the ISCN metadata by riding on the IPFS network. And those functionalities will also be implemented and released in the upcoming update of LikeCoin chain. Now, let’s take a look what the whole process when retrieving ISCN metadata through IPFS network.

For example, LikeCoin chain contains the following ISCN metadata:

Type is ISCN kernel, and CID is z4gAY85sHVv5P1CnxGPPYvL3yFxdZXrJ1j4j8PJLntyPfnbJpwQ ->

{
  "context": "https://iscn.io/schema/iscn-v1",
  "id": "1/25G6rDEXF2SVLaqMHjX5jur4nnNFJgGoZvms8Hx8LP2C",
  "timestamp": "2020-01-01T12:34:56Z",
  "version": 1,
  "rights": {
    "/": "/ipfs/z4gc5ex1gES8s5vDfVAFXfpvNssKaQ4YKf2agiZN5UDsXDx9QXo"
  },
  "stakeholders": {
    "/": "/ipfs/z4h3dBp1iw52EY8KKUX7kWK6sP7GKfw8AJEnJoqoNtSo6vKJ9wW"
  },
  "content": {
    "/": "/ipfs/z4hviFYTSJE27xnwc8f6XttU6tEbgJqqp7KLvxNfKoPzNvyZJxF"
  }
}

For the rights, a Merkle link links to z4gc5ex1gES8s5vDfVAFXfpvNssKaQ4YKf2agiZN5UDsXDx9QXo ->

{
  "context": "https://iscn.io/schema/rights-v1",
  "rights": [
    {
      "holder": {
        "/": "/ipfs/z4hVAigQdpFadDiGMf3NY8iucisUvaWwPhzfLzrY61DANWWuhHk"
      },
      "type": "license",
      "terms": {
        "/": "/ipfs/QmbgTwBpvX5SNW2hYCw6umrxsKRWxyVmefn9fa9aE25GRR"
      }
    }
  ]
}

First of all, we can download an IPFS client and have a try:

$ wget https://dist.ipfs.io/go-ipfs/v0.5.1/go-ipfs_v0.5.1_darwin-amd64.tar.gz
$ tar xvfz go-ipfs_v0.5.1_darwin-amd64.tar.gz
$ cd go-ipfs
$ ipfs init
$ ipfs daemon

And install the IPLD-plugin:

$ git clone https://github.com/likecoin/iscn-ipld.git
$ cd iscn-ipld
$ ./set-target.sh v0.5.1
$ make install

We can have a try now:

$ ./ipfs dag get z4gAY85sHVv5P1CnxGPPYvL3yFxdZXrJ1j4j8PJLntyPfnbJpwQ | jq .
{
  "content": {
    "/": "/ipfs/z4hviFYTSJE27xnwc8f6XttU6tEbgJqqp7KLvxNfKoPzNvyZJxF"
  },
  "context": "https://iscn.io/schema/iscn-v1",
  "id": "1/25G6rDEXF2SVLaqMHjX5jur4nnNFJgGoZvms8Hx8LP2C",
  "rights": {
    "/": "/ipfs/z4gc5ex1gES8s5vDfVAFXfpvNssKaQ4YKf2agiZN5UDsXDx9QXo"
  },
  "stakeholders": {
    "/": "/ipfs/z4h3dBp1iw52EY8KKUX7kWK6sP7GKfw8AJEnJoqoNtSo6vKJ9wW"
  },
  "timestamp": "2020-01-01T12:34:56Z",
  "version": 1
}

See, we get back the ISCN metadata, and by the Merkle link, we can even query the rights like this:

% ./ipfs dag get z4gAY85sHVv5P1CnxGPPYvL3yFxdZXrJ1j4j8PJLntyPfnbJpwQ/rights | jq .
{
  "context": "https://iscn.io/schema/rights-v1",
  "rights": [
    {
      "holder": {
        "/": "/ipfs/z4hVAigQdpFadDiGMf3NY8iucisUvaWwPhzfLzrY61DANWWuhHk"
      },
      "terms": {
        "/": "/ipfs/QmbgTwBpvX5SNW2hYCw6umrxsKRWxyVmefn9fa9aE25GRR"
      },
      "type": "license"
    }
  ]
}

Only two simple installations, we can immediately query the ISCN metadata, and you do not even know the existence of LikeCoin chain.

IPNS

You may notice that querying data from IPFS network can only use the CID, but the ISCN metadata is using the ISCN-ID as an identifier. Additionally, multiple CIDs may point to the same ISCN due to the different versions of ISCN metadata. Therefore, if the content distributor only shows the ISCN-ID is not enough to query the ISCN metadata and if only shows the CID is also not good enough as the CID is not unique for a given digital content. Up to this moment, if a user only gets an ISCN-ID on hand, he/she can only exchange a CID from LikeCoin chain, and this situation is going back to our original problem which distributing ISCN metadata is not a strength of LikeCoin chain.

To tackle this problem, we will introduce another component from IPFS stacks that call InterPanetary Name System (IPNS). LikeCoin chain will then be not only an IPFS datastore for ISCN metadata but will also hold multiple IPNSs to serve the ISCN metadata. Precisely, IPNS keeps a linkage between a name and the corresponding CID; hence LikeCoin chain can set up an ISCN-ID that maps to its CID and the user can directly query ISCN-ID by IPFS now. We will do a proof of concept on this topic in the future and will share our findings after it.

IPFS is a distributed file system that guarantees the availability of the querying service, and the use of CID can let the end-user do not need to worry about the integrity of the data. Furthermore, if the IPNS is one day deploys on the LikeCoin chain, then for the users who want to query the ISCN metadata, they will only need to deal with IPFS and do not even need to know the existence of LikeCoin chain.

After combining LikeCoin chain and IPFS, the ISCN metadata can be effectively stored for and distributed to the content creators and all the audience of the digital content.