Creating a private Ethereum blockchain and using it as a model

Recently there’s been a lot of fascination surrounding blockchains, and how they can be used to improve long-standing industry technologies. Because of this, we’ve been investigating the applications of blockchains in messaging platforms. I’ve developed a blockchain-based messaging web-app which treats the blockchain as the main data model to gain some hands-on experience working with a blockchain.

In this post I’ll detail my process and rationale behind creating a private Ethereum blockchain and a web-app which interacts with it, with the hopes of guiding anyone who’d like to try it for themselves. No Ethereum experience necessary!

You can check out my proof-of-concept web-app here, which contains the finished web-app and a simple readme with instructions for running the Ethereum network.

Creating a private Ethereum network

The first thing that’s needed to start playing around with Ethereum is a way to leverage the Ethereum protocol, which gives access to the tools needed to interact with Ethereum networks. There are C++ and Python implementations of the protocol, but I’ll be using Geth (the Go implementation) as it can be interacted with through the CLI, making it quick and easy to use. Even though I’m using Geth, any implementation can be used as they should provide similar capabilities.

Before we can create the Ethereum network, we’ll first need to create its blockchain (starting with a genesis block).

Creating the genesis block

A blockchain is a data structure which builds upon itself over time, which means that it needs an initial structure to build upon - the genesis block! This block is defined by a genesis state file, which describes both the block’s parameters and some parameters which affect the operation of the Ethereum network.

I initialized the genesis block using a state file named genesis.json in my working directory. Here’s what my genesis.json file contains:

{
    "config": {  
        "chainId": 731, 
        "homesteadBlock": 0
    },
    "difficulty": "0x400",
    "gasLimit": "0x8000000",
    "alloc": {}
}

This is a fairly minimal genesis state file, as I’ve only defined the parameters that I need for my particular setup. I’ll give some explanation on what each parameter does:

• config: An object containing the configuration of Ethereum. It doesn’t configure anything specific to the genesis block itself, but it defines how the Ethereum network will run.

  • chainId: Identifies which blockchain Ethereum will be using. There are a few different important chain IDs, like the ID of the mainnet’s blockchain (1). I chose a random 3 digit ID.
  • homesteadBlock: 0: Ensures that we’ll be using the latest release of Ethereum, homestead.
• difficulty: Determines how difficult it is for a block to be mined. The value is the reciprocal of the probability that mining the block succeeds (in hexadecimal). The value I chose (0x400, 1024 in denary) is fairly low, as quicker mining means that transactions can be committed to the blockchain with less latency.
• gasLimit: The maximum number of computations any following block can support (in hexadecimal), chosen as an arbitrarily high number that shouldn’t be reached.
• alloc: An object allowing you to allocate Ether to specific accounts. I didn’t pre-allocate any Ether, as my blockchain was set up to have no transaction fees and my transactions won’t contain any Ether.

Initializing the blockchain

Once you have a genesis state file, setting up the blockchain is easy-peasy. I ran the following command in my working directory to create the blockchain based on my genesis state file in a directory named blockchain:

geth -datadir ./blockchain init ./genesis.json

If all goes well, the output looks something like this:

INFO [06-19|13:12:14] Maximum peer count                       ETH=25 LES=0 total=25
INFO [06-19|13:12:14] Allocated cache and file handles         database=<my-working-dir>\blockchain\geth\chaindata cache=16 handles=16
INFO [06-19|13:12:14] Writing custom genesis block
INFO [06-19|13:12:14] Persisted trie from memory database      nodes=0 size=0.00B time=0s gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [06-19|13:12:14] Successfully wrote genesis state         database=chaindata                                                                  hash=d1a12d…4c8725
INFO [06-19|13:12:14] Allocated cache and file handles         database=<my-working-dir>\blockchain\geth\lightchaindata cache=16 handles=16
INFO [06-19|13:12:15] Writing custom genesis block
INFO [06-19|13:12:15] Persisted trie from memory database      nodes=0 size=0.00B time=0s gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [06-19|13:12:15] Successfully wrote genesis state         database=lightchaindata                                                                  hash=d1a12d…4c8725

Creating the network

After creating the blockchain data, running a private network for the blockchain is simple thanks to Geth. Here’s the command I used to initialize my private Ethereum network (replacing <device name> with my device’s name):

geth --datadir ./blockchain --networkid 7314 --ws --wsorigins "http://localhost:8080,http://<device name>" --wsapi="eth,web3,personal,miner" --gasprice 0

I’ll explain why I passed those specific parameters below (see here for a full list of viable options):

• datadir: The directory containing the blockchain data (created by initializing the blockchain).
• networkid: Identifies the Ethereum network, I chose a random 4 digit ID to minimise clashing with other networks (see here for a list of public network IDs).
• ws: Enables the WS-RPC server, which allows (insecure) websocket connections to be opened with this server. I’ll be using websockets to communicate between the web-app and the blockchain as it can run easily in a browser. To use secure websockets, you’d need a proxy node or similar which responds to secure websocket connections and forwards requests, as Ethereum doesn’t provide support for secure websockets.
• wsorigins: A list of origins from which to accept websocket requests. I’ll be serving my web-app on port 8080, so requests from that port needed to be accepted or else the app wouldn’t be able to communicate with the network. My device’s name also needs to be accepted so that I can connect to the network through the Geth console. Passing * as a parameter will allow any connection, but this is less secure as it allows anyone access to the network.
• wsapi: A list of APIs to allow over websocket communication. I need web3 to expose the Ethereum protocol to my web-app, eth to manage transactions, personal to manage accounts, and miner to mine transactions.
• gasprice: The minimum gas price of a transaction. Transaction fees are calculated as gasprice * gaslimit and paid in Ether. As I want no transaction fees on my transactions, I set the gas price to 0 to allow fee-free transactions to be sent/mined.

With this, the private Ethereum network is running and transactions can be sent and mined to the blockchain! When running successfully, your ouput should look something like this:

INFO [06-19|13:45:55] Maximum peer count                       ETH=25 LES=0 total=25
INFO [06-19|13:45:55] Starting peer-to-peer node               instance=Geth/v1.8.10-stable-eae63c51/windows-amd64/go1.10.2
INFO [06-19|13:45:55] Allocated cache and file handles         database=<my-working-dir>\blockchain\geth\chaindata cache=768 handles=1024
INFO [06-19|13:45:55] Initialised chain configuration          config="{ChainID: 731 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: <nil> EIP155: <nil> EIP158: <nil> Byzantium: <nil> Constantinople: <nil> Engine: unknown}"
INFO [06-19|13:45:55] Disk storage enabled for ethash caches   dir=<my-working-dir>\blockchain\geth\ethash count=3
INFO [06-19|13:45:55] Disk storage enabled for ethash DAGs     dir=C:\AppData\Ethash                                                     count=2
INFO [06-19|13:45:55] Initialising Ethereum protocol           versions="[63 62]" network=7314
INFO [06-19|13:45:55] Loaded most recent local header          number=0 hash=d1a12d…4c8725 td=1024
INFO [06-19|13:45:55] Loaded most recent local full block      number=0 hash=d1a12d…4c8725 td=1024
INFO [06-19|13:45:55] Loaded most recent local fast block      number=0 hash=d1a12d…4c8725 td=1024
INFO [06-19|13:45:55] Regenerated local transaction journal    transactions=0 accounts=0
INFO [06-19|13:45:55] Starting P2P networking
INFO [06-19|13:45:57] UDP listener up                          self=enode://939c12fbde8d140e55800327c8b5fcd71a81a32647b95a89840580f67628cf4a650974af1c28588c1df476400cbaa6d514b23d0d14919f04ece8da52e6f7fcf5@[::]:30303
INFO [06-19|13:45:57] RLPx listener up                         self=enode://939c12fbde8d140e55800327c8b5fcd71a81a32647b95a89840580f67628cf4a650974af1c28588c1df476400cbaa6d514b23d0d14919f04ece8da52e6f7fcf5@[::]:30303
INFO [06-19|13:45:57] IPC endpoint opened                      url=\\.\pipe\geth.ipc
INFO [06-19|13:45:57] WebSocket endpoint opened                url=ws://127.0.0.1:8546

Interacting with the network

With the server running, the network should be accessible through Geth. You can attach a Geth console to the network with the following command:

geth attach ws://localhost:8546

This attaches to the network through the websocket served at port 8546 (the default port for WS-RPC) and provides a javascript console for interacting with it.

If you see the error message: Fatal: Unable to attach to remote geth: bad status accompanied with the following warning in the network output: WARN [06-19|13:58:58] origin 'http://<device name>' not allowed on WS-RPC interface, then you need to pass the given device name to the -wsorigins option when running your network.

Now that we have a console attached to the network, we’ll need to create an account before we can do certain things on the network (like mining and sending transactions). This can be done with the function personal.newAccount(), which prompts twice for a passphrase and then outputs the new account’s address (which acts like an ID for our purposes). The address you are allocated should look something like this: 0xedfc0cea37e9ed3bc7927a81f0dcd393bc57c0e6.

Sending and mining a transaction

To test that the network is set up correctly, we’ll send a dummy transaction and mine it. First, your account needs to be unlocked so that you can send a transaction from it. Calling the function personal.unlockAccount("<your account address>") and entering the passphrase when prompted will unlock the given account, otherwise an error will be thrown.

Next, we’ll send a transaction. The function ethereum.sendTransaction({from: "<address>"}) will send a transaction on the network from a given address. For the full details of what the configuration object can hold see here, but for now you only need to provide the required from field to send a basic transaction. The ouput from the function is the transaction hash (i.e. all of the transaction data hashed together).

After sending a transaction, you can check that it’s visible to the network by calling eth.getBlock("pending"), which outputs the pending block (a virtual block containing transactions which haven’t been committed to the blockchain). If the transactions list present in the block contains the dummy transaction’s hash, that means the transaction is waiting to be mined. You can mine the transaction by calling miner.start(<number-of-threads>) (but remember to call miner.stop() unless you want to be mining forever). While mining, the network output should look like this:

INFO [06-19|14:34:39] Starting mining operation
INFO [06-19|14:34:39] Commit new mining work                   number=1 txs=1 uncles=0 elapsed=0s
INFO [06-19|14:34:43] Successfully sealed new block            number=1 hash=0d5b6c…978671
INFO [06-19|14:34:43] 🔨 mined potential block                  number=1 hash=0d5b6c…978671
INFO [06-19|14:34:43] Commit new mining work                   number=2 txs=0 uncles=0 elapsed=0s
INFO [06-19|14:34:43] Successfully sealed new block            number=2 hash=62f2bd…351578
INFO [06-19|14:34:43] 🔨 mined potential block                  number=2 hash=62f2bd…351578**

After mining for a (very) short while, the transaction should be committed to the blockchain. You can check individual blocks with eth.getBlock(<block-number>) to find the transaction manually (it should be in block 1 if you didn’t mine until after sending the transaction), or you can loop through the possible block numbers to find any non-empty transactions array. If all has gone well, you should find the same transaction hash you saw before in one of the blocks’ transactions arrays. This means that your network is set up correctly!

Creating a lazy miner

The web-app we’ll be making will use transactions as a vehicle for data, so we’ll need to mine new blocks whenever transactions are sent or the data will never be stored on the blockchain. We don’t really want to run a miner non-stop as this is a waste of energy, so I’ve written the script lazyMine.js to subscribe to updates to the set of pending transactions and mine them when necessary:

function checkPendingTransactions() {
    if (eth.getBlock("pending").transactions.length > 0) {
        if (eth.mining) return;

        console.log("Mining pending transactions...\n");
        miner.start(1);
    } else {
        if (!eth.mining) return;

        miner.stop();
        console.log("Mining stopped.\n");
    }
}

eth.filter("latest", checkPendingTransactions);
eth.filter("pending", checkPendingTransactions);
checkPendingTransactions();

I’ll break down how this script works now:

• The function checkPendingTransactions checks if any transactions are pending, and then:

  • If transactions are pending and we’re not already mining, it starts mining on 1 thread.
  • If no transactions are pending and we’re mining, it stops mining.

• eth.filter causes the given function to be called on the specified block (latest is the latest block added to the blockchain, pending is the virtual block of non-committed transactions) when that block is updated.

  • When the latest block updates it means that new transactions may have been mined, so we may have to stop mining if no pending transactions are left.
  • When the pending block updates it means that new transactions may have been sent, so we may have to start mining.
• We then call checkPendingTransactions to mine any non-committed transactions when the script is first run.

We can run this script in a Geth console attached to the network using the following command:

geth attach ws://localhost:8546 --preload ./lazyMine.js

You can test if the script is working correctly with a similar method to how we tested that the network was set up correctly. I recommend leaving this script running in the background for now, as it won’t mine anything until transactions are actually sent.

Now that we have our blockchain and Ethereum network fully set up, it’s time to make the web-app that’ll communicate with it!

Creating the web-app

Since we’re interested in messaging here at MindLink, I’ll be building an MVC messaging app which uses the blockchain as a model. I won’t explain all of the nitty-gritty implementation details behind the app, but I’ll explain everything you should need to interact with the Ethereum network. If you want to see the full codebase for my web-app you can access it here.

Server and packaging

I used webpack to package my web-app, mainly as we use it for development of MindLink and so anything I create in webpack is potentially compatible with our systems. It also has some convenient features/plugins, like webpack-dev-server, which will help speed up development of the app.

Web-app overview

I’ve written a very simple messaging web-app, which allows messages to be represented by individual transactions on the blockchain. In order to stay focused on Ethereum, the web-app is a barebones html/javascript implementation with packages needed to run the server in webpack and nothing more. The application has the following features:

• It can unlock accounts with a given account’s address & passphrase
• It subscribes to blockchain updates (i.e. new transactions)
• It can list all of the accounts on the network
• It displays new transactions as messages
• It can send messages in the form of transactions

When the web-app is first accessed, the user must provide their account’s credentials, which will be used to unlock the account (or create a new one). The account address will also be used as the from field when sending messages. Make sure not to input any important credentials, as they are not encrypted at this stage! After this, the user is greeted with a basic messaging application, containing a message list and a message input, as shown in the screenshot below.

Interacting with Ethereum in Javascript

In order to access the Ethereum protocol in Javascript, I’m using Web3 - a package which provides a Javascript implementation for Ethereum. Thankfully it’s very easy to set it up! You can import the package through npm (or in my case, yarn) and then import and instantiate it wherever you need to use it, like so:

import Web3 from "web3";

const web3 = new Web3("ws://localhost:8546");

This instantiates a new Web3 object, which you can use to interact with the Ethereum network through the given websocket. We can call methods on web3 to interact with the network.

If you’re not using websockets, you can find out what arguments to pass the Web3 constructor here. The reason I decided on using websockets is that my application code is run entirely on the client’s browser, which means that we can’t use IPC as it requires the NPM net package (which isn’t browser compatible).

Ethereum account management with Web3

In order to send transactions, we’ll need to unlock an account first (just like when we sent a transaction from the Geth console). To do this with Web3, you’ll need to obtain the user’s credentials and then pass them to the following function:

web3.eth.personal.unlockAccount(address, passphrase);

This is an asynchronous function (as are any which require a response from the network), so if you need the result before continuing then make sure to use await beforehand. The function will throw an error if the wrong credentials are given, so you should ideally have some error handling in place if your credentials are input by the user.

If you don’t already have an account set up (or you just want to create a new account), you can do this with Web3 as well! First you’ll need to obtain a passphrase, and then you can pass it onto the following async function (which returns the new account address if successful):

web3.eth.personal.newAccount(passphrase);

It can also be useful to get a list of all accounts on the Ethereum network, allowing users to select addresses when needed instead of typing them (as they’re not human-readable). In my messaging web-app, I used this to allow the user to select the recipient of their message from a drop-down box. You can get a list of accounts in the Ethereum network with the following async function:

web3.eth.personal.getAccounts();

Sending transactions with Web3

Sending transactions with Web3 is quick and easy! First you’ll need to construct a transaction object, which contains the fields used by the transaction (see here for the full list of available fields). The only required field is the from field, but I’ll go through all of the fields I used in my web-app to treat a transaction as a message-sending mechanism:

• from: The address the transaction is sent from (must be unlocked). I use the account the user provided for this, and treat it as the message’s sender field.
• to: The address the transaction is sent to. I use this as the message’s recipient field, and obtain the value from a drop-down box (populated by all account addresses in the network) which the user can select.
• value: The amount of Ether transferred for the transaction. I set this value to 0 so that all messaging can be effectively free (and therefore unlimited).
• data: This field can take any hexadecimal string, which is great for sending additional data!

To send my messages in the data field, I first create an object with a message field (which contains the message text). This is useful as if I wanted to send any other data along with the message text, I could simply add another field. I then use JSON.stringify to convert the object into a string, and then web3.utils.asciiToHex to convert the string to hexadecimal. This final value is what I pass in to the transaction’s data field.

Once you have your transaction object, you can send the transaction with the following asynchronous function:

web3.eth.sendTransaction(transaction)

This returns the transaction hash upon completion. The transaction will have to be mined after this, so it won’t appear on the blockchain for a little while. If you still have the lazy miner running then it should appear quickly with no outside intervention. How convenient!

Receiving transactions with Web3

Receiving transactions requires a little more work than sending them. There’s no function to list all transactions, so instead we’ll have to make our own! In order to get all the transactions from the blockchain, we’ll have to iterate over each block and process the transactions array they contain. Here’s the method I use to process transactions (which I’ll explain below):

async function getMessages() {
    let startBlockNumber = lastBlockRead + 1;
    let endBlockNumber = await web3.eth.getBlockNumber();

    for (var i = startBlockNumber; i <= endBlockNumber; i++) {
        var block = await web3.eth.getBlock(i, true);
        if (block && block.transactions !== null) {
            for (let tx of block.transactions) {
                handleTransaction(tx);
            }
            lastBlockRead = block.number;
        }
    }
}

Firstly, the function checks which block number it should start from (using a variable lastBlockRead). This is useful as there’s no point re-processing blocks you’ve already processed, so it’s good to keep track of where you got up to. The variable is initialized at -1 (as the first block number is 0). The function then checks where it can read up to using web3.eth.getBlockNumber, which returns the highest block number on the blockchain. You could also read blocks up until a block is undefined, and use that as a loop termination condition.

The function then iterates over the blocks in the blockchain, getting each one with web3.eth.getBlock(i, true). The first parameter to getBlock is the block number, and the second is a boolean which determines whether the transactions array should contain transaction objects (true) or hashes (false). We need the transaction objects to process the data they store, so we must pass in true. Once we have the block, the function simply loops through the available transactions and processes them.

This is great for manually processing all unread transactions, but what if we want to process transactions as they occur? For this, we’ll need to subscribe to certain blockchain events, so that this function is called whenever a new block is committed. To do this we can use the following asynchronous function:

web3.eth.subscribe("newBlockHeaders", callback)

This function takes a string and a callback function. The string defines what kind of events we are subscribing to - "newBlockHeaders" means the callback is called whenever new blocks are committed (see here for the list of event types). The callback is called whenever an event of the given kind occurs, and is passed a block header object or an error. I pass getMessages as my callback function so that the list of messages updates whenever a new block is committed.

With that, you should have all you need to send and receive transactions from a web-app! If you’d like to see my blockchain-integrated web-app, click here.

Conclusions

As you can see, once you have an Ethereum network and blockchain set up it’s surprisingly easy to interact with it from a web-app. Everything you need to use the blockchain as a model is provided through Web3, as you can commit and read back data in the form of transactions. This makes it quite easy to use and quick to set up.

When it comes to the efficacy of an Ethereum blockchain as a data models, there are some obvious problems. The main issue is latency, as any transactions being sent to the blockchain must be mined first, which is not particularly quick in comparison to more conventional data stores.

Another big issue I found was that all data on the blockchain is freely available to anyone with access to the network, and so any user (with or without an account) could read any other user’s data easily. While there are ways to work around this restriction, there are methods built into conventional data stores which handle this issue for you, so this makes using a blockchain as a store look much less appetizing.

I’ve yet to find any practical benefits to using the blockchain as a model. While there are unique systems like mining/transaction fees and smart contracts, I’m not sure that they provide any tangible benefit that couldn’t be replicated on other types of model.

Overall, while this was a worthwhile proof-of-concept to gain experience working with Ethereum, it seems like there aren’t many reasons to use the blockchain as a model in the first place.