Hive Punks - Enhanced Market History

A while ago I put the Hive Punks Market History update on hold due to Hive Engine not having complete information to faciliate it.

Hive Engine has built-in market history that I use to feed the current https://punks.usehive.com/markethistory page but it is truncated every 24 hours. This is why the market history is limited to the last 24 hours.

I've always planned to replace this with my own market history but when I looked into doing this I realized Hive Engine does not provide the necessary information to do this in the NFT Market settlement operation.

For example, this is a recent Hive Punk sale as reported by Hive Engine.

{
  "transactions": [
    {
      "block_num": 14259607,
      "ref_steem_block_num": 61246825,
      "trx_id": "8c8335f66ca381188c113980e2c11e3333a5ff21",
      "sender": "handtalk5",
      "contract": "nftmarket",
      "action": "buy",
      "payload": {
        "symbol": "PUNK",
        "nfts": [
          "765"
        ],
        "market_account": "blockheadgames",
        "exp_price": "17.50000000",
        "exp_price_symbol": "SWAP.HIVE",
        "is_signed_with_active_key": true
      },
      "logs": {
        "events": [
          {
            "contract": "tokens",
            "event": "transfer",
            "data": {
              "from": "handtalk5",
              "to": "blockheadgames",
              "symbol": "SWAP.HIVE",
              "quantity": "0.70000000"
            }
          },
          {
            "contract": "tokens",
            "event": "transfer",
            "data": {
              "from": "handtalk5",
              "to": "blockheadgames",
              "symbol": "SWAP.HIVE",
              "quantity": "0.17500000"
            }
          },
          {
            "contract": "tokens",
            "event": "transfer",
            "data": {
              "from": "handtalk5",
              "to": "supportbase",
              "symbol": "SWAP.HIVE",
              "quantity": "16.62500000"
            }
          },
          {
            "contract": "nft",
            "event": "transfer",
            "data": {
              "from": "nftmarket",
              "from_type": "c",
              "to": "handtalk5",
              "to_type": "u",
              "symbol": "PUNK",
              "id": "765"
            }
          },
          {
            "contract": "nftmarket",
            "event": "hitSellOrder",
            "data": {
              "symbol": "PUNK",
              "price_symbol": "SWAP.HIVE",
              "account": "handtalk5",
              "owned_by": "u",
              "sellers": [
                {
                  "account": "supportbase",
                  "owned_by": "u",
                  "nft_ids": [
                    "765"
                  ],
                  "payment_total": "16.62500000"
                }
              ],
              "payment_total": "16.62500000",
              "market_account": "blockheadgames",
              "fee_total": "0.70000000",
              "agent_account": "blockheadgames",
              "agent_fee_total": "0.17500000"
            }
          }
        ]
      },
      "executed_code_hash": "a3060c1e7365ee3f2d2356de4898c6c4a6e31c7416ea5c5697ba8d88f3c1ab0d815ad74975ef2099fb0d332f42eb5503e27c52c78002f3d408fb0ca67bc37edb815ad74975ef2099fb0d332f42eb5503e27c52c78002f3d408fb0ca67bc37edb815ad74975ef2099fb0d332f42eb5503e27c52c78002f3d408fb0ca67bc37edb389babf88e5a3f8966d947525fc60bee5048ec4fa6abf3724198bffa30f7e5a9",
      "hash": "880649ae06f9adf242073e3f0270fb16210d422306f83cb3933016c19c1ab6e7",
      "database_hash": "ded0e813cdac6c98ff9a815f022b9a5d2409d07dce1319f6d668af88de3964b7",
      "timestamp": "2022-01-25T21:05:24.000Z"
    }
  ],
  "query": {
    "start": "2022-01-25T23:41:16.032+00:00",
    "elapsed": 0.000123584,
    "params": {
      "controller": "transactions",
      "action": "show",
      "trx_id": "8c8335f66ca381188c113980e2c11e3333a5ff21",
      "format": "json"
    }
  }
}

Looking at this is pretty overwhelming, but we are mostly only interested in the hitSellOrder operation in the nftmarket contract.

You can see this section here:

          {
            "contract": "nftmarket",
            "event": "hitSellOrder",
            "data": {
              "symbol": "PUNK",
              "price_symbol": "SWAP.HIVE",
              "account": "handtalk5",
              "owned_by": "u",
              "sellers": [
                {
                  "account": "supportbase",
                  "owned_by": "u",
                  "nft_ids": [
                    "765"
                  ],
                  "payment_total": "16.62500000"
                }
              ],
              "payment_total": "16.62500000",
              "market_account": "blockheadgames",
              "fee_total": "0.70000000",
              "agent_account": "blockheadgames",
              "agent_fee_total": "0.17500000"
            }
          }

From here I can collect the seller, buyer, NFT ID, and price. Here is where the problem comes in though, the price is specified as payment_total, this is the amount that goes to the seller without factoring fees.

If you are a developer, you will also notice "sellers" is an array, meaning multiple Punks can be in the same transaction. Yet the fee (both market and agent) are in the parent section of the transaction bundle, meaning it is shared with all the NFT sold. In English, this means it is possible someone can buy two Hive Punks in the same transaction with a single set of fees for both NFTs.

Almost all Hive Punks sold are sold one at a time, so this shouldn't be a problem, but it is possible more than one is sold at once, as such it has to be accounted for.

You would think you could just add the payment_total for all the NFT's sold, figure out what percentage of total each NFT is, and then split the fees that way. This in theory should be fine, but it is possible two Punks were put on the market through two different interfaces and have different market fees. There is a minimum market fee allowed by the contract but other interfaces can change this. Also Punks launched with a 10% market fee but are now at 5%. If one user put two Punks into their basket and they have different market fees, there is no way to calculate the final sell price for either. This situation isn't likely, but it is possible, so it has to be handled.

Because of this complexity, I haven't been able to do an improved market history as I was originally hoping for. I also didn't expect all Punks to sell out in 36 hours so I figured I'd have a lot more time to add in features.

I recently started to revisit dealing with the market history problem, and come up with a solution. I managed to get a dump of all Hive Engine blocks since the launch of Hive Punks (~2 months) and modified my market history fetcher to process these blocks and strip out the Hive Punk market sales. To date, there have been 1,218 Punks sold on the market. I am going to process these market orders and enter them into a database to drive the new Market History.

Did you know 2 months of Hive Engine blocks came out to 59 Gigabytes?

As I build my own local database for the market history, a lot of new possibilies open up. I can do things like show market volume for more than just 24 hours, average sell price, and potentially even more complex analytics like average sell price per rarity.

I was planning on analyzing the current market sales when I put them into a database and see if there have been any trades that would prevent using the payment_total + market fee + agent fee formula to calculate the final price accurately. If not, I will be able to continue to use the final sale price on the new Market History.

I also got word there is a pull request for Hive Engine that will this problem going forward by including the final sale price in the settlement operation.

This change means you don't have to calculate the final sale price and hope all NFTs sold in the transaction have the same market fee percentage. The change isn't live yet and only affects transactions going forward, and will not affect any in the past. I will still need to use my current code to get the new Market History database up to date.

This change will also help other dapps that use NFTs to have this data as well.

If you haven't gathered already, this is a new Market History coming out that will go back further than 24 hours, I will let everyone know when it is ready.

_{Thanks @doze!}

Posted Using LeoFinance ^Beta