Solana Hackathon Submission for Half Baked
Half Baked is a no-code oracle publishing tool and public library for Solana built with Rust and React.
Last week, I went into detail about the story behind this project and the vision for the Solana hackathon submission. Basically, I wanted to learn more about on-chain programs and decided to use the concept of an oracle for the price of weed as the example use case. I was surprised how complicated it was to get data from cluutch.io into a Solana data account.
My submission for the hackathon is a proof of concept web3 app where users can create oracles without using any code.
Problem
After conducting a review of existing oracle solutions on Solana, a consistent weakness was identified across the options. All existing tools require some level of coding to create a new data source. Secondly, the data oracles are geared towards order book price data and not arbitrary blobs of data.
Solution
The Half Baked oracle populates oracles with data requested from the user’s browser.
Most existing oracles on Solana are resolved through a consensus process involving several decentralized oracle agents. That level of tamper resistance can be critical but is not necessary in all situations. This solution optimizes for ease-of-use at the cost of centralization.
For clarity, cluutch.io is the project name for the effort to create an index for weed prices. Half Baked is our general purpose Solana oracle library. Half Baked was built with the use case of getting cluutch’s data onto Solana in mind.
Code:
Design considerations
The requirements for this project include:
Store a single field on Solana as a string from the most recent API request made to a public server on the Internet.
A single program instance can be referenced on-chain. Users do not have to deploy anything.
Assume trust is not a primary a concern for users. Reputation of data publisher is sufficient.
Users should interact with the Solana program through a web browser.
Last week, I went into detail about alternative designs considered, including treating individual API responses as NFTs, requiring API servers to sign API responses with their Solana key, and piggybacking off of HTTPS certificates. Those solutions would have been more elegant but are beyond my technical understanding at the moment.
In our chosen solution, the user fills out a form with information about the API endpoint they want to create an oracle for, which includes:
API endpoint URL
JSONPath location* of field to extract from JSON response
Description
Image URL
* Credit to Switchboard for introducing me to this standard.
The user’s browser then makes a call to the API and uses that response to create a transaction that the user signs in order to send the information to the Half Baked oracle on Solana.
Half Baked must keep track of two types of information in order to do its job: information about each oracle and meta data about the entire program’s state.
Oracle data accounts
In addition to the endpoint URL, description, JSON path, and icon URL
mentioned above, we also need to store the most recent API response on-chain. The String
data type was chosen in order to give the most flexibility to what can be stored.
This flexibility comes at the cost of simplicity. Every UTF-8 character is made up of one or more 8-bit units. This means that a fixed byte-length will not always correspond to the same readable character length. For purposes of this proof of concept, a higher bound than needed, 100 bytes, was chosen for the size of each field stored by the oracle. Any unused space is padded with 0s.
The address for API data accounts are a combination of:
Base = The user’s public key
Owner = The Half Baked program
Seed = API endpoint URL
This provides a predictable way to lookup oracle information based on the literal endpoint being queried. Many users can also create their own oracles for the same API endpoint without collisions.
Half Baked meta data
Oracle data accounts contain everything we need to know about a given oracle. But when a user first visits the website, they want to be able to browse all existing oracles without having to know the address to any specific one.
The Half Baked meta data account is just a list of the public keys (addresses) of the individual API data accounts for all registered oracles. Each public key is 32 bytes and a 1 byte counter has been added to make it easier to identify the end of the list.
The address for the metadata account is a combination of:
Base = The user’s public key
Owner = The Half Baked program
Seed = half-baked-meta-data
Running the code
Assumes Solana SDK installed. The repo come with a sample keychain and binary. Its recommended that you use these existing files. Instructions to recreate your own are marked with [NOT RECOMMENDED].
Get code
✗
git clone
✗
cd half-baked
Start local validator
✗solana-test-validator
[NOT RECOMMENDED]: Generate a new account. Instead, use the existing admin account in the git repo for consistency.
✗
solana-keygen new -o admin.json
=> HS4DFksToksNYF3fyU8Rxmg2B7sAq5JYhHKzitaVbDCp
Airdrop to new account
✗
solana airdrop --url http://127.0.0.1:8899 --keypair admin.json 1000
Start website. Observe what it looks like with dummy data. `Sync from Solana` will not do anything yet.
✗
yarn start
[NOT RECOMMENDED]: Compile Half Baked oracle
✗
cd program
✗
cargo build-bpf
Deploy oracle
✗ cd ..
✗
solana program deploy \
--url http://127.0.0.1:8899 \
--keypair admin.json \
program/target/deploy/half_baked.so
Prepare the meta data account
✗
cd cli
✗
cargo run -- create-meta-data-account --keypair ../admin.json --url http://127.0.0.1:8899
=> EtmqeZo4r1pFXonVCegK8S5YwcnmG9eEXKj4BVbR5Sun
Prepare the API response data account
✗
cargo run -- create-cluutch-data-account --keypair admin.json --url http://127.0.0.1:8899
=> 52iJqwHvBsUhEaNg3wJJLzNSBzksaVC6kFSV18w215ZJ
Seed the oracle with some data through the CLI.
✗ cargo run -- process-api-payload --keypair ../admin.json --url http://127.0.0.1:8899
From the website, click `Sync from Solana`. You will now see the UI load the entry from on-chain.
OK, so this proof of concept admittedly falls short of our original goals.
Not sending data from browser. I was able to send hardcoded transactions to the oracle from the browser but didn’t have time to link to the web form. Querying the right API endpoint is trivial.
Data account creation also happening outside the web experience.
Meta data account is not fully implemented. The on-chain account currently only supports a single data source.
Should hash endpoint before supplying it as public key seed.
Things I’ve learned since last week
I’ve learned a lot in the past week, trying to finalize this submission. I’m pretty tired, so here is a list of factoids, mostly for my own future reference.
RPC as a service: https://rpcpool.com/instructions/
Transaction serialization in JS: https://solana-labs.github.io/solana-web3.js/classes/transaction.html#serialize
Transaction serialization in Rust: https://docs.rs/solana-sdk/1.6.8/solana_sdk/transaction/struct.Transaction.html#trait-implementations
Upcoming NFT metadata support: https://github.com/solana-labs/solana-program-library/pull/1544
Great source for public APIs: https://github.com/public-apis/public-apis/blob/master/README.md
Future work
If I had more time, I would look into:
Data poster gets paid every time oracle data accessed.
Integration with Name Service for registration.
Oracle as a service. Pay Cluutch to poll the data on your behalf, even when your browser is closed.
Store a longer history. Currently only storing the most recent API response.
Customize what types of fields are stored. Currently assuming a single string value.
Thanks for reading. I’ll see you in two weeks with new content. Time for a break.