Search for a command to run...
By the time this article was written, some of the modules were not yet stable - This is why you will find in HelloPeers package.jsons a couple of dependency references from git - but we have plans of editing this guide so that it mostly uses the core Holepunch Modules. The only 3rd party library we intend to leave as is is hyperdeebee (Because we have no intention of doing or hyperbee indexing from scratch).
Your comment also indicates that it might not be obvious how to find the link as instructed above. We have therefore gone ahead and explicitly included a section called The Repo in which we pasted the link and gave a few instructions on how to test it. Again we Appreciate your reactions. Thanks Nick Thomas
You are handing over the keys to your entire digital life just to send a single PDF. And it’s time to stop.
What our first week actually showed—and what it did not prove.
This is why we built SEYFR: you deserve to stay in charge of your most important files.
The Bank Card and Bank Account Analogy
Letting Child Components Talk to Parents
Memorial Day was bliss for our team, we found a ward into a dimension of software development we'd all thought was fictitious before - The World of Local-first P2P software development on the web. A world where your data is yours despite the cloud, a world of real collaboration, ownership and full availability of your data. We found Holepunch!
Holepunch is an app development platform that makes it easy to build, ship, and update fully P2P desktop/mobile/(even web) apps without using any cloud hosting servers. As you might have probably heard, working with Holepunch's tools in the browser is not common. Many believe it's impossible. Vih and I set out to debunk this when we built a demo to-do web app called hello-peers. You should clone it and run it (both the server and client) to see it in action!
I am happy to show you, but I strongly recommend that you check out the tech stack description in the GitHub repo. Here, I will take a precise approach to explain the demo's implementation and point out the specific semantics upon which the demo was built so you can follow it to build anything you want.
Create a Hyperswarm DHT and add relay it over framed streams when a WebSocket connection is established. Your WebSocket implementation is up to you.
import Stream from '@hyperswarm/dht-relay/ws';
import { relay } from '@hyperswarm/dht-relay';
import DHT from 'hyperdht';
const dht = new DHT();
// .... websocket server implementaion => wss
wss.on('connection', function (ws) {
relay(dht, new Stream(false, ws));
})
On the client side, grab the DHT from the WebSocket connection (whatever implementation) and create a HyperSDK to use for the rest of your hyper-core implementation on the client.
import DHT from '@hyperswarm/dht-relay';
import Stream from '@hyperswarm/dht-relay/ws';
import * as SDK from 'hyper-sdk';
const dht = new DHT(new Stream(true, socket));
const sdk = await SDK.create({
storage: false, // by choice we did not persist any data
autoJoin: false, // we wanted fine control over peer discovery
swarmOpts: {
dht // passed the dht to the hyperswarm constructor
}
})
And that's it. You got yourself a Hyper SDK that combines all the lower-level pieces of the Hyper stack into high-level APIs that you can use across platforms (web inclusive) and can now focus on your application's business logic.
Managing this is a choice we made when we turned autoJoin off on our SDK. Your use case may be different but if like us you asked for fine-grained control over peer discovery, here is how you would do it. Otherwise, just flip autoJoin to true.
There are several ways to discover nodes, you can do that by name, DNS, or a 32-byte buffer token. In our case, we used the later
//...
// @ts-ignore
import goodbye from 'graceful-goodbye';
// import * as BufferSource from 'buffer/'
import b4a from 'b4a';
// ... sdk generation code above
const topicBuffer = await crypto.subtle.digest('SHA-256', b4a.from('say a good hello', 'hex')).then(b4a.from);
// discovery here is a hypercore
const discovery = await sdk.get(topicBuffer);
discovery.on('peer-add', peerInfo => {
console.log('new peer, peer:', peerInfo, 'peer count:', discovery.peers.length);
});
// const db = await createMultiWriterDB(sdk, discovery)
// goodbye(async () => {
// await db.close()
// await discovery.close()
// await sdk.close()
// })
// const todoCollection = db.collection('todo')
// await todoCollection.createIndex(['text'])
// await todoCollection.createIndex(['done', 'text'])
// resolveReady()
//
// createWatcher()
sdk.joinCore(discovery).then(() => console.log('discovering'));
A Hypercore is a secure, distributed append-only log built for sharing large datasets and streams of real-time data but we use the Hyperbee which is a b-tree running on a Hypercore and makes operations on it more efficient. Both Hypercore and Hyperbee can only have a single writer on a single machine; their creator is the only person who can modify it because they're the only ones with the private key, this is where Autobase comes in handy to transform these higher-level data structures into multi-writer data structures which facilitate two way features like collaboration.
For hello-peers, each peer needed an input and output core but since their roles are interlinked, we needed to remove the responsibility of managing custom storage/replication code from these independent higher-level modules so they are easier to manage so we used CoreStore; a Hypercore factory that makes it easier to manage large collections of named Hypercores and created an Autobase off that as you can see below:
import Autobase from 'autobase';
import Hyperbee from 'hyperbee'
// ....
export async function createMultiWriterDB (sdk, discoveryCore, { extPrefix, name } = defaultMultiWriterOpts) {
//Get back a namespaced Corestore instance
const IOCore = await sdk.namespace(name);
/*
UPGRADE:
No more there is no need to get the input and output cores
as autobase does that for you now.
const localInput = IOCore.get({ name: 'local-input' });
const localOutput = IOCore.get({ name: 'local-output' });
goodbye(async () => {
await Promise.all([localInput.close(), localOutput.close()]);
})
await Promise.all([localInput.ready(), localOutput.ready()]);
const autobase = new Autobase({ localInput, inputs: [localInput], localOutput })
*/
/*
If loading an existing Autobase then set bootstrap
to base.key (as second argument),otherwise pass bootstrap as null.
*/
const autobase = new Autobase(IOCore,null,{
apply: async (batch, view, base) => {
// Add .addWriter functionality
for (const node of batch) {
const op = node.value
if ('id' in op) {
console.log('\rAdding writer', op.key)
await base.addWriter(b4a.from(op.key, 'hex'))
}
}
// Pass through to Autobee's apply
await Autodeebee.apply(batch, view, base)
},
open: (store)=> {
const core = store.get(name)
return new Hyperbee(core, {
extension: false,
})
}
});
const localBee = new Autobee(autobase)
await localBee.ready() // multi-writer bee here
// ...
class Autobee {
constructor (autobase, opts = {}) {
this.autobase = autobase
this.opts = opts
/* UPGRADE: Needed here anymore - the view was created in Autobas'e open mthod
// use autobase to transform its hyperbees into multi-writer bee
if (!opts.sub) {
this.autobase.start({
unwrap: true, // this.bee.get calls will return full nodes
// The apply function to create the linearized Hypercore (this.bee)
apply: applyAutobeeBatch,
view: (core) =>
new Hyperbee(core.unwrap(), {
...this.opts,
extension: false
})
})
*/
this.bee = this.autobase.view
}
}
// MOVED: Into Autobee
// A real apply function would need to handle conflicts, beyond last-one-wins.
static async apply (batch, view) {
const b = view.batch({ update: false })
for (const node of batch) {
if('id' in node.value) continue;
const op = BSON.deserialize(node.value)
const bufKey = getKeyBufferWithPrefix(op.key.buffer || op.key, op.prefix.buffer || op.prefix)
if (op.type === 'put') {
await b.put(bufKey, op.value.buffer)
}
if (op.type === 'del') {
await b.del(bufKey)
}
}
await b.flush()
}
addInput (input, opts={indexer
: true}) {
const key = input.key;
return this.autobase.addWriter(key, opts)
}
removeInput (key) {
return this.autobase.removeWriter(key)
}
put (key, value, /*opts={}*/) {
const op = b4a.from(
BSON.serialize({ type: 'put', key, value, prefix: this.bee.prefix })
)
return this.autobase.append(op)
}
del (key, /*opts = {}*/ ) {
const op = b4a.from(
BSON.serialize({ type: 'del', key, prefix: this.bee.prefix })
)
return this.autobase.append(op)
}
// ... methods to manage the multi writer bee
}
/**
A linearized view is a "merged" view over the inputs giving you a way of interacting with the N input Hypercores as though it were a single, combined Hypercore.
*/
function getKeyBufferWithPrefix (key, prefix) {
return prefix ? b4a.concat([b4a.from(prefix), b4a.from(key)]) : b4a.from(key)
}
Hyperbee is super useful for storing data and searching for data using its ordered key search. When you set up your keyspace just right, you can query a small subset of a large database and quickly load what you need without having to download the entire dataset or traverse it. There is a module dedicated to explaining how to manually set up these indices for JSON data right but what is the reason for going with the manual process when there is a MongoDB-like database built on top of Hyperbee with support for indexing following the same design? We didn't have any so we just decided to use it. It's called Hyperbeedeebee. What a name! Anyway, you can look into the hello-peers repo to see how we use it through the app and or check out its documentation.
To see the magic. Get a peer outside the network you are connected to and go to the hello-peers repo by clicking https://github.com/jermsam/hello-peers.
Clone the repo, install dependencies for both the server and client, run the server and the client and start collaborating.
We believe that you now have an understanding of how the pieces of Holepuch modules fit together and are ready to dive in and enjoy the excitement with the rest of us. For more information and discussion about Peer to Peer, let's hang out in the Holepunch discord servers. You can also check out our Personal Jitpomi Discord Server if you want to explore more exciting state-of-the-art technologies and maybe join our coding school (JSchool)