My teardown of Beaker and the Dat network.
We’re probably all scrambling to figure out a future for ourselves—either by
hunting around for some shred of promising technology or by throwing up our
hands—shouting and retreating and dreading the shadow of the next corporate titan
descending from the sky. The prolonged obsolescence of distributed protocols like
Bitcoin and torrents means we’re maybe skeptical or jaded about any new
protocols. Maybe we’re just hunkered down in some current online bucket.
And I’ve felt this too—ActivityPub and Secure Scuttlebutt are too complicated.
Tim Berner-Lee’s Solid is—well, I can’t even tell what it is. I don’t want to
hear about blockchain: do we really need a GPU mining war at the center of our new
Web? These are all someone’s idea of a distributed Web, but not mine. IPFS is
really cool—but how do I surf it?
After discovering theBeaker Browser,
a Web browser for the distributed Dat
network, I felt that there was some real promise
here. It was simple—load up the browser, create a website, pass your link
around. There’s not much new to learn. And the underlying technology is solid: a
binary protocol very similar to Git. (As opposed to Secure Scuttlebutt, which is
tons of encrypted JSON.)
I spent four months using Beaker actively: running this website on the network,
messing with the different libraries, trying out the different
I hit a wall. Had a good time, for sure. And I kept seeding my Dats—kept my
sites on the network. The technology was just lovely.
But: you can’t yet edit a website from a different browser (like on a
different computer). This is calledmulti-writersupport—and there is some
talk about this landing by the end of the year. But this is, from what I can
see, the single major omission in Beaker. (It’s not a problem with Dat itself—you
can use a Hyperdb for that—but Beaker hasn’t settled the details.)
So I left Dat alone. I figured: they need time to work this problem out. Beaker
has remained remarkably effortless to use—I’d hate for multi-writer to be tacked on,
complicating the whole thing.
An Imperfect Dat—Cool?
Recently, it occured to me that maybe I don’t need multi-writer. And maybe I
shouldreallybe sure that the rest of Dat is as perfect as I think it is.
So I started working on a limited (but full-featured) app for Beaker, with the
intention of writing up a full ‘review’/‘teardown’ of everything I discover
in the process.
This is my review—and the app is
It occured to me that a Muxtape clone would be a perfecttracer
bulletfor me to push Beaker.
(Muxtape was a 2008 website for sharing mixtapes—minimal design, suddenly
became very prominent, and then was promptly DEMOLISHED by the music industry.)
Muxtape was shut down because it was centralized.If Muxtape had been
distributed, it would be much more difficult (perhaps impossible) to
Muxtape did some file processing.Reading music file metadata
(title, artist’s name) and loading music into the browser’s music player.
Could the app handle this?
The Muxtape home page listed recent mixtapes.This would give me a chance
to use datPeers—a way of talking to others that are using the same site.
Storing song information and order.I don’t have a database, so where
do I put this stuff?
A more general question:What if I upgrade the code?How do I handle
upgrading the mixtapes too?
I also didn’t want to think in terms of social networks. Many of Beaker’s most
advanced apps (like Fritter and Rotonde) are ‘messaging’/‘social’ apps. I
specifically wanted a creation tool that spit out something that was easy to
How would Beaker do with that kind of tool?
A Teardown of The Network
Ok, so how does Dat work exactly? It is simply aunique addressattached to
a folder of files (kind of like a ZIP file.) You then share that folder on the
network and others can sync it to their system when they visit the unique
In the case of Duxtape, the address is
The full folder contents can be viewed
So when you visitDuxtape, all that stuff is
downloaded. Beaker will show you theindex.html, which simply lets you
create a new mixtape and lists any that you’ve encountered.
Now, you can’t edit my Dat—so how do you create a mixtape?? And how does it
keep track of other mixtapes?? Teardown time!
- CREATING A MIXTAPE
This creates a new Dat (new folder on your computer)
with justindex.htmlinside. I actually copy thetape.html
from my Dat into that folder, your mixtape. That HTML file will load its
upgrade my original Dat—and upgradeYOURDat automatically—cool, but…
- DISCOVERING A MIXTAPE
home page in an HTML iframe—passing the link to that page. The link is then
stored in ‘localStorage’ for that page. So, those are kept in a kind of a
cookie. Nothing very server-like about any of that.
But furthermore: when you are on the Duxtape homepage, your browser will
connect to other browsers (using datPeers) that are viewing the homepage.
And you will trade mixtapes there. Think about this: you can only discover
those who happen to be around when you are! It truly acts like a street
corner for a random encounter.
- ALTERING A MIXTAPE
Where are song titles and song ordering kept? Well, heh—this is just kept in
the HTML—in yourindex.html. Many Beaker apps keep stuff like this in a
JSON file. But I felt that there was no need for duplication. (I think the
IndieWeb has fully corrupted me.) When I want to read the mixtape title, I
load theindex.htmland find the proper tags in the page.
(Like:span.tape-title, for instance.)
- PUBLISHING A MIXTAPE
Beaker has a special technique you can use for batching up edits before you
publish them. (See the
Basically, you can create a temporary Dat, make your changes to it, then
either delete it or publish it.
However, I didn’t go this route. It turned out that I could batch up all my
changes in the browser before saving them. This includes uploaded files! I
can play files in the browser and read their data without copying them to the
Dat. So no need to do this. It’s a neat feature—for a different app.
So this allows you to work on your mixtape, add and delete songs, get it
perfect—then upload things to the network.
This all worked very well—though I doubt it would work as well if you had
1,000 songs on your mixtape. In that case, I’d probably recommend using a
database to store stuff rather than HTML. But it still might work well for 1,000
songs—and maybe even 1,000,000. This is another advantage to not having a
server as a bottleneck. There is only so much that a single person can do to
overload their browser.
For reading song metadata, I used the
library—yes, I actually parse the MP3 and OGG files right in the browser! This
the server, now all of that good stuff can move into the browser and the whole
app doesn’t need a server—in fact, WebAssembly makes Dat even more compelling.
Special Feature: The DatArchive Object
Lastly, here are some calls that I used which are specific to the Beaker
Browser—these are the only differences between running Duxtape in plain Chrome
and running it distributed:
stat: I use this to
check if a song file has already been uploaded.
readFile: To read
theindex.htmlwhen I need to get song information.
writeFile: To save
changes to songs—to publish theindex.htmlfor your mixtape.
unlink: To delete
songs—NOTE:that songs are still in the Dat’s history and may be downloaded.
isOwner: The getInfo() above also tells me if you are the owner of this
mixtape. This is crucial! I wanted to highlight this—I use this to enable
mixtape editing automatically. If you don’t own the mixtape, you don’t see
this. (All editor controls are removed when theindex.htmlis saved back
So this should give you a good idea of what Dat adds. And I just want to say:
I have been wondering for awhile why Dat has its own special format rather than
just using something like Git. But now I see: that would be too complex. I am
so glad that I don’t have topull()andcommit()and all that.
I spent most of my time working on the design and on subtle niceties—and
that’s how it should be.
Peeling Back the Good and Bad
It’s clear that there are tremendous advantages here:Dat is apps without
death.Because there is no server, it is simple to both seed an app (keep it
going) and to copy it (re-centralize it). I have one central Duxtape right now
(duxtape.kickscondor.com), but you could easily fork that one (using Beaker’s
‘make editable copy’ button) and improve it, take it further.
The roots of ‘view source’ live on, in an incredibly realized form. (In Beaker, you
can right-click on Duxtape and ‘view source’ for the entire app. You can do this
for your mixtapes, too. Question:When was the last time you inspected the code
hosting your Webmail, your blog, your photo storage?Related question:When was
the first time?)
In fact, it now becomes HARD:IMPOSSIBLE to take down an app. There is no app
store to shut things down. There is no central app to target. In minutes, it can
be renamed, rehashed, reminified even (if needed)—reborn on the network.
This has a fascinating conflict with the need to version and centralize an app.
Many might desire to stay with the authoritative app—to preserve their data,
to stay in touch with the seeders of that central app. But this is a good
tension, too—itINSISTSon backwards compatibility. I am pressured to keep
Duxtape’s conventions, to preserve everyone’s mixtapes. It will be difficult to
upgrade everything that is possibly out there.
This same pressure is reminiscent of the Web’s own history: HTML that ran in
1995 often still runs today—Flash and Quicktime are quite the opposite, as
will be all of the native apps of today. (Think of apps you’ve bought that are
already outmoded.) The ‘view source’ keeps compatibility in check. If Beaker
is able to keep their APIs firm, then there is real strength here.
Still, Dat is limited. Where is it short? Can we accept these?
- It truly RESISTS centralization.This becomes starkly apparent when you are
working on your app—you cannot connect to a REST web service. You need to
rethink everything. This is good—but it is painful.
- Discovery suffers.This is related: I cannot just advertise published
mixtapes to a central web server that stays up all night showing off how
busy things are. But, as I mentioned above (in the PUBLISHING A MIXTAPE section,) the
datPeers feature has really helped assuage this sore spot.
- Not everything can be stored in the browser.How does a search engine
work on this network? Or is this type of centralization something we should
resist? (I do offer search on my Dat-version of this website, by leaning on
- Inter-app communication is hard.Earlier I mentioned that I need to use
an HTML iframe to communicate with the Duxtape home page—there is no need
to use theFetch
Beaker, ever. DatArchive deprecates it. (Though I would be interested to see
a use for the Fetch API—if a Dat could house a GUI-less service, to
negate the need for iframes.)
- The multi-writer problem.Again: you cannot edit a Dat from a second
But—think about this: I don’t have to take on cloud hosting! I don’t need to
scale the app! This is a huge relief. URGENT QUESTION:Why are we trying to even do this?
I also mentioned not needing the multi-writer feature. Obviously, multi-writer
demands some centralization. A central Dat needs to authorize other Dats. But I
think this centralization could be moved to the DNS resolution—basically, if
I edit Duxtape on a second machine, it will have a newunique address—and I
can pointduxtape.kickscondor.comto that new address. This means I can never
get locked out of the Dat—unless I am locked out of the DNS. (So there is a
way forward without any new features.)
Still, these downsides are pure side effects of a distributed Web. These are the
realities we’re asking for—for me, it’s time to start accepting them.
Several months had passed since I last used Dat—how was it doing with adoption?
Well, it seems, no different. But it’s hard to say for a distributed network.
Every Dat runs in secret—they are difficult to find. The discovery problems
are perhaps the most urgent ones.
But there is good recent work:
- Cabal: Not a browser project. Just an IRC-like network
on Dat. There is very active work on all of
- Data Terra Nemoand theDWeb Camp
show that ‘distributed web’ stuff has momentum. Beaker seems to have a solid
presence at these.
- Unwalled Garden: The developer of Beaker is
dabbling with… social networks. This is probably needed, though. But I would
hope for more work on multi-writer, on sparse downloading, on different modes
of seeding (like it would be cool to have a ‘vacuum’ type mode—where you
only seed the latest,) or on BitTorrent integration. (I wish I could just
serve large files with BitTorrent and mix dat: and magnet: links!)
- The Dat project’sbloghas all kinds of
academic and hobbyist work going on.
These are all cool—but Dat has a long way to go. With thecorpypastas (or CorpASAs)taking
up all the attention, adoption is terribly slow. What Beaker may need most of
all is a mobile version. But, hey, I’ll write my article here and make my
dent—if you feel stimulated to noise about, then please join in. I mean: using
a new web browser is just very low effort—perhaps the lowest. You need to use
HTTPS has proven itself well for the centralized stuff. Perhaps there is a
future for HTTPS as simply a collection of centralized REST APIs for things
like search and peer discovery. I think the remaining apps could migrate to this
fertile garden emerging on the Dat network.