The promise of grid computing.

I was reading my weekly haul of industry articles when I came across this article on InformationWeek. They mention in the article how IBM “will provide researches and scientists with the ability to harness supercomputer ower, allowing them to simultaneously analyse in one day a volume of cancer biopsy specimens, or tissue microarrays, that would take 13 decades to do with standard PCs”.

Doing the math, thats nearly 5000 times their current computing power. The interesting thing about this isn’t that its 5000 times faster, but that it can be done in one day. When 130 years of raw analysis can be done in one day it means that scientists have the scope to be more speculative about how they spend that computing time – want to try a different calculation? Its only going to cost you a day – not the rest of your life.

It would be interesting to understand the kinds of calculations that are being performed and how easily it could be distributed on gridenvy.com. One of the things that I’ve always had in the back of my head with this thing is “you might help cure cancer”. Given that the grid hasn’t really grown much I doubt it, but I wonder if people would be more keen to connect if the grid was doing something for the good of humanity?

I might be getting my wish.

I’ve been pretty quiet this week working interstate doing the kind of work that pays the bills. While I’ve been mobile I’ve been checking in every day to see how the grid is growing/shrinking over the previous 24 our period.

As you can see from the graph below the 1st of August was the stronges day with a smidge over 250,000 uploads. This is still far short of what my computer can pump by itself.

My current thinking is that maybe I am going to get my wish and that this grid idea is going to fail fast, mostly because I’ve found it difficult to drive the traffic required to start getting a significant amount of processing power. There have been quite a few suggested improvements to the system which I hope to tackle soon, it will be interesting to see whether they make more people want to join the grid.

Grid sizer code updated.

Just a quick note to let folks know that I’ve made a change to the JavaScript code that is running in the grid. It should provide a more even scrolling experience, perform more processing and use less traffic. On the downside it will probably completely floor the work upload numbers for tomorrow!

The grid, 29 hours in.

Well, it has been 29 hours since I first started up the grid. As the backend dishes out work it also records some basic statistics about the amount of work being done. Probably the most relevant is the number of work uploads that have taken place at specific time intervals. The graph below shows the number of work uploads each our since we went live.

That isn’t terribly impressive and we aren’t anywhere near reaching the computational power of a single desktop computer yet. A big part of the problem is that my program isn’t particularly efficient from a distributed algorithm point of view. The most expensive operation is sending data across the network, and in our case the network is the entire network. In order to nullify the effect of the network inbetween the nodes and the central site the processing that is done at the node needs to exceed the cost of actually sending and receiving the piece of work from it.

I’m working on making the algorithm more computationally intensive on the browser end, but I obviously need to be careful about the impact that has on the browser user.

JavaScript Cluster at Mininova

Looks like the folks at Mininova had a similar idea around building a JavaScript cluster to harness the potential processing power of those visiting their web-site. The comments on the post are possibly the most interesting thing with about 1/3 liking the idea, 1/3 not really caring much either way and the remain 1/3 not being so keen.

From what I can tell the Mininova guys aren’t running the JavaScript cluster anymore so I wonder whether there was a real technical problem or they just lost interest. If someone from Mininova is reading this post I’d love to hear from you!

Fix for iframe bug.

A few hours after launching this evening I got two e-mails, one from Frank Arrigo and another from Geoff Appleby letting me know that the iframe code that I was generating was not correct.

According to HTML 4.0 Transitional it must be a binary tag whereas the code I was rendering was a unary tag. I’ve fixed up the code generator on the join page – if you have already pasted this code into your site I recommend that you update it as soon as possible as it can cause problems with rendering on pages.

Thanks again to Frank and Geoff for getting me this feedback!

What is gridenvy.com?

A little while back I had an idea about how it might be possible to build a massive grid computer harnessing the power of the transient Internet user population. In essence the idea was to use the almost universally deployed JavaScript runtime embedded in modern web-browsers to perform small processing tasks which are part of a larger distributed processing algorithm (I can just hear those bozo bits flipping).

I’ll grant you that its a pretty wacky idea, but is it possible? If it is, what kind of processing could a grid with this architecture perform?

Anyway, one night I got bored and decided that I would prototype the idea, and within a few hours I had a functional prototype. Of course, it was only of limited use because I didn’t have enough machines to really get processing large amounts of work, but the code executed and it was time to show it to some of my peers.

I contacted a few of my colleagues and decided that it was probably worth showing to some of the executives within the company. Over the course of a few weeks it was decided that we would file a patent for the idea and then expose it to the world. The patent process takes months just to get to the provisional patent stage and several times I was close to just bailing on the process and throwing it out there – but I didn’t, and here we are today.

Over the past two days I’ve been taking the prototype code that I wrote and plugging it into a web-site called gridenvy.com. What is gridenvy.com? It is an experiment.

The Experiment

The problem with my idea was that in order to test whether it was useful it requires a massive amount of exposure and the cooperation of those individuals that run web-sites (large and small). When you visit the gridenvy.com site your web-browser starts executing a small piece of JavaScript code – that JavaScript code sends and receives small fragments of XML to and from the central gridenvy.com site.

If you are a content publisher (blogger, major site, other) then you can join your web-site to the grid by embedding an <iframe /> into the site. That frame points to the gridenvy.com site so that whenever someone visits your site they start processing work items from the central site. If people get into the spirit of it then those embedded <iframe /> elements should create an amazing amount of leverage in accessing more processing power.

At the moment the grid isn’t doing anything really exciting, the code simply performs a series of SHA1 hashes on GUIDs and returns the result, but by running this simple program I can start to measure how effective the grid is at accessing the processing power of today’s modern web-browsers.

The Future

The purpose of creating this blog is to create a dialog with folks that are interested in this idea, perhaps from a purely academic perspective, and others from a commercial perspective. Over the coming weeks I will make rolling updates to the code-base at gridenvy.com and of course keep folks abreast of changes here on this blog.

I see two outcomes, either its brilliant idea and it’ll capture peoples imagination or it is a completely crazy idea and it’ll fail. If thats the case I want it to fail fast so that I can move onto wacky idea #12,604.

Your thoughts? Leave there here has a comment, or if you would prefer shoot me an e-mail at mitch.denny@readify.net, or call me on +61 (414) 610–141 (Australia).

Hello, grid!

Sometimes I wonder what the correct output of the trusty old “hello world” program actually is – should there be a comma, an exclaimation mark, what capitalisation style should be used. Which one is it? I’ve seen it implemented several times in each of the ways above. Of course, the title of this blog post is completely wrong, who in their right mind would implement “Hello, grid!”?

Well – that would be me, and this blog is all about learning lessons the hard way. A little while ago I had an idea, and I built a prototype of that idea which showed enough promise to take that idea further. I want to talk about this idea on this blog (soon) and enlist peoples help in seeing if the idea has merit or is simply the product of the unstable imagination of a deranged software developer.

I can’t tell you much about the idea just yet, but I can point you at the rather sad looking launch site and encourage you to subscribe to this feed. Over the coming weeks I will be able to start revealing more about my idea.

P.S. It has something to do with grid computing!