Jedi/Sector One's random thoughts - Home

Twonav for iPhone is now available

2009-12-29T21:44:00Z

CompeGPS Twonav is an awesome piece of GPS navigation software, featuring both onroad and offroad navigation.

The iPhone version is now available on the AppStore and it definitely looks like the best GPS navigation app for iPhone ever.

Oh and it features a well-known FTP server :)

Get it while it's hot, it's really worth the price.

Application-controlled browser cache using local storage.

2009-09-12T18:41:00Z

In order to reduce latency and improve the user experience, using client caching is the web development 101.

Setting correct HTTP server-side headers in order to instruct the web browser to keep the content in its local cache, has become straightforward with modern web servers (for static content), frameworks and middleware layers (Rack!).

However, this is only a hint. The actual caching policy remains up to the web browser. Setting HTTP headers doesn't give applications any control nor feedback about what's really going on client-side.

Specifically:

You can't tell whether a resource is already in the client cache or not (ok, you can, but by making extra requests, which is, outside the scope of a demonstration, exactly what you're trying to avoid),
You can't tell whether what you asked the browser to cache was actually cached or not
You can't invalidate a cached resource
You have no way to prioritize cached content. A huge Javascript should probably stay in the client cache as long as possible, while a 20 bytes-long transparent GIF image is no biggie if it has to eventually get reloaded. The only knob you have on the browser cache is a deadline (which is almost always, and should be, "immediately" or "never"), and it doesn't allow any kind of real prioritization.
Some web browsers, like Safari (and UIWebView components) on the iPhone have a ridiculously small cache.
Watching your web server delivering so many 304 replies is driving you nuts.

Fortunately, the HTML5 specs bring an exciting feature: client-side storage. Thanks to localStorage et al., web browsers now provide a convenient way to permanently store big chunks of data, giving applications full control of the data store. While the primary target was offline web applications, client-side storage can also be extremely helpful in a bunch of other situations.

When you think about it, what's the difference between local storage and cached resources? Not much, except that local storage is fully application-controlled, and can solve every issue listed above!

A good example might be a huge Javascript that you really would love to keep cached. It doesn't mean that additional content like pictures shouldn't get cached as well, but you know that this specific script is critical and that users won't be able to see anything but a blank page till this one isn't ready for action.

Here's the trick. Instead of using a script element, this very script can be loaded using XHR, then eval()'d. Since the script is available as a regular string, storing it as an entry in the browser data store is a piece of cake. Checking whether the script is already in the cache is also as easy as checking a key for existence. Invalidation is as easy as deleting the entry. This way, we get a totally user-controlled browser cache.

Here's a real-life example: Geobar

Geobar was specifically designed for iPhone and Android devices. It relies on the OpenLayers scripts + some custom Geoportail scripts. All packed together, we get 1 Mb worth of data that are absolutely mandatory for the page to display. And this is where local storage works wonders.

    function load_scripts() {
        if (window.localStorage["geoapi_js"]) {
            window.eval(window.localStorage["geoapi_js"]);
        } else {
            var xhr = new XMLHttpRequest;
            xhr.onreadystatechange = function() {
                if (this.readyState != 4) {
                    return;
                }
                if (this.status != 200 && this.status != 0) {
                    alert("Error: " + this.status);
                    return;
                }
                try {
                    window.localStorage["geoapi_js"] = this.responseText;
                } catch (e) { }
                window.eval(this.responseText);            
            };
            xhr.open("GET", "geoapi.js", true);
            xhr.send();
        }
    }

This is enough to keep the script in the data store instead of relying on the traditional HTTP cache.

Why not use an HTML5 manifest instead, you may ask? To start with, the manifest still doesn't give you much control about the cached content. And further HTTP requests made from a resource loaded this way have no Referer, which might be a showstopper, as it was the case here.

Of course, the same trick can be used in order to store other kind of resources like stylesheets, and, why not, hex-encoded images.

While HTML5 makes local storage easy and powerful, almost every major web browser out there has some kind of support for local storage for ages: userData (since IE 5.0), globalStorage (since Firefox 2), and for those running behind, Flash comes to the rescue with its Shared Objects.

Is Twitter a blog killer?

2009-09-12T18:35:00Z

Long time, no see.

Granted, this very blog hasn't been updated for ages.

How come?

Well, the main culprit is called... Twitter.

Twitting random crap on the spur of the moment is so much easier than writing blog articles that I didn't really feel like posting new articles here.

Incidentally, it looks like a lot of big twitters I'm following are also somehow slowing down the pace of their posts on their blog.

Is it just because Twitter is the shit, or because people are getting lazy?

Maybe both.

Enter webhooks

2009-06-29T19:59:00Z

From igvita.com:

"With all the recent buzz about real-time web, surely this is the year XMPP/AMQP Publish-Subscribe (PubSub) makes it to the big leagues! Or maybe not. Ejabberd (XMPP), RabbitMQ (AMQP) and other pubsub server implementations have come a long way but they remain cumbersome to setup and maintain, and perhaps more importantly, the clients require special libraries and a steep learning curve. That is not to say that either XMPP or AMQP are doomed for failure, in fact, they will continue to thrive, but there is a great case for a simplified PubSub implementation to cover the ad-hoc cases where a dedicated TCP channel might be an overkill: enter Webhooks.

The best part about Webhooks is that most of us are already familiar with them: callbacks over HTTP. Pioneered by PayPal and Subversion as a way to send real-time notifications to the client, they have found their way into many dozens of products we all use every day. Need pre or post commit hooks for your SVN or Git repository? Both GitHub and SVN support HTTP callbacks. Need a payment alert from PayPal, or an alert when a wiki page is modified? There are webhooks for that too. This simple mechanism allows us to build web services that work together via a simple and ubiquitous protocol we can all understand: HTTP!"

Elliptics Network 2.5.0 has been released

2009-06-25T06:54:00Z

[Elliptics Network 2.5.0] has just been released.

"This is a major milestone in the elliptics network roadmap. System got full support of all essential operations needed for the fully self-contained distributed hash table storage creation.

Elliptics network is an object based distributed storage which supports different kinds of object replication, data deduplication, high-level file-based API and low-level object-based one. All logically complex parts are hidden behind provided API including failover connection processing, routing table maintenance, joining and synchronization protocols, merge strategies and IO itself.

Example applications contain a full-featured IO server and client capable of data replication and parallel reading and failover processing, system statistics gathering tool, notification receiver and history dump utility."

I'm dying to give it a try.

PHP-FPM might get merged into PHP

2009-06-24T17:43:00Z

If you're running a highly loaded web site powered by PHP, you must be using php-fpm, don't you?

While new releases of php-fpm always immediately follow PHP releases, it's still a PITA to always patch the PHP source code at every release.

How come php-fpm hasn't been merged into PHP at the first place? The main reason is an incompatible license. Or rather... was.

Andrei just announced that the license of php-fpm had been changed. It's now the PHP license, and php-fpm can now technically get officially merged into PHP.

Here's a relevant post of the High-performance PHP group.

PHP is still going to suck, but faster :)

Stealing your browser history without Javascript

2009-06-13T22:23:00Z

Here is a cool demo of a clever technique that displays your browser history without using Javascript. Yep, not without any single line of Javascript.

Well done.

Bayon, a fast clustering tool

2009-06-10T21:16:00Z

Just released: Bayon, a simple and fast hard-clustering tool, with support for repeated bisection clustering and K-means clustering.

Feed him a list of documents, optionally with weighted terms, ask for any number of groups you want it to output, and Bayon will do its best to assign documents to groups.

That kind of tool can bring a lot of benefits to Ning-like web sites, forums, etc.

And it seems to work just as advertised.

Yet another gem by Mixi.

Yet some more good presentations

2009-06-06T07:49:00Z

Presentations of the Los Angeles Ruby Conference 2009 are now online.

Here are my picks:

Resource-oriented architectures, and Google Waves
Scaling 'most popular' lists - a very common issue
Mobilize your Rails app
Flying Robot: Unmanned Aerial Vehicles Using Ruby and Arduino
Johnson - is it Javascript, is it Ruby, or? WTF?
PoolParty

Embracing events

2009-05-30T22:39:00Z

Straight away from Railswaycon 2009, here's a great presentation about event-driven and fibers-driven development: Embracing events in Ruby. Neverblock is amazing!

Python zealots can also read the Python counterpart.

Meanwhile, PHP zealots can keep swearing by PHP 6, that's gonna introduce innovative brain-blasting technology: unicode support (woah!) and basic "goto" support (re-woah!).

Coroutines puts an end to latency in SOA and database-driven apps, where apps keep being stupidly stuck waiting for a remote server to answer instead of keeping the ball rolling.

As demonstrated in this presentation, web developpers still have loads to learn from games coders. But it seems to sink in slowly.

ICANN to open up the TLD namespace

2009-05-29T08:34:00Z

It was just announced that the following TLDs are likely to emerge soon:

.RADIO
.ECO (Ecological)
.GREEN (Ecological)
.MOVIE (Movie/Film Industry)
.FAM (Family)
.MUSIC (Music)
.HEALTH (dot health)
.SPORT (dot Sport)
.INDIGI (for indigenous peoples)
.NYC (New York City)
.BERLIN (Berlin Germany)
.PARIS (Paris France)
.BZH (Brittany, a region in France)
.ENG (England, a kingdom in the U.K.)
.GAL (Galicia, a region in Spain)
.MED (Mediterranean)
.LLI (Leonese Language and Leonese Culture)
.GAY
.WEB
.POST
.MAIL (for emails and to control spam)
.GEO (generic geographical locations)
.XXX (Adult Entertainment)
.BCN (Barcelona)
.LAT (Latin America)

Are those really useful or is it just a way for the ICANN to make a quick buck?

A sneak peek at the Google IO conference

2009-05-29T06:38:00Z

Some videos of what happened at the Google IO conference are now online.

Wave really looks like a giant step forward.

Side note about spam prevention in Wave: "In the press conference right after the keynote, a reporter asked about spam prevention. Lars Rasmussen responded that it hasn't been given much thought yet, since it is a closed developer's preview for now, but also mentioned that most likely Wave would use a whitelist option, where you'd have to add a friend/coworker before they could send/invite you to Waves."

Meanwhile, Microsoft just launched a real-time focused version of IE8 using OneRiot components.

UCARP 1.5.1

2009-05-27T22:34:00Z

A new release of UCARP is now available for download.

As a workaround for some OS / setups, that new version adds an option (--nomcast) to use broadcast advertisements instead of multicast.

Thanks a lot to Steve Kehlet and Juan Antonio for bringing in and testing that new feature.

Hardening MacOSX against the Java vulnerability

2009-05-26T08:38:00Z

Marc Schoenefeld writes:

"regarding the regarding OSX java threat CVE-2008-5353 you can either join the current panic, or fix the issue in five minutes yourself.

If you belong to the second group of people you can follow the steps listed here, and also on http://www.illegalaccess.org

Basically the approach takes non-vulnerable classes from a fixed java version (like sun jdk 1.5.0_18 and makes it available to the OSX java class loader, which then fixes the issue). It is a non-intrusive fix, so it does not impair any patch (if any) that apple will rollout.

Get the src.zip of a recent non-OSX java distribution (like Sun Java 5/JDK 1.5.0_18 for Linux)
unzip src.zip java/util/Calendar.java
javac java/util/Calendar.java
zip /somepath/FixedCalendar.jar java/util/Calendar*.class
In ~/Library/Caches/Java/deployment.properties set option deployment.javapi.jre.1.5.0.args=-Xbootclasspath/p\:/somepath/FixedCalendar.jar
Start up a browser, browse to http://www.java.com/en/download/help/testvm.xml, see the dancing duke, open Java Console, press s, you should now see FixedCalendar.jar in the sun.boot.class.path
If you are brave, try the PoC exploit on http://landonf.bikemonkey.org/static/moab-tests/CVE-2008-5353/hello.html, it should give you a bootstrap failure now"

New benchmarks of Elliptics Network published

2009-05-21T22:58:00Z

Elliptics Network is a a fault tolerant distributed hash table object storage, made by the genius who already brang POHMELFS.

A new benchmark of Elliptics Network has been published, and it demonstrates its parallel scalability.

Pretty good. Elliptics Network is yet another project to keep an eye on.

Rados, from the Ceph project is also moving on, btw, but it's still not production-ready yet.