Jedi/Sector One's random thoughts

Skyrock Spot is an iPhone app whose UI mixes native components and web views.

And something was definitely wrong with the latter. Browsing web pages was dog slow, although they all share the same CSS and images.

How come? Nginx was properly configured to send caching headers with static content. And scripts producing dynamic content were correctly sending caching headers as well. To top it off, on the development platform, Webkit's web inspector and other tools didn't find any significant issue with cacheability.

But on the production platform, it was a mess.

For some reason, elements were sometimes served as a 302 code, and on a subsequent request, like one second later, they were not. Even though the expiration date was way off. And the pattern looked like random.

Here were the two root causes.

Anyone with a clue knows that deploying code with something like "svn update" plenty sucks and can have loads of pesky side effects. But the fact that it can annihilate HTTP caching, hence ruin performance, is not a widely documented one.

The static files of the Spot application are served by two hosts. After a new release, the "svn update" command is run on both. Consistency apart (something trivial to fix with a symbolic link), it might sound acceptable.

But there's a catch. While "svn update" effectively deploys the same data on every host it is run on, it doesn't keep the metadata. And in particular, the modification time can differ. And they did.

Here's how cacheable elements ended up as being served with a 200 code:

1st request:

• Client, to Load balancer: "hey, gimme /main.css"
• Load balancer, to Host A: "hey, gimme /main.css"
• Host A: "Here it is. One last thing: Last-Modified at 13:29:12"

2nd request:

• Client, to Load balancer: "hey, gimme /main.css If-Modified-Since 13:29:12"
• Load balancer, to Host B: "hey gimme /main.css If-Modified-Since 13:29:12"
• Host B: "Here's the complete thing, since the local file's modification date is 13:30:23 here"

A simple "touch" command with the same reference date on both hosts immediately solved the caching issues.

Lesson learned:

• deploying code means that data AND METADATA should be identical on every host you deploy on.

• configure your load balancer for stickiness. Both for static and for dynamic content. If, for some reason, an application server has its clock one second off, the caching mess is bound to arise, just like it does with static content.

Here's a big picture of what data structures you can get of some NoSQL data stores, in a JSON representation.

Before picking a data store, try to model your data to these constraints and see what the best fit is.

SQL

{
    "table 1": {
        "key 1": {
            "property 1": "string",
            "property 2": "numerical value"
        },
        "key 2": {
            "property 1": "string",
            "property 2": "numerical value"
        }, ...
    },
    "table 2": {
        "key 3": {
            "property 3": "date"
        },
        "key 4": {
            "property 3": "date"
        }, ...
    }, ...
}

• The set of properties is fixed for every record of a table.
• Any key and/or property can be indexed.

Cassandra

Columns:

{
    "column family 1": {
        "key 1": {
            "property 1": "value",
            "property 2": "value"
        },
        "key 2": {
            "property 1": "value",
            "property 4": "value",
            "property 5": "value"
        }
    }, ...
}

and supercolumns:

{
    "column family 2": {
        "super key 1": {
            "key 1": {
                "property 1": "value",
                "property 2": "value"               
            },
            "key 2": {
                "property 1": "value",
                "property 4": "value",
                "property 5": "value"           
            }, ...
        }, ...
        "super key 2": {
            "key 1": {
                "property 4": "value",
                "property 5": "value"               
            },
            "key 2": {
                "property 1": "value",
                "property 6": "value",
                "property 7": "value"           
            }, ...
        }, ...
    }, ...
}

• Columns families are constrained by a schema. You need to predefine their names, their types (simple columns or supercolumns), and how the keys and properties will be sorted. Changing the schema requires a server restart in < 0.7 versions.
• A database ("keyspace") can mix simple columns and supercolumns.
• Range queries are supported.
• Support for expiration times in >= 0.7
• Distributed, fault-tolerant.
• Never locks, even for writes. Efficiently uses multiple cores.
• Runs on a lot of operating systems (wherever a JVM exists).
• No REST client interface. Uses Thrift serialization.
• Can run Hadoop map/reduce jobs.

MongoDB

{
    "namespace 1": any json object,
    "namespace 2": any json object,
    ...
}

To elaborate, here's an example:

{
    "namespace 1": [
        {
            "_id": "key 1",
            "property 1": "value",
            "property 2": {
                "property 3": "value",
                "property 4": [ "value", "value", "value" ]
            }, ...
        },
        {
            "_id": "key 2",
            "property5": {
                "property3": "value",
                "property7": { "question": "6x9", "answer": 42, "list": [ 3, 5 ] }
            }, ...      
        }, ...
    ]
}

• No schema, the model is totally flexible. Any property can be added to any object any time.
• Any property can be indexed, at any depth (for example, "property3" could be indexed).
• A wide range of operations can be performed.
• Collections can be capped to a fixed number of bytes or elements.
• Will be distributed and fault-tolerant through a proxy, but this one is still in early stage.
• Write operations lock the whole database. Efficiently uses multiple cores.
• Written in C++ but currently runs on a limited number of operating systems.
• Uses its own protocol, but basic REST proxies do exist (in Python and NodeJS).

Riak

{
    "bucket 1": {
        "key 1": document + content-type,
        "key 2": document + content-type,
        "link to another object 1": URI of other bucket/key,
        "link to another object 2": URI of other bucket/key,        
    },
    "bucket 2": {
        "key 3": document + content-type,
        "key 4": document + content-type,
        "key 5": document + content-type
        ...
    }, ...
}

• Keys are indexed.
• Range queries are not supported.
• Retrieving multiple keys in a single query isn't supported (must use map/reduce).
• Never locks, even for writes. Efficiently uses multiple cores.
• Distributed and fault tolerant.
• Records can be traversed through links.
• Runs on any platform Erlang runs on.
• REST client interface.

Pincaster

{
    "layer 1": {
        "key 1": {
            {
                "property 1": "value",
                "property 2": "value",
                "property 3": "value", ...
            }
        },
        "key 2": {
            {
                "property 1": "value",
                "property 4": "value",
                "geographic location": "latitude, longitude"
            }
        }, ...
    }, ...
}

• Keys and geographic locations are indexed.
• Keys are lexically ordered and prefix matching queries are supported.
• Efficient storage of a lot of small entries.
• Support for expiration times on stored keys.
• High throughput with a single host.
• Writes can block reads, but efficiently uses multiple cores.
• Written in portable C, works on most operating systems.
• No distribution nor fault tolerance.
• REST client interface.

Redis

{
    database number: {
        "key 1": "value",
        "key 2": [ "value", "value", "value" ],
        "key 3": [
            { "value": "value", "score": score },
            { "value": "value", "score": score },
            ...         
        ],
        "key 4": {
            "property 1": "value",
            "property 2": "value",
            "property 3": "value", ...
        }, ...
    }
}

• A database can mix different types of keys.
• Keys and scores are indexed.
• Efficient storage of a lot of small entries.
• High throughput with a single host.
• Support for expiration times on stored keys.
• Collections can be lists or hold distinct elements (sets).
• Rich set of atomic operations.
• No distribution (but through some client libraries) nor fault tolerance.
• Only takes advantage of a single CPU core (but you can run multiple instances on the same host)
• Written in portable C. Runs on most operating systems.
• Uses its own protocol.

Elliptics Network

KumoFS

Flare

{
    "key 1": "value",
    "key 2": "value",
    ...
}

• No range queries.
• Keys are indexed.
• Efficient storage of a lot of small entries.
• KumoFS supports the CAS operation.
• High throughput with a single host.
• Uses Tokyo Cabinet for on-disk storage. Writes can block reads.
• Distributed and fault tolerant.
• Written in portable C and C++, works on most operating systems.
• Memcache (KumoFS, Flare) or REST (Elliptics Network) client interface.

CompeGPS Twonav is an awesome piece of GPS navigation software, featuring both onroad and offroad navigation.

The iPhone version is now available on the AppStore and it definitely looks like the best GPS navigation app for iPhone ever.

Oh and it features a well-known FTP server :)

Get it while it's hot, it's really worth the price.

In order to reduce latency and improve the user experience, using client caching is the web development 101.

Setting correct HTTP server-side headers in order to instruct the web browser to keep the content in its local cache, has become straightforward with modern web servers (for static content), frameworks and middleware layers (Rack!).

However, this is only a hint. The actual caching policy remains up to the web browser. Setting HTTP headers doesn't give applications any control nor feedback about what's really going on client-side.

Specifically:

• You can't tell whether a resource is already in the client cache or not (ok, you can, but by making extra requests, which is, outside the scope of a demonstration, exactly what you're trying to avoid),
• You can't tell whether what you asked the browser to cache was actually cached or not
• You can't invalidate a cached resource
• You have no way to prioritize cached content. A huge Javascript should probably stay in the client cache as long as possible, while a 20 bytes-long transparent GIF image is no biggie if it has to eventually get reloaded. The only knob you have on the browser cache is a deadline (which is almost always, and should be, "immediately" or "never"), and it doesn't allow any kind of real prioritization.
• Some web browsers, like Safari (and UIWebView components) on the iPhone have a ridiculously small cache.
• Watching your web server delivering so many 304 replies is driving you nuts.

Fortunately, the HTML5 specs bring an exciting feature: client-side storage. Thanks to localStorage et al., web browsers now provide a convenient way to permanently store big chunks of data, giving applications full control of the data store. While the primary target was offline web applications, client-side storage can also be extremely helpful in a bunch of other situations.

When you think about it, what's the difference between local storage and cached resources? Not much, except that local storage is fully application-controlled, and can solve every issue listed above!

A good example might be a huge Javascript that you really would love to keep cached. It doesn't mean that additional content like pictures shouldn't get cached as well, but you know that this specific script is critical and that users won't be able to see anything but a blank page till this one isn't ready for action.

Here's the trick. Instead of using a script element, this very script can be loaded using XHR, then eval()'d. Since the script is available as a regular string, storing it as an entry in the browser data store is a piece of cake. Checking whether the script is already in the cache is also as easy as checking a key for existence. Invalidation is as easy as deleting the entry. This way, we get a totally user-controlled browser cache.

Here's a real-life example: Geobar

Geobar was specifically designed for iPhone and Android devices. It relies on the OpenLayers scripts + some custom Geoportail scripts. All packed together, we get 1 Mb worth of data that are absolutely mandatory for the page to display. And this is where local storage works wonders.

    function load_scripts() {
        if (window.localStorage["geoapi_js"]) {
            window.eval(window.localStorage["geoapi_js"]);
        } else {
            var xhr = new XMLHttpRequest;
            xhr.onreadystatechange = function() {
                if (this.readyState != 4) {
                    return;
                }
                if (this.status != 200 && this.status != 0) {
                    alert("Error: " + this.status);
                    return;
                }
                try {
                    window.localStorage["geoapi_js"] = this.responseText;
                } catch (e) { }
                window.eval(this.responseText);            
            };
            xhr.open("GET", "geoapi.js", true);
            xhr.send();
        }
    }

This is enough to keep the script in the data store instead of relying on the traditional HTTP cache.

Why not use an HTML5 manifest instead, you may ask? To start with, the manifest still doesn't give you much control about the cached content. And further HTTP requests made from a resource loaded this way have no Referer, which might be a showstopper, as it was the case here.

Of course, the same trick can be used in order to store other kind of resources like stylesheets, and, why not, hex-encoded images.

While HTML5 makes local storage easy and powerful, almost every major web browser out there has some kind of support for local storage for ages: userData (since IE 5.0), globalStorage (since Firefox 2), and for those running behind, Flash comes to the rescue with its Shared Objects.

Long time, no see.

Granted, this very blog hasn't been updated for ages.

How come?

Well, the main culprit is called... Twitter.

Twitting random crap on the spur of the moment is so much easier than writing blog articles that I didn't really feel like posting new articles here.

Incidentally, it looks like a lot of big twitters I'm following are also somehow slowing down the pace of their posts on their blog.

Is it just because Twitter is the shit, or because people are getting lazy?

Maybe both.

From igvita.com:

"With all the recent buzz about real-time web, surely this is the year XMPP/AMQP Publish-Subscribe (PubSub) makes it to the big leagues! Or maybe not. Ejabberd (XMPP), RabbitMQ (AMQP) and other pubsub server implementations have come a long way but they remain cumbersome to setup and maintain, and perhaps more importantly, the clients require special libraries and a steep learning curve. That is not to say that either XMPP or AMQP are doomed for failure, in fact, they will continue to thrive, but there is a great case for a simplified PubSub implementation to cover the ad-hoc cases where a dedicated TCP channel might be an overkill: enter Webhooks.

The best part about Webhooks is that most of us are already familiar with them: callbacks over HTTP. Pioneered by PayPal and Subversion as a way to send real-time notifications to the client, they have found their way into many dozens of products we all use every day. Need pre or post commit hooks for your SVN or Git repository? Both GitHub and SVN support HTTP callbacks. Need a payment alert from PayPal, or an alert when a wiki page is modified? There are webhooks for that too. This simple mechanism allows us to build web services that work together via a simple and ubiquitous protocol we can all understand: HTTP!"

Read more

[Elliptics Network 2.5.0] has just been released.

"This is a major milestone in the elliptics network roadmap. System got full support of all essential operations needed for the fully self-contained distributed hash table storage creation.

Elliptics network is an object based distributed storage which supports different kinds of object replication, data deduplication, high-level file-based API and low-level object-based one. All logically complex parts are hidden behind provided API including failover connection processing, routing table maintenance, joining and synchronization protocols, merge strategies and IO itself.

Example applications contain a full-featured IO server and client capable of data replication and parallel reading and failover processing, system statistics gathering tool, notification receiver and history dump utility."

I'm dying to give it a try.

If you're running a highly loaded web site powered by PHP, you must be using php-fpm, don't you?

While new releases of php-fpm always immediately follow PHP releases, it's still a PITA to always patch the PHP source code at every release.

How come php-fpm hasn't been merged into PHP at the first place? The main reason is an incompatible license. Or rather... was.

Andrei just announced that the license of php-fpm had been changed. It's now the PHP license, and php-fpm can now technically get officially merged into PHP.

Here's a relevant post of the High-performance PHP group.

PHP is still going to suck, but faster :)

Here is a cool demo of a clever technique that displays your browser history without using Javascript. Yep, not without any single line of Javascript.

Well done.

Just released: Bayon, a simple and fast hard-clustering tool, with support for repeated bisection clustering and K-means clustering.

Feed him a list of documents, optionally with weighted terms, ask for any number of groups you want it to output, and Bayon will do its best to assign documents to groups.

That kind of tool can bring a lot of benefits to Ning-like web sites, forums, etc.

And it seems to work just as advertised.

Yet another gem by Mixi.

Presentations of the Los Angeles Ruby Conference 2009 are now online.

Here are my picks:

• Resource-oriented architectures, and Google Waves
• Scaling 'most popular' lists - a very common issue
• Mobilize your Rails app
• Flying Robot: Unmanned Aerial Vehicles Using Ruby and Arduino
• Johnson - is it Javascript, is it Ruby, or? WTF?
• PoolParty

Straight away from Railswaycon 2009, here's a great presentation about event-driven and fibers-driven development: Embracing events in Ruby. Neverblock is amazing!

Python zealots can also read the Python counterpart.

Meanwhile, PHP zealots can keep swearing by PHP 6, that's gonna introduce innovative brain-blasting technology: unicode support (woah!) and basic "goto" support (re-woah!).

Coroutines puts an end to latency in SOA and database-driven apps, where apps keep being stupidly stuck waiting for a remote server to answer instead of keeping the ball rolling.

As demonstrated in this presentation, web developpers still have loads to learn from games coders. But it seems to sink in slowly.

It was just announced that the following TLDs are likely to emerge soon:

• .RADIO
• .ECO (Ecological)
• .GREEN (Ecological)
• .MOVIE (Movie/Film Industry)
• .FAM (Family)
• .MUSIC (Music)
• .HEALTH (dot health)
• .SPORT (dot Sport)
• .INDIGI (for indigenous peoples)
• .NYC (New York City)
• .BERLIN (Berlin Germany)
• .PARIS (Paris France)
• .BZH (Brittany, a region in France)
• .ENG (England, a kingdom in the U.K.)
• .GAL (Galicia, a region in Spain)
• .MED (Mediterranean)
• .LLI (Leonese Language and Leonese Culture)
• .GAY
• .WEB
• .POST
• .MAIL (for emails and to control spam)
• .GEO (generic geographical locations)
• .XXX (Adult Entertainment)
• .BCN (Barcelona)
• .LAT (Latin America)

Are those really useful or is it just a way for the ICANN to make a quick buck?

Some videos of what happened at the Google IO conference are now online.

Wave really looks like a giant step forward.

Side note about spam prevention in Wave: "In the press conference right after the keynote, a reporter asked about spam prevention. Lars Rasmussen responded that it hasn't been given much thought yet, since it is a closed developer's preview for now, but also mentioned that most likely Wave would use a whitelist option, where you'd have to add a friend/coworker before they could send/invite you to Waves."

Meanwhile, Microsoft just launched a real-time focused version of IE8 using OneRiot components.

A new release of UCARP is now available for download.

As a workaround for some OS / setups, that new version adds an option (--nomcast) to use broadcast advertisements instead of multicast.

Thanks a lot to Steve Kehlet and Juan Antonio for bringing in and testing that new feature.