Speaker: How to balance dictatorship and anarchy…

OpenSocial 0.8 has been release, see http://opensocial.org

Enhancements to JS and XML definitions, RESTful APIs, Templating language

JS/XML:
Main thing: Cleanup and convenience (Gadgets XML, gedgets.* opensocial.*)
InlinedMessage bundles (languages, i18ln)

Server rendering Gadgets can preload data (ServerSide), improves perceived performance since Gadget doesn’t need to make async XHR to query data

OAuth supported

John Panzer on stage for talk on RESTful APis

Access to new environments (Mobile, etc)
Discovery via XRDS-Simple (Same as OpenID)
Authentication + Authorization: OAuth (Authorizing third party code to act on behalf of user)

Data: People, Activites, AppData, Groups
Fomats: JSON, Atom XML

RESTful API provides facilities for: Messaging, Concurrency, Batching

Templating Language and tags:
Goals: Easy to develop, Secure (XSS Security preventions), Fast (Pre-Process server side), OpenSocial-Style, Can build an entire application

Template mime type is text/os-template — defines inline template section, template is valid xml snippet , uses JSP expression language.

Containers can provide their own template tag libraries…(YouTube could provide its own tags for exmaple.

The first call will occur via a ‘Skypecast’, at the following time:

1500GMT Wednesday May 28th
(8a San Francisco / 11a WashingtonDC / 4p London)

The call will open with a brief overview of what’s new in the latest
Heritrix releases (1.14.0 and 2.0.0), and some of the priorities for
future releases. Then, it will be an open discussion of whatever
participants want to talk about.

http://webteam.archive.org/confluence/display/Heritrix/Community+Calls

Mike is (implicitly) comparing Blekko to Cuill. Cuill (pronounced “cool”) also in stealth mode claims to have a much cheaper, and more efficient way of indexing the web compared to Google. I think this is a bit premature since no one really know what Cuill or Blekko are doing.

Links

Blekko raises round from Marc Andreessen

Google Challenger: Blekko

It will be interesting to see how competitor Hakia responds.

I’m a little underwhelmed with powerset, especially since it is searching semi-structured data where some of the relationships are more explicit than out in the wild.

http://20bits.com/2008/05/12/powerset-launches-verdict-meh/

http://venturebeat.com/2008/05/12/powerset-opens-to-everyone-now-whats-next/

http://www.techcrunch.com/2008/05/11/powerset-launches-showcase-for-user-search-experience/

http://gigaom.com/2008/05/11/powerset-is-live/

If you were to make a request to Slashdot and examine the Http headers using either HttpLiveHeaders or FireBug you would notice one of two unusual headers: X-Bender, and X-Fry with quirky messages. These are pop-culture references to the show “Futurama“.

There are a few ways you could accomplish this — if you were already working with mod_perl, mod_python, or PHP it would be trivial to manipulate the headers programatically. However, it’s also possible to do so with a simple one liner in your Apache conf.

First you’ll need to enable mod_headers — On my Debian server this was done with the following command: a2enmod headers, after which I had to restart Apache.

Then simply add something like the following line in your conf, of course replacing “HeaderNameHere” and “Header Value” with whatever you want.

Header add HeaderNameHere "Header Value here"

The “Header” directive has a number of possible actions including: set, unset, append, add and echo. The documentation is fairly well written and can be found here: mod_headers documentation

One useful application would be in a load-balanced  apache setup, you could  set the actual hostname as a header for debugging purposes. You can echo environment variables like so: %{FOOBAR}e  which would display the “FOOBAR” variable.

Have fun tweaking Apache headers, hope this helps.

15 handy wordpress plugins (for power users) 

I’m a huge fan. I check our site a few times a day. However, there is one thing that has really been bugging me. The maintenance page. Whenever you throw up that oh-so-useful page full of links to the favorite sites of your employees… you return an 200 status code.

Think of me not as a big fan, but rather an eager web-crawler from Google or some new startup. When I find my way to your page, and your error page is return a 200 Status (OK) — I take it to mean that everything is hunky dory, and this content is really what you are all about.

Let me school you on this radical bit of information: You can return other status codes — such as 500 — and return a pretty html document containing all the links to your favorite sites.

Speaking of documents…

There is one document in particular I would like to refer(get it…) you to: rfc 2616. Section 10 is particularly relevant titled “Status Code definitions.” The most relevant among this list are the following status code: 200, 304, and 503. I mention these because there seems to be a certain level of confusion among your engineers.

200: OK — This is what we meant to serve out, and feel free to associate this context with our URI

304: Not Modified: A cache setting, meaning “No Update” use cached version

503: Service Unavailable: Bingo! Doesn’t that sound like something we would want to return as our status when we have our maintenance page up? For extra browny points, consider sending Retry-After header with some amount of time…

Here is the output from Live HTTP headers on one of these requests.

Thanks again!

Yousef

GET / HTTP/1.1
Host: digg.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13)
Gecko/20080311 Firefox/2.0.0.13
Accept:
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plai
n;q=0.8,image/png,*/*;q=0.5
Accept-Language: en
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

HTTP/1.x 200 OK
Date: Mon, 31 Mar 2008 23:38:53 GMT
Server: Apache
Last-Modified: Mon, 31 Mar 2008 20:52:50 GMT
Etag: “714c87-15e9-449c1d5b88c80″
Accept-Ranges: bytes
Keep-Alive: timeout=5, max=10000
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Cache-Control: private
Content-Encoding: gzip
Content-Length: 2069
———————————————————-
http://digg.com/img/down.png

GET /img/down.png HTTP/1.1
Host: digg.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13)
Gecko/20080311 Firefox/2.0.0.13
Accept: image/png,*/*;q=0.5
Accept-Language: en
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://digg.com/

HTTP/1.x 200 OK
Age: 2027
Date: Mon, 31 Mar 2008 23:04:19 GMT
Connection: Keep-Alive
Via: NS-CACHE: 100
Etag: “720139-89b-41414d62cbc00″
Server: Apache
Last-Modified: Thu, 18 May 2006 19:13:52 GMT
Accept-Ranges: bytes
Content-Length: 2203
Keep-Alive: timeout=5, max=9998
Content-Type: image/png
———————————————————-
http://digg.com/favicon.ico

GET /favicon.ico HTTP/1.1
Host: digg.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13)
Gecko/20080311 Firefox/2.0.0.13
Accept: image/png,*/*;q=0.5
Accept-Language: en
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

HTTP/1.x 200 OK
Age: 2932
Date: Mon, 31 Mar 2008 22:49:13 GMT
Connection: Keep-Alive
Via: NS-CACHE: 100
Etag: “2884541-47e-418dfb6884480″
Server: Apache
Last-Modified: Tue, 18 Jul 2006 18:57:06 GMT
Accept-Ranges: bytes
Content-Length: 1150
Keep-Alive: timeout=5, max=9997
Content-Type: image/x-ico

Evernote is an exciting startup — when I day-dream about the future of the web — this is the sort of thing I imagine.

It’s a service that lets you take pictures of anything, and upload them. Simple enough. The twist is that it has out-of-this-world optical character recognition and will find snippets of text wherever they appear on the image — and then evernote creates a full-text search engine (think google) of that text –
This is a totally amazing startup, and I won’t be surprised if they get snapped up by one of the big boys.

Keep your eye on their site:  http://www.evernote.com/

Download Statusbar

Firebug

Forcastfox

Google Browser Sync

Live HTTP Headers

Web Developer

YSlow

This is definitely a project worth watching.  It’s just starting (was accepted to apache on the 28th) so if you are looking to jump in, now is the time.

1) Foowi, the “Social” aspects have gone open source. See: http://svn.swlabs.org/foowi

2) I’ve released a few patches for the re-designed grub crawler, that aim to make things easier to use. The patch only slightly breaks things. See here: http://lists.wikia.com/pipermail/grub-dev/2008-January/thread.html

3) The Nutch stuff is supposed to be open sourced in the coming days.

4) I really want someone to explain the “interaction” between Visvo, whom I’ve previously blogged about here:  http://lists.wikia.com/pipermail/grub-dev/2008-January/thread.html and Wikia. Denis Kubes, the CTO of Visvo is doing a lot of Wikia work. Hats off. But still, are they two and the same? What is the deal here.

I’m late on my next FSM article, which is going to be the coolest yet I believe.

I’m sleepy. More tomorrow.

-Yousef

http://lists.wikia.com/pipermail/grub-dev/2007-November/000010.html

It is relatively easy to setup and run (I manage my own Zimbra server), and it is by far the best cross-browser web-mail experience — it is fast, and works for me not against me (I’m looking at you Yahoo Mail 2.0)

This all sounds well and good, what could be the problem you ask? Two things: 1) Java, and 2) It’s standalone nature.

The two together make it a real pain in general.  Java imposes higher memory requirement (don’t even think about running it on a machine with 512M ram) and the fact that is is pre-integrated with postfix/spamassasin…and meant to be run as a standalone solution makes it very hard to run with other software. It assumes it is the only thing running and binds by default to port 80, and 443…etc.

What would be better would  be a “componentized” version, meaning I already have postfix/mysql/spamassasin setup, and I want Zimbra for web-mail and shared calendering — drop a war into a tomcat webapps directory and presto, you have a great web-mail package.

My Current solution is just to run Xen and give it its own instance…but when thinking about moving to shared hosting or a dedicated server it is…restricting to say the least.

I love Zimbra, I still use it… but keep this in mind if you are considering it.

The timing of this is a little suspicious. Ask has been running a large, and mostly failed advertising campaign “Who killed the Algorithm” — perhaps a Sopranos reference?

The interesting thing in the back for my mind is: what will Microsoft do now that IAC is ripe for the picking?

Microsoft has been on a bender snapping up smaller companies (the cash must be burning holes in Ballmers pockets). In early 2007 there was a stream of rumors that Microsoft was in serious talks to Acquire Yahoo. If those fell through, or were never legitimate to begin with will be a matter for the history books — but now, the number four search destination is open.

The ability to build a web search index while challenging is not impossible, something even smaller startups can do see: Visvo Search

And what would Microsoft be buying? The experience to deliver on internet products, something Microsoft sorely lacks. Heck, it can barely deliver on its core competencies, desktop operating systems.

Junpiers Security team shouldn’t really be doing stuff like this, and if they were they should be much more open about it, using a proper User Agent, and including the URL to a project page with legitimate information.

The shelob v1.0 bot didn’t even check robots.txt, so this one is definately getting blocked with mod_security.

If anyone from Jupiter is reading this, feel free to post an explanation, but I won’t hold my breath.

Resources:

http://en.wikipedia.org/wiki/Shelob

http://security-lab1.juniper.net/

The Dallas based startup explains its name as the Sanskrit word for “universe”, and their tag line is: “its our universe”. After the customary grandiose and unqualified statements about how advanced they are Visvo claims to have developed the “automatic categorization search engine for web content”. While this implementation of this wasn’t immediately obvious to me after trying their search engine, I decided I like them as a company for other reasons.

I immediately did my favorite search — a vanity search on my full name (should be fairly unique). The Visvo engine returned two (and once three) results that were highly relevant: The “Debian as a Desktop System” article I wrote for the Free Software Magazine, and a ping back for a blog entry I wrote about the upcoming FreeBSD installer “finstall” . Here is a screen shot of the results page for my vanity search above (Click on image for full resolution):

Each search result has an “explain” link (see screen shots ) where so-inclined users can view the technical details of how the search result was ranked by the Visvo engine. It shows factors such as the tfidf score of the document, which is probably the most used statistical model for measuring the frequency of a term in a document relative to documents in the rest of the corpus. (Click on screen shot for full resolution)

The fact that they are building their own index is relevant, todays search giants have amassed too much power — in essence a monopoly on who finds what — and my hope for the future is that startups like Visvo can challenge the entrenched incumbents. The fact that they also open up their ranking is important, and this level of transparency is something Google could learn from.

The technical presentation by Dennis Kubes, founder and CTO of Visvo (linked at the bottom of the page) gives us some juicy details about their architecture. Visvo is indeed using Hadoop, the open source Map-Reduce implementation, occasionally supplements its dedicated hardware with Amazons EC2 hardware service.

I’m impressed so far. I wish them success, as their success is also the success of the open source community, more so than with Google.

I encourage everyone to give it a go, although the product is still in Alpha, so probably won’t be ready for your day to day search needs for a while.

Resources:

http://www.visvo.com

http://www.visvo.com/bot.html

Visvo Technical Presentation

I recently had the need to download all the images on Wikipedia, and an excellent project– wikix — was brought to my attention, which is the “best-practice” way of downloading Wikipedia image data.

It is an application written in C that parses the Wikipedia XML, extracts all the image links, and then creates a set of bash shell scripts that use common Unix utilities such as curl to actually fetch the images, while remaining respecting Wikipedia’s guidelines on bots.

Compiling WikiX

Compiling wikix is pretty straight forward. It is important to note that the package is bzipped, then gzipped, so you’ll run something similar to the following commands:

ftp://www.wikigadugi.org/wiki/MediaWiki/wikix.tar.gz.bz2

bzip2 -d wikix.tar.gz.bz2

tar xzvf wikix.tar.gz

cd wikix

chmod 775 * (Note: the downloaded files have very restrictive permissions by default, this opens them up a bit)

make all && make install

If you get the following error: cc1: error: unrecognized command line option “-Wno-pointer-sign” — it probably means you are trying to compile using gcc3 and not the newer gcc4. There are two options:

1) If you have both installed you can update the CC environment variable

2) you can comment you the CFLAGS and CFLAGS_LIB lines, and uncomment the already commented out versions at the top of the file, so the Makefile goes from looking like this:

#CFLAGS = -g
#CFLAGS_LIB = -g -c
CFLAGS = -Wno-pointer-sign -g
CFLAGS_LIB = -Wno-pointer-sign -g -c

To looking like this:

CFLAGS = -g
CFLAGS_LIB = -g -c
#CFLAGS = -Wno-pointer-sign -g
#CFLAGS_LIB = -Wno-pointer-sign -g -c

Downloading Wikipedia Images

To get started, first download the xml database dump. At the time of writing, I issued the following command:

wget http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2

Then unpack the bzipped xml with:

bzip2 -d enwiki-latest-pages-articles.xml.bz2

Then run wikix, specifying the -p flag if you want the scripts to be parallelized ( I did ) or omit it if you don’t:

wikix -p < enwiki-20070802-pages-articles.xml &

This took about 12 minutes on the machine I am working on. However the machine is a Dual Dual-Core Opterons (model 254 @ 2.8 Ghz), with 16G ram, and 4+ Terabytes of storage — so your mileage may vary.

If you need to put the images in a different directory then your current working directory simply edit image_sh and changed the “OUTPUT” variable to the path where you want your images.

Then to actually start sucking down images over the internet run:

./image_sh — this should be in the directory you were in when you ran wikix.

An added bonus is that the wikix script creates a file called “image.log” which contains each image found, one per line, which is an ideal format for writing a quick script to insert all those image names in a database, such as mysql.

The approximate size of all the images as of October, 2007 is approximately 406 gigabytes. So make sure you have lots of disk space!

A good place to start when looking for the dumps would be the official Wikipedia Database Dump page.

Resources

http://en.wikipedia.org/wiki/Wikipedia:Database_download

http://meta.wikimedia.org/wiki/Wikix