apt-get install solr-tomcat5.5

Note, I’ve already noticed one bug with the get-properties link in the solr admin and created a bug report on Launchpad: https://bugs.launchpad.net/ubuntu/+source/solr/+bug/218419

1. Apache on Ubuntu leaves a legacy (init.d broken)
2. trac on 64bit Gusty Broken
3. Flash on 64bit Ubuntu broken

FAIL

If Hardy Heron isn’t any better I’m going to go back to Debian w/custom compiled debs for my server needs — though I’ll probably stick with Ubuntu on the Desktop –

Ubuntu Wall of Shame

1. Apache on Ubuntu leaves a legacy (init.d broken)
2. trac on 64bit Gusty Broken
3. Flash on 64bit Ubuntu broken
4. Why I’ve stopped reporting bugs to Ubuntu

Long story short — it will cause funkiness w/your system and will eventually prevent apache from starting / restarting normally, spewing errors like these (among others potentially).

You can easily verify if your system is suffering your problem by doing the following:

1) verify apache is not running: netstat -lntp (you shouldn’t see anything w/apache in the PID/Program column) and “pgrep apache” should also display nothing.

2) start apache normally: /etc/init.d/apache2 start

3) stop apache normally: /etc/init.d/apache2 stop

Now normally at this point there should be no apache related processes running, but if you are suffering from this bug you should see a slew of apache processes from: “pgrep apache” — and that list only grows with each restart, or stop and start. Eventually leading to errors like these:

Wed Mar 05 21:49:24 2008] [notice] mod_python: using mutex_directory /tmp
[Wed Mar 05 21:49:24 2008] [error] (28)No space left on device: mod_python: Failed to create global mutex 6 of 8 (/tmp/mpmtx68006).
[Wed Mar 05 21:49:24 2008] [error] mod_python: We can probably continue, but with diminished ability to process session locks.
[Wed Mar 05 21:49:24 2008] [error] mod_python: Hint: On Linux, the problem may be the number of available semaphores, check ’sysctl kernel.sem’
[Wed Mar 05 21:49:25 2008] [emerg] (28)No space left on device: Couldn’t create accept lock (/var/lock/apache2/accept.lock.6800) (5)

The official launchpad bug report has a few good suggestions on possible fixes, including this that caught my attention, from Juergen Kreileder :

The cleaner work-around is to just replace “$APACHE2CTL graceful-stop” with “$APACHE2CTL stop”.

This is in reference to the apache_stop() function, with the specific line currently at 102 (/etc/init.d/apache2) — So these lines (with context)

apache_stop() {
if `$APACHE2 -t > /dev/null 2>&1`; then
# if the config is ok than we just stop normaly
$APACHE2CTL graceful-stop

would look like this instead:

apache_stop() {
if `$APACHE2 -t > /dev/null 2>&1`; then
# if the config is ok than we just stop normaly
$APACHE2CTL stop

I run Ubuntu gusty gibbon on both my server and workstation — but this is just the latest on a growing list of bugs that are Ubuntu specific — and I’m starting to lose patience.

Ubuntu Wall of Shame

1. Apache on Ubuntu leaves a legacy (init.d broken)
2. trac on 64bit Gusty Broken
3. Flash on 64bit Ubuntu broken

TracError: ClearSilver not installed (/usr/lib/python2.5/site-packages/neo_cgi.so: undefined symbol: Py_InitModule4)

Rant

I’m going to outline the steps I found to fix the problem, but first I’m going to rail at Canonical / Ubuntu for a minute. This new issue isn’t the first bug affecting 64 bit systems that Ubuntu took *Months* an eternity in the software industry to fix. The last time it was with the flash plugin in their repositories see “Firefox flash plugin broken on 64bit systems“. These issues both came after many of us read at the frustration members of the community were experiencing with the Ubuntu bug reporting and tracking community, aka “Why I’ve stopped reporting bugs to Ubuntu“. Ubuntu is an OK distribution. I use it. However it’s not the savior of the Desktop Linux movement it claims to be. It’s not easy enough for non-technical users to install (neither is Windows for that matter). It still takes months to fix bugs on mainstream architectures.

Fix

Follow these steps to download the latest clearsilver, compile it from source, install it, but also copy one of the generated shared objects over the existing Ubuntu one.

Note: The original calls for installing build-dep, which no longer exists so I replaced it with build-essential

$ sudo apt-get build-essential python-clearsilver
$ sudo apt-get install python-dev

Download the 0.10.4 version (as it contains fixes for python2.5)
$ wget http://www.clearsilver.net/downloads/clearsilver-0.10.4.tar.gz
$ tar xzvf clearsilver-0.10.4.tar.gz

Edit the configure files as listed by Antonio Censi above

Build the new version
$ cd clearsilver-0.10.4
$ ./configure
$ make
$ sudo make install

Copy the new shared object over the packaged version
$ sudo cp -bi python/neo_cgi.so /usr/lib/python2.5/site-packages/neo_cgi.so

Resources:
Official trac wiki page — contains cookbook version of fix: http://trac.edgewall.org/wiki/TracOnUbuntu

Ubuntu bug report page: https://launchpad.net/ubuntu/+source/clearsilver/+bug/86685

Specific comment with fix-cookbook, this is the orignal that was propagated over the inter-tubes:

https://launchpad.net/ubuntu/+source/clearsilver/+bug/86685/comments/14

Ubuntu Wall of Shame

1. Apache on Ubuntu leaves a legacy (init.d broken)
2. trac on 64bit Gusty Broken
3. Flash on 64bit Ubuntu broken

lighthttpd and its fans are no different. There is an interesting thread in the wikipedia mailing list inquiring about the “advertised” use of lighthttpd — pronounced “lighty” — on wikipedia servers.

The truth of the matter is that Apache is running the “wiki” portion, while lighthttpd serves up static content like CSS and images. And that is what they are both good at, lighty built for speed is dedicated to serving static files like CSS and images, and apache for generating dynamic content.

1) More people know apache, can configure it, and properly secure  it

2)  It is really not geared towards generating dynamic content (think php…)

3)  Has fewer plugins.

So, no, you should not apply lighthttpd everywhere. Apply it where it will shine, serving static files quickly.  Use Apache for everything else.

Ok, this comes not long after my gripes about desktop virtualization on linux. This won’t fix it either. The problem is not that parallels is hard to install — it’s really not (dpkg -i parallels.deb) — it’s that integration sucks with any open source desktop such as kde or gnome. It’s that parallels only releases 32 bit versions of its linux software rendering any system with more than 3 gigs of ram useless. Its that the “parallels workstaiton” for linux just in not as good as “parallels” for mac os x, here are some reasons.

1) Poor integration. The OS X version runs like a bat out of hell, and has one particular feature called “Unity” that allows OS X users to run windows applications almost as stand alone applications — no need for the clunky interface, just click on the IE logo and it starts and runs like a native application — behind the scenes parallels is starting a virtualized instance and all that jazz — but it is highly integrated.

2) Speed. Oh my god, having seen paralles and vmware fusion, no virtualization package for linux runs as fast, both VMWare and Parallels are clunky and slow. There are performance tweaks that provide a marginal boost (5%) — but OS X virtualization products take full advantage for the modern intel cpu’s in new macs that support VMT, intels virtualization optimizations and extensions, while for linux they seem to code to the lowest comment denominator.

3) Graphical Accelerations. OS X users can run games and graphic programs programmed towards the windows direct x 8.1 api. Why can’t they do this for linux as well?

If you are looking for insight into the processes inside a java virtual machine try: “jps” — which will print a list of java processes to STDOUT.

I had several versions of the jdk installed including the one I wanted “sun-java6-jdk” however the gcj one was the default system.

There are two steps required to change the default JVM.

1) Edit /etc/jvm, and make sure the desired JVM is first on the list

2) sudo update-java-alternatives -s java-6-sun will update all the “jdk” binaries, or you can only point to specific tools like the java compiler (‘javac’) like so update-alternatives –set javac /usr/lib/jvm/java6-sun/bin/javac

There, you should be all set. There might be cleaner ways of doing this — you could of course export JAVA_HOME and set that at the top of your $PATH list…

If you’ve accidentally locked yourself out either via an accidental crtl-c or system restart and you were in the middle of editing /etc/sudoers via visudo there are two steps to take to restore it to a normal state (aka un-lock yourself)

I’ve done the following on an Ubuntu GNU/Linux system — but I’m assuming it is fairly cross-platform.

1) rm /etc/sudoers.tmp — this is the lockfile visudo creates

2) Double check permissions. If the /etc/sudoers file is not 440 (r–,r–,—) sudoers will barf. (Edit: Originally I had said to chmod u+x /etc/sudoers, but this was not correct, and could prevent you from sudoing)

VMWare Desktop is slow, even on my q6600 quad core with 4gigs of ram. The integration blows, it’s only good for syncing my ipod at best. iTunes will play sound just fine, but if I try to watch a movie I’ve bought or a video cast the sound stops working after about 10 seconds, though the video continues to play.  It doesn’t look or feel as fast as the Mac OS X equivalents on the same or even better hardware.

Vmware server is a joke for desktop stuff, don’t even get me started. VMWare server 2.0, currently in beta is supposed to be bring improved performance and para-virtualization to but it only has a web-interface, not a desktop interface like vmware server 1.0 and vmware workstation.

I believe a large part of the recent apple revolution has to do with the ease of use of running windows applications at near native speeds. Why doesn’t this exist for GNU/Linux desktops — I would pay for this.   I still have to dual-boot for the occasional intensive app (Blackberry IDE for BB development).

The virtualization options alone are enough to have me seriously thinking about making the “switch” –

This new iso-spin includes many patches so users who use this new install cd won’t have to download approximately 600 MBs of patches (it’s been out for a while now).

The mailing list has more here:  https://lists.ubuntu.com/archives/ubuntu-announce/2008-January/000107.html

The problem is that Adobe has released a new version of the plugin, however the ubuntu team has still not updated the deb package to contain the updated md5sum checksum, so after apt-get dutifully downloads the plugin the file integrity check fails, however worse still the installation continues and registers the flashplugin-nonfree package as installed WITHOUT actually being installed (which is very confusing)…

The problem is that this fails silently behind the scenes as most users will just use the ubuntu-fox modification to automagically install the appropriate firefox plugin, if however you installed it from the command line with something like apt-get install flashplugin-nonfree you would indeed see the following error:

Download done.
md5sum mismatch install_flash_player_9_linux.tar.gz
The Flash plugin is NOT installed.

Until the updated .deb package is pushed out upstream you can build your own plugin with the following steps:

Note: The new flash plugin has a new dependency the “cdbs” package — install it with the following command.

sudo apt-get install cdbs

wget http://launchpadlibrarian.net/10756602/flashplugin-nonfree_9.0.115.0ubuntu2.tar.gz
tar -zxvf flashplugin-nonfree_9.0.115.0ubuntu2.tar.gz
cd flashplugin-nonfree-9.0.115.0ubuntu2
dpkg-buildpackage -b -rfakeroot

cd ../

dpkg -i flashplugin-nonfree_9.0.115.0ubuntu2_amd64.deb

For the incurably curious here is the full bug report: https://bugs.launchpad.net/ubuntu/+source/flashplugin-nonfree/+bug/173890

Ubuntu Wall of Shame

1. Apache on Ubuntu leaves a legacy (init.d broken)
2. trac on 64bit Gusty Broken
3. Flash on 64bit Ubuntu broken

I have not installed this yet, but will soon and I’ll write it up here.

Check it out here: http://packages.debian.org/testing/text/solr-common

This is a big deal in my opinion and plays well into Canonicals server ambitions — many small shops who don’t have a lot of time could use this server to synchronize any software they need that isn’t installed via a default repository across multiple machines. This would normally be done by either compiling from source several times, or rolling your own deb packages.

Here is the link to the Canonical announcment:  http://www.ubuntu.com/news/launchpad-ppa

We left of having configuring the Windows side of the equation, having set up a separate hardware profile and installed the VMware SCSI drivers. All we have left to do is create a virtual machine that reads the already installed Windows XP, which is for the most part a simple process. There is are a few places that can be time sinks, so I’ve included screen shots and a step-by-step walk through to save you time.

I will not cover installing VMware workstation or server, has how you choose to do so depends on your system and what you want. I’ll leave that up to you.

To begin, make sure the user that will be running vmware is in the “disk” group (on my Ubuntu system) — which allows the user access to the raw disk.

I am using VMware workstation so these screen shots are geared towards that, but are mostly relevant to VMware server.

1) Fire up VMware — you should see something like the Screen Shot Below — A standard VMWare splash screen

2) Start the process of creating a new virtual machine, either by going to File->New or right clicking inside the inventory pane (the far left pane). The key thing here is to select a “Custom” virtual machine, instead of the “Typical” virtual machine.

3) The Guest operating system to select is obviously Windows XP — I have not tried this out with Vista yet, but it should probably work.

4) Give your new Virtual Machine a name and configure where the configuration files will be stored on disk.

5) Select the number of processors you want to allocate to this virtual machine.

6) Configure the amount of RAM you want allocated to this virtual machine.

7) Configure the networking, I selected “bridged networking” which means the virtual machine gets its own IP, instead of using the Hosts via NAT.

8 ) Select the I/O Adapter type — this is critical, make sure to select the “Buslogic” option, which should be selected by default.

9) Next you will be prompted to “Select a Disk” — you obviously want to choose “Use a physical disk” (this is the premise of this article after all?)

10) You will then be prompted to configure which physical disk you want to use. I used /dev/sda, and that should be enough for you, the only time this becomes a problem is when you have your linux boot partition on another disk, but GRUB installed on the MBR of the first disk. In that case after the wizard is complete, simply had a second physical device to the virtual machine and you should be set.

11) Finish!

Have fun with your new Windows partition running under linux. This setup has allowed me the flexibility of booting into my favorite operating system most of the time, and only requiring a reboot when I need to use “Hardware Intensive” applications (Read: games).

Cheers!

This is part One, part Two is here: VMWare running native XP on SATA disk, part 2.

The trick is to use install VMwares SCSI disk drivers under a windows hardware profile. I assume you already have your GNU/Linux environment setup already as you’ll need it to unpack the SCSI driver. Here are the exact steps:

1) Download the VMware SCSI drivers (On your Linux box)

2) mount -o loop vmscsi-1.2.0.4.flp /mnt/tmp (Do this as root via sudo or su)

3) cp /mnt/tmp /where/ever (This is the equivalent of unpacking a zip file). Ideally /where is the location where you mounted your windows partition read-write via something like ntfs3g, so you can just boot into windows and see your files. Otherwise see the next step.

4) Get the files from the floopy image onto Windows, copy them onto a CD, or over a network share. I’ll leave this up to you.

5) Create a Windows Hardware Profile by right clicking on “My Computer” -> Properties -> Hardware -> Hardware Profiles. You should see something like: “Profile 1 (Current)”. Copy that, and name the copy “Virtualized”. Rename the original “Physical” (or whatever you think you’ll remember). Also, on that same screen enable “Wait Until I select a hardware profile”.

6) Reboot your computer. When you start booting Windows you should be presented with a DOS-Like menu. Select the “Virtualized” profile. Start -> Settings -> Control Panel -> Add Hardware -> Yes, I have already connected the hardware -> You’ll then be presented with a list, scroll to the bottom “Add a new hardware device” -> Install the hardware that I manually select from a list -> SCSI and RAID Controllers -> Have Disk -> Browse to where ever you copied the files we unpacked from the floppy image under Linux. Select the driver. After the driver is installed, you might get a warning message about the device not functioning properly — Ignore it, obviously the VMWare SCSI device will only be present when running the OS under VMWare.

7) You are Done!

Tomorrow I’ll cover the Linux portion of things, how to configure VMWare to use the installed Windows XP partition.

Hope this helps, and see you tomorrow.
Resources:

http://vmware.com/download/server/drivers_tools.html

http://digg.com/linux_unix/Running_a_Windows_Partition_in_VMware

http://forums.gentoo.org/viewtopic-t-246371-start-25.html?sid=ae86f6d7c87732ca4661ca0cf2731d35

http://ubuntuforums.org/showthread.php?p=3642297

http://sanbarrow.com/phpBB2/viewtopic.php?t=930&start=0&sid=2bb0f1e76a0e57cf7d6a0547b3deaf20

Warnings:

http://www.vmware.com/support/ws4/doc/disks_dualboot_ws.html#1046312 “Caution: Running a Windows XP guest from a raw disk is not supported. You should not test a Windows XP raw disk configuration in a production environment.”

There are a few risks and reasons to block certain bots 1) If you use apache as a proxy spammers may try to use your Apache to “fake” clicks on advertisements on their pages, 2) to send plain old email spam and 3) to steal content from your site and use it as filler on their sites used only to serve out ads. mod_security can prevent all of this.

The key is to leverage mod_security’s abilities to perform certain actions based on the the HTTP protocol requests it receives from the client. In this case clients are both legitimate web-browsers, search engine crawlers, and nefarious bots looking to mirror content on their own sites and make advertising money.

The key field in the Http request is called the “User-Agent”. This is the string that clients use to identify themselves, for example when the Google spider fetches a page it declares its user agent to be “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” , The Yahoo spider identifies itself as “Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)”, and finally my firefox browser currently version 2.0.0.8 identifies itself as “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.8) Gecko/20071008 Firefox/2.0.0.8″.

It is worth noting that clients have full control over the way they identify themselves in the User-Agent header, and many choose to mask their existence by looking like a standard web browser. However, many quick and dirty bots, and scrapers either fail to do so, either because they don’t care or because their authors don’t know enough about the HTTP protocol to know why they would want to.

There are two approaches: 1) Block everything but a pre-approved white list, 2) Allow everything except a pre-denied black list. The first approach doesn’t really work, because we don’t want to have to update our apache configuration every time a new web-browser comes out, so we are left with only blocking a set of the most egregious bots.

Ideally we want our configuration to fail gracefully when mod_security isn’t available (if we move servers for examples).

Below is the configuration I use on my Apache host.

I block bots that try to proxy to other sites (presumably to fake ad clicks) and ones that try to connect to Port 25 — the email port to send spam. I comment the configuration inline (the lines that begin with #), but here is a quick overview:

The <IfModule> tells apache only to run the following configuration if the mod_security module is present. This allows me to copy this around to different hosts where I may or may not have mod_security without breaking the apache config.

The begin just sets up mod_security, and configures where we will log traffic we have blocked.  The first interesting configuratin directive “REQUEST_METHOD” blocks two HTTP request types that probe the server (TRACE) or can connect to a mail server to relay spam (CONNECT). The next line “REQUEST_URI_RAW” blocks requests that are not relative to the server (such as requestion /help.html) but contain the full protocol which means the request is probably trying to access a remote site. The remaining  “REQUEST_HEADERS” lines block access depending on the headers in the HTTP request. We must specificy which header we want to examine, in our case the “User Agent” and what identifier we want to look for.

Hope this helps.

<IfModule mod_security2.c>
# Basic configuration options
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess Off

SecUploadKeepFiles Off

# Debug log
SecDebugLog /var/log/apache2/modsec_debug.log
SecDebugLogLevel 0

# Serial audit log
SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus ^5
SecAuditLogParts ABIFHZ
SecAuditLogType Serial
SecAuditLog /var/log/apache2/modsec_audit.log

# Block HTTP requests that have http://, proxy spam, or ones that go to port 25

SecRule REQUEST_METHOD “^((?:connect|trace))$” “log,drop”
SecRule REQUEST_URI_RAW “http:/” “log,drop”
SecRule REQUEST_HEADERS:User-Agent “VadixBot” “log,drop”
SecRule REQUEST_HEADERS:User-Agent “radianrss-1.0″ “log,drop”
SecRule REQUEST_HEADERS:User-Agent “Python-urllib/2.5″ “log,drop”
</IfModule>

Resources:

http://www.modsecurity.org/

However on Gusty Gibbon amd64, which I’m running on my dell laptop, the vmware-install.pl script ran to completion without any patch!

Gusty is truly an awesome release, and I highly encourage everyone to check it out.

I’ll also be doing a “Virtualization Roundup” at some point, covering major virtualization, and para-virtualization software packages.

sudo apt-get install network-manager-pptp

You may, or may not need to restart dbus for the plugin to be loaded. If you click on the network manager, and it doesn’t look like the screenshot below, simply issue the following command:

/etc/init.d/dbus restart

After that you may need to reconnect to your network.

Update: There is a known bug with the pptp package and mmpe, see the bug report here (it is unclear if this affects Ubuntu as well): http://www.nabble.com/-Bug-34844–NEW:-PPTP-connection-impossible-t4636741.html