shelob the evil bot (spider from juniper networks)

Posted on November 10th, 2007 by Yousef Ourabi

I just noticed a new spider in my servers logs: “shelob v1.0″ coming from host 208.223.208.181 which resolved to security-lab1.juniper.net — Per this site: http://ella.slis.indiana.edu/~pwelsch/shelob/ — shelob stands for “ Shelob Helps Examine Links on Blogs”
For those of you who are keen Tolkien fans, you’ll remember Shelob is the “evil spider”. This story gets weird when I browsed to “http://security-lab1.juniper.net/” and found an open apache directory containing some images and an executable (note: I didn’t click or download the executable, I’m not that brave, and I suggest you don’t either).

Junpiers Security team shouldn’t really be doing stuff like this, and if they were they should be much more open about it, using a proper User Agent, and including the URL to a project page with legitimate information.

The shelob v1.0 bot didn’t even check robots.txt, so this one is definately getting blocked with mod_security.

If anyone from Jupiter is reading this, feel free to post an explanation, but I won’t hold my breath.

Resources:

http://en.wikipedia.org/wiki/Shelob

http://security-lab1.juniper.net/

  • Digg
  • del.icio.us
  • StumbleUpon
  • Technorati
  • Reddit

Filed under: Semantic Web


« »
  • My page http://ip.needthe.net contains Google AdSense ads and referers. Some of them could be related to Juniper Networks Inc. equipment and/or services. When I for 1st time received click from ads on this page, "shelob v1.0" entry for 1st time appeared in weblog. Juniper comes and checks pages where their ads/referers appear. Nothing harmful. They go directly to page where their ads are shown and nowhere else therefore no need to check robots.txt .
  • Rob
    I have no ads on my site and shelob v1.0 is crawling my site www.patientfirst.com.
  • Lain
    Well, I was just hit by this bot, and my bot trap captured it within 3 page indexes, it hit my index page, one more then went straight for the "bad bot trap". My logs show a blank referrer, and now this bot will never be allowed on my server again.

    Its shameful to think that a so called "security" company will blatantly skip the robots.txt file, then seek the section it should not be indexing to begin with. I bet this company would have me arrested if I disregarded the private property sign on its building and started looking over their documents. My bot trap automatically sends the bot to 403 land upon capture.

    The robots.txt file needs to be a legally binding agreement, like a "no trespassers" sign.
  • I just got hit by the bot also. It got stuck on a few 404 error pages. A security agency shouldn't be making evil spider bots.
  • Erik
    I'm not from Juniper, but security-lab1.juniper.net is a Tor exit node, so it is possible another party was running their bot through Junipers hidden service.
blog comments powered by Disqus