How to develop WhatWeb plugins

by Andrew Horton aka urbanadventurer. MorningStar Security http://www.morningstarsecurity.com/ This is a live document and is often updated.

Introduction to WhatWeb
Introduction to WhatWeb plugins
1. General aims of a plugin
2. Methods to identify systems
3. Important files and folders
4. Anatomy of a plugin
Research background information
Collect samples
1. Website Showcases
2. Using Search Engines
3. Forums for website development with the cms
Analyze samples
1. Read the source of a couple of samples
2. Collect HTML and HTTP headers from samples
3. Remove incorrectly identified samples
4. Examine the samples with WhatWeb
5. Remove more incorrectly identified samples with the whatweb report
6. Use find-common-stuff to automatically identify common strings in the samples
7. Analyse HTTP headers and cookies
8. Read more HTML source
Review of unique patterns identified
Write the plugin
Closing notes
Resources

1. Introduction to WhatWeb

WhatWeb lets you identify content management systems (CMS), blogging platforms, stats/analytics packages, javascript libraries, servers and more. When you visit a website in your browser the transaction includes many unseen hints about how the webserver is set up and what software is delivering the webpage. Some of these hints are obvious, eg. "Powered by XYZ" and others are more subtle. WhatWeb recognises these hints and reports what it finds.

WhatWeb has many plugins and needs community support to develop more. Plugins can identify systems with obvious identifying hints removed by also looking for subtle clues. For example, a WordPress site might remove the tag but the WordPress plugin also looks for "wp-content" which is less easy to disguise. Plugins are flexible and can return any datatype, for example plugins can return version numbers, email addresses, account ID's and more.

There are both passive and aggressive plugins, passive plugins use information on the page, in cookies and in the URL to identify the system. A passive request is as light weight as a simple GET / HTTP/1.1 request so it is suitable for large scale scanning of websites. Aggressive plugins guess URLs and request more files.

2. Introduction to WhatWeb Plugins

Plugins are easy to write, you don't need to know ruby to make them but it helps.

General aims of a plugin

Most plugins have a primary aim which is to identify a type of system based on signatures. The system could be a:

Content Management System
Javascript Library
HTTP Server
Application Framework

Secondary aims include:

Detect version, return in :version, :model, :firmware
Detect modules, return in :module
Detect usernames, return in :account

Some plugins do not have the aim to identify a specific type of system. Instead they try to give information that can be used to identify unanticipated systems or can be used for all types of websites. Some examples are:

Title - The HTML Page Title
MD5 hash
Header hash
Footer hash
Meta generator tag name
Uncommon HTTP headers
Tag-hash (Hash of the HTML tag pattern)

Methods to identify systems

There are 4 main methods to identify a web application. They are:

Matching patterns in the HTTP headers and HTML of a simple webpage request
Testing for URLs and identifying patterns in the HTML
Testing for URLs and recognising the MD5 hash
Testing for URLs and simply noting they exist or return an HTTP status 200 code.

WhatWeb supports all 4 methods however the 1st method is the most useful in large scale scanning. It is also the most efficient by trading off knowledge for network bandwidth and time.

Support for the first method is the most developed method within WhatWeb and is discussed in detail in this document.

Future development of WhatWeb will add more user friendly support for methods 2 through 4 which come under the purview of aggressive plugins. Use of method 4 is discouraged in favor of method 3 due to mod_rewrite type behaviour where 200 OK is returned even when the requested resource does not exist.

Important files and folders

The important folders to plugins are:

my-plugins/
plugin-development/
plugin-development/tests/
plugins/

All .rb files in the plugins/ folder are loaded by WhatWeb. To disable a plugin, move it into another folder or rename it to an extension that is not .rb.

The plugin-development folder contains some tools that are occasionally useful in developing plugins. The tools are:

find-common-stuff - This searches for common strings among a set of HTML files
wget-list - This downloads a list of example websites
get-pattern - This generates a tag pattern from a given URL for use in a plugin

The plugin-development/tests folder contains example webpages of CMS's to study. The wget-list will create two files for each example webpage. A .html file and a .meta file.

Anatomy of a plugin

This is a typical plugin. It identifies the Drupal framework and it's split into sections and given line numbers.

Header

This section declares the plugin.

1. Plugin.define "Drupal" do
2. author "Andrew Horton"
3. version "0.1"
4. description "Drupal is an opensource CMS written in PHP. Homepage: http://www.drupal.org"

Line 1. has the name. This name can be referred to on the commandline in a case insensitive way.

For example, the following works:

$ ./whatweb -p drupal www.example.com

Line 2. has the author. Just fill in your name between the double quotes.

Line 3. contains the version number. It's up to you what number to choose.

Line 4. Contains the description. This should contain a description of what the plugin identifies that anyone can understand. It can be many lines but must start and end with double quotes.

Note that the author, version and description follow the format:

field-name field-content

On the left is the name of the variable and on the right, separated by a space is the value. This type of variable declaration isn't ruby code, it's specific to the plugins and only works for certain variable names.

The list of variable names that can be declared in a plugin in this manner are:

author
version
description
examples
matches
dorks

Examples

5. # hard to identify
6. #<a href="http://drupal.org"><img src="/dagboek/misc/powered-black-80x15.png" alt="Powered by Drupal, an open source content management system" title="Powered by Drupal, an open source content management system" width="80" height="15" /></a> </div>
7. # <script type="text/javascript" src="/misc/drupal.js"></script>
8. # <script type="text/javascript" src="/main/misc/drupal.js"></script>
9. # @import "/misc/drupal.css";
10. # Set-Cookie: SESS6bdd09d4debccdc3a0f49becc449e8d5=2sq674vjn6vig48e3podh3j8e2; expires=Fri, 11 Dec 2009 15:37:52 GMT; path=/; domain=.moby.com
11. # Set-Cookie: SESS9795bcd4ea70e3f846e84f29f9491636=57eafcca6400d894772a136fb5889b92; expires=Fri, 11-Dec-2009 15:38:25 GMT; path=/; domain=.save-your-future.com
12. 
13. 
14. examples %w| amnesty.org/ appel.nasa.gov/ beta.worldbank.org/ entergy.pewclimate.org/ labs.divx.com/ lindenlab.com/ littlestarprints.com moby.com/ myplay.com/ sequelnaturals.com/ teen.secondlife.com/ www.artwaves.de www.asys.com.br/ www.atomicbop.net www.cristal.com.pe/?adulto=si www.dutchbutnotfromholland.eu/ www.elespectador.com/ www.ensembles.com.ph/ www.foxsearchlight.com/index.php www.freshbrain.org/ www.icsalabs.com/ www.johnnycashonline.com/ www.journalismcenter.org/ www.jovenscriativos.com.br/ www.koalafoundation.org.au/ www.la2day.com/ www.moove.be www.mtv.co.uk/channel/flux www.mulinobianco.it/ www.multiways.com/ www.nowpublic.com/ www.pravda.lt/ www.realismssoftware.com/ www.save-your-future.com www.shock.com.co/ www.sosojuicy.com/ www.spreadfirefox.com/ www.tidningenresultat.se www.ubuntu.com/ www.universitytowers.net/ www.warnerbrosrecords.com |
15.

Lines 5 through to 11 are comments. Each commented line must begin with a # character as this is a standard ruby way to comment code.

Line 14 is a list of example websites. The examples prefix of %w| means an array of elements separated by whitespace. The individual examples are URLs. If they are missing the http:// or https:// then http:// is assumed.

If you prefer you can list the examples like this:

examples %w|
http://www.example.com
http://www.example2.com
http://www.site.com/blah/
|

Matches

This section is a list of patterns to match against the webpage. Matches is an array and each element of the array is a hash and is surrounded by {} brackets. Notice that each pattern has a comma after it except for the last one. This is the normal ruby method of defining an array except that there is whitespace between matches and the content.

16. matches [
17. {:name=>"/misc/drupal.js",
18. :regexp=>/<script type="text\/javascript" src="[^\"]*\/misc\/drupal.js[^\"]*"><\/script>/},
29. 
20. {:name=>"Powered by link",
21. :regexp=>/<[^>]+alt="Powered by Drupal, an open source content management system"/},
22.
23. {:name=>"/misc/drupal.css",
24. :regexp=>/@import "[^\"]*\/misc\/drupal.css"/},
25.
26. {:text=>'jQuery.extend(Drupal.settings,'},
27. 
28. {:text=>'Drupal.extend('}
29. ]

Lines 17 through 18 define the first pattern.

Line 17 defines the pattern name. The name can be anything that describes what it's matching. The pattern name is optional, if included it should be unique.

Line 18 contains the pattern to match. It is a regular expression but could be any of the following:

:regexp - Regular Expression. Standard ruby regular expression surrounded by slashes.
:text - Simple string of text surrounded by " or ' quotes
:ghdb - Google Hacking Database. This is a google-like query that supports a few parameters.
- inurl: - the following string is in the URL
- intitle: - the following string is between the <title> </title> tags
- filetype: - the following string is the file extension, eg. PDF, JPG, RB, etc.
- - the following string is not matched on the page
:md5 - MD5 hash of the requested page
:tagpattern - A pattern of the HTML tags
:version - If set to a regular expression it extracts and returns the string, if set to a string, it returns the string.
- Used for reporting the version of the target software.
:string - If set to a regular expression it extracts and returns the string, if set to a string, it returns the string.
- Used for reporting interesting information, such as URLs, software build dates or MAC addresses for embedded devices.
:filepath - If set to a regular expression it extracts and returns the string, if set to a string, it returns the string.
- Used for reporting local file paths, usually revealed by system errors or configuration pages.
:account - If set to a regular expression it extracts and returns the string, if set to a string, it returns the string.
- Used for reporting usernames, often discovered on login pages and member lists.
:module - If set to a regular expression it extracts and returns the string, if set to a string, it returns the string.
- Used for reporting modules of the software, for example a webserver may have a ruby module or an embedded device may have a file sharing module.
:model - If set to a regular expression it extracts and returns the string, if set to a string, it returns the string.
- Used for reporting the model of a device.
:firmware - If set to a regular expression it extracts and returns the string, if set to a string, it returns the string.
- Used for reporting the firmware version of a device.

The match used on Line 18 is regexp and the pattern is:

/<script type="text\/javascript" src="[^\"]*\/misc\/drupal.js[^\"]*"><\/script>/

The slash needs to be escaped with a backslash. That is why "text/javascript" is written as "text/javascript". This is a standard ruby regular expression which differs slightly from regular expressions in other languages. To learn to write regular expressions visit http://rubular.com/ where you can copy & paste some HTML into the box then test out different regular expressions to see if they match.

38. def passive
39. m=[]
40. #SESS 9795bcd4ea70e3f846e84f29f9491636 =6b74f8aff4bf7d34d181a6a380d1ec7b; expires=Tue, 15-Dec-2009 15:21:24 GMT; path=/; domain=.save-your-future.com
41. m << {:name=>"SESS Drupal Cookie", :certainty=>75 } if @meta["set-cookie"] =~ /^SESS[a-z0-9]{32}=[a-z0-9]{32}/
42. m
43. end

44. end

Lines 38 through 43 defined the passive function. This function is called every time the plugin is matched against a web page.

Accessible Variables

Functions are able to access the following variables:

@body - The HTML body
@meta - The HTTP Headers (including cookies)
@status - The HTTP status code. 200 is successful, 404 is not found.
@base_uri - The URL
@md5_sum - The MD5 hash of the page
@tagpattern - The pattern of HTML tags opening and closing
@ip - The IP address of the webpage as a string

@body.

This is the HTML page, it includes the <html> and </html> tags. It isn't to be confused with the <body> tag.

@meta

This contains the HTTP headers as a hash. Remember, all hash keys are converted to lower case.

To view the HTTP headers sent by digg.com, use the curl command with the -I parameter.

$ curl -I digg.com/
HTTP/1.1 302 Found
Date: Sat, 16 Oct 2010 22:45:35 GMT
Server: Apache
X-Powered-By: PHP/5.2.9-digg8
Location: /news
Cache-Control: no-cache,no-store,must-revalidate
Pragma: no-cache
Set-Cookie: traffic_control=1152940230047891456%3A99; expires=Mon, 15-Nov-2010 22:45:35 GMT; path=/; domain=digg.com
Set-Cookie: d=ece93573ab6a5336e5e60c3063fdf4289db1118ebdce277255514a2977786cd3; expires=Fri, 16-Oct-2020 08:53:15 GMT; path=/; domain=.digg.com
X-Digg-Time: D=12827 10.2.130.111
Vary: Accept-Encoding
Content-Type: text/html;charset=UTF-8

Notice that above there are two lines to set cookies and below these are combined. All the hash keys are converted to lower case, the hash values remain the same. Hashes are also not ordered. The @meta behaviour is not a WhatWeb convention but is defined by ruby substrate code.

{"x-powered-by"=>"PHP/5.2.9-digg8",
 "connection"=>"Keep-Alive",
 "content-type"=>"text/html;charset=UTF-8",
 "x-digg-time"=>"D=116165 10.2.130.111",
 "date"=>"Sat, 16 Oct 2010 22:44:41 GMT",
 "server"=>"Apache",
 "set-cookie"=>
  "traffic_control=1152940230047891456%3A99; expires=Mon, 15-Nov-2010 22:44:41 GMT; path=/; domain=digg.com, d=3e304961a45164e0558290357e9ff861631e86ce46406743917419bce369d0ea; expires=Fri, 16-Oct-2020 08:52:21 GMT; path=/; domain=.digg.com",
 "cache-control"=>"no-cache,no-store,must-revalidate",
 "vary"=>"Accept-Encoding",
 "keep-alive"=>"timeout=5, max=9997",
 "pragma"=>"no-cache",
 "transfer-encoding"=>"chunked"}

@status

This is a number representing the HTTP status code. Some codes are: 200 = OK, 403 = Forbidden, 404 = Not Found

@base_uri

This contains a URI object of the current page. If the URI is http://www.reddit.com/r/netsec, the @base_uri.path returns "/r/netsec", @base_uri.host returns "www.reddit.com", @base_uri.port returns 80, and you can convert it to a string with @base_uri.to_s.

@md5_sum

This contains a md5 hash of the webpage source stored as a string. This is used to compare and confirm that a page has exactly the same content as another page. When scanning you may find many webpages that return the same page, using the md5 hash you can easily group these.

You can verify the md5sum of page using curl:

 curl http://about.digg.com/faq | md5sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 49893    0 49893    0     0  38547      0 --:--:--  0:00:01 --:--:-- 45152
cf164b74696ad2b7d5266369deb934c7  -

WhatWeb also returns a hash of the md5 page in the md5 plugin.

@tagpattern

This is the pattern HTML tags opening and closing stored as a string.

@ip

The IP address of the current page stored as a string.

The passive plugin on line 39 creates an empty array called m. On line 42 it returns that array. The m array will either be empty or will have the same fields as the patterns in the match array.

Line 40 is a comment which contains a sample session cookie.

Line 41 adds the hash to m if the @meta array element set-cookie matches the regular expression /^SESS[a-z0-9]{32}=[a-z0-9]{32}/

This regexp means a line that starts with SESS followed by 32 lowercase letters or numbers followed by the equals sign which is followed by 32 lowercase letters or numbers.

Line 44 ends the plugin which was started on line 1.

Plugin functions can be:

passive
aggressive
startup
shutdown

3. Research background information

Go to the homepage of the software or CMS you are researching and learn about it.

Look for:

Requirements, eg. the type of web server and languages it requires
Demo sites
Website showcases and portfolios
Download links
Documentation

Some of this information will help in writing the plugin description and some will be useful in collecting samples.

The information I gathered:

The SilverStripe homepage is http://www.silverstripe.com/
The opensource CMS software is at http://silverstripe.org/
Documentation of requirements is at http://doc.silverstripe.org/doku.php?id=server-requirements
A project showcase at http://www.silverstripe.com/project-showcase/
A demo site at http://demo.silverstripe.com/

Using the information found I wrote the following plugin description: "SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"

Advanced hint: If you intend to make an aggressive plugin then you may wish to download multiple versions of the software.

4. Collect samples

Your website samples should be representative of all SilverStripe installations. Take care not to just collect samples that are recently developed. Try to collect samples from a variety of sources and with a range of configurations.

Methods to find samples:

Search Engines
Website Showcases and Design Portfolios
Forums for website development with the cms

Website Showcases

A website showcase is a collection of websites that show off the abilities of the web designers and the potential of the CMS. Try to find showcases that have websites designed by more than one web developer. Sites that are made by the same developer are not properly representative of all sites and may include the designers idiosyncrasies.

While reading the background information I found this showcase on the official homepage: http://www.silverstripe.com/project-showcase/

By Googling for "silverstripe showcase" I found the official community showcase at http://www.silverstripe.org/community-showcase.

Googling for "webdesign portfolio silverstripe" found some web designers with links to SilverStripe websites.

The portfolio at http://smartplugsdesign.com/portfolio/ contains the following SilverStripe sites:

http://www.lisamarieelliott.com/
http://www.moonlitekustoms.com/
http://www.textiprints.com/
http://www.intandemtheatre.org/
http://www.stillrunnin.com/
http://www.enamaine.org/

The best source of samples is the community showcase because it contains a variety of websites made by different webdesigners and the websites are included in the portfolio over a period of time. Websites created over a wide period of time are useful as samples because they will run different versions of the SilverStripe software. There are 98 portfolio pages so I collected samples from pages 1, 25, 50, 75 and 98.

The samples collected from the community portfolio:

http://www.holistichealth.com/
http://www.verus.com.tr/
http://www.latenightdisco.com/
http://www.arprostatecancer.org/
http://www.cavendishimaging.com/
http://beatone.co.uk/
http://www.loguitos.com/
http://www.easycash4life.com/
http://www.gsbc.edu/
http://www.bradyinc.com/
http://www.monjasantner.de/
http://www.robert80.de/
http://customcanvas.fritzandandre.com/
http://www.idee-cruises.de/
http://www.maklerservice-greiz.de/
http://www.kitesurfnelson.co.nz/
http://www.moto-racepaint.com/
http://www.hutmacherin.com/
http://www.fuel.ie/silverstripe
http://www.infinitestillness.ie/ss
http://www.peterpanvakantieclub.nl/
http://www.chapmansurfboards.com/
http://www.fairtradenap.net/
http://www.benpearce.co.nz/
http://www.wend.nl/
http://www.resoba.com/
http://maungataniwha.co.nz/
http://www.gyo.co.nz/
http://www.firstgalaxies.org/
http://www.clockwork.co.nz/
http://www.upstreamgroup.com/
http://www.moerakihavenmotel.co.nz/
http://www.thelightboxdesigns.com/
http://www.nadabakery.co.nz/
http://comtel.com.au/
http://victoriaoruwari.com/
http://www.demconvention.com/
http://www.whileyouwait.co.nz/
http://omb.cl/
http://www.executivemediasearch.com/
http://www.naciondnb.com/
http://www.thecelebritytruth.com/
http://www.frussian.com.ar/
http://unbounded.org/
http://www.rcaforum.org.nz/
http://charcoalinteriors.com.au/
http://www.rcaforum.org.nz/
http://www.andrewking.co.nz/
http://www.elijahlofgren.com/silverstripe/
http://www.silverstripe.com/

This may seem like a large number of samples to collect but I assume that some of these websites will no longer be running SilverStripe or may no longer exist at all.

Using Search Engines

Introduction

Google-dorks are strings that can be used with Google to discover specific systems. There is an extensive database of google-dorks in the Google Hacking Database:

http://www.hackersforcharity.org/ghdb/
http://www.exploit-db.com/google-dorks/

Example: "Powered by Vsns Lemon" intitle:"Vsns Lemon"

Using search engines to discover samples with google-dorks must not be the sole method used as these websites do not represent all sites on the internet running the system you are searching for. This method suffers from a selection bias. Webmasters have an incentive to remove the identifying strings discovered by google-dorks to reduce it's discoverability by malicious hackers.

Some WordPress installations include the text in the footer Powered by WordPress while this makes an excellent string to search for to find some installations, most WordPress sites do not include this string.

SilverStripe example

First I searched for known google-dorks for SilverStripe by googling for "silverstripe google dork" and "silverstripe google hacking".

We won't know how to search for SilverStripe sites until we analyze some of the sample sites. Note that Google doesn't index html fragments, instead it just indexes words, titles, and urls.

I pick one sample to check, www.cavendishimaging.com. By reading the HTML source code I notice that the following line is included: <meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >

This positively identifies the site is made with SilverStripe but this content won't be indexed by google. At first glance nothing else on the page looks as though it can identify it was made with SilverStripe.

Forums for website development with the cms:

Webdesign forums often have links to websites provided by the web designers. Some of these websites will be of lower quality than found in a portfolio and some will also be in the default setup. Such websites would not be included in an offical portfolio.

By Googling for "silverstripe webdesign forum" I found the official SilverStripe Forum: http://silverstripe.org/connect-with-other-silverstripe-members/show/256356

Some website samples collected from the forum are:

http://hungryhearts.no
http://weonline.in
http://belitsky.info/work/hartmann
http://kunstforum.as/
http://www.choidoco.com/demo/
http://www.tobychampion.co.uk/
http://www.silverstripe.org.pl/

5. Anaylze samples

I need to analyze the samples I have collected to find similarities that can be used to identify these websites as SilverStripe. First I will search for identifying features in the webpages and HTTP headers.

In step 2 I collected 62 SilverStripe samples. I assume that some of these websites are incorrectly listed as SilverStripe so I will keep that in mind.

http://beatone.co.uk/
http://belitsky.info/work/hartmann
http://charcoalinteriors.com.au/
http://comtel.com.au/
http://customcanvas.fritzandandre.com/
http://hungryhearts.no
http://kunstforum.as/
http://maungataniwha.co.nz/
http://omb.cl/
http://unbounded.org/
http://victoriaoruwari.com/
http://weonline.in
http://www.andrewking.co.nz/
http://www.arprostatecancer.org/
http://www.benpearce.co.nz/
http://www.bradyinc.com/
http://www.cavendishimaging.com/
http://www.chapmansurfboards.com/
http://www.choidoco.com/demo/
http://www.clockwork.co.nz/
http://www.demconvention.com/
http://www.easycash4life.com/
http://www.elijahlofgren.com/silverstripe/
http://www.enamaine.org/
http://www.executivemediasearch.com/
http://www.fairtradenap.net/
http://www.firstgalaxies.org/
http://www.frussian.com.ar/
http://www.fuel.ie/silverstripe
http://www.gsbc.edu/
http://www.gyo.co.nz/
http://www.holistichealth.com/
http://www.hutmacherin.com/
http://www.idee-cruises.de/
http://www.infinitestillness.ie/ss
http://www.intandemtheatre.org/
http://www.kitesurfnelson.co.nz/
http://www.latenightdisco.com/
http://www.lisamarieelliott.com/
http://www.loguitos.com/
http://www.maklerservice-greiz.de/
http://www.moerakihavenmotel.co.nz/
http://www.monjasantner.de/
http://www.moonlitekustoms.com/
http://www.moto-racepaint.com/
http://www.naciondnb.com/
http://www.nadabakery.co.nz/
http://www.peterpanvakantieclub.nl/
http://www.rcaforum.org.nz/
http://www.resoba.com/
http://www.robert80.de/
http://www.silverstripe.com/
http://www.silverstripe.org.pl/
http://www.stillrunnin.com/
http://www.textiprints.com/
http://www.thecelebritytruth.com/
http://www.thelightboxdesigns.com/
http://www.tobychampion.co.uk/
http://www.upstreamgroup.com/
http://www.verus.com.tr/
http://www.wend.nl/
http://www.whileyouwait.co.nz/

Read the source of a couple of samples

Select at random 2 or 3 websites and read the HTML source carefully. Look for anything that isn't generic or anything that you wouldn't find on any website. Good places to scrutinise are headers, footers, URL structures, filenames of javascript libraries and CSS files, and div naming schemes.

A fast visual inspection only identifies the meta generator tag: <meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >

A 2nd sample shows the following tag which includes a version number. <meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >

I also notice the URL format of some images is interesting, eg. "/assets/galleries/cakes/_resampled/Banner-Nada-090.jpg"

The div id names appear generic, eg. <div id="BgContainer">, <div id="Footer"> and <div class="footerTop">. At this stage they aren't interesting because I expect these div names to change with themes.

Collect HTML and HTTP headers from samples

Make a separate folder for the plugin you are analyzing. I will make the folder, plugin-development/tests/silverstripe/

    $ cd whatweb-0.4.6/plugin-development/tests
    $ mkdir silverstripe
    $ cd silverstripe

Create a file in the silverstripe folder that contains the list of samples. I have called the file 'list'.

    $ ../../wget-list
    Usage: ../../wget-list <file with list of urls>
    downloads each URL's html and headers into the current directory

In the plugin-development/ folder there is a script called wget-list. Use the script to download the samples into the silverstripe folder.

    $ ../../wget-list ./list
    --2010-03-04 17:03:09-- http://beatone.co.uk/
    Resolving beatone.co.uk... 84.45.68.168
    Connecting to beatone.co.uk|84.45.68.168|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: unspecified [text/html]
    Saving to: `beatone.co.uk-.html'

    [ <=> ] 10,785 19.2K/s in 0.5s

This takes a few minutes to complete. The script creates 2 files for each sample, an HTML and a META file which contains the HTTP headers.

    $ ls
    beatone.co.uk-.html www.demconvention.com-.meta www.moerakihavenmotel.co.nz-.meta
    beatone.co.uk-.meta www.easycash4life.com-.html www.monjasantner.de-.html
    belitsky.info-work-hartmann.html www.easycash4life.com-.meta www.monjasantner.de-.meta
    belitsky.info-work-hartmann.meta www.elijahlofgren.com-silverstripe-.html www.moonlitekustoms.com-.html
    charcoalinteriors.com.au-.html www.elijahlofgren.com-silverstripe-.meta www.moonlitekustoms.com-.meta
    charcoalinteriors.com.au-.meta www.enamaine.org-.html www.moto-racepaint.com-.html
    comtel.com.au-.html www.enamaine.org-.meta www.moto-racepaint.com-.meta
    comtel.com.au-.meta www.executivemediasearch.com-.html www.naciondnb.com-.html
    customcanvas.fritzandandre.com-.html www.executivemediasearch.com-.meta www.naciondnb.com-.meta
    customcanvas.fritzandandre.com-.meta www.fairtradenap.net-.html www.nadabakery.co.nz-.html
    hungryhearts.no.html www.fairtradenap.net-.meta www.nadabakery.co.nz-.meta
    hungryhearts.no.meta www.firstgalaxies.org-.html www.peterpanvakantieclub.nl-.html
    kunstforum.as-.html www.firstgalaxies.org-.meta www.peterpanvakantieclub.nl-.meta
    kunstforum.as-.meta www.frussian.com.ar-.html www.rcaforum.org.nz-.html
    list www.frussian.com.ar-.meta www.rcaforum.org.nz-.meta
    maungataniwha.co.nz-.html www.fuel.ie-silverstripe.html www.resoba.com-.html
    maungataniwha.co.nz-.meta www.fuel.ie-silverstripe.meta www.resoba.com-.meta
    omb.cl-.html www.gsbc.edu-.html www.robert80.de-.html
    omb.cl-.meta www.gsbc.edu-.meta www.robert80.de-.meta
    unbounded.org-.html www.gyo.co.nz-.html www.silverstripe.com-.html
    unbounded.org-.meta www.gyo.co.nz-.meta www.silverstripe.com-.meta
    victoriaoruwari.com-.html www.holistichealth.com-.html www.silverstripe.org.pl-.html
    victoriaoruwari.com-.meta www.holistichealth.com-.meta www.silverstripe.org.pl-.meta
    weonline.in.html www.hutmacherin.com-.html www.stillrunnin.com-.html
    weonline.in.meta www.hutmacherin.com-.meta www.stillrunnin.com-.meta
    www.andrewking.co.nz-.html www.idee-cruises.de-.html www.textiprints.com-.html
    www.andrewking.co.nz-.meta www.idee-cruises.de-.meta www.textiprints.com-.meta
    www.arprostatecancer.org-.html www.infinitestillness.ie-ss.html www.thecelebritytruth.com-.html
    www.arprostatecancer.org-.meta www.infinitestillness.ie-ss.meta www.thecelebritytruth.com-.meta
    www.benpearce.co.nz-.html www.intandemtheatre.org-.html www.thelightboxdesigns.com-.html
    www.benpearce.co.nz-.meta www.intandemtheatre.org-.meta www.thelightboxdesigns.com-.meta
    www.bradyinc.com-.html www.kitesurfnelson.co.nz-.html www.tobychampion.co.uk-.html
    www.bradyinc.com-.meta www.kitesurfnelson.co.nz-.meta www.tobychampion.co.uk-.meta
    www.cavendishimaging.com-.html www.latenightdisco.com-.html www.upstreamgroup.com-.html
    www.cavendishimaging.com-.meta www.latenightdisco.com-.meta www.upstreamgroup.com-.meta
    www.chapmansurfboards.com-.html www.lisamarieelliott.com-.html www.verus.com.tr-.html
    www.chapmansurfboards.com-.meta www.lisamarieelliott.com-.meta www.verus.com.tr-.meta
    www.choidoco.com-demo-.html www.loguitos.com-.html www.wend.nl-.html
    www.choidoco.com-demo-.meta www.loguitos.com-.meta www.wend.nl-.meta
    www.clockwork.co.nz-.html www.maklerservice-greiz.de-.html www.whileyouwait.co.nz-.html
    www.clockwork.co.nz-.meta www.maklerservice-greiz.de-.meta www.whileyouwait.co.nz-.meta
    www.demconvention.com-.html www.moerakihavenmotel.co.nz-.html

The folder whatweb-0.4/plugin-development/tests/silverstripe now contains many .html and .meta files.

    $ head beatone.co.uk-.html
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
    <html>
    <head>
    <base href="http://beatone.co.uk/" ><!--[if IE 6]></base><![endif]-->
    <title>Be At One - London Bar, Bookings Central London, Great Cocktails London </title>
    <meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" >
    <meta http-equiv="Content-Language" content="en-US">

    <link rel="shortcut icon" href="/favicon.ico">

This is a standard HTML file, this is the same as what you see when you select 'View Source' in a web browser.

$ cat beatone.co.uk-.meta
    HTTP/1.1 200 OK
    Date: Thu, 04 Mar 2010 04:04:35 GMT
    Server: Apache/2.2.3 (Debian) PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c
    X-Powered-By: PHP/5.2.0-8+etch16
    Expires: Thu, 19 Nov 1981 08:52:00 GMT
    Cache-Control: no-cache, max-age=0, must-revalidate
    Pragma: no-cache
    Vary: Accept,User-Agent,Accept-Encoding
    Content-Type: text/html; charset=utf-8
    Via: 1.1 bc2
    Connection: Keep-Alive
    Set-Cookie: PHPSESSID=4d463f54abb74031c117569ca3aa3c61; path=/

These are the HTTP Headers the webserver sends before the HTML. Look for unusual cookie names and non-standard HTTP headers.

Remove incorrectly identified samples

To identify some samples that are not SilverStripe I grep for the generator tag and remove all occurances that include the word Silver. This leaves me with the following:

$ grep generator *html | grep -v Silver
    omb.cl-.html: <meta name="generator" content="dospuntocero.cl" >
    www.andrewking.co.nz-.html:<meta name="generator" content="WordPress 2.8.4" />
    www.easycash4life.com-.html:<meta name="generator" content="WordPress 2.8.2" />
    www.idee-cruises.de-.html:<meta name="generator" http-equiv="generator" content="cms.Koncepts - http://www.koncepts.de" />
    www.thecelebritytruth.com-.html:<meta name="generator" content="WordPress 2.8.4" />

These websites are obviously not SilverStripe so I delete their files and remove them from the list.

$ rm omb.cl-.* www.andrewking.co.nz-.* www.easycash4life.com-.* www.idee-cruises.de-.* www.thecelebrity truth.com-.*

How many sites can we be certain are SilverStripe?

There are 57 website samples left:

    $ ls *html | wc -l
    57

Of these samples, 38 include the term silverstrip in the HTML

    $ grep -li silverstripe *html | wc -l
    38

Examine the samples with WhatWeb

Using whatweb before the plugin is written may show some interesting information. In this case it has identified the meta generator tag. However notice that some of these websites do not have the meta generator tag. Reasons could be that the webmaster has removed it or the website is no longer running SilverStripe.

This is also useful to find more samples that are not SilverStripe

$ ./whatweb -i ./plugin-development/tests/silverstripe/list

http://belitsky.info/work/hartmann [301] md5[c112335e6a56038ca4ba4b906d6aee05], redirect-location[http://belitsky.info/work/hartmann/], server-header[Apache], title[301 Moved Permanently]
http://belitsky.info/work/hartmann/ [200] index-of, md5[21577203b9abc6091d99203295712f0c], server-header[Apache], title[Index of /work/hartmann]
http://charcoalinteriors.com.au/ [200] md5[bff9a28ebdc1cdfdb80743458606df1d], server-header[Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8], title[Home -Charcoal Interiors], x-powered-by-header[PHP/5.2.10]
http://customcanvas.fritzandandre.com/ [200] JQuery, Mailto, md5[452d4fd540f07c98e0288094d0bf959f], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ruby/1.2.6 Ruby/1.8.7(2008-08-11) mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Home], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://comtel.com.au/ [200] Google-Analytics-GA[1388941], probably Joomla[com_search], md5[050d35f63e2d9cd46065cde83f876989], server-header[Apache/2.0.55 (Ubuntu) PHP/5.1.2], title[Comtel - Telephone Radio & Data Systems | Comtel], x-powered-by-header[PHP/5.1.2]
http://beatone.co.uk/ [200] Google-Analytics-GA[11953167], JQuery, md5[b85a0567c28cfc3ba050ccbc95c899c4], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Be At One - London Bar, Bookings Central London, Great Cocktails London], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://kunstforum.as/ ERROR: Socket error getaddrinfo: Name or service not known
http://hungryhearts.no [200] Google-Analytics-GA[2984373], JQuery, Mailto, md5[1156688f57dfb37d853d3d7326daaadc], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3.41 (Unix) PHP/5.2.6 mod_psoft_traffic/0.2 mod_ssl/2.8.31 OpenSSL/0.9.7a mod_macro/1.1.2], title[The Hungry Hearts. Pin-up performance band.], x-powered-by-header[PHP/5.2.6]
http://unbounded.org/ [200] Google-Analytics-urchin[97930], md5[34d6c3cfc3b9ba0c66a8942a490c259d], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache/2.2.3 (Debian) DAV/2 PHP/5.2.6-1+lenny3 with Suhosin-Patch mod_ssl/2.2.3 OpenSSL/0.9.8g], title[unbounded], x-powered-by-header[PHP/5.2.6-1+lenny3]
http://www.arprostatecancer.org/ [200] Google-Analytics-GA[2447233], Mailto, md5[dc1a9f02efc52b1ebc72f2c8a0b03ae6], server-header[Apache/1.3.41 (Unix) PHP/5.2.6 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a], title[Arkansas Prostate Cancer Foundation], x-powered-by-header[PHP/5.2.6]
http://weonline.in [200] Google-Analytics-GA[8297705], Mailto, md5[f9d784e8b18c942fe7ea4ed8273630c9], meta-generator[SilverStripe 2.3.1 - http://www.silverstripe.com], server-header[Apache], title[Home. Weonline web design group. We love to do beautiful stuff for the web.], x-powered-by-header[PHP/5.2.9]
http://maungataniwha.co.nz/ [200] Google-Analytics-GA[3842018], JQuery, md5[58fbcc50e566ceaab813ecd38098e1df], server-header[Apache/2.2], title[Maungataniwha Lodge | New Zealand | Home]
http://victoriaoruwari.com/ [200] md5[2afe04307f5e3503efbe1c2c75b62511], server-header[Apache/1.3.41 (Unix) mod_ssl/2.8.31 OpenSSL/0.9.7a PHP/5.2.8 mod_perl/1.29 FrontPage/5.0.2.2510], title[Victoria Oruwari - Home], x-powered-by-header[PHP/5.2.8]
http://www.benpearce.co.nz/ [200] Google-Analytics-GA[1362535], JQuery, Mailto, Prototype, md5[4b1abe022c1c40c6091366c71c367cca], server-header[Apache/2.0.54 (Debian GNU/Linux) PHP/5.2.3-0.dotdeb.0 with Suhosin-Patch mod_ssl/2.0.54 OpenSSL/0.9.7e], title[Ben Pearce - artist], x-powered-by-header[PHP/5.2.3-0.dotdeb.0]
http://www.chapmansurfboards.com/ [200] Google-Analytics-GA[419314], JQuery, md5[9036620b334c3d2e7f6722e946277d29], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache], title[Dale Chapman Surf Designs], x-powered-by-header[PHP/5.2.10]
http://www.cavendishimaging.com/ [200] Google-Analytics-GA[11469477], JQuery, md5[ef8fbfae1dec5750a55478ed295cb34d], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Dentomaxillofacial Imaging & Anatomical Model Specialists - Cavendish Imaging], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://www.demconvention.com/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.clockwork.co.nz/ [200] md5[70e4b614d4ae85161014d05939ebd073], server-header[Apache], title[clockwork.co.nz]
http://www.choidoco.com/demo/ [200] md5[ccfe1940e9651416af58ff3f4a3eff77], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.11 mod_perl/2.0.4 Perl/v5.8.8], title[home], x-powered-by-header[PHP/5.2.11]
http://www.bradyinc.com/ [200] Google-Analytics-GA[13121212], Prototype, md5[34d5c1f763e3ceffaeae07de7de98f3d], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3.41 (Unix) FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7m], title[Staffing Productivity Benchmarks » Brady & Associates], x-powered-by-header[PHP/5.2.11]
http://www.executivemediasearch.com/ [404] md5[588da43361637cd97f3096ab9ce70183], server-header[Apache], title[Error 404 - Not found]
http://www.fairtradenap.net/ [200] Google-Analytics-GA[1362535], Mailto, md5[f4f36e648290b8fb818ea7fb297a4944], server-header[Apache], title[Home], x-powered-by-header[PHP/5.2.9]
http://www.elijahlofgren.com/silverstripe/ [404] Google-Analytics-urchin[2328965], maybe Mambo, md5[89a0d093054c83ef60a842b2aa7ff48f], meta-generator[CMS Made Simple - Copyright (C) 2004-6 Ted Kulp. All rights reserved.], powered by...[CMSMS], server-header[lighttpd/1.4.22], title[404 Error - Elijah Lofgren's Website]
http://www.enamaine.org/ [200] Google-Analytics-GA[3359251], Mailto, md5[57cb1e27c565acff11ce5f8103696696], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Maine ENA Home | Maine ENA], x-powered-by-header[PHP/5.2.9]
http://www.fuel.ie/silverstripe [301] md5[cbee7d5cfda4e161caffa892cc08558a], redirect-location[http://www.fuel.ie/silverstripe/], server-header[Zeus/4.3], title[Error 301 Moved Permanently]
http://www.firstgalaxies.org/ [200] Google-Analytics-urchin[777185], md5[fde922a270e0b438789eef03f2bbc064], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[A Resource for Research on the Most Distant Galaxies], x-powered-by-header[PHP/5.2.11]
http://www.frussian.com.ar/ [200] md5[54b8d6e69f5be47d29f8225126bf92da], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635], title[Home], x-powered-by-header[PHP/5.2.9]
http://www.gyo.co.nz/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.gsbc.edu/ [200] Google-Analytics-GA[276990], JQuery, Prototype, md5[dc166f92e1ed43f5435f60743c0d272a], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8l DAV/2 mod_auth_passthrough/2.1 FrontPage/5.0.2.2635], title[Home » Golden State Baptist College], x-powered-by-header[PHP/5.2.11]
http://www.fuel.ie/silverstripe/ [200] Mailto, md5[c63c8ea00aa0624fb4df7989c92b172e], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Zeus/4.3], title[The Fuel/Silverstripe Demo Site » The Fuel/Silverstripe Demo Site]
http://www.infinitestillness.ie/ss [301] md5[cbee7d5cfda4e161caffa892cc08558a], redirect-location[http://www.infinitestillness.ie/ss/], server-header[Zeus/4.3], title[Error 301 Moved Permanently]
http://www.holistichealth.com/ [200] Google-Analytics-GA[6289330], md5[b59e703ec114ea8cd99da42af210f1a1], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3.41 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a], title[Holistic Health International - Where Science and Caring Meet], x-powered-by-header[PHP/5.2.6]
http://www.hutmacherin.com/ [301] md5[d41d8cd98f00b204e9800998ecf8427e], redirect-location[/start], server-header[Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g]
http://www.infinitestillness.ie/ss/ [200] Mailto, md5[77b579a3d6752d45bcd8718d906fe5c3], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Zeus/4.3], title[Infinite Stillness | Ki Massage & Reiki Healing Dublin 4]
http://www.hutmacherin.com/start [200] md5[a89fbf86cf798314dba620717b1d99b8], server-header[Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g], title[Start | Isabell von Maltzahn | Hutmacherin aus Berlin]
http://www.intandemtheatre.org/ [200] Google-Analytics-GA[6603467], Prototype, md5[133a6893f85c4c41034ced6a7aec3e75], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Welcome | In Tandem Theatre], x-powered-by-header[PHP/5.2.9]
http://www.latenightdisco.com/ [200] Google-Analytics-GA[768894], md5[b81b6b18af21e3d5f7d0749aa483da64], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.4 with Suhosin-Patch], title[Experience Central Arkansas hottest night club Discovery], x-powered-by-header[PHP/5.2.4-2ubuntu5.4]
http://www.lisamarieelliott.com/ [200] Google-Analytics-GA[3359251], md5[d844eb0937b79fc02c347930522fe490], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Home], x-powered-by-header[PHP/5.2.9]
http://www.loguitos.com/ [200] Joomla[1.0], maybe Mambo, md5[0eed48ee56f6a2b784ea010b1bfa15b8], meta-generator[Joomla! - Copyright (C) 2005 - 2007 Open Source Matters. All rights reserved.], server-header[Apache/2.2.10 (Unix) mod_ssl/2.2.10 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635], title[Dise?o de logotipos - Loguitos - Dise?adores de logos profesionales - Inicio], x-powered-by-header[PHP/5.2.6]
http://www.moonlitekustoms.com/ [200] md5[cf3d07ae32e645d04a1acad6560ce668], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[In The Shop - Moonlite Kustoms], x-powered-by-header[PHP/5.2.9]
http://www.maklerservice-greiz.de/ [200] Google-Analytics-GA[10587433], Prototype, md5[9ac4c5710cf3e694917e2a3949680fc7], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/1.3 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.8f AuthPG/1.3 FrontPage/5.0.2.2635], title[Ihr Maklerservice in Greiz: Steiniger Versicherungsmakler], x-powered-by-header[PHP/5.2.9]
http://www.moerakihavenmotel.co.nz/ [200] md5[12e0be3129dea7738500b21aeab0ff96], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], powered by...[:], server-header[Apache], title[Moeraki Haven Motel, Moreaki Motel Accommodation, Otago Motel Accommodation.], x-powered-by-header[PHP/5.2.11]
http://www.monjasantner.de/ [200] Google-Analytics-GA[10599117], JQuery, md5[c30a4c5e6b9aad2a7f435621deb2ef40], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Monja Santner » Home], x-powered-by-header[PHP/5.2.6-1+lenny6]
http://www.nadabakery.co.nz/ [200] Google-Analytics-GA[4761582], md5[68f3dd581e40f370927205103ca6903a], meta-generator[SilverStripe 2.3.1 - http://www.silverstripe.com], server-header[Apache/2], title[Home | Nada - New Zealand's Greatest Bakery], x-powered-by-header[PHP/5.2.12]
http://www.naciondnb.com/ [200] md5[d443c89dc137c8286aaa173ebc806176], server-header[Apache], title[NacionDNB]
http://www.moto-racepaint.com/ [200] Google-Analytics-GA[1912569], JQuery, md5[7c1db45c5801d09dac4539725c6cf658], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4], title[MotoRacePaint Home], x-powered-by-header[PHP/5.2.11]
http://www.rcaforum.org.nz/ [200] Google-Analytics-GA[4693659], Mailto, Prototype, md5[5cbb333d29fb180f42142a9551e105a2], meta-generator[SilverStripe 2.3.0 - http://www.silverstripe.com], server-header[Apache], title[Homepage - RCA Forum], x-powered-by-header[PHP/5.2.1]
http://www.peterpanvakantieclub.nl/ [200] Google-Analytics-GA[1010482], JQuery, md5[011e5aa0e0e9599d10cd2913270959ea], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache/2], title[Peter Pan Vakantieclub | Home], x-powered-by-header[PHP/5.2.4]
http://www.robert80.de/ [200] Lightbox, md5[e12ecd879e7a24eaf6630d0097f6d0c8], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[Robert Müller 80 Freunde » Robert Müller 80 Freunde helfen], x-powered-by-header[PHP/5.2.13]
http://www.silverstripe.com/ [200] Google-Analytics-urchin[84547], JQuery, md5[25a7a15b8e40a9443ed6bd896705b7ba], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.11 (Debian) PHP/5.2.6-1+lenny2 with Suhosin-Patch mod_ssl/2.2.11 OpenSSL/0.9.8g], title[SilverStripe.com - Open Source CMS / Framework], x-powered-by-header[PHP/5.2.6-1+lenny2]
http://www.stillrunnin.com/ [200] Google-Analytics-GA[3359251], md5[bd3f631ef6075169b66c44031e526184], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635], title[Still Runnin Magazine - Online Gearhead Ezine], x-powered-by-header[PHP/5.2.9]
http://www.silverstripe.org.pl/ [200] Google-Analytics-GA[8121843], md5[1d8badb4df0d05a72c665de18087ae2b], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.8], title[Serwis polskiej spolecznosci SilverStripe » SilverStripe.org.pl], x-powered-by-header[PHP/5.2.8]
http://www.thelightboxdesigns.com/ [200] Mailto, md5[763842fb73491b1ccc9090219e299ed1], server-header[Apache], title[Graphic and Web Site Design Services in Brownsville, TX, McAllen, Harlingen and the Rio Grande Valley :: The Lightbox Designs]
http://www.textiprints.com/ [200] Google-Analytics-GA[10064088], md5[65c429cfbad7a5acccddb8eda664f13c], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache], title[TextiPrints - Digital Garment Printer - Ormond Beach, Florida], x-powered-by-header[PHP/5.2.9]
http://www.tobychampion.co.uk/ [500] md5[c5ea88dc871f71751f932a8a9bed884b], server-header[Apache/2.0.63 (FreeBSD) DAV/2 SVN/1.5.2 mod_python/3.3.1 Python/2.5.1 PHP/5.2.6 with Suhosin-Patch mod_ssl/2.0.63 OpenSSL/0.9.7e-p1 mod_fastcgi/2.4.6 mod_perl/2.0.3 Perl/v5.8.8], title[GET /], x-powered-by-header[PHP/5.2.12]
http://www.upstreamgroup.com/ [200] Google-Analytics-GA[3522744], md5[085964a490aa87a8d061828ebd63f35b], meta-generator[SilverStripe 2.3.1 - http://www.silverstripe.com], server-header[Apache], title[Upstream Group: Clarity, Perspective, Knowledge], x-powered-by-header[PHP/5.2.11]
http://www.wend.nl/ [200] Google-Analytics-GA[1010482], JQuery, md5[e08dcd1b16e7210f1762b49bb45d58ff], server-header[Apache/2], title[Wend - Home], x-powered-by-header[PHP/5.2.4]
http://www.verus.com.tr/ [200] Google-Analytics-GA[7233761], md5[54bfe62b41b88b472d9739d2ed47b30f], meta-generator[SilverStripe - http://www.silverstripe.com], server-header[Apache/2.2.8 (Ubuntu) mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8], title[VERUS » ETKINLIK ÇÖZÜMLERI » IS ÇÖZÜMLERI » WEB ÇÖZÜMLERI], x-powered-by-header[PHP/5.2.4-2ubuntu5.10]
http://www.whileyouwait.co.nz/ [200] Mailto, md5[05df26cf45f8410dea2272f6c0ff269b], meta-generator[SilverStripe 2.0 - http://www.silverstripe.com], server-header[Apache], title[While You Wait Studios - Christchurch, New Zealand - PTFOTO], x-powered-by-header[PHP/5.2.11]
http://www.kitesurfnelson.co.nz/ [200] Google-Analytics-GA[1921819], md5[f7618ec4d8da26579e5df948d80ba5b8], server-header[Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8], title[Kite Surf Nelson - kitesurfing lessons, equipment sales and advice - in Nelson, New Zealand], x-powered-by-header[PHP/5.2.6]
http://www.resoba.com/ ERROR: EOF error end of file reached

Remove more incorrectly identified samples with the whatweb report

    http://belitsky.info/work/hartmann [301] md5[c112335e6a56038ca4ba4b906d6aee05], redirect-location[http://belitsky.info/work/hartmann/], server-header[Apache], title[301 Moved Permanently]
    http://belitsky.info/work/hartmann/ [200] index-of, md5[21577203b9abc6091d99203295712f0c], server-header[Apache], title[Index of /work/hartmann]

This matches the 'Index of' plugin. By loading this URL into a web browser I can see that this webpage isn't a CMS, instead it's a directory listing so I can delete http://belitsky.info/work/hartmann from the list.

    http://charcoalinteriors.com.au/ [200] md5[bff9a28ebdc1cdfdb80743458606df1d], server-header[Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8], title[Home -Charcoal Interiors], x-powered-by-header[PHP/5.2.10]

There's no way to be sure if this is SilverStripe because it has the meta generator tag removed.

    http://comtel.com.au/ [200] Google-Analytics-GA[1388941], probably Joomla[com_search], md5[050d35f63e2d9cd46065cde83f876989], server-header[Apache/2.0.55 (Ubuntu) PHP/5.1.2], title[Comtel - Telephone Radio & Data Systems | Comtel], x-powered-by-header[PHP/5.1.2]

This appears to be powered by the Joomla CMS. A website cannot be two CMSs at the same time with the same URL so if Joomla is present then SilverStripe cannot be. Note that other plugin matches such as Jquery identify a javascript library and will be found with many different CMSs including SilverStripe. Manual verification by testing http://comtel.com.au/administrator proves it is Joomla.

    http://kunstforum.as/ ERROR: Socket error getaddrinfo: Name or service not known

This website doesn't exist anymore.

    http://www.arprostatecancer.org/ [200] Google-Analytics-GA[2447233], Mailto, md5[dc1a9f02efc52b1ebc72f2c8a0b03ae6], server-header[Apache/1.3.41 (Unix) PHP/5.2.6 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a], title[Arkansas Prostate Cancer Foundation], x-powered-by-header[PHP/5.2.6]

I can't be sure if this is SilverStripe yet.

I deleted the files of the remaining websites that are identified as something other than SilverStream. $ wc -l list 49 list

Now I have just 49 SilverStripe samples. I know that some of these samples might not be SilverStripe.

Use find-common-stuff to automatically identify common strings in the samples

I can use a simple tool called find-common-stuff to find certain types of common strings in samples.

find-common-stuff will identify and count the occurances of:

complete HTML tags
strings enclosed in double quotes

It has threshold setting that adjusts how many uncommon things are displayed.

    $ ../../find-common-stuff
    Usage: find-common-stuff FILES
    --threshold, -t The lowest % of files an item occurs in to display. Eg. 0.25 and 0.50

    $ ../../find-common-stuff *html
    imported 49 files
    counted 3324 tags
    [["<script type=\"text/javascript\">", 35],
    ["<![endif]-->", 30],
    ["<meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />",
    28],
    ["<style type=\"text/css\">", 21],
    ["<!--[if IE 7]>", 21],
    ["<!--[if IE 6]>", 21],
    ["<meta name=\"generator\" http-equiv=\"generator\" content=\"SilverStripe - http://www.silverstripe.com\" />",
    20],
    ["<div id=\"footer\">", 16],
    ["<link rel=\"shortcut icon\" href=\"/favicon.ico\" />", 16],
    ["<div class=\"clear\">", 16],
    ["<meta http-equiv=\"Content-Language\" content=\"en-US\"/>", 14],
    ["<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.1//EN\" \"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd\">",
    14],
    ["<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">",
    14],
    ["<div class=\"typography\">", 14],
    ["<meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" >",
    13]]
    counted 1874 quoted texts
    [["\"text/javascript\"", 35],
    ["\"link\"", 24],
    ["\"text/css\"", 23],
    ["\"typography\"", 19],
    ["\"current\"", 19],
    ["\"clear\"", 18],
    ["\"footer\"", 17],
    ["\"en\"", 16],
    ["\"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd\"", 14],
    ["\"http://www.w3.org/TR/html4/strict.dtd\"", 14],
    ["\"header\"", 14]]

In this case, the automated tool, find-common-stuff has failed to identify anything I didn't noticed while reading the HTML source.

Analyse HTTP headers and cookies

The .meta files contain the HTTP headers and any cookies set by the websites. Use the cat command to display all of them.

$ cat *meta
HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:35 GMT
Server: Apache/2.2.3 (Debian) PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8+etch16
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,User-Agent,Accept-Encoding
Content-Type: text/html; charset=utf-8
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=4d463f54abb74031c117569ca3aa3c61; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:41 GMT
Server: Apache/2.2.13 (Unix) mod_ssl/2.2.13 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8
X-Powered-By: PHP/5.2.10
Content-Type: text/html
Via: 1.1 bc6
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:30 GMT
Server: Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ruby/1.2.6 Ruby/1.8.7(2008-08-11) mod_ssl/2.2.9 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc1
Connection: Keep-Alive
Set-Cookie: PHPSESSID=1b627d14a21e4475c49e7089c01e11b8; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:04:49 GMT
Server: Apache/1.3.41 (Unix) PHP/5.2.6 mod_psoft_traffic/0.2 mod_ssl/2.8.31 OpenSSL/0.9.7a mod_macro/1.1.2
X-Powered-By: PHP/5.2.6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=kibbf9utq9decif304ml5lgu01; path=/

HTTP/1.1 200 OK
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Date: Thu, 04 Mar 2010 04:05:09 GMT
Server: Apache/2.2
Content-Type: text/html; charset="utf-8"
Pragma: no-cache
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=tmufihalkg02p4t0vi3mdf3k94; path=/
Set-Cookie: X-Mapping-caklakng=293486F930CB5202C46B0E5EFB41C64E; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:29 GMT
Server: Apache/2.2.3 (Debian) DAV/2 PHP/5.2.6-1+lenny3 with Suhosin-Patch mod_ssl/2.2.3 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny3
Set-Cookie: PHPSESSID=1b0b76cadaefb10b5973b8bb622d9669; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
Vary: Accept,Accept-Encoding
Content-Type: text/html; charset=utf-8

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:35 GMT
Server: Apache/1.3.41 (Unix) mod_ssl/2.8.31 OpenSSL/0.9.7a PHP/5.2.8 mod_perl/1.29 FrontPage/5.0.2.2510
X-Powered-By: PHP/5.2.8
Expires: Mon, 21 Jun 2010 11:11:02 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Sat, 14 Nov 2009 21:00:08 GMT
Vary: Accept
Content-Type: text/html; charset=utf-8
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=b255ace17a48eadebd5de6ec6b7f3dc6; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:35 GMT; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:37 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=5nfa7b62smslu5qo660bmtmk32; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:34 GMT
Server: Apache/1.3.41 (Unix) PHP/5.2.6 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a
X-Powered-By: PHP/5.2.6
Content-Type: text/html
Via: 1.1 bc7
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:46 GMT
Server: Apache/2.0.54 (Debian GNU/Linux) PHP/5.2.3-0.dotdeb.0 with Suhosin-Patch mod_ssl/2.0.54 OpenSSL/0.9.7e
X-Powered-By: PHP/5.2.3-0.dotdeb.0
Set-Cookie: PHPSESSID=d627a8e8613eecd6b0c2a41b0ec79dd4; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:50 GMT
Server: Apache/1.3.41 (Unix) FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7m
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.11
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=9508d9fa7a9d869e745c286ff9799d40; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:05:54 GMT
Server: Apache/2.2.3 (Debian) PHP/5.2.0-8+etch16 mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Powered-By: PHP/5.2.0-8+etch16
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,User-Agent,Accept-Encoding
Content-Type: text/html; charset=utf-8
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=8d9b19f46268342f8a0823bd79599cd2; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:01 GMT
Server: Apache
X-Powered-By: PHP/5.2.10
Expires: Sat, 13 Mar 2010 04:54:33 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Tue, 23 Feb 2010 03:17:29 GMT
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=4askloa60f26kvftt1cfiao344; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:06:01 GMT; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:13 GMT
Server: Apache/2.2.12 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.11 mod_perl/2.0.4 Perl/v5.8.8
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, must-revalidate, max-age=0, proxy-revalidate, no-transform
Pragma: no-cache
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8
X-Cache: MISS from sv25.byethost25.org
Via: 1.0 sv25.byethost25.org:80 (squid/2.7.STABLE7), 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=f0c426eaeef84f38c04624a59a98edd4; path=/demo/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:17 GMT
Server: Apache
Expires: Thu, 29 Oct 1998 17:04:19 GMT
Last-Modified: Thu, 04 Mar 2010 04:06:17 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: clockwork_co_nz=4040a20413b27678f6a5b225bd85f612; expires=Tue, 03-Mar-2015 04:06:17 GMT; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:46 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=6e9a4bb3b41cdd968202a19fbb91b3c0; path=/

HTTP/1.1 404 Not Found
Date: Thu, 04 Mar 2010 04:06:48 GMT
Server: Apache
Content-Type: text/html
Via: 1.1 bc1
Content-Length: 0
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:06:49 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=ec0a2031ab05b371baf412429859bbdf; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:01 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Vary: Accept,Accept-Encoding
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset=utf-8
Via: 1.1 bc4
Connection: Keep-Alive
Set-Cookie: PHPSESSID=5e61do18cag74urkrq65eqkcf3; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:07 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.9
Cache-Control: max-age=86400, must-revalidate
Pragma:
Expires: Wed, 14 Mar 2012 07:10:47 GMT
Vary: Accept
Last-Modified: Fri, 22 Feb 2008 01:03:27 GMT
Content-Type: text/html; charset=utf-8
Via: 1.1 bc3
Connection: Keep-Alive
Set-Cookie: PHPSESSID=3fb099aeb12584c9a2a308667e960553; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:07:07 GMT; path=/

HTTP/1.1 301 Moved Permanently
Date: Thu, 04 Mar 2010 04:07:14 GMT
Location: http://www.fuel.ie/silverstripe/
Server: Zeus/4.3
Content-Type: text/html
Via: 1.1 bc3
Content-Length: 212
Connection: Keep-Alive
Set-Cookie: X-Mapping-enlokcai=E05A570E7E395D6AB72BEA6FC2D4D8D4; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:19 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8l DAV/2 mod_auth_passthrough/2.1 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc1
Connection: Keep-Alive
Set-Cookie: PHPSESSID=05b506d2ba68f9057269c4ff70ff9643; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:11 GMT
Server: Apache/1.3.41 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.31 OpenSSL/0.9.7a
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.6
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=f7aec63e8640dcd5d42a0476301a065f; path=/

HTTP/1.1 301 OK
Date: Thu, 04 Mar 2010 04:07:27 GMT
Server: Apache/2.2.9 (Debian) DAV/2 SVN/1.5.1 PHP/5.2.6-1+lenny6 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: /start
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc1
Connection: Keep-Alive
Set-Cookie: PHPSESSID=0e75b5e83a844f8902e361394c39c39f; path=/

HTTP/1.1 301 Moved Permanently
Date: Thu, 04 Mar 2010 04:07:36 GMT
Location: http://www.infinitestillness.ie/ss/
Server: Zeus/4.3
Content-Type: text/html
Via: 1.1 bc3
Content-Length: 212
Connection: Keep-Alive
Set-Cookie: X-Mapping-enlokcai=E05A570E7E395D6AB72BEA6FC2D4D8D4; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:48 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=75cd9c74439efc3bca8a6f6a8fd429f4; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:48 GMT
Server: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_perl/2.0.4 Perl/v5.8.8
X-Powered-By: PHP/5.2.6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc3
Connection: Keep-Alive
Set-Cookie: PHPSESSID=e5fa1ae0a7149cb25f36f4cf63e55603; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:07:54 GMT
Server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.4 with Suhosin-Patch
X-Powered-By: PHP/5.2.4-2ubuntu5.4
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=4c256b274d52c2746423786faadd30c4; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:00 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=bb34c4a17da632e7aefab6246e7704ab; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:06 GMT
Server: Apache/1.3 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.8f AuthPG/1.3 FrontPage/5.0.2.2635
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.9
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc2
Connection: Keep-Alive
Set-Cookie: PHPSESSID=416bd54449181a0b0839ae7dcda95902; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:06 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Cache-Control: max-age=86400, must-revalidate
Pragma:
Expires: Sat, 09 Oct 2010 04:35:26 GMT
Vary: Accept
Set-Cookie: PHPSESSID=c977eedf73ae811cabbe26bc8e0313c5; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:08:06 GMT; path=/
Last-Modified: Tue, 28 Jul 2009 03:40:46 GMT
Content-Type: text/html; charset=utf-8

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:11 GMT
Server: Apache
X-Powered-By: PHP/5.2.6-1+lenny6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=e0a6ca2c8e29af432c8a1266e992e129; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:17 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=0423b5407d22301ff1e4b3d2f41b9970; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:14 GMT
Server: Apache/2.2.14 (Unix) mod_ssl/2.2.14 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc5
Connection: Keep-Alive
Set-Cookie: PHPSESSID=6ad778ad438fca511d10c7f438d181dd; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:20 GMT
Server: Apache
Last-Modified: Mon, 17 Aug 2009 04:07:41 GMT
ETag: "7c8c026-3aa-8b0b3d40"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Type: text/html
Via: 1.1 bc1
Content-Length: 938
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:20 GMT
Server: Apache/2
X-Powered-By: PHP/5.2.12
Set-Cookie: PHPSESSID=78e42396dc632a12633ef0f065b979be; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:09:47 GMT
Server: Apache/2
X-Powered-By: PHP/5.2.4
Expires: Fri, 19 Mar 2010 21:22:18 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Tue, 16 Feb 2010 10:57:16 GMT
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=2055abb25c60b3a28fc9810dd601fe13; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:09:47 GMT; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:08:24 GMT
Server: Apache
X-Powered-By: PHP/5.2.1
Set-Cookie: PHPSESSID=b6cc6c9f2dc42acb0dca07bd778b031c; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:26 GMT
Server: Apache
Cache-Control: no-cache, max-age=0, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/5.2.13
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc3
Connection: Keep-Alive
Set-Cookie: PHPSESSID=40555488da9765a46c63a96625ad28b6; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:27 GMT
Server: Apache/2.2.11 (Debian) PHP/5.2.6-1+lenny2 with Suhosin-Patch mod_ssl/2.2.11 OpenSSL/0.9.8g
X-Powered-By: PHP/5.2.6-1+lenny2
Set-Cookie: PHPSESSID=1f41c7f80dfad6ef9e9594963612472f; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Vary: Accept,Accept-Encoding
Content-Type: text/html; charset=utf-8

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:13 GMT
Server: Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.8
X-Powered-By: PHP/5.2.8
Content-Type: text/html
Via: 1.1 bc5
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:35 GMT
Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8i DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=f0ca06da67663004b31d8f14f721daab; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:40 GMT
Server: Apache
X-Powered-By: PHP/5.2.9
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc6
Connection: Keep-Alive
Set-Cookie: PHPSESSID=98f72bf1ed2792b143f47308b3b5f099; path=/

HTTP/1.1 403 Forbidden
Date: Thu, 04 Mar 2010 04:28:42 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1
Via: 1.1 bc5
Connection: Keep-Alive

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:45 GMT
Server: Apache/2.0.63 (FreeBSD) DAV/2 SVN/1.5.2 mod_python/3.3.1 Python/2.5.1 PHP/5.2.6 with Suhosin-Patch mod_ssl/2.0.63 OpenSSL/0.9.7e-p1 mod_fastcgi/2.4.6 mod_perl/2.0.3 Perl/v5.8.8
X-Powered-By: PHP/5.2.12
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=860147a02be38690829b458077eabd7c; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:47 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=t4962e1dunk93crufg8qil4375; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:53 GMT
Server: Apache/2.2.8 (Ubuntu) mod_python/3.3.1 Python/2.5.2 PHP/5.2.4-2ubuntu5.10 with Suhosin-Patch mod_ssl/2.2.8 OpenSSL/0.9.8g mod_perl/2.0.3 Perl/v5.8.8
X-Powered-By: PHP/5.2.4-2ubuntu5.10
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-cache, max-age=0, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset="utf-8"
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=5a5baaf3c09bd237859f534039e52f03; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:30:20 GMT
Server: Apache/2
X-Powered-By: PHP/5.2.4
Expires: Tue, 08 Jun 2010 22:50:34 GMT
Cache-Control: max-age=86400, must-revalidate
Pragma:
Last-Modified: Fri, 27 Nov 2009 10:10:06 GMT
Vary: Accept,Accept-Encoding,User-Agent
Content-Type: text/html; charset=utf-8
Via: 1.1 bc7
Connection: Keep-Alive
Set-Cookie: PHPSESSID=2089013901ea310463177f049f4a92b4; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:30:20 GMT; path=/

HTTP/1.1 200 OK
Date: Thu, 04 Mar 2010 04:28:55 GMT
Server: Apache
X-Powered-By: PHP/5.2.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept
Set-Cookie: PHPSESSID=b63d92df808dc2f1edafa1c1e8830409; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/
Content-Type: text/html; charset=utf-8

I noticed two uncommon HTTP headers. The expiry date in 1981 and the cookie, PastVisitor.

Expires: Thu, 19 Nov 1981 08:52:00 GMT
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/

Googling for 'cookie "PastVisitor"' turns up results referring directly to SilverStripe and results referring to websites running SilverStripe. This cookie name while generic sounding appears to be only used by SilverStripe and makes a good plugin pattern.

Googling for 'Thu, 19 Nov 1981 08:52:00 GMT' turns up many results. This date relates to the PHP language and is not useful in identifying SilverStripe.

6. Review of unique patterns identified

Pattern 1 - Meta generator tag

Examples:

$ grep -hi 'name="generator' *html | sed 's/^[ \t]*//g' | sort -u
    <meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
    <meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
    <meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
    <meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
    <meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
    <meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
    <meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />

The meta generator tag is most likely to be removed by a web developer. It sometimes has the version number which is useful. I will give this pattern a certainty of 100%.

Pattern 2 - Cookie PastVisitor

Googling for 'cookie "PastVisitor"' turns up results referring directly to SilverStripe and results referring to websites that turn out to be running SilverStripe. This cookie name, while generic sounding appears to be only used by SilverStripe and will make a good plugin match.

Examples:

$ grep -h PastV *meta
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:35 GMT; path=/
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:06:01 GMT; path=/
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:07:07 GMT; path=/
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:08:06 GMT; path=/
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:09:47 GMT; path=/
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:30:20 GMT; path=/
    Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/

I will give this pattern a certainty of 100% because I couldn't find any examples of a non-SilverStripe website using the cookie name.

Pattern 3 - 3 CSS files, layouts.css, typography.css and form.css

Many of the samples have at least 3 css files named layout.css, typography.css and form.css. The use of these names is not exclusive to SilverStripe and is considered best practice for making CSS frameworks but the order of their appearance combined with the folder structure is unique enough for a 'probable' plugin match.

    <link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/layout.css?m=1266347738" />
    <link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/typography.css?m=1266347623" />
    <link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/form.css?m=1247030621" />

Not all samples have the ?m= after the css filename. Here are examples without the ?m=

$ grep -h -A 2 layout.css plugin-development/tests/silverstripe/*html |fgrep -v "?m="

    <link rel="stylesheet" type="text/css" href="themes/dcsd/css/layout.css" />
    <link rel="stylesheet" type="text/css" href="themes/dcsd/css/form.css" />

    <link rel="stylesheet" type="text/css" href="tutorial/css/layout.css" >
    <link rel="stylesheet" type="text/css" href="tutorial/css/typography.css" >
    <link rel="stylesheet" type="text/css" href="tutorial/css/form.css" >

    <link rel="stylesheet" href="/themes/firstgalaxies/css/layout.css" type="text/css">
    <link rel="stylesheet" href="/themes/firstgalaxies/css/typography.css" type="text/css">
    <link rel="stylesheet" href="/themes/firstgalaxies/css/form.css" type="text/css">

I will give this pattern a certainty of 75% because these three CSS filenames are considered best practice.

Pattern 4 - image assets url structure

<img src="/assets/.*/_resampled/.*.jpg"

Examples:

At first it doesn't appear to be a very unique match but a google query for "/assets/ _resampled/" returned almost entirely SilverStripe websites. I will give this pattern a certainty of 75% because I found at least 1 non-SilverStripe example with Google.

7. Write the plugin

Build on the plugin template

The plugin template is found in the plugin-development/ folder. Copy this into the plugins/ folder with the name of your choosing. All plugin names have the .rb extension.

The template:

    Plugin.define "Plugin-Template" do
    author "Enter Your Name"
    version "0.1"
    description "Describe what the plugin identifies. Include the homepage of the software package"
    examples %w| include-some.net example-websites.com here.com |

    \# a comment block here is a good place to make notes for yourself and others

    \# There are four types of matches: regexp, text, ghdb
    \# Matches are enclosed in {} brackets and separated by commas
    matches [
    {:name=>"a brief description of the match, eg. powered by in footer",
    :certainty=>100, # 100 is certain, 75 is probably and 25 is maybe. if omitted, it defaults to 100.
    :regexp=>/This page was generated by <a href="http:\/\/www.genericcms.com\/en\/products\/generic-cms\/">Generic CMS<\/a>/ },

    {:name=>"title",
    :certainty=>75,
    :text=>"<title>Generic Homepage</title>" }
    ]
    end
``

Fill in the plugin name, author, version, description and examples fields. The examples are a ruby array delimited by whitespace and the http:// prefix is optional.
```text
    Plugin.define "SilverStripe" do
    author "Andrew Horton"
    version "0.1"
    description "SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"

    examples %w|http://beatone.co.uk/ http://charcoalinteriors.com.au/ http://customcanvas.fritzandandre.com/ http://hungryhearts.no http://maungataniwha.co.nz/ http://unbounded.org/ http://victoriaoruwari.com/ http://weonline.in http://www.arprostatecancer.org/ http://www.benpearce.co.nz/ http://www.bradyinc.com/ http://www.cavendishimaging.com/ http://www.chapmansurfboards.com/ http://www.choidoco.com/demo/ http://www.clockwork.co.nz/ http://www.enamaine.org/ http://www.executivemediasearch.com/ http://www.fairtradenap.net/ http://www.firstgalaxies.org/ http://www.frussian.com.ar/ http://www.fuel.ie/silverstripe http://www.gsbc.edu/ http://www.holistichealth.com/ http://www.hutmacherin.com/ http://www.infinitestillness.ie/ss http://www.intandemtheatre.org/ http://www.kitesurfnelson.co.nz/ http://www.latenightdisco.com/ http://www.lisamarieelliott.com/ http://www.maklerservice-greiz.de/ http://www.moerakihavenmotel.co.nz/ http://www.monjasantner.de/ http://www.moonlitekustoms.com/ http://www.moto-racepaint.com/ http://www.naciondnb.com/ http://www.nadabakery.co.nz/ http://www.peterpanvakantieclub.nl/ http://www.rcaforum.org.nz/ http://www.robert80.de/ http://www.silverstripe.com/ http://www.silverstripe.org.pl/ http://www.stillrunnin.com/ http://www.textiprints.com/ http://www.thelightboxdesigns.com/ http://www.tobychampion.co.uk/ http://www.upstreamgroup.com/ http://www.verus.com.tr/ http://www.wend.nl/ http://www.whileyouwait.co.nz/ |

    \# a comment block here is a good place to make notes for yourself and others

    \# There are four types of matches: regexp, text, ghdb
    \# Matches are enclosed in {} brackets and separated by commas
    matches [
    {:name=>"a brief description of the match, eg. powered by in footer",
    :certainty=>100, # 100 is certain, 75 is probably and 25 is maybe. Optional
    :regexp=>/This page was generated by <a href="http:\/\/www.genericcms.com\/en\/products\/generic-cms\/">Generic CMS<\/a>/ },

    {:name=>"title",
    :certainty=>75,
    :text=>"<title>Generic Homepage</title>" }
    ]
    end

Match Pattern 1 - Meta generator tag

Review the examples you have collected for match 1 and decide on what type of match is best suited to this pattern.

<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />

Match types are: regexp - Ruby regular expressions that start and end with / characters text - A text string enclosed in ' or " characters ghdb - Google Hacking Database. This uses some Google query parameters. It currently supports intitle:, filetype:, inurl:

For this plugin to match we will look for a meta tag with the name 'generator' and a content parameter that starts with "SilverStripe". Later we can extract the version number. A regular expression match is best suited for this.

The following regular expression will match the tag. /<meta name="generator"[^>]*content="SilverStripe/

Notice how I haven't tried to match the http-equiv="generator" part of the tag or the website URL in the content field. Those parts of the tag are irrelevant and may change in future versions of SilverStripe.

Note: If you don't understand regular expressions you could make a text match like: :text=>'<meta name="generator" http-equiv="generator" content="SilverStripe'

The plugin now looks like:

    Plugin.define "SilverStripe" do
    author "Andrew Horton"
    version "0.1"
    description "SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"

    examples %w|http://beatone.co.uk/ http://charcoalinteriors.com.au/ http://customcanvas.fritzandandre.com/ http://hungryhearts.no http://maungataniwha.co.nz/ http://unbounded.org/ http://victoriaoruwari.com/ http://weonline.in http://www.arprostatecancer.org/ http://www.benpearce.co.nz/ http://www.bradyinc.com/ http://www.cavendishimaging.com/ http://www.chapmansurfboards.com/ http://www.choidoco.com/demo/ http://www.clockwork.co.nz/ http://www.enamaine.org/ http://www.executivemediasearch.com/ http://www.fairtradenap.net/ http://www.firstgalaxies.org/ http://www.frussian.com.ar/ http://www.fuel.ie/silverstripe http://www.gsbc.edu/ http://www.holistichealth.com/ http://www.hutmacherin.com/ http://www.infinitestillness.ie/ss http://www.intandemtheatre.org/ http://www.kitesurfnelson.co.nz/ http://www.latenightdisco.com/ http://www.lisamarieelliott.com/ http://www.maklerservice-greiz.de/ http://www.moerakihavenmotel.co.nz/ http://www.monjasantner.de/ http://www.moonlitekustoms.com/ http://www.moto-racepaint.com/ http://www.naciondnb.com/ http://www.nadabakery.co.nz/ http://www.peterpanvakantieclub.nl/ http://www.rcaforum.org.nz/ http://www.robert80.de/ http://www.silverstripe.com/ http://www.silverstripe.org.pl/ http://www.stillrunnin.com/ http://www.textiprints.com/ http://www.thelightboxdesigns.com/ http://www.tobychampion.co.uk/ http://www.upstreamgroup.com/ http://www.verus.com.tr/ http://www.wend.nl/ http://www.whileyouwait.co.nz/ |

    #<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
    #<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
    #<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
    #<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
    #<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
    #<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
    #<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />

    matches [
    {:name=>"meta generator tag",
    :regexp=>/<meta name="generator"[^>]*content="SilverStripe/}

    ]
    end

I have included the meta generatator tag examples within the plugin as comments because this is a good place to refer to them later.

Plugin Testing

It's good practice to test your plugin while writing it to make sure it works.

If you want to ensure your plugin is loaded, run the following command: $ ./whatweb -l

This shows all loaded plugins displayed along with the version number.

Test your current plugin on the SilverStripe samples you have collected. The whatweb parameters used are: -v Verbose. This shows us which matches are being found -p Plugins. Only load the SilverStripe plugin

$ ./whatweb -v -psilverstripe ./plugin-development/tests/silverstripe/*html

./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html []
Identifying: ./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html
HTTP-Status:

./plugin-development/tests/silverstripe/beatone.co.uk-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/beatone.co.uk-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:certainty=>100}]]]

./plugin-development/tests/silverstripe/hungryhearts.no.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/hungryhearts.no.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:certainty=>100}]]]

./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html []
Identifying: ./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html
HTTP-Status:

./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
:name=>"meta generator tag",
:certainty=>100}]]]

We notice that charcoalinteriors.com.au and maungataniwha.co.nz aren't matching. After viewing the HTML files we notice they do not include the meta generator tag. Our first match is working correctly.

Match Pattern 2 - Cookie PastVisitor

Review the examples you have collected and decide on what type of match is best suited.

Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:35 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:06:01 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:07:07 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:08:06 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:09:47 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:30:20 GMT; path=/
Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:28:56 GMT; path=/

Matching the cookie cannot be done within the matches array. Create a function called passive that will be triggered whenever the plugin is used.

def passive
    m=[]

    m
end

This code creates a empty array called m and returns the value of that array. This array will contain hashes in the same format as the matches section.

def passive
    m=[]
    m << {:name=>"PastVisitor Cookie" } if @meta["set-cookie"] =~ /PastVisitor=[0-9]+.*/
    m
end

Now the function is checking the @meta array element "set-cookie" is see if it contains a regular expression that begins with PastVisitor= then has some numbers.

We cannot test this against the saved HTML files in our plugin-development/silverstripe/ folder. Instead we will test it using example sites. Note the whatweb parameter -e uses the examples in the loaded plugins as targets.

$ ./whatweb -v -psilverstripe -e
http://charcoalinteriors.com.au/ [200]
Identifying: http://charcoalinteriors.com.au/
HTTP-Status: 200

http://beatone.co.uk/ [200] SilverStripe
Identifying: http://beatone.co.uk/
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100}]]]

http://maungataniwha.co.nz/ [200]
Identifying: http://maungataniwha.co.nz/
HTTP-Status: 200

http://hungryhearts.no [200] SilverStripe
Identifying: http://hungryhearts.no
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100}]]]

http://customcanvas.fritzandandre.com/ [200] SilverStripe
Identifying: http://customcanvas.fritzandandre.com/
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100}]]]

http://unbounded.org/ [200] SilverStripe
Identifying: http://unbounded.org/
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100},
{:name=>"PastVisitor Cookie"}]]]

http://weonline.in [200] SilverStripe
Identifying: http://weonline.in
HTTP-Status: 200
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100}]]]

http://www.arprostatecancer.org/ [200]
Identifying: http://www.arprostatecancer.org/
HTTP-Status: 200

http://victoriaoruwari.com/ [200] SilverStripe
Identifying: http://victoriaoruwari.com/
HTTP-Status: 200
[["SilverStripe", [{:name=>"PastVisitor Cookie"}]]]

Our plugin has recognised the PastVistitor cookie for unbounded.org and victoriaoruwari.com so we know that it works.

Using just the two matches so far would be insufficient. Notice how some sites match only the meta generator tag, other match only the cookie and some aren't matched at all.

Match Pattern 3 - 3 CSS files, layouts.css, typography.css and form.css

Review the examples you have collected and decide on what type of match is best suited.

    <link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/layout.css?m=1266347738" />
    <link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/typography.css?m=1266347623" />
    <link rel="stylesheet" type="text/css" href="http://www.textiprints.com/themes/textiprints/css/form.css?m=1247030621" />

    <link rel="stylesheet" href="/themes/firstgalaxies/css/layout.css" type="text/css">
    <link rel="stylesheet" href="/themes/firstgalaxies/css/typography.css" type="text/css">
    <link rel="stylesheet" href="/themes/firstgalaxies/css/form.css" type="text/css">

I decided earlier to give this pattern a certainty of 75% because using this set of filenames for CSS files is considered best practice. I also discovered an example where the typography.css file wasn't included. I chose to not match that because three names in order is a more likely unique match.

A regular expression is the best choice to match match these three css files in order: /<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/

Including this regular expression gives us the following matches array in our plugin. Notice how a comma is included after the first match.

matches [
{:name=>"meta generator tag",
:certainty=>100,
:regexp=>/<meta name="generator"[^>]*content="SilverStripe/},

{:name=>"layout, typography, form css files",
:certainty=>75,
:regexp=>/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/}
]

Test the regular expression to make sure it works.

$ ./whatweb -v -psilverstripe ./plugin-development/tests/silverstripe/*html
./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html []
Identifying: ./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html
HTTP-Status:

./plugin-development/tests/silverstripe/beatone.co.uk-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/beatone.co.uk-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100}]]]

./plugin-development/tests/silverstripe/hungryhearts.no.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/hungryhearts.no.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100}]]]

./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html [] probably SilverStripe
Identifying: ./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>
/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/,
    :name=>"layout, typography, form css files",
    :certainty=>75}]]]

./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html [] SilverStripe
Identifying: ./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html
HTTP-Status:
[["SilverStripe",
[{:regexp=>/<meta name="generator"[^>]*content="SilverStripe/,
    :name=>"meta generator tag",
    :certainty=>100},
{:regexp=>
/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/,
    :name=>"layout, typography, form css files",
    :certainty=>75}]]]

3 of the first 5 didn't match the CSS files, charcoalinteriors.com.au, beatone.co.uk, and hungryhearts.no.html. I manually inspected the files to see what their CSS files were. None of them include the 3 CSS files in order so our regular expression works.

Match Pattern 4 - image assets url structure

Review the examples you have collected and decide on what type of match is best suited.

<img src="/assets/magazine/sr6/toc/_resampled/croppedimage220165-02-Two Vietnam Vets.jpg" alt="" />
<img class="left noborder" src="/assets/Uploads/services/icons/fundraisers-icon.jpg" alt="Fundraisers Icon" />

This match is best found with a regular expression.

Earlier I thought it didn't appear to be a unique match but a google query for "/assets/ _resampled/" returned almost entirely SilverStripe websites. I'm giving this pattern a certainty of 75% because I found at least 1 non-SilverStripe example with Google.

The following regular expression will work, <img src="/assets/[^/]+/_resampled/[^"]+.jpg" . In plain english this is read as:

<img src="/assets/anything but a slash/_resampled/anything but double quotes.jpg"

Our plugin matches array now looks like:

matches [
    {:name=>"meta generator tag",
    :certainty=>100,
    :regexp=>/<meta name="generator"[^>]*content="SilverStripe/},

    {:name=>"layout, typography, form css files",
    :certainty=>75,
    :regexp=>/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/},

    {:name=>"<img src="/assets/something/_resampled/something.jpg"",
    :certainty=>75,
    :regexp=>/<img src="/assets/[^/]+/_resampled/[^"]+.jpg"/}
]

Next I tested the plugin to confirm that it matches some of the samples.

Test the plugin against all the examples

Test the plugin against all the example websites to test how effectively it idenfies them. Instead of testing against the saved HTML files I will test against the example URLs in the plugin so that the match that checks the cookies will work.

$ ./whatweb -psilverstripe -e
http://unbounded.org/ [200] SilverStripe
http://charcoalinteriors.com.au/ [200]
http://customcanvas.fritzandandre.com/ [200] SilverStripe
http://beatone.co.uk/ [200] SilverStripe
http://maungataniwha.co.nz/ [200] probably SilverStripe
http://hungryhearts.no [200] SilverStripe
http://www.benpearce.co.nz/ [200]
http://victoriaoruwari.com/ [200] SilverStripe
http://www.arprostatecancer.org/ [200]
http://weonline.in [200] SilverStripe
http://www.choidoco.com/demo/ [200] SilverStripe
http://www.cavendishimaging.com/ [200] SilverStripe
http://www.bradyinc.com/ [200] SilverStripe
http://www.clockwork.co.nz/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.chapmansurfboards.com/ [200] SilverStripe
http://www.fairtradenap.net/ [200] probably SilverStripe
http://www.executivemediasearch.com/ [404]
http://www.fuel.ie/silverstripe [301]
http://www.firstgalaxies.org/ [200] SilverStripe
http://www.fuel.ie/silverstripe/ [200] SilverStripe
http://www.enamaine.org/ [200] SilverStripe
http://www.frussian.com.ar/ [200] SilverStripe
http://www.infinitestillness.ie/ss [301]
http://www.holistichealth.com/ [200] SilverStripe
http://www.gsbc.edu/ [200] SilverStripe
http://www.hutmacherin.com/ [301]
http://www.infinitestillness.ie/ss/ [200] SilverStripe
http://www.hutmacherin.com/start [200] probably SilverStripe
http://www.latenightdisco.com/ [200] SilverStripe
http://www.kitesurfnelson.co.nz/ [200] probably SilverStripe
http://www.intandemtheatre.org/ [200] SilverStripe
http://www.moerakihavenmotel.co.nz/ [200] SilverStripe
http://www.lisamarieelliott.com/ [200] SilverStripe
http://www.moonlitekustoms.com/ [200] SilverStripe
http://www.naciondnb.com/ ERROR: Socket error getaddrinfo: Name or service not known
http://www.maklerservice-greiz.de/ [200] SilverStripe
http://www.monjasantner.de/ [200] SilverStripe
http://www.nadabakery.co.nz/ [200] SilverStripe
http://www.rcaforum.org.nz/ [200] SilverStripe
http://www.moto-racepaint.com/ [200] SilverStripe
http://www.peterpanvakantieclub.nl/ [200] SilverStripe
http://www.robert80.de/ [200] SilverStripe
http://www.silverstripe.com/ [200] SilverStripe
http://www.stillrunnin.com/ [200] SilverStripe
http://www.textiprints.com/ [200] SilverStripe
http://www.thelightboxdesigns.com/ [200]
http://www.silverstripe.org.pl/ [200] SilverStripe
http://www.whileyouwait.co.nz/ [200] SilverStripe
http://www.tobychampion.co.uk/ [500]
http://www.wend.nl/ [200] SilverStripe
http://www.upstreamgroup.com/ [200] SilverStripe
http://www.verus.com.tr/ [200] SilverStripe

Most of the sites with a HTTP 301 status redirect to a page with a status of 200 and are identified as SilverStripe. Some sites are no longer active so I removed them from the examples list.

Of the 45 live websites, 43 are identified as SilverStripe. It is accurate to say our plugin, using only passive matches identifies about 95% of SilverStripe websites.

Extract version numbers

The meta generator tag sometimes contains version numbers which we want to detect.

<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />

To extract the version number I need to write some custom ruby code in the passive function.

if @body =~ /<meta name="generator"[^>]*content="SilverStripe [0-9\.]+/
    v=@body.scan(/<meta name="generator"[^>]*content="SilverStripe ([0-9\.]+)/)[0].to_s
    m << {:name=>"meta generator version", :certainty=>100, :version=>v }
end

Make sure that the regex patterns on the first two lines are the same.

Alternatively, as of WhatWeb 0.4.6 you can use:

{ :version=>/<meta name="generator"[^>]*content="SilverStripe ([0-9\.]+)/, :offset=>0 },

This code checks that the regular expression that has the SilverStripe version within the meta generator tag is in the HTML body. If so, it copies it into a variable called v then includes it in a hash which is put into the array of matches.

Testing the code shows that version numbers are being extracted. Note that if SilverStripe were to include letters after version number, eg. 2.3.5b that the letter wouldn't be recognised.

    $ ./whatweb -psilverstripe ./plugin-development/tests/silverstripe/*html
    ./plugin-development/tests/silverstripe/charcoalinteriors.com.au-.html []
    ./plugin-development/tests/silverstripe/beatone.co.uk-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/maungataniwha.co.nz-.html [] probably SilverStripe
    ./plugin-development/tests/silverstripe/hungryhearts.no.html [] SilverStripe
    ./plugin-development/tests/silverstripe/customcanvas.fritzandandre.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/victoriaoruwari.com-.html []
    ./plugin-development/tests/silverstripe/weonline.in.html [] SilverStripe[2.3.1]
    ./plugin-development/tests/silverstripe/unbounded.org-.html [] SilverStripe[2.0]
    ./plugin-development/tests/silverstripe/www.benpearce.co.nz-.html []
    ./plugin-development/tests/silverstripe/www.choidoco.com-demo-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.bradyinc.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.cavendishimaging.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.chapmansurfboards.com-.html [] SilverStripe[2.0]
    ./plugin-development/tests/silverstripe/www.arprostatecancer.org-.html []
    ./plugin-development/tests/silverstripe/www.executivemediasearch.com-.html []
    ./plugin-development/tests/silverstripe/www.enamaine.org-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.firstgalaxies.org-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.fairtradenap.net-.html [] probably SilverStripe
    ./plugin-development/tests/silverstripe/www.fuel.ie-silverstripe.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.holistichealth.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.hutmacherin.com-.html [] probably SilverStripe
    ./plugin-development/tests/silverstripe/www.gsbc.edu-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.clockwork.co.nz-.html []
    ./plugin-development/tests/silverstripe/www.frussian.com.ar-.html [] SilverStripe[2.0]
    ./plugin-development/tests/silverstripe/www.infinitestillness.ie-ss.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.lisamarieelliott.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.latenightdisco.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.kitesurfnelson.co.nz-.html [] probably SilverStripe
    ./plugin-development/tests/silverstripe/www.intandemtheatre.org-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.moerakihavenmotel.co.nz-.html [] SilverStripe[2.0]
    ./plugin-development/tests/silverstripe/www.maklerservice-greiz.de-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.monjasantner.de-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.naciondnb.com-.html []
    ./plugin-development/tests/silverstripe/www.moto-racepaint.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.nadabakery.co.nz-.html [] SilverStripe[2.3.1]
    ./plugin-development/tests/silverstripe/www.moonlitekustoms.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.peterpanvakantieclub.nl-.html [] SilverStripe[2.0]
    ./plugin-development/tests/silverstripe/www.robert80.de-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.rcaforum.org.nz-.html [] SilverStripe[2.3.0]
    ./plugin-development/tests/silverstripe/www.silverstripe.org.pl-.html []
    ./plugin-development/tests/silverstripe/www.thelightboxdesigns.com-.html []
    ./plugin-development/tests/silverstripe/www.silverstripe.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.textiprints.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.stillrunnin.com-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.tobychampion.co.uk-.html [] SilverStripe
    ./plugin-development/tests/silverstripe/www.upstreamgroup.com-.html [] SilverStripe[2.3.1]
    ./plugin-development/tests/silverstripe/www.whileyouwait.co.nz-.html [] SilverStripe[2.0]
    ./plugin-development/tests/silverstripe/www.wend.nl-.html [] probably SilverStripe
    ./plugin-development/tests/silverstripe/www.verus.com.tr-.html [] SilverStripe

We can see that only some websites include the version number in the meta generator tag.

The final plugin

Plugin.define "SilverStripe" do
author "Andrew Horton"
version "0.1"
description "SilverStripe is an opensource CMS written in PHP. It can run on Apache, IIS or lighthttpd. Homepage: http://www.silverstripe.com"

examples %w|http://beatone.co.uk/ http://charcoalinteriors.com.au/ http://customcanvas.fritzandandre.com/ http://hungryhearts.no http://maungataniwha.co.nz/ http://unbounded.org/ http://victoriaoruwari.com/ http://weonline.in http://www.arprostatecancer.org/ http://www.benpearce.co.nz/ http://www.bradyinc.com/ http://www.cavendishimaging.com/ http://www.chapmansurfboards.com/ http://www.choidoco.com/demo/ http://www.enamaine.org/ http://www.executivemediasearch.com/ http://www.fairtradenap.net/ http://www.firstgalaxies.org/ http://www.frussian.com.ar/ http://www.fuel.ie/silverstripe http://www.gsbc.edu/ http://www.holistichealth.com/ http://www.hutmacherin.com/ http://www.infinitestillness.ie/ss http://www.intandemtheatre.org/ http://www.kitesurfnelson.co.nz/ http://www.latenightdisco.com/ http://www.lisamarieelliott.com/ http://www.maklerservice-greiz.de/ http://www.moerakihavenmotel.co.nz/ http://www.monjasantner.de/ http://www.moonlitekustoms.com/ http://www.moto-racepaint.com/ http://www.nadabakery.co.nz/ http://www.peterpanvakantieclub.nl/ http://www.rcaforum.org.nz/ http://www.robert80.de/ http://www.silverstripe.com/ http://www.silverstripe.org.pl/ http://www.stillrunnin.com/ http://www.textiprints.com/ http://www.thelightboxdesigns.com/ http://www.tobychampion.co.uk/ http://www.upstreamgroup.com/ http://www.verus.com.tr/ http://www.wend.nl/ http://www.whileyouwait.co.nz/ |

#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.0 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe 2.3.1 - http://www.silverstripe.com" />
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" >
#<meta name="generator" http-equiv="generator" content="SilverStripe - http://www.silverstripe.com" />

matches [
    {:name=>"meta generator tag",
    :certainty=>100,
    :regexp=>/<meta name="generator"[^>]*content="SilverStripe/}, #" I have included a comment with double quotes for the benefit of syntax hilighting in gedit

    {:name=>"layout, typography, form css files",
    :certainty=>75,
    :regexp=>/<link[^>]*stylesheet[^>]*layout.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*typography.css[^>]*>[^<]*<link[^>]*stylesheet[^>]*form.css[^>]*>/},

    {:name=>'<img src="/assets/something/_resampled/something.jpg"',
    :certainty=>75,
    :regexp=>/<img src="\/assets\/[^\/]+\/_resampled\/[^"]+.jpg"/} #"

]

# Set-Cookie: PastVisitor=1; expires=Wed, 02-Jun-2010 04:05:29 GMT; path=/
def passive
    m=[]
    m << {:name=>"PastVisitor Cookie"  } if @meta["set-cookie"] =~ /PastVisitor=[0-9]+.*/

    if @body =~ /<meta name="generator"[^>]*content="SilverStripe [0-9\.]+/
        v=@body.scan(/<meta name="generator"[^>]*content="SilverStripe ([0-9\.]+)/)[0].to_s
        m << {:name=>"meta generator version", :certainty=>100, :version=>v }
    end
    m
end

end

8. Closing Notes

I have shown you the process of how to write a simple WhatWeb plugin. The most important part of the process is the complete research that happens before writing any matches.

Some people will be tempted to write a pattern for the meta-generator tag then stop. Such an approach would identify about 75% of SilverStripe sites. Futhermore there is a generic meta generator tag plugin so such an effort would be of little practical use.

Our final plugin identifies about 95% of SilverStripe websites using only passive matches. An aggressive plugin that guesses URLs would increase the effectivenses to 100% however aggressive plugins are not stealthy, they use more bandwidth and so are less suitable for large scale website identification. However aggressive plugins are useful during penetration testing to identify frameworks. Writing aggressive plugins is a more advanced topic and will be covered in another tutorial.

Please submit your plugins to andrew [at] morningstarsecurity.com to be included in the next release of WhatWeb.

9. Resources

The best way to learn how to develop plugins is by reading the plugins bundled with WhatWeb.

To learn and test regular expressions visit: http://rubular.com/ WhatWeb homepage: http://www.morningstarsecurity.com/research/whatweb

Visit MorningStar Security for the best Information Security news at http://www.morningstarsecurity.com/news

10. Convert plugins from the old format to the new

After version 0.5.0 plugins now define their name within the Plugin as a variable.

sed -i -r "s#Plugin\.define\s+(\"[^\"]+\")\s+do#Plugin.define do\nname \1#" *.rb

How to develop WhatWeb plugins

How to develop WhatWeb plugins

Contents

1. Introduction to WhatWeb

2. Introduction to WhatWeb Plugins

General aims of a plugin

Methods to identify systems

Important files and folders

Anatomy of a plugin

Accessible Variables

@body.

@meta

@status

@base_uri

@md5_sum

@tagpattern

@ip

3. Research background information

4. Collect samples

Website Showcases

Using Search Engines

Introduction

SilverStripe example

Forums for website development with the cms:

5. Anaylze samples

Read the source of a couple of samples

Collect HTML and HTTP headers from samples

Remove incorrectly identified samples

How many sites can we be certain are SilverStripe?

Examine the samples with WhatWeb

Remove more incorrectly identified samples with the whatweb report

Use find-common-stuff to automatically identify common strings in the samples

Analyse HTTP headers and cookies

Read more HTML source

6. Review of unique patterns identified

Pattern 1 - Meta generator tag

Pattern 2 - Cookie PastVisitor

Pattern 3 - 3 CSS files, layouts.css, typography.css and form.css

Pattern 4 - image assets url structure

7. Write the plugin

Build on the plugin template

Match Pattern 1 - Meta generator tag

Plugin Testing

Match Pattern 2 - Cookie PastVisitor

Match Pattern 3 - 3 CSS files, layouts.css, typography.css and form.css

Match Pattern 4 - image assets url structure

Test the plugin against all the examples

Extract version numbers

The final plugin

8. Closing Notes

9. Resources

10. Convert plugins from the old format to the new

I - Basics

II - Development

III - Bugs

Clone this wiki locally