Tracking Downloads To Your Eclipse Plug-in Update Site

 Source

Eclipse Update Site
artifacts.xsl

Introduction

My company wanted to add a plug-in to Eclipse for our internal bug-tracking system. No problem I say and I build that sucker. I used Spring Tool Suite 3.5.1 (which is just Eclipse 4.3 with a fancy paint job) with Maven, and deployed to a Linux Red Hat server, mostly following the vogella tutorials. But now they want to know adoption rates and that will be part of my annual review! Holy shit I better find out how to get some metrics quick! Oh look here is the p2 documentation: Equinox p2 download stats

Umm... okay well thanks for nothing really. The idea is to add 2 properties to artifacts.xml, which is packaged inside of artifacts.jar, that tells the update site to perform an HTTP HEAD request (it's like a HTTP GET request but without body content) whenever your plug-in is downloaded with the url you provide. However if you use Eclipse (for your Eclipse plug-in) then we are immediately going to run into some problems. The p2 instructions would like us to modify artifacts.xml, however that file is generated automajically and there are no way to set properties in your project and have it appear in artifacts.xml. Well not yet at least: Eclipse Project 4.5 M1 - New and Noteworthy. Sonofabitch!
So what is suggested is to perform the following steps:

1. Unpack artifacts.jar
2. Modify artifacts.xml and add the 'p2.statsURI' and 'download.stats' properties.
3. Repackage artifacts.jar

 Configuring the Eclipse Update Site

Aw fuck, so I have to waste time and write scripts now? Don't bang your head on the table I've already figured it out. That's what I'm here for.  First we have to create an XSLT transform to deal with adding the properties to artifacts.xml that point to your url. The transform will have 2 parameters, which can be passed on the command-line, that will set the value of the url that gets called when the plug-in is downloaded. That url is ultimately a combination of the 'p2.statsURI' and 'download.stats' properties.  So if set

• p2.statsURI=www.foo.com/updatesite/stats/
•  download.stats=index.jsp

The website that will get called is 'www.foo.com/updatesite/stats/index.jsp'

So in your update site project create a file called 'artifacts.xsl'. For my example I created a subfolder called 'xslt' and placed artifacts.xsl there.

artifacts.xsl

<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <xsl:output method="xml" version="1.0" encoding="UTF-8"  indent="yes" />

  <xsl:param name="p2.statsURI"></xsl:param>
  <xsl:param name="download.stats"></xsl:param>

  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()" />
    </xsl:copy>
  </xsl:template>

  <xsl:template match="/repository/properties">
    <xsl:element name="properties">
      <xsl:attribute name="size">
        <xsl:value-of select="@size + 1" />
      </xsl:attribute>
      <xsl:apply-templates select="node()" />
      <xsl:element name="property">
        <xsl:attribute name="name">p2.statsURI</xsl:attribute>
        <xsl:attribute name="value">
          <xsl:value-of select="$p2.statsURI" />
        </xsl:attribute>
      </xsl:element>
    </xsl:element>
  </xsl:template>

  <xsl:template match="artifact[@classifier='org.eclipse.update.feature']">
    <xsl:element name="artifact">
      <xsl:attribute name="classifier">
        <xsl:value-of select="@classifier"></xsl:value-of>
      </xsl:attribute>
      <xsl:attribute name="id">
        <xsl:value-of select="@id"></xsl:value-of>
      </xsl:attribute>
      <xsl:attribute name="version">
        <xsl:value-of select="@version"></xsl:value-of>
      </xsl:attribute>
      <xsl:apply-templates select="node()" mode="artifact" />
    </xsl:element>
  </xsl:template>
  
  <xsl:template match="properties" mode="artifact">
    <xsl:element name="properties">
      <xsl:attribute name="size">
        <xsl:value-of select="@size + 1" />
      </xsl:attribute>
      <xsl:apply-templates select="node()" mode="artifact" />
      <xsl:element name="property">
        <xsl:attribute name="name">download.stats</xsl:attribute>
        <xsl:attribute name="value">
          <xsl:value-of select="$download.stats" />
        </xsl:attribute>
      </xsl:element>
    </xsl:element>
  </xsl:template>
  
  <xsl:template match="property" mode="artifact">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()" />
    </xsl:copy>
  </xsl:template>

</xsl:stylesheet>

Cool. Next we need to modify the update site's pom.xml to add the Maven Ant plugin. This will allow us to hook into a maven life-cycle phase and run some Ant tasks that we specify. OBEY ME ANT! MAVEN IS NO LONGER YOUR MASTER! Right. So in a nutshell we will demand Ant to do the following after the update site has been compiled and packaged into their jars.

1. Unpack artifacts.jar in a temporary folder called 'output'. Artifacts.jar contains only 1 file, artifacts.xml.
2. Modify artifacts.xml and add the 'p2.statsURI' and 'download.stats' properties using the transform above, artifacts.xsl. In the param nodes set your site's values in the expression attributes.
1. For 'p2.statsURI' we will be eventually creating a subfolder in your update site's location called 'stats' that will contain the files used to track downloads. Don't forget the slash at the end.
2. For 'download.stats' this will be the page that will process the HTTP HEAD request. For my example I will use JSP but you can use PHP, Perl, or whatever I don't care I'm not your boss.
3. Repackage artifacts.jar.
4. Remove the temp folder 'output'.

pom.xml

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-antrun-plugin</artifactId>
  <version>1.7</version>
  <executions>
    <execution>
      <id>prepare</id>
      <phase>package</phase>
      <configuration>
        <tasks>
          <echo message="Add download stats to artifacts.jar" />
          <unzip src="${project.build.directory}/repository/artifacts.jar" dest="output/" />
          <xslt in="output/artifacts.xml" out="output/artifacts2.xml" style="${basedir}/xslt/artifacts.xsl">
            <param name="p2.statsURI" expression="http://yoursite/updatesite/stats/"/>
            <param name="download.stats" expression="insert.jsp"/>
          </xslt>
          <move file="output/artifacts2.xml" tofile="output/artifacts.xml" overwrite="true" />
          <zip destfile="${project.build.directory}/repository/artifacts.jar"  basedir="output" update="true" />
          <delete dir="output"/>
        </tasks>
      </configuration>
      <goals>
        <goal>run</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Processing the HTTP Request

Yay we're done! Not so fast. We're not done. Not at all. Now we have to process that page the update site will call when your plug-in is downloaded. My example will be in JSP with a MySQL database to store the data. Also here is an example in PHP: p2-stats-recorder

First let's set up the MySQL database to hold the information of the folks who download the plug-in.  Create a new database called 'p2_stats'.

CREATE DATABASE p2_stats;

Create a user account named 'p2' with the password 'p2' (or you can be smart and name it something stronger, please be smart!)

CREATE USER 'p2'@'localhost' IDENTIFIED BY 'p2';

Next let's create our lone table.

CREATE TABLE `installations` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`package` VARCHAR(255),
`version` VARCHAR(255),
`os` VARCHAR(255),
`host` VARCHAR(255),
`created_on` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY(`id`));

Finally let's give the 'p2' user access to the 'installations' table.

GRANT INSERT, SELECT, UPDATE, DELETE ON `installations` TO 'p2'@'localhost';

That's it. This is identical to schema.sql in the p2-stats-recorder, except I use 'created_on' as the name of my timestamp column, removed the ui column that I personally won't need, and added a primary key column... because... you know... database normalization.

Now on your server in your update site directory create a folder called 'stats'.  This is where all of our scripts to track downloads will be.

Next is to create the script that will process the HTTP HEAD request. As I mentioned before I am using JSP which requires the MySQL Java connector to be installed. Just go here Download Connector/J, unzip that guy and place the jar in a your updatesite folder in the location 'WEB-INF\lib', with 'WEB-INF' being a sibling of 'features', 'plugins', and your newly created 'stats' folder. MySQL says to fuck around with the path but fuck that shit as this method is preferred for so many reasons. Also don't be stupid like me and waste hours trying to figure out why a connection to MySQL cannot be established and first make sure that your MySQL server is started.

Create a new file in your 'stats' folder called 'insert.jsp' or whatever value you gave your 'download.stats' xslt parameter. Essentially what we want to do is establish a connection to our 'p2_stats' MySQL database, then insert a new record with data gathered from the HTTP header. Here is the code for that guy.:

insert.jsp

<%@ page import="java.sql.*" %> 
<%@ page import="java.io.*" %> 

<html>
<head>
<title>My Eclipse Plug-in Download Counter</title>
<body>
<h2>Insert download record</h2>

<%

try {
  String connectionURL = "jdbc:mysql://localhost:3306/p2_stats";
  Connection connection = null; 
  PreparedStatement pstatement = null;

  Class.forName("com.mysql.jdbc.Driver").newInstance(); 
  connection = DriverManager.getConnection(connectionURL, "p2", "p2");
  if(!connection.isClosed()) {    
    String queryString = "INSERT INTO installations(package, version, os, host) VALUES(?, ?, ?, ?)";                   
    pstatement = connection.prepareStatement(queryString);
    pstatement.setString(1, request.getParameter("project.name"));
    pstatement.setString(2, request.getParameter("project.version"));
    pstatement.setString(3, request.getParameter("os.name"));
    pstatement.setString(4, request.getRemoteHost());
                
    int updateQuery = pstatement.executeUpdate();
    if (updateQuery != 0) {
      out.println("Successfully inserted record.<BR/>");
      out.println("project.name: " + request.getParameter("project.name") + "<BR/>");
      out.println("project.version: " + request.getParameter("project.version") + "<BR/>");
      out.println("os.name: " + request.getParameter("os.name") + "<BR/>");
      out.println("host: " + request.getRemoteHost() + "<BR/>");
    }
  }
  connection.close();
}catch(Exception ex){
    out.println("Unable to connect to database. " + ex);
}

%> 

</body>
</html>

Now we can test the page by opening a browser and using the url:
http://yoursite/updatesite/stats/insert.jsp

You can ensure a record was created by looking at your 'installations' table in the database. You can also check your server logs. For example I am using Tomcat so in the directory '/var/log/tomcat' I can look in the file 'localhost_access_log' and I should see an entry:

"HEAD /eclipse-updatesite/stats/insert.jsp HTTP/1.1" 200 240

Hooray it worked! You now have a new record in your database. Right now some of the columns are blank and I'm sorry I may have deceived you... but look the basics are working and I promise I will fix that in a later "advanced reporting" section.

That Later Advanced Reporting Section

AW FUCK THIS SECTION IS WRONG. COME BACK TOMORROW AND I'LL HAVE IT FIXED.

Let's say you want to track some information about who and what was downloaded. Seems reasonable. All that has to be done is the following:

1. Add url parameters to the download.stats page in the pom.xml 
2. Capture those parameters in our download.stats page
3. Insert the parameters into the database. 

Luckily the code to capture the parameters and insert them into the database were already included above but I saved this part for a later section because we want to be smart about it and URL encode the values we pass and that requires further details. First thing we need to do is add project properties to the update site's pom.xml so we can modify them with URL encoded values. For example, let's say we want to track the project name, the project version, and the user's operating system. Then we can add some properties that will hold those values which we can then manipulate later.

pom.xml

  <properties>
    <project.name.url>${project.name}</project.name.url>
    <project.version.url>${project.version}</project.version.url>
  </properties>

Cool. Next let's add a new plugin to run groovy scripts called gmaven-plugin which will allow us to run Java commands. So we will take our properties we added above and be able to URL encode them or perform any other string manipulation you would need. You might notice I am adding the os.name.url variable here and didn't include it above. That is because the os.name is a System property and any the other guys are Maven properties which will be null at this point and that is why we added them above.

pom.xml

      <plugin>
        <groupId>org.codehaus.gmaven</groupId>
        <artifactId>gmaven-plugin</artifactId>
        <version>1.5</version>
        <executions>
          <execution>
            <id>setup-groovy</id>
            <phase>initialize</phase>
            <goals>
              <goal>execute</goal>
            </goals>
            <configuration>
              <source>
                project.properties["project.name.url"] = java.net.URLEncoder.encode(project.properties['project.name.url'])
                project.properties["project.version.url"] = java.net.URLEncoder.encode(project.properties['project.version.url'])
                project.properties["os.name.url"] = java.net.URLEncoder.encode(System.properties['os.name'])                
              </source>              
            </configuration>
          </execution>
        </executions>
      </plugin>

Finally let's append URL parameters to the download.stats xslt parameter.  So find that guy again in your pom.xml and make the change from this:

<param name="download.stats" expression="insert.jsp" />

to this:

<param name="download.stats" expression="insert.jsp?project.name=${project.name.url}&amp;project.version=${project.version.url}&amp;os.name=${os.name.url}" />

And that's it! You can add as many parameters as you like, just remember to separate them with & and not just an & since we are working with xml.  Then whatever parameter you add, alter your p2_stats table to add a new column that you want to map to it, and change the sql statement in your download.stats to insert your new parameter to your new column.

I just want to say this took me two days to work through.  If this saved you any time I suggest you stop what you are doing and enjoy whatever time I saved you with friends, family, a good book, binge watch that Netflix show, whatever.