Programming For The Rest Of Us

Saturday, August 1, 2020

DayZ Modding: Adding Custom Cars

Step 1: Install mod

This tutorial isn't focused on installing a mod, if you need information I'll gather links to various tutorials, but you'll need the mod installed on your DayZ server.

For our comprehensive tutorial, we'll be using the mod [CrSk] BMW 525i E34, which adds a 90 BMW and is near production-level mod.

So go install the mod or the mod of your choice.

Step 2: Prepare your class names

In order to add the items in your mod to your DayZ server, you'll first need to tell DayZ what will be added. Most mods will list class names (unique identifiers for your mod items) in their description or have an XML file in the install folder to help get you quickly started.

For example the ARMA 2 ACR Weapons Pack has an XML folder in the install directory with the files to append (not replace.) If your mod has these files then skip to Step 4.

At this point if you don't know your mod class names to add, you'll need an administration mod like ZomBerry Admin Tools or VPPAdminTools to be installed, which you'll probably need anyway.

Step 3: Gather class names

Using your admin tool, go to the Spawn menu and search for items in your mod, in our example I will search for "bmw" which will give you a list of the class name and then a description in brackets.

We are interested in the class name like, "CrSk_BMW_525i_E34".

The list seems long but these is an extreme example because this mod has 4 variants of the same car, a gray, black, red, and beater (wrecked) version.

So let's just focus on the base CrSk_BMW_525i_E34 variant.

BMW Class Name	Description
CrSk_BMW_525i_E34	car
BMWE34_bagazhnik	trunk
BMWE34_kapot	hood
BMWE34_dver_1_1	driver's side front door
BMWE34_dver_1_2	pasenger's side front
BMWE34_dver_2_1	driver's side rear door
BMWE34_dver_2_2	pasenger's side rear door

Now that we have our class names we can start editing the files.

Step 4: Backup your files

You are not perfect. You will make mistakes. There are several files we're going to be editing, and one mistake can cause hours of painful searching.

Back up your files every time before editing them.

This could mean copying the files into a new backup folder, or better yet, create a GitHub account and use git to version control your files.

I also recommend using a free editor such as Notepad++, Visual Studio Code, Eclipse, or any IDE to help validate your XML while you edit. Some more powerful IDEs, like Visual Studio Code and Eclipse, integrate with GitHub.

Step 5: Edit types.xml

The file types.xml is a list of every item in the game. So it's a good first place to start.

Browse to "mpmissions\dayzOffline.enoch\db", assuming you are using Livonia, and open the file "types.xml" in your editor. Do not use Widows Notepad as it has a habit of converting the file encoding and is not XML aware.

If your mod has a types.xml already, open it up and copy all of the <type> nodes into the bottom of your file, but inside the root node. If you're not familiar with XML, each file has a root node (in this case <types>) that contain all of the child nodes, so your insertion must be below the last type node and before the closing </types> tag.

Because it is such a big file, let's pretend "..." is all of the already existing content of the file. Here I copied the contents of the Arma 2 DLC Weapon Pack after the last type node, and before the closing </types> tag.


<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<types>
    ...
    ... (existing content)
    ...
    <type name="ZucchiniSeedsPack">
        <nominal>30</nominal>
        <lifetime>3600</lifetime>
        <restock>0</restock>
        <min>20</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="food"/>
        <usage name="Farm"/>
    </type>

    <!-- @ArmA 2 DLC WEAPON PACK -->
    <type name="A2DLC_CZ805A2_StndBttstck">
        <nominal>2</nominal>
        <lifetime>7200</lifetime>
        <restock>1800</restock>
        <min>2</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="weapons"/>
        <usage name="Military"/>
        <value name="Tier3"/>  
        <value name="Tier4"/>    
    </type>
    <type name="A2DLC_CZ805A2_SnprBttstck">
        <nominal>2</nominal>
        <lifetime>7200</lifetime>
        <restock>1800</restock>
        <min>2</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="weapons"/>
        <usage name="Military"/> 
        <value name="Tier4"/>    
    </type> 
</types>

However if your mod has no types.xml to import, like the BMW mod, then you'll need to create these nodes yourself. But don't worry we can just find a similar item, copy that XML, and replace the name attribute of the type node with our class name.

Since the BMW is a sedan, we can find the already existing "CivilianSedan" node, then copy the nodes for the doors, hood, trunk, and wheels, and paste them after the last type node and before the closing </types> tag, and replace the CivilianSedan parts with our BMW parts.

CivilianSedan Class Name	BMW Class Name	Description
CivilianSedan	CrSk_BMW_525i_E34	car
CivSedanTrunk	BMWE34_bagazhnik	trunk
CivSedanHood	BMWE34_kapot	hood
CivSedanDoors_Driver	BMWE34_dver_1_1	driver's side front door
CivSedanDoors_CoDriver	BMWE34_dver_1_2	pasenger's side front
CivSedanDoors_BackLeft	BMWE34_dver_2_1	driver's side rear door
CivSedanDoors_BackRight	BMWE34_dver_2_2	pasenger's side rear door

Your file would look like this with "..." representing the already existing content of the file.

Note: In Livonia, "vehiclesparts" category items do not spawn for some reason, so I changed the category name to "tools" and it worked.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<types>
    ...
    ... (existing content)
    ...
    <type name="ZucchiniSeedsPack">
        <nominal>30</nominal>
        <lifetime>3600</lifetime>
        <restock>0</restock>
        <min>20</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="food"/>
        <usage name="Farm"/>
    </type>

    <!-- @[CrSk] BMW 525i E34 -->
    <type name="CrSk_BMW_525i_E34">
        <nominal>0</nominal>
        <lifetime>3888000</lifetime>
        <restock>1800</restock>
        <min>0</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
    </type>
    <type name="BMWE34_bagazhnik">
        <nominal>20</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>10</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_kapot">
        <nominal>20</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>10</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_dver_2_2">
        <nominal>20</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>10</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_dver_1_2">
        <nominal>20</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>10</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_dver_2_1">
        <nominal>20</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>10</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_dver_1_1">
        <nominal>20</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>10</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    
</types>

Next we copy these nodes above for our red, black, and beater variants of the BMW, just add "_red", "_black", and "_beater" to the end of each class.

CivilianSedan Class Name	BMW Class Name	BMW Black Class Name	BMW Red Class Name	BMW Beater Class Name	Description
CivilianSedan	CrSk_BMW_525i_E34	CrSk_BMW_525i_E34_black	CrSk_BMW_525i_E34_red	CrSk_BMW_525i_E34_beater	car
CivSedanTrunk	BMWE34_bagazhnik	BMWE34_bagazhnik_black	BMWE34_bagazhnik_red	BMWE34_bagazhnik_beater	trunk
CivSedanHood	BMWE34_kapot	BMWE34_kapot_black	BMWE34_kapot_red	BMWE34_kapot_beater	hood
CivSedanDoors_Driver	BMWE34_dver_1_1	BMWE34_dver_1_1_black	BMWE34_dver_1_1_red	BMWE34_dver_1_1_beater	driver's side front door
CivSedanDoors_CoDriver	BMWE34_dver_1_2	BMWE34_dver_1_2_black	BMWE34_dver_1_2_red	BMWE34_dver_1_2_beater	pasenger's side front
CivSedanDoors_BackLeft	BMWE34_dver_2_1	BMWE34_dver_2_1_black	BMWE34_dver_2_1_red	BMWE34_dver_2_1_beater	driver's side rear door
CivSedanDoors_BackRight	BMWE34_dver_2_2	BMWE34_dver_2_2_black	BMWE34_dver_2_2_red	BMWE34_dver_2_2_beater	pasenger's side rear door

Next let's do the same for our wheels. I'm going to copy the type node for "CivSedanWheel" for each of the wheel variants, and then "CivSedanWheel_Ruined" for the shtamp wheel variants, and paste them below our "BMWE34_dver_1_1" type node.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<types>
    ...
    ... (existing content)
    ...
 
    <type name="BMWE34_koleso_shtamp_black">
        <nominal>4</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>1</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_koleso_shtamp">
        <nominal>4</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>1</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_koleso_style37">
        <nominal>8</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>4</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_koleso_style21_black">
        <nominal>8</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>4</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_koleso_style21">
        <nominal>8</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>4</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_koleso_style16">
        <nominal>8</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>4</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_koleso_style2">
        <nominal>8</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>4</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    <type name="BMWE34_koleso_style5">
        <nominal>8</nominal>
        <lifetime>10800</lifetime>
        <restock>0</restock>
        <min>4</min>
        <quantmin>-1</quantmin>
        <quantmax>-1</quantmax>
        <cost>100</cost>
        <flags count_in_cargo="0" count_in_hoarder="0" count_in_map="1" count_in_player="0" crafted="0" deloot="0"/>
        <category name="vehiclesparts"/>
        <tag name="floor"/>
        <usage name="Industrial"/>
    </type>
    
</types>

Step 6: Edit cfgspawnabletypes.xml

For most mods you are done, sort of. Skip to Step 10. However for more complex items, such as cars, we'll need to do some more setup.

The cfgspawnabletypes.xml file will configure how your car will spawn. Will it have wheels and doors? Will it have a spark plug? Will it contain other loot in its trunk? This is where you will set those values.

Open the file, browse to "mpmissions\dayzOffline.enoch", assuming you are using Livonia, and open the file "cfgspawnabletypes.xml".

Did you back up the file first?

We'll use the same trick where we will find a similar item, copy its type node, and paste it after the last type node and before the closing </spawnabletypes> tag. The type name attribute here is the same as the class in types.xml.

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

<spawnabletypes>
    <damage min="0.3" max="0.7" />
    ...
    ... (existing content)
    ...
    <!-- @[CrSk] BMW 525i E34 -->
    <type name="CrSK_BMW_525i_E34">
        <attachments chance="1.00">
            <item name="BMWE34_koleso_style2" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_koleso_style2" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_koleso_style2" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_koleso_shtamp" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="CarRadiator" chance="0.40" />
        </attachments>
        <attachments chance="1.00">
            <item name="CarBattery" chance="0.40" />
        </attachments>
        <attachments chance="1.00">
            <item name="SparkPlug" chance="0.40" />
        </attachments>
        <attachments chance="1.00">
            <item name="HeadlightH7" chance="0.40" />
        </attachments>
        <attachments chance="1.00">
            <item name="HeadlightH7" chance="0.40" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_dver_1_1" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_dver_2_1" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_dver_1_2" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_dver_2_2" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_kapot" chance="0.60" />
        </attachments>
        <attachments chance="1.00">
            <item name="BMWE34_bagazhnik" chance="0.60" />
        </attachments>
    </type>

</spawnabletypes>

Next go through and edit the chance values from 0.00 to 1.00, with 0 being never spawning as an attachment and 1 being always spawn as an attachment.

What value you choose is up to your playstyle. Should it be harder for a user to build a car or should it be pretty easy to drive away once you spot a car. This will take some trial-and-error on your part.

Step 7: Edit events.xml

Again, for most items like clothes and guns you're already done, but for more complex items like cars you'll need to manually place them in your map.

To do that first you'll need to create an "event" in the events.xml file located in the folder "mpmissions\dayzOffline.enoch\db", assuming you are using Livonia.

I've read some documentation that there is a convention for the naming of events. If you look you'll see some events start with "Static", animals start with "Animal", and car events start with "Vehicle", and the names use title case with no spaces, hyphens, or underscores, so you should probably follow this convention to avoid any issues.

Did you backup your file?

We will again find a similar event node, such as "VehicleSedan02" for our car, and copy and paste the node after the last event node and before the </events> closing tag.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<events>
    ...
    ...
    ...
    <!-- @[CrSk] BMW 525i E34 -->
    <event name="VehicleCrSKBMW525iE34">
        <nominal>10</nominal>
        <min>7</min>
        <max>13</max>
        <lifetime>3888000</lifetime>
        <restock>0</restock>
        <saferadius>500</saferadius>
        <distanceradius>500</distanceradius>
        <cleanupradius>200</cleanupradius>
        <flags deletable="0" init_random="0" remove_damaged="1"/>
        <position>fixed</position>
        <limit>mixed</limit>
        <active>1</active>
        <children>
            <child lootmax="0" lootmin="0" max="5" min="3" type="CrSK_BMW_525i_E34_Red"/>
            <child lootmax="0" lootmin="0" max="5" min="3" type="CrSK_BMW_525i_E34_Black"/>
            <child lootmax="0" lootmin="0" max="5" min="3" type="CrSK_BMW_525i_E34"/>
        </children>
    </event>
</events>

Here I've named the copied event "VehicleCrSKBMW525iE34", and what we're looking for is having one child node per variant. So we have three child nodes for the grey, black, and red variant.

I could add the beater variant as a fourth child and that's perfectly fine, but in another tutorial I'll go over creating BMW crash sites players can grab doors and wheels from which will have different spawn numbers and spawn points.

Yes we have to manually set spawn points. Let's go over that next.

Step 8: Edit cfgeventspawns.xml

So more complex items like cars need to know where to spawn, and those locations are set in this file. All other items like clothes and guns will spawn depending on their location you set in types.xml and you do not need to do anything in this step.

It may not be as difficult as you expect. We can simply take coordinate locations from existing events such as the VehicleCivilianSedan, and move them into our event for our car, this would help keep the ratio of functioning cars on your server the same. Or we can go and scout new locations for our car making your server highly unique.

To set your own custom spawn locations for your car, you'll need an administration mod like ZomBerry Admin Tools or VPPAdminTools to be installed, which has functions to locate your coordinates on the game map.

Go to the Map tab, put your mouse over a location and the admin tool will give you its X and Y coordinates. You'll notice the 0, 0 coordinates for each map are on the bottom left of the map.

Gather your list of X, Y coordinates and we're ready to add them.

Browse to the folder "mpmissions\dayzOffline.enoch", assuming you are using Livonia, and open the file "cfgeventspawns.xml" in your editor.

Did you backup your file?

Same technique here where we grab a similar event and paste it after the last event and before the closing </eventposdef> tag.

The name of the event will be the same event name you gave in events.xml, which in my example I named ""VehicleCrSKBMW525iE34".

Two things to note, the Y coordinate has the attribute name "z", and the "a" attribute I believe is not altitude but azimuth, or the rotation since no value is greater than 360. I haven't verified yet but for your custom locations you can set it to "0" and the value works fine.

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<eventposdef>
    ...
    ...
    ...
    <event name="VehicleCivilianSedan">
        <pos x="8475.086914" z="12021.825195" a="98.171761" />
        <pos x="6400.875" z="11015.832" a="87.982" />
        <pos x="1454.454834" z="7873.223632" a="169.339111" />
        <pos x="7979.476074" z="11590.756836" a="306.358276" />
        <pos x="1962.798462" z="6963.157226" a="89.255974" />
        <pos x="2026.577026" z="7376.600586" a="280.835632" />
        <!--
        <pos x="8110.324218" z="11124.857422" a="239.048874" />
        <pos x="5941.000488" z="4073.796875" a="224.205597" />
        <pos x="1807.832031" z="7316.355957" a="353.48465" />
        -->
        <pos x="1158.611328" z="7232.993652" a="184.390396" />
        <pos x="3419.759277" z="12219.115234" a="210.244019" />
        <pos x="5706.550293" z="4009.871582" a="348.438843" />
        <pos x="6538.852539" z="11313" a="197.654129" />
        <pos x="1524.120727" z="9755.583985" a="275.054993" />
        <pos x="6463.915039" z="11295.385742" a="18.860342" />
        <pos x="6101.026855" z="4090.22168" a="314.294312" />
        <pos x="6569.03418" z="11228.957032" a="328.941681" />
        <pos x="1437.497437" z="9759.655273" a="119.077553" />
        <pos x="4956.69629" z="9913.737305" a="30.127554" />
        <pos x="3177.355713" z="12065.053711" a="291.759857" />
        <pos x="6435.437988" z="11455.860352" a="316.596741" />
        <pos x="1689.447754" z="7440.379395" a="322.377319" />
        <pos x="3699.560547" z="11819.297851" a="122.567703" />
        <pos x="8442.847656" z="11842.393554" a="177.814026" />
        <pos x="10650.541016" z="11054.634765" a="142.799698" />
        <pos x="11064.290039" z="4239.735352" a="339.768036" />
        <!--
        <pos x="10657.114258" z="11214.108398" a="144.355301" />
        <pos x="11562.433594" z="429.821716" a="54.768456" />
        <pos x="10961.392578" z="11193.723633" a="281.178528" />
        <pos x="11489.000977" z="4625.76709" a="7.299194" />
        <pos x="11148.257812" z="11417.144532" a="302.733154" />
        -->
        <pos x="11581.658203" z="9631.269532" a="326.835175" />
        <pos x="11245.519531" z="9564.609374" a="148.678238" />
        <pos x="10872.585938" z="940.378479" a="112.660118" />
        <pos x="5559.256836" z="3846.330811" a="331.978912" />
        <pos x="11513.013672" z="470.985016" a="322.622253" />
        <pos x="6192.668458" z="4165.701172" a="49.918659" />
        <pos x="11644.21289" z="9636.391602" a="252.961456" />
        <pos x="11236.445312" z="4365.956543" a="221.51123" />
        <pos x="11050.731445" z="4385.636231" a="328.794769" />
    </event>
    <event name="VehicleCrSKBMW525iE34">
        <!-- Coordinates grabbed from VehicleCivilianSedan -->
        <pos x="8110.324218" z="11124.857422" a="239.048874" />
        <pos x="5941.000488" z="4073.796875" a="224.205597" />
        <pos x="1807.832031" z="7316.355957" a="353.48465" />
        <pos x="10657.114258" z="11214.108398" a="144.355301" />
        <pos x="11562.433594" z="429.821716" a="54.768456" />
        <pos x="10961.392578" z="11193.723633" a="281.178528" />
        <pos x="11489.000977" z="4625.76709" a="7.299194" />
        <pos x="11148.257812" z="11417.144532" a="302.733154" />
 <!-- Custom coordinate locations -->
        <pos x="11515.3" z="475.762" a="0" />
        <pos x="5732.29" z="4032.79" a="0" />
    </event>
</eventposdef>

I included examples of both techniques. I commented out the values and copied them into my custom event, and created some custom positions.

Step 9: Launch your server

Run your server, check for any crashes in the logs, and have fun driving in 1990's style.

Friday, December 11, 2015

Converting Nested Set Data to Tree Object

In a project at work we decided to use the nested set model. It fit very well with our needs. We set up the database schema and also created some procedures to help with inserting and deleting nodes. The next step is now reading the data from a client. However we want to read the data and make sure the objects are represented in a tree structure. So how do we take tabular data and convert it into a tree? Easy, recursion. We can sort the data by the left value, which should give us the ability to step through using a depth-first approach.

I'm not going to go into what the nested set model is, I'm assuming that since you are here you know something about it. If not then I recommend the de facto blog here: http://mikehillyer.com/articles/managing-hierarchical-data-in-mysql/

Data Structures

First step is we need to define a data structure for our nodes. It will contain the id, name, and list of child nodes. You'll notice I set the properties for left and right to internal. This is because there is no value giving users access to these properties so they should only be accessible within the project.

public class Node
{
 private double category_id = 0;
 private string name = "";
 private int left;
 private int right;
 private List<Node> nodes = new List<Node>();

 public Node()
 {
 }

 public Node(double category_id, string name, int left, int right)
 {
  this.CategoryId = category_id;
  this.Name = name;
  this.Left = left;
  this.Right = right;
 }

 public double CategoryId
 {
  get { return category_id; }
  set { category_id = value; }
 }

 public string Name
 {
  get { return name; }
  set { name = value; }
 }

 internal int Left
 {
  get { return left; }
  set { left = value; }
 }

 internal int Right
 {
  get { return right; }
  set { right = value; }
 }

 public List<Node> Nodes
 {
  get { return nodes; }
  set { nodes = value; }
 }
}

Okay this looks good.

Public Method

Next we can create the public method to open a connection to the database, load the data into a list, then pass that list off to a recursive function called "BuildNodes()" that will generate the node hierarchy, which we will then return. Notice we are sorting the results the by left value in the SQL command.

public Node GetNodes()
{
 Node root = null;

 try
 {
  List<Node> nodes = new List<Node>();

  // connection 
  using (MySqlConnection connection = new MySqlConnection(_connectionstring))
  {
   //Open connection
   connection.Open();

   //Create command
   using (MySqlCommand command = new MySqlCommand("SELECT * FROM nested_category ORDER BY lft", connection))
   {
    //Read the table row
    using (MySqlDataReader reader = command.ExecuteReader())
    {
     //Read row values
     Node node;
     while (reader.Read())
     {
      node = new Node(); 
      node.CategoryId = Convert.ToDouble(reader["id"]);
      node.Text = reader["text"];
      node.Left = Convert.ToInt(reader["lft"]);
      node.Right = Convert.ToInt(reader["rgt"]);

      nodes.Add(node);
     }
    }
   }
  }

  if (nodes.Count == 0)
   return null;

  //Build the node hierarchy
  int nodeIndex = 0;
  root = BuildNodes(nodes, ref nodeIndex);
 }
 catch (Exception ex)
 {
  throw new Exception(ex);
 }

 return root;
}

Recursive Method

The last step is the recursive method to step through the list and convert it to a tree structure. The trick here is to track 3 values, the row index and then the left and right values of the node in that row index. The flow is hierarchy

If a leaf node (a node without children) then return the node.
If an internal node (a node with children) then

Continue to step through the list and add children while the child's right value is less than the node's right value.

private Node BuildNodes(List<Node> nodes, ref int nodeIndex)
{
 
 Node node = nodes[nodeIndex];
 if ((node.Right - node.Left) == 1)
 {
  //Leaf
  return node;
 }
 else
 {
  //Internal node     
  Node child;
  int childIndex = node.Left++;
  while (childIndex + 1 < node.Right)
  {
   nodeIndex++;
   child = BuildNodes(nodes, ref nodeIndex);
   if (child != null)
   {
    childIndex = child.Right++;
    node.Nodes.Add(child);
   }
  }    
 }

 return node;
}

This will build up the hierarchy converted from a list. It assumes that the left and right values are continuous and there are no numbers missing. If your list filters the tree and there are gaps in the left and right values then good luck and let me know what you find.

Wednesday, December 9, 2015

Inserting and Deleting In a Nested Set Model

Storing tree data in a database can be tricky. There are multiple approaches you can take depending on your project's needs. For one of my projects I added and removed nodes so infrequently (like once or twice a year) but required thousands of requests per day. The nested set model was the perfect fit and I got an excited for the chance to play with a new technique that was such a shift in thinking to adjacency lists. I'm not going to go into what the nested set model is, I'm assuming that since you are here you know something about it. If not then I recommend the de facto blog here: http://mikehillyer.com/articles/managing-hierarchical-data-in-mysql/

I find using procedures for these tasks to be much easier than requiring users to wrap their head around the nested set model or to try and translate the code to the language in your client.

Inserting

One item missing from the blog is the actual code, sure it gives you snippets but not the entire picture. For example inserting nodes is different if you are dealing with a leaf or internal node. Just a small oversight that I thought I would resolve by creating a procedure. It takes as input the id of the parent node that we want to insert and we pass the values for the new node. The procedure tests if the parent is a leaf node and branches accordingly.

DELIMITER;;
CREATE PROCEDURE `node_insert`(parent_category_id INT, name VARCHAR(255))
BEGIN
 DECLARE parent_left, parent_right INT DEFAULT 0;
 SELECT lft INTO parent_left FROM nested_category

  WHERE category_id = parent_category_id;
 SELECT rgt INTO parent_right FROM nested_category

  WHERE category_id = parent_category_id;

 IF parent_right = (parent_left + 1) THEN
  SELECT @my_left := lft FROM nested_category

   WHERE id = parent_category_id;
  UPDATE nested_category SET rgt = rgt + 2

   WHERE rgt > @my_left;
  UPDATE nested_category SET lft = lft + 2

   WHERE lft > @my_left;
  INSERT INTO nested_category(`name`, `lft`, `rgt`) 
   VALUES(name, @my_left  + 1, @my_left + 2);
 ELSE
  SELECT @my_right := rgt FROM nested_category

   WHERE id = parent_category_id;
  UPDATE nested_category SET rgt = rgt + 2

   WHERE rgt >= @my_right;
  UPDATE nested_category SET lft = lft + 2

   WHERE lft >= @my_right;
  INSERT INTO nested_category(`name`, `lft`, `rgt`) 
   VALUES(name, @my_right, @my_right + 1);
 END IF;
END ;;
DELIMITER ;

To call the procedure we will need the id of the parent node and values. For example if we were to add a node for smartwatches to the PORTABLE_ELECTRONICS that has an id of 6 we can call the procedure like this:

CALL node_insert(6, 'Smartwatch');

Deleting

This one is pretty easy. We take as input the id of the node we want to delete.

CREATE PROCEDURE `node_delete`(category_id INT)
BEGIN
 SELECT @my_left := `lft`, @my_right := `rgt`, @my_width := `rgt` - `lft` + 1
  FROM `nested_category`
  WHERE `category_id` = category_id;

 DELETE FROM `nested_category`

  WHERE `lft` BETWEEN @my_left AND @my_right;

 UPDATE `nested_category` SET `rgt` = `rgt` - @my_width

  WHERE `rgt` > @my_right;
 UPDATE `nested_category` SET `lft` = `lft` - @my_width

  WHERE `lft` > @my_right;
END

To call the procedure we just need the id of the node. For example if we decided that CD players were now obsolete we can delete the node "CD PLAYERS" which has an id of 9.

CALL node_delete(9);

Monday, October 13, 2014

A Real World Example of GWT 2.x, MySQL, and MVP: Part 4

Part 1: Getting Started
Part 2: Putting the Framework Together
Part 3: Finally Making the Page Content
Part 4: User Authentication

User Authentication

Now that we have our website up and running let's protect certain pages and restrict functionality that we wouldn't want to give to all users who visit your site with user authentication by using a Role Based Access Control (RBAC) approach. Don't know what RBAC is? You're best bet is to Google some tutorials and purchase some books because it can get real complicated real fast. Because there are so many approaches I will take the most basic and general approach which you can modify for whatever variant you wish to use. Now at this time let me say it is VERY IMPORTANT that access to your site should be done over SSL, that means your website should be "https" and not "http". Information like passwords will be sent over in plain text and using SSL will fix a huge security layer on top of your website.

There are 3 basic principles of Role Based Access Control that we will need to know first:

User - An person (or service account) who can perform functions on your webpage.
Role - A job function that defines authority level.
Permission - The authority to someone to perform an operation.

The relationship between these principles looks like this:

Image source: http://www.isaca.org/Journal/Past-Issues/2008/Volume-3/Pages/JOnline-Role-Engineering-The-Cornerstone-of-RBAC1.aspx

Basically:

A User has one or many Roles
A Role has one or many Permissions

Next we need to decide what roles we will need and what type of access we would like to give to each of our roles. Having a good solid strategy ahead of time will make your security stronger and save you time later on. In a real website we could add roles for super users, power users, super power users, each with their own levels of access, but for this example I will keep things very simple and have only 2 types of roles:

Administrator
User

Finally we should decide what each of these roles can and can't do. For example we don't want an average user able to change things we don't want them to. Remember a user can have multiple roles, so there is no need to duplicate some permissions.

Administrator

View admin page

Add search engine
Edit search engine
Remove search engine
Edit user settings

User

Log-in
Edit own user settings

The basic idea is to be able to check to see if a user has permission to perform an action, not what role they have. We will do this in 2 ways. On the database side we will create a system that will use triggers and functions to check if a user has authority to perform an action, and on the client side we will store the list of permissions with the user object and perform checks on that list for a permission.

Create database tables

I love it when a plan comes together! Now let's get moving. To set this up in a database we will need 6 tables, 5 of them will store everything we will need for RBAC authentication, and 1 table to store user sessions. A session is how we will actually be authenticating a user after he or she logs into the system. A session should be unique per user and very difficult to predict and have an end date requiring the user to re-login at some point. You can create each table manually or also download a SQL script here.

Table 1: users
Stores user information and any metadata you want to keep. Things like, address, city, state, gender, etc... Heck be as creative as you want because if you're lucky you can eventually sell this information to advertisers for a profit! Evil laugh! MWA-HA-HA-HA! (Seriously though don't be a jerk and do that.)

CREATE TABLE `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `username` varchar(99) NOT NULL COMMENT 'SYMC username',
  `password` varchar(99) NOT NULL COMMENT 'User password',
  `first_name` varchar(99) DEFAULT NULL COMMENT 'User first name',
  `last_name` varchar(99) DEFAULT NULL COMMENT 'User last name',
  `email` varchar(99) DEFAULT NULL COMMENT 'User email address',
  `created_by` int(11) NOT NULL COMMENT 'Created by',
  `created_on` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Date created on',
  `updated_by` int(11) DEFAULT NULL COMMENT 'Updated by',
  `updated_on` timestamp NULL DEFAULT NULL COMMENT 'Date updated on',
  `active` tinyint(1) DEFAULT '1' COMMENT 'Is the user active',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`)
) ENGINE=InnoDB COMMENT='User of the system';

Table 2: roles
Stores the roles that can be assigned to a user.

CREATE TABLE `roles` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(45) DEFAULT NULL,
  `description` varchar(45) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB COMMENT='User roles';

Table 3: permissions
Permissions hold the whole security together. They tell us that user X can do Y with object Z. The name will be a short name to description what the permission allows and will be our alternate key, action will be something like "add", "remove", "modify", and presenter will be the webpage (object) we want to access.

CREATE TABLE `permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(99) DEFAULT NULL,
  `action` varchar(45) DEFAULT NULL,
  `presenter` varchar(45) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB COMMENT='User permissions';

Table4: user_roles
This will contain many-to-many references for users and their assigned roles. We'll also add some constraints to delete rows if a user or a role is ever deleted from their respective tables. That should keep this table nice and tidy.

CREATE TABLE `user_roles` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'Unique identifier',
  `user_id` int(11) DEFAULT NULL COMMENT 'User identifier',
  `role_id` int(11) DEFAULT NULL COMMENT 'Role identifier',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `FK_user_roles_users_idx` (`user_id`),
  KEY `FK_user_roles_roles_idx` (`role_id`),
  CONSTRAINT `FK_user_roles_roles` FOREIGN KEY (`role_id`) REFERENCES `roles` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION,
  CONSTRAINT `FK_user_roles_users` FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION
) ENGINE=InnoDB COMMENT='User roles';

Table 5: role_permissions
Ah, ah, ah, ah, table 5! Table 5! Not a Simpsons fan, eh? Never-mind then. This will contain many-to-many references for roles and their permissions. Again we'll add constraints to keep the table neat.

CREATE TABLE `role_permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'Unique identifier',
  `role_id` int(11) DEFAULT '0' COMMENT 'Role identifier',
  `permission_id` int(11) DEFAULT '0' COMMENT 'Permission identifier',
  PRIMARY KEY (`id`),
  UNIQUE KEY `id_UNIQUE` (`id`),
  KEY `FK_role_permissions_roles_idx` (`role_id`),
  KEY `FK_role_permissions_permissions_idx` (`permission_id`),
  CONSTRAINT `FK_role_permissions_permissions` FOREIGN KEY (`permission_id`) REFERENCES `permissions` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION,
  CONSTRAINT `FK_role_permissions_roles` FOREIGN KEY (`role_id`) REFERENCES `roles` (`id`) ON DELETE CASCADE ON UPDATE NO ACTION
) ENGINE=InnoDB COMMENT='Role permissions';

Table 6: user_sessions
This table will contain user sessions. It will help us in our system security to track when a user has logged in, if the session is still active, and also provide some mechanism for metrics at a later date if we so wish.

CREATE TABLE `user_sessions` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT 'Unique identifier',
  `user_id` int(11) NOT NULL DEFAULT '0' COMMENT 'User identifier',
  `session_id` varchar(45) DEFAULT NULL COMMENT 'Unique client session identifier',
  `ip_address` varchar(45) NOT NULL COMMENT 'User IP address',
  `created_on` timestamp NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Date created on',
  `ended_on` timestamp NULL DEFAULT NULL COMMENT 'Date session will end on',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB;

Create Some Triggers

If you are not familiar with databases this might start getting a bit complicated, but don't worry it really isn't. We are going to create a trigger on the "users" table to automatically assign any new user the role "user" which will give them the ability to log into the system, edit their own user settings, and any other permissions we give the user role. This will save us the time from having to add this role manually every time a new user signs up or is registered in the system.

DELIMITER ;;
CREATE TRIGGER `users_AINS` AFTER INSERT ON `users` FOR EACH ROW
BEGIN
    -- Get the user role id
    DECLARE role_id INT;
    SET role_id = (SELECT `id` FROM `roles` WHERE name = 'user'); 

    -- Insert the new user with a user role
    INSERT INTO `user_roles` (`user_id`, `role_id`) VALUES (NEW.id, role_id);
END ;;
DELIMITER ;

Insert Test Data

For the sake of getting something out of this tutorial what I will do is create a test user and set up our roles and permissions directly within the database. What should really happen is we create some administrator pages in GWT to manage users, roles, and permissions... and we will get to that (I think.) In the mean-time let's set up some test data:

Insert 1: roles

INSERT INTO `roles` VALUES 
(1,'administrator','Administrator of the system'),
(2,'user','User of the system');

Insert 2: permissions

INSERT INTO `permissions` VALUES 
(1,'allow_login','login','login'),
(2,'allow_show_admin','show','admin'),
(3,'allow_modify_engine','modify','engine'),
(4,'allow_add_engine','add','engine'),
(5,'allow_remove_engine','remove','engine'),
(6,'allow_modify_user','modify','user'),
(7,'allow_modify_self','modify','self');

Insert 3: users

INSERT INTO `users` VALUES 
(1,'admin','test','Admin','Test','admin@mom.com',1,NULL,NULL,NULL,1),
(2,'user','test','User','Test','user@mom.com',1,NULL,NULL,NULL,1);

STOP RIGHT THERE! I wanna know right now. Before we go any further do you now have rows in the "user_roles" table? If so it should look like this:

id	user_id	role_id
1	1	2
2	2	2

This means that when we created the users above in "Insert 3" they were automatically assigned the "user" role by the trigger 'users_AINS'. Now we don't have to worry about making sure every user can log in when they register at our site. Cool!

Insert 4: user_roles
We already have some user roles thanks to the trigger, but now we just have to give the 'admin' user the 'administrator' role.

INSERT INTO `user_roles` (`user_id`, `role_id`) VALUES 
(1,1);

Insert 5: role_permissions
Do the following insert and it will assign the permissions to the roles that we decided above. Again normally we would have pages in an Administrator section on our website to manage what permissions belong to what roles, and I highly recommend building those, but once again let's just get some test data so we can play with this tutorial.

Administrator (role_id: 1)

View admin page (permission_id: 2)

Add search engine (permission_id: 4)
Edit search engine (permission_id: 3)
Remove search engine (permission_id: 5)
Edit user settings (permission_id: 6)

User (role_id: 2)

Log-in (permission_id: 1)
Edit own user settings (permission_id: 7)

INSERT INTO `role_permissions` VALUES 
(1,1,2),
(2,1,3),
(3,1,4),
(4,1,5),
(4,1,6),
(5,2,1),
(6,2,7);

I think that gets us to a place where we have a normal user and an administrator. Sweet. Now let's start adding to our website.

Refactoring the Server

I'm glad you stepped through my tutorial on MySQL and GWT (if you haven't then right now might be a good time right now since that is where all the code I will be referencing is) but now that we are working on a much larger and scalable application we need to make some tiny adjustments to my previous code. Here is the dilemma, the authentication stuff and the database stuff are going to be used together a lot. What I want to avoid is having multiple RPC guys on my client, because that's not where I want to do all of the business logic for authentication. I want to check to see if a user has permission to insert a record or get certain records before sending that data back to the client. So what I want to avoid is this:

Instead I want to add another layer of abstraction by creating a class to group all off the back-end services on the server-side in order to deal with all of the business logic, such as the authentication crap we're working on now and the data layer. This will have the benefit of separation of concerns and saving time with RPC calls.

To start we need a few things. I will first create a custom Exception called "NotAuthenticatedException" which will help communicate specific failures from my server to my client. Of course you can skip this and send everything as a plain-old Exception, but what if you want to branch your logic depending on exactly what the problem was on the server-side? Maybe I want to limit the number of times a user has to authenticate? It will be easier to add this up by catching NotAuthenticatedExceptions than looking at something like the error message. So here we go:

NotAuthenticatedException.java

package com.example.searchengines.shared;

public class NotAuthenticatedException extends Exception {

    private static final long serialVersionUID = 1L;

    public NotAuthenticatedException() {
        this("Login failed; Invalid username or password");
    }

    public NotAuthenticatedException(String arg0) {
        super(arg0);
    }

    public NotAuthenticatedException(Throwable arg0) {
        super(arg0);
    }

    public NotAuthenticatedException(String arg0, Throwable arg1) {
        super(arg0, arg1);
    }
}

Now let's add modify and add new methods to our DBConnection and DBConnectionAsync services that we created in my previous blog to check user authentication and log a user out of the system:

DBConnection.java

package com.rappa.lapizza.client.service;

public interface DBConnection extends RemoteService
{
    /**
     * Authenticates the User with the specified username and password
     * @param username User's name
     * @param password User's password
     * @return User
     * @throws Exception
     */
    public User authenticateUser(String username, String password) throws NotAuthenticatedException, Exception;

    /**
     * Logs out a user session
     * @param jSessionId User session to log out
     * @throws Exception 
     */
    public void logout(String jSessionId) throws Exception;

DBConnectionAsync.java

package com.example.searchengines.client.service;

public interface DBConnectionAsync
{
    /**
     * Authenticates the User with the specified username and password
     * @param username User's name
     * @param password User's password
     * @param callback Async return of the User
     * @return User
     * @throws Exception
     */
    public void authenticateUser(String username, String password, AsyncCallback<User> callback) throws NotAuthenticatedException, Exception;

    /**
     * Logs out a user session
     * @param jSessionId User session to log out
     * @param callback Nothing
     */
    public void logout(String jSessionId, AsyncCallback<Void> callback) throws Exception;

Sorry MySQLConnection but we are stripping you of your rank of RemoteServiceServlet. Hand over your badge and your gun. Let's remove the "extends RemoteServiceServlet" but keep "implements DBConnection" in order to no longer use this class as our remote service.

MySQLConnection.java

package com.example.searchengines.server;

public class MySQLConnection extends RemoteServiceServlet implements DBConnection {

Now let's add a new class to act as a repository for the multiple back-end server processes. If I thought ahead with the previous tutorials I would maybe have named this better, but whatever fuck-it, I'll change it all later. This class will basically wrap the MySQLConnection class and pass most of the results on.

DBConnectionImpl.java

package com.example.searchengines.server;

/**
 * @author Austin_Rappa
 *
 */
public class DBConnectionImpl extends RemoteServiceServlet implements DBConnection {

    private MySQLConnection mysql = new MySQLConnection();
    
    /**
     * Authenticates the User with the specified username and password
     * @param username User's name
     * @param password User's password
     * @return User
     * @throws Exception
     */
    public User authenticateUser(String username, String password) throws Exception, NotAuthenticatedException {
        User user = null;
        
        user = mysql.authenticateUser(username, password);
        if (user != null) {
            //Store session id
            HttpServletRequest httpServletRequest = this.getThreadLocalRequest();
            HttpSession session = httpServletRequest.getSession(true);
            String jSessionId = session.getId();
            session.setAttribute("JSESSIONID", jSessionId);
            
            //Insert database session
            mysql.insertUserSession(user, jSessionId, getThreadLocalRequest().getRemoteAddr());
        }
        
        return user;
    }

    /**
     * Logs out a user session
     * @param jSessionId User session to log out
     * @throws Exception 
     */
    public void logout(String jSessionId) throws Exception {
        //Remove session id from http header
        HttpServletRequest httpServletRequest = this.getThreadLocalRequest();
        HttpSession session = httpServletRequest.getSession();
        session.removeAttribute("JSESSIONID");
        
        //Logout from database session
        mysql.logout(jSessionId);
    }

    /**
     * Gets the SearchEngines
     * @return SearchEngine
     * @throws Exception
     */
    public List<SearchEngine> getSearchEngines() throws Exception {
        return mysql.getSearchEngines();
    }
}

Finally we have to update the web.xml file to use the new DBConnectionImpl.java class instead of MySQLConnection.java class.

web.xml

    <servlet>
        <servlet-name>mySQLConnection</servlet-name>
        <servlet-class>com.example.searchengines.server.DBConnectionImpl</servlet-class>
    </servlet>

Permissions in the Website

First thing I am going to do is create an enum with my list of permissions. You don't have to do this and in-fact can be a pain as your website grows and strings work just fine as permission names shouldn't change. For me I don't mind. Add an enum to your 'shared' package called 'Permission.java'.

Permissions.java

package com.example.searchengines.shared;

public enum Permission {
    ALLOW_LOGIN,
    ALLOW_SHOW_ADMIN,
    ALLOW_MODIFY_ENGINE,
    ALLOW_ADD_ENGINE,
    ALLOW_REMOVE_ENGINE,
    ALLOW_MODIFY_USER,
    ALLOW_MODIFY_SELF
}

Next edit your user class in the "shared" package to ad a list of permissions. If you chose to not use an enum for your permissions just change the List type to String and be on your way. What we will do is store a list of the permissions assigned to the user. This will allow our presenter classes to check to see if this user has permission to perform an action or show a control.

User.java

package com.example.searchengines.shared;

public class User implements IsSerializable {

    private List<Permission> permissions;

    ...

    /**
     * Gets the user permissions
     * @return the permissions
     */
    public List<Permission> getPermissions() {
        return permissions;
    }

    /**
     * Sets the users permissions
     * @param permissions the permissions to set
     */
    public void setPermissions(List<Permission> permissions) {
        this.permissions = permissions;
    }

Let's standardize on the length of time we would like the session to be active. 24 hours seems good so let's go with that. If you want to change to a week just change that "1" at the end of the formula to a "7".

Property.java

package com.example.searchengines.client;

public class Property {
    
    public static final long SessionDuration = (1000 * 60 * 60 * 24) * 1; //duration remembering login.

The last step for the server-side code is adding methods to the MySQLConnection.java class to connect to the database, look up the username and password (which is right now stored as plain-text but we will change later) and return a User object with the user's information, otherwise raise an NotAuthenticatedException if we could not authenticate.

MySQLConnection.java

package com.example.searchengines.server;

public class MySQLConnection implements DBConnection {

    /**
     * Authenticates the User with the specified username and password
     * @param username User's name
     * @param password User's password
     * @return User
     * @throws Exception
     */
    public User authenticateUser(String username, String password) throws Exception, NotAuthenticatedException {        
        Connection conn = null;
        PreparedStatement pstmt = null;
        ResultSet result = null;
        User user = null;

        try {
            conn = getConnection();
            
            boolean authenticated = true;
            //TODO: Authenticate encrypted password.
            
            if (authenticated) {
                pstmt = conn.prepareStatement("SELECT * FROM users WHERE username = ? AND password = ?");
                pstmt.setString(1, username);
                pstmt.setString(2, password);
                result = pstmt.executeQuery();
                
                //Check for null results
                if (result.isBeforeFirst() ) {
                    //Loop through results
                    while (result.next()) {
                        //Create return object
                        user = new User();
                        user.setId(result.getDouble("id"));
                        user.setUserName(result.getString("username"));
                        user.setFirstName(result.getString("first_name"));
                        user.setLastName(result.getString("last_name"));
                        user.setEmail(result.getString("email"));
                        user.setPermissions(this.getUserPermissions(result.getDouble("id")));
                    }
                }
                else {
                    throw new NotAuthenticatedException();
                }
            }
            else {
                throw new NotAuthenticatedException();
            }
        } catch (SQLException sqle) {
            logger.error("SQL error in authenticateUser(" + username + "): " + sqle.getMessage() + "\n" + sqle.getStackTrace().toString());
            throw new Exception(sqle);
        } finally {
            result.close();
            pstmt.close();
            conn.close();
        }

        return user;
    }
    
    /**
     * Logs a user login session
     * @param user User who logged into session
     * @param session_id Session identifier
     * @param ip_address User's IP address
     * @throws Exception     
     */
    void insertUserSession(User user, String session_id, String ip_address) throws Exception {
        Connection conn = null;
        PreparedStatement pstmt = null;

        try {
            // Get database connection
            conn = getConnection();            

            // Insert new user
            pstmt = conn.prepareStatement("INSERT INTO user_sessions (user_id, session_id, ip_address, created_on, ended_on) VALUES (?, ?, ?, ?, ?)", Statement.RETURN_GENERATED_KEYS);
            pstmt.setDouble(1, user.getId());
            pstmt.setString(2, session_id);
            pstmt.setString(3, ip_address);
            pstmt.setTimestamp(4, new Timestamp(new java.util.Date().getTime()));
            pstmt.setTimestamp(5, new Timestamp(new java.util.Date().getTime() + Property.SessionDuration));
             
            //Execute sql statement
            if (pstmt.executeUpdate() == 0) {            
                logger.error("Error in insertUserSession(): Bad log of session.");
            }
        } catch (SQLException sqle) {
            logger.error("Error in insertUserSession(): " + sqle.getMessage() + "\n" + sqle.getStackTrace().toString());
            throw new Exception(sqle);
        } catch (Exception e) {
            throw new Exception(e);
        } finally {
            // Cleanup
            pstmt.close();
            conn.close();
        }
    }
    
    /**
     * Gets the permission list of a user
     * @param user_id User identifier
     * @throws Exception     
     */
    private List<Permission> getUserPermissions(Double user_id) throws Exception {
        Connection conn = null;
        PreparedStatement pstmt = null;
        ResultSet result = null;
        List<Permission> permissions = new ArrayList<Permission>();

        try {
            // Get database connection
            conn = getConnection();            

            //Get user permissions
            pstmt = conn.prepareStatement("SELECT permissions.name FROM user_roles LEFT JOIN role_permissions ON role_permissions.role_id = user_roles.role_id LEFT JOIN permissions ON permissions.id = role_permissions.permission_id WHERE user_roles.user_id = ?");
            pstmt.setDouble(1, user_id);
             
            //Execute sql statement
            result = pstmt.executeQuery();
            while (result.next()) {
                Permission permission = Permission.valueOf(result.getString("name").toUpperCase());
                permissions.add(permission);
            }
        } catch (SQLException sqle) {
            logger.error("Error in getUserPermissions(): " + sqle.getMessage() + "\n" + sqle.getStackTrace().toString());
            throw new Exception(sqle);
        } catch (Exception e) {
            throw new Exception(e);
        } finally {
            // Cleanup
            result.close();
            pstmt.close();
            conn.close();
        }
        
        return permissions;
    }
    
     /**
      * Logs out a user session
      * @param jSessionId User session to log out
      * @throws Exception 
      */
    public void logout(String jSessionId) throws Exception {
        Connection conn = null;
        PreparedStatement pstmt = null;

        try {            
            // Get database connection
            conn = getConnection();            

            // Insert new user
            pstmt = conn.prepareStatement("UPDATE user_sessions SET ended_on = NOW() WHERE session_id = ?");
            pstmt.setString(1, jSessionId);
             
            //Execute sql statement
            if (pstmt.executeUpdate() == 0) {
                throw new Exception("Bad log of session.");
            }
        } catch (SQLException sqle) {
            logger.error("Error in logout(): " + sqle.getMessage() + "\n" + sqle.getStackTrace().toString());
            throw new Exception(sqle);
        } catch (Exception e) {            
            logger.error("Error in logout(): " + e.getMessage() + "\n" + e.getStackTrace().toString());
            throw new Exception(e);
        } finally {
            // Cleanup
            pstmt.close();
            conn.close();
        }   
    }

Modifying the User Interface

Let's create some events! Events in GWT are used to broadcast a notification to the rest of the code in your project and any code that is listening for that notification will be called. What we want to do is to notify our header page (and any other page that is listening) that the user has successfully logged-in and take actions to show a different collection of buttons, like a log out button. In your "client" package created a sub-package called "event" and create the following classes:

LogInEvent.java

package com.example.searchengines.client.event;

import com.google.gwt.event.shared.GwtEvent;

public class LogInEvent extends GwtEvent<LogInEventHandler>{
    
      public static Type<LogInEventHandler> TYPE = new Type<LogInEventHandler>();
      
      public LogInEvent() {
      }

      @Override
      public Type<LogInEventHandler> getAssociatedType() {
        return TYPE;
      }

      @Override
      protected void dispatch(LogInEventHandler handler) {
        handler.onLogIn(this);
      }
}

LogInEventHandler.java

package com.example.searchengines.client.event;

import com.google.gwt.event.shared.EventHandler;

public interface LogInEventHandler extends EventHandler {
      void onLogIn(LogInEvent event);
}

LogOutEvent.java

package com.example.searchengines.client.event;

import com.google.gwt.event.shared.GwtEvent;

public class LogOutEvent extends GwtEvent<LogOutEventHandler>{
    
      public static Type<LogOutEventHandler> TYPE = new Type<LogOutEventHandler>();
      
      public LogOutEvent() {
      }

      @Override
      public Type<LogOutEventHandler> getAssociatedType() {
        return TYPE;
      }

      @Override
      protected void dispatch(LogOutEventHandler handler) {
        handler.onLogOut(this);
      }
}

LogOutEventHandler.java

package com.example.searchengines.client.event;

import com.google.gwt.event.shared.EventHandler;

public interface LogOutEventHandler extends EventHandler {
      void onLogOut(LogOutEvent event);
}

This is where we add the most important piece, the guy that ties this all together on the client-side, the "REMEMBER_ME" cookie. Create a presenter for the log-in page in your "presenter" package. This will hold the log-in page logic. For our view we'll need a text box for the username and password, a button to accept the form, and a button to take the user to a password recovery page. If the user authentication comes back from the server okay then we will create a cookie called "REMEMBER_ME" that has an expiration date, this way if a user returns to your site we can check if that session is still active and retrieve the session.

LogInPresenter.java

package com.example.searchengines.client.presenter;

public class LoginPresenter implements Presenter {

    public interface Display {
        HasClickHandlers getLogin();
        HasClickHandlers getForgotPassword();
        HasKeyPressHandlers getKeyPressUsername();
        HasKeyPressHandlers getKeyPressPassword();
        HasText getUsername();
        HasText getPassword();
        Widget asWidget();
        void showError(String errormessage);
        void hideError();
    }

    private final DataFactory factory;
    private final Display display;

    /**
     * 
     * @param factory
     * @param view
     */
    public LoginPresenter(DataFactory factory, Display view) {
        this.factory = factory;
        this.display = view;
    }
    
    /**
     * 
     */
    public void bind() {

        display.getLogin().addClickHandler(new ClickHandler() {
            public void onClick(ClickEvent event) {
                try {
                    if (!display.getUsername().getText().isEmpty() && !display.getPassword().getText().isEmpty()) {
                        factory.getRpcService().authenticateUser(display.getUsername().getText(), display.getPassword().getText(), new AsyncCallback<User>() {
                            public void onFailure(Throwable caught) {
                                display.showError("Error authenticating user. " + caught.getMessage());
                            }
    
                            public void onSuccess(final User user) {
                                if (user != null) {
                                    //Set session cookie for 1 day expire. Set secure to false while in development but true in production.
                                    String sessionID = Cookies.getCookie("JSESSIONID");
                                    Date expires = new Date(System.currentTimeMillis() + Property.SessionDuration);
                                    Cookies.setCookie("REMEMBER_ME", sessionID, expires, null, "/", false);
                                    
                                    //Set user in the data factory
                                    factory.setUser(user);

                                    //Broadcast log-in notification. Presenters like the HeaderPresenter are listening for this.                            
                                    factory.getEventBus().fireEvent(new LogInEvent());              
                                    
                                    //Go to home page
                                    History.newItem("home");
                                }
                                else {
                                    display.showError("Username or password is invalid.");
                                }
                                    
                            }                        
                        });
                    }
                    else {
                        display.showError("Please enter your username and password.");
                    }
                } 
                catch (Exception caught) {
                    display.showError("Error authenticating user. " + caught.getMessage());
                }
            }            
        });
        
        display.getKeyPressUsername().addKeyPressHandler(new KeyPressHandler() {
            public void onKeyPress(KeyPressEvent event) {
                if (event.getNativeEvent().getKeyCode() == KeyCodes.KEY_ENTER) {
                    display.getLogin().fireEvent(new ClickEvent(){});
                }
            }            
        });
        
        display.getKeyPressPassword().addKeyPressHandler(new KeyPressHandler() {
            public void onKeyPress(KeyPressEvent event) {
                if (event.getNativeEvent().getKeyCode() == KeyCodes.KEY_ENTER) {
                    display.getLogin().fireEvent(new ClickEvent(){});
                }
            }            
        });        
    }
    
    /**
     * 
     */
    public void go(final HasWidgets container) {
        bind();
        container.clear();
        container.add(display.asWidget());
    }
}

Now let's create the view that will contain our display widgets.

LoginView.java

package com.example.searchengines.client.view;

public class LoginView extends View implements LoginPresenter.Display {
    
    private Button forgotpasswordButton = new Button(constants.forgotpassword());
    private Button loginButton = new Button(constants.login());
    private TextBox usernameTextBox = new TextBox();
    private PasswordTextBox passwordTextBox = new PasswordTextBox();
    private Label errorLabel = new Label();

    public LoginView() {    
    
        //Main panel
        VerticalDivPanel mainPanel = new VerticalDivPanel();
        mainPanel.addStyleName("login");
        initWidget(mainPanel);    
        
        //Define controls
        Label usernameLabel = new Label(constants.loginUsername());
        usernameLabel.setStyleName("bodytext-bold");
        Label passwordLabel = new Label(constants.loginPassword());
        passwordLabel.setStyleName("bodytext-bold");
                
        loginButton.setStyleName("loginbutton");
        loginButton.setTitle("Log in");
        
        forgotpasswordButton.setStyleName("hyperlink");
        
        //Form table
        FlexTable loginTable = new FlexTable();
        loginTable.setStyleName("loginform");
        
        //Add controls to form
        loginTable.setWidget(0, 0, usernameLabel);
        loginTable.setWidget(0, 1, usernameTextBox);
        loginTable.setWidget(1, 0, passwordLabel);                
        loginTable.setWidget(1, 1, passwordTextBox);
        loginTable.setWidget(2, 1, loginButton);
        loginTable.setWidget(3, 1, forgotpasswordButton);
        mainPanel.add(loginTable); 
        
        Scheduler.get().scheduleDeferred(new Command() {
            public void execute() {
                usernameTextBox.setFocus(true);
            }
          });
        
        //Error message
        this.hideError();
        errorLabel.setStyleName("errortext");
        mainPanel.add(errorLabel);        
    }
    
    public Widget asWidget() {
        return this;
    }        

    public HasClickHandlers getLogin() {
        return this.loginButton;
    }

    public HasText getUsername() {
        return this.usernameTextBox;
    }

    public HasText getPassword() {
        return this.passwordTextBox;
    }

    public void showError(String errormessage) {
        errorLabel.setText(errormessage);
        errorLabel.setVisible(true);
    }

    public void hideError() {
        errorLabel.setText("");
        errorLabel.setVisible(false);
    }

    public HasKeyPressHandlers getKeyPressUsername() {
        return this.usernameTextBox;
    }    
    
    public HasKeyPressHandlers getKeyPressPassword() {
        return this.passwordTextBox;
    }

    public HasClickHandlers getForgotPassword() {
        return this.forgotpasswordButton;
    }
}

Now that we have our Log-in header and presenter let's modify our project so our users can browse to those pages when they click the LogIn button on the header. Open the HeaderPresenter class and change "TODO 10" with the highlighted code below that inserts a new browser history and highlights the LogIn header button.

HeaderPresenter.java

package com.example.searchengines.client.presenter;

public class HeaderPresenter implements Presenter {

    ...
    
    /**
     * Bind events and actions
     */
    public void bind() {
        
        ...
        
        display.getLogIn().addClickHandler(new ClickHandler() {
            public void onClick(ClickEvent event) {
                History.newItem("login");
                display.setSelected();
            }
        });

For the last part of the log-in page we need to tell the AppController class to call the LoginPresenter when the Log-in header button is clicked and your site's url changes with the "login" token added to the history.

AppController.java

package com.example.searchengines.client;

public class AppController implements Presenter, ValueChangeHandler<String> {

    ...

    public void onValueChange(ValueChangeEvent<String> event) {
        ...        
            
            // For every page you create set the presenter here
            if (token.getPage().equals("home"))    {
                presenter = new HomePresenter(factory, new HomeView());
            }        
            ...
            }
            else if (token.getPage().equals("login")) {
                presenter = new LoginPresenter(factory, new LoginView());
            }

If you debug the page, now you should be able to see the log-in page when you click the button on the header. Also notice the "login" token in your url. That is a result of adding the "History.newItem()" in the HeaderPresenter and the AppController class picking up that change in the history, branching on that history token value, and displaying the login page.

Admin only pages

Let's say we want to give users who have elevated privileges access to areas of your website where changes to the site's data can be made. Modify HeaderPresenter.java and add functionality for a new button for a future administrator page and to listen for log-in and log-out events. If you followed the previous tutorials (which you should have) then this will replace the TODO #4 for the log-out button and TODO #5 for listening for log-in events. We'll also add a check if the data factory's user object is not null.

HeaderPresenter.java

public class HeaderPresenter implements Presenter {

    public interface Display {
        ...
        HasClickHandlers getAdmin();
        ...
    }

    public void bind() {
        ...
        display.getAdmin().addClickHandler(new ClickHandler() {
            public void onClick(ClickEvent event) {
                History.newItem("admin");
                display.setSelected();
            }
        });

        display.getLogOut().addClickHandler(new ClickHandler() {
            public void onClick(ClickEvent event) { 
                doLogout();
            }
        });
        
        factory.getEventBus().addHandler(LogInEvent.TYPE, new LogInEventHandler() {
            public void onLogIn(LogInEvent event) {
                doLogIn(factory.getUser());
            } 
        });
    }

    public void go(final HasWidgets container) {
        bind();
        container.clear();
        container.add(display.asWidget());
        if (factory.getUser() != null) {
            this.doLogIn(factory.getUser());
        }
        display.setSelected();
    }

    private void doLogIn(User user) {
        display.setData(factory.getUser());
    }

    private void doLogout() {
        try {
            String sessionId = Cookies.getCookie("REMEMBER_ME");
            factory.getRpcService().logout(sessionId, new AsyncCallback() {
                public void onFailure(Throwable caught) {
            }

            public void onSuccess(Void result) {
                Cookies.removeCookie("REMEMBER_ME");
                factory.setUser(null);
                display.setData();
                factory.getEventBus().fireEvent(new LogOutEvent());
                History.fireCurrentHistoryState();  
            }});
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

Nearing the home stretch. Let's just modify the HeaderViewto show our new "Admin" button and to display it only if the user has the "allow_show_admin" permission. Oh wow see how that all comes together now? You can have 50 roles that have the "allow_show_admin" permission and instead of listing them all we just need to know if that user can get to the admin page and that is all. Sweet!

HeaderView.java

package com.example.searchengines.client.view;

public class HeaderView extends View implements HeaderPresenter.Display {
            
    private final Button adminButton = new Button(constants.admin());
    
    public void setData(User user) {
        //Clear right panel
        rightPanel.clear();
        
        //Hello user
        Label userLabel = new Label(constants.welcome() + " " + user.getFirstName());
        userLabel.setStyleName("headerpaneltext");
        rightPanel.add(userLabel);
        
        Label seperatorLabel = new Label("|");
        seperatorLabel.setStyleName("headerpaneltext");
        rightPanel.add(seperatorLabel);
        
        if (user.getPermissions().contains(Permission.ALLOW_SHOW_ADMIN)) {
            adminButton.setStyleName("headerbutton");
            rightPanel.add(adminButton);
            
            Label seperatorLabelAdmin = new Label("|");
            seperatorLabelAdmin.setStyleName("headerpaneltext");
            rightPanel.add(seperatorLabelAdmin);
        }
            
        logoutButton.setStyleName("headerbutton");
        rightPanel.add(logoutButton);
    }


    public HasClickHandlers getAdmin() {
        return this.adminButton;
    }
}

Almost forgot. Add a new constant to SearchEnginesConstants.java for the display text of our administrator button.

SearchEnginesConstants.java

public interface SearchEnginesConstants extends Constants {
        
    ...
    @DefaultStringValue("Admin")
    String admin();
    ...

Let's take a quick look what our website can do so far with authentication:

Display log-in page
Authenticate a user
Retrieve information about that user
Assign permissions to a user based on their role
Display a new admin button on the header when user has permission
When admin header button is clicked, the "admin" token is added to the history.

Now let's take the last step and display the admin page, only if the user has permission to do so. In the AppControler, the class that controls what page to display in our webpage, we will add some logic to show an Administrator page, only if the user who has logged in, has the "ALLOW_SHOW_ADMIN" permission, otherwise send them to the home page.

package com.example.searchengines.client;

public class AppController implements Presenter, ValueChangeHandler<String> {

    ...

    public void onValueChange(ValueChangeEvent<String> event) {
        ...        
            
            // For every page you create set the presenter here
            if (token.getPage().equals("home"))    {
                presenter = new HomePresenter(factory, new HomeView());
            }        
            ...
            }
            else if (token.getPage().equals("admin")) {
                if (factory.getUser() != null && factory.getUser().getPermissions().contains(Permission.ALLOW_SHOW_ADMIN)) {
                    //TODO 9: Create administrator page
                }
                else {
                    presenter = new HomePresenter(factory, new HomeView());
                }
            }

Now when you debug the page as a Web Application try authenticating with your test user and you should not see an "Admin" button on the header but have a "Log Out" button available. However if you authenticate with your test admin user then not only should you see the "Admin" button on the header but you can click that button and your site's url will change with an "admin" token now on the history like so:

http://127.0.0.1:8888/SearchEngines.html?gwt.codesvr=127.0.0.1:9997#admin

If you see this then that will indicate that your test admin user does indeed have permission to view the administrator page, we just haven't built it yet. So in the next tutorial let's build the administrator page, a user registration page, add some more roles and permissions, and learn how to encrypt user passwords in the database.

References

http://csrc.nist.gov/rbac/rbac-std-ncits.pdf
http://code.google.com/p/google-web-toolkit-incubator/wiki/LoginSecurityFAQ
https://www.owasp.org/index.php/Authentication_Cheat_Sheet
https://www.owasp.org/index.php/Session_Management_Cheat_Sheet
https://code.google.com/p/yfs-health/source/browse/trunk/yfs/src/com/varun/yfs/server/login/