Friday, March 7, 2014

Mysql 5.6.16 on OSX throws connection errors with innodb_file_per_table option

Description:
Under heavy workload mysql 5.6.16 (replicated on 5.6.13 and higher)
throws connection errors "Lost connection to MySQL server at 
'sending authentication information', system error: 32". 
This was tested on OSX 10.8 and 10.9. The issue is not seen 
on windows or other unix servers like centos.

How to repeat:
This was tested and the issue was repeatable with a simple database 
import through the mysql client.The database dump had around 1000 tables
 and 200 MB in size. While the dump is being sourced other mysql client(s)
 trying to connect (even if it were a different user than the one running
 the import) you get the authentication error.

"mysql -uroot -p
Enter password: 
ERROR 2013 (HY000): Lost connection to MySQL server at 
'sending authentication information', system error: 32"

Also observed that passing in the host causes mysql to 
throw a different exception:

"mysql -uroot -p -h127.0.0.1
Enter password: 
ERROR 2013 (HY000): Lost connection to MySQL server at 
'reading authorization packet', system error: 0"

Running mysql under debug (mysql-dubug) and generating a trace 
shows the below when the issue occurs:

[login_connection: info: login_connection called by thread 11
login_connection: info: New connection received on TCP/IP (1089)
login_connection: info: Host: unknown host  ip: 127.0.0.1
acl_authenticate: info: com_change_user_pkt_len=0
my_message_sql: error: error: 1158  message: 
'Got an error reading communication packets'
my_message_sql: error: error: 1043  message: 'Bad handshake'
one_thread_per_connection_end: info: thd 0x7f824d245200 block_pthread 1 
 
The interesting piece of trace being "login_connection: info: 
Host: unknown host  ip: 127.0.0.1" where it sees the host as 
"unknown host" which appears to be the cause of the authentication error.

This could also be seen under the performance schema where the HOST
 is seen as NULL while multiple clients try to connect:

mysql> select * from hosts;
+-----------+---------------------+-------------------+
| HOST      | CURRENT_CONNECTIONS | TOTAL_CONNECTIONS |
+-----------+---------------------+-------------------+
| NULL      |                  18 |                35 |
| localhost |                   1 |                 2 |
+-----------+---------------------+-------------------+
2 rows in set (0.00 sec)

The counter "Aborted_connects" keeps increasing.

From the JDBC side, tested this with the recent connector/j and mariadb
java client and both fail with similar exceptions:

[Caused by: java.io.EOFException: 
unexpected end of stream, read 0bytes from 4
 at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully
(ReadUtil.java:84) [mariadb-java-client.jar:]
 at org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully
(ReadUtil.java:92) [mariadb-java-client.jar:]
 at org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket
(RawPacket.java:77) [mariadb-java-client.jar:]
 at org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket
(SyncPacketFetcher.java:67) [mariadb-java-client.jar:]
 at org.mariadb.jdbc.internal.mysql.MySQLProtocol.connect
(MySQLProtocol.java:468) [mariadb-java-client.jar:]
 ... 69 more"]

Also observed that bouncing the mysql instance clears up the problem. 
It was also seen that terminating all the mysql clients like jdbc 
(shutting down the app servers) also helps with the issue and you are
 able to login once again.

my.cnf that was used to test this:
[
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.6/en/server-configuration-defaults.html

[mysqld]
interactive_timeout=180
wait_timeout=180
max_connect_errors=9999
skip-name-resolve
query_cache_size=0
bind-address=127.0.0.1

# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
innodb_buffer_pool_size = 2G

# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin

# These are commonly set, remove the # and set as required.
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
port = 3306
server_id = kris_dev
socket = /tmp/mysql.sock 

# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
join_buffer_size = 128M
sort_buffer_size = 2M
read_rnd_buffer_size = 2M 

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES 

[client]
default-character-set=utf8

]

 
Workaround
 
This was tested on machine specs ranging from 4GB, 8GB and 16 GB and 32 GB RAM.
Things were a little better on the higher specs (16GB and above) but looked severe
on anything 8GB and below.

Further debugging and tracing has led to the mysql option "innodb_file_per_table" 
which is turned on by default on Mysql 5.6 to be causing this issue. Turning off 
this option has revealed positive results so far and no occurrence of 
system error: 32. 
 
It was also noticed that increasing open file limits (ulimit -n) 
on the higher spec machines helped with better results while running with
"innodb_file_per_table" turned on. 


Tools used: mysql-debug (trace option), dtruss

Logged this under: http://bugs.mysql.com/bug.php?id=71960

Hope this helps!

Tuesday, August 21, 2012

Managed Deployments in Jboss AS 7

This post is targeted at Jboss AS 7 users/developers who are struggling with automated managed deployments ever since the hue architectural change in AS7 that implements a parallel deployer as opposed to serial or sequential/ordered deployer in prior versions like Jboss 5.1 and Jboss 4.0.x.

No doubt that the parallel deployer and the architecture around this has been more aligned towards achieving blazing startup times through clear modularity. This works great for developers building new java EE applications agaionst AS7. What about folks that are trying to migrate their legacy applications to AS7?

In this document I am going to focus more on a particular use case that is widely used in servers before AS7.

Let's say we have an application core-app.ear that is the core framework, API, Engine whatever you want to call it. Some may refer this to as shared or common code. And there are a suite of application's bundled as .ear that leverage core-app.ear or depend on the common code or framework beans, spring beans and so forth.  In the AS7 world this model is a problem largely because of the parallel loading sequence. The consumer applications need to start-up after the core-app.ear has started and fully initialized.

The Jboss AS7 documentation doesn't throw too much light on this except for setting up dependencies in your class-path or defining your modular dependencies in jboss-deployment-structure.xml. There are some ways on how one could extend the Jboss AS7 subsystem and provide custom implementations but nothing obvious (I still have to explore this more) . 

Here is a that I could pull this off without too many changes and might be the way to go for some of you unless AS7 has a way to switch to serial deployers:

For my applications I leverage the springframework so you would see a lot of references to Spring beans and classes that are used. I also use little JMX to get me some AS7 properties etc.

Modes

This document assumes we are running under the Standalone mode (Domain mode is largely the same except that it uses a server-group to associate deployments).

The Deployment Scanner

Its important to understand the deployment scanner and how to configure this before we deep dive into the implementation. If you haven't looked at this before, I would highly recommend that you read up: https://docs.jboss.org/author/display/AS7/Deployment+Scanner+configuration before you read up any further.

In the Jboss 4.0.5 we used to have a deploy directory that is scanned by Jboss automatically and deployed on startup. Anything else that needed to be scanned could be configured under jboss-service.xml. Withing a folder configured for scanning the older versions of Jboss used an aphabetical sorter to determine the loading sequence. I would say it was pretty much flexible and managed to a large extent.

I tried to achieve the same here with AS7.


As you see, I have two deployment scanners that I refer to as auto_deployer and managed_deployer respectively. The auto_deployer as the name suggests we have this setup to deploy applications on container start up. The managed_deployer deals with the applications that are dependent on the applications that the auto_deployer picks up and deploys but does not attempt to auto-deploy at container startup.


File Markers

https://docs.jboss.org/author/display/AS72/Application+deployment


Application Startup and Managed Deployer's using File Markers

In the jboss 4.0.x world I used to leverage a JbossNotificationListener that used to be ntified on container startup. I yet have to find the replacement in AS7.  But there are a few ways we could do this.  We could use a MBeanServerDelegate and register for registration events. For the scope of this post I am leveraging Spring's ContextLoaderListener.

Here is how it would look under your Deployment Descriptor:



FooSpringContextLoaderListener.java



This way we also leverage the parallel deployer for the "ears" deployment_scanner.   As soon as we run through teh folder and create marker files for ears we need to deploy, AS7 kicks off a parallel deployer and thus helps us achieve the best of both worlds.

We may implement business logic to selectively deploy applications on your business needs and services.

We may also use CLI to achieve the same but this example tries to implement the same programmatically and achieve seamless deployment with less management overhead outside of Jboss AS7.


References that maybe useful to you:


1. 5 ways to deploy applications on AS7: http://www.youtube.com/watch?v=OuFaYYQVJP0
2. More on MBeans: http://middlewaremagic.com/jboss/?tag=mbeans
3. CLI Client API : http://jamezp.blogspot.com/2011/07/jboss-application-server-deployment.html

Tuesday, November 22, 2011

Upgrading from Mysql 5.0 to 5.5

Having dealt with Application server upgrades, library upgrades and more in the past, I have to say that this has been one of the smoother component upgrades I have dealt with so far.

Here are some brief findings and code changes I had to go through to get our applications upgraded from Mysql 5.0[58]/ Mysql 5.0[77] to Mysql 5.5[15]

1. INNODB definition

a. The TYPE attribute is unsupported as of Mysql 5.5

b. Schema needs to be updated to use the ENGINE instead.

c. Example: ENGINE=InnoDB (unsupported: TYPE=InnoDB)


2. Timestamp

a. In very old versions of MySQL (prior to 4.1), the TIMESTAMP data type supported a display width, which was silenty ignored beginning with MySQL 4.1. This is deprecated in MySQL 5.1, and removed altogether in MySQL 5.5. These changes in behavior can lead to two problem scenarios when trying to use TIMESTAMP(N) columns with a MySQL 5.5 or later server:

i. When importing a dump file (for example, one created using mysqldump) created in a MySQL 5.0 or earlier server into a server from a newer release series, a CREATE TABLE or ALTER TABLE statement containing TIMESTAMP(N) causes the import to fail with a syntax error.

ii. To fix this problem, edit the dump file in a text editor to replace any instances of TIMESTAMP(N) with TIMESTAMP prior to importing the file. Be sure to use a plain text editor for this, and not a word processor; otherwise, the result is almost certain to be unusable for importing into the MySQL server.


3. GROUP_CONCAT(N)

GROUP_CONCAT when used with Integers used to give us back a byte [] as of Mysql 5.0. Mysql 5.5 treats this differently and does an auto conversion to String. So any code that expected a byte [] will have to be updated.

Here is an example:

SELECT GROUP_CONCAT(ROW_NUMBER ORDER BY ROW_NUMBER) AS ROW_NUMBER FROM test_table;

TO get this to work consistently we will have to make code changes that involve Casting object types in SQL and always returning a String. Any code that read them back as a byte [] will have to be refactored to use a String instead.

For example:

SELECT CAST(GROUP_CONCAT(ROW_NUMBER ORDER BY ROW_NUMBER) AS BINARY) AS ROW_NUMBER FROM test_table;



4 Mysqldump – This has been largely backward compatible, except for one of the options that we have been using, which is -–all. This option was deprecated a while ago and somewhere between 5.1.x and 5.5.x this was taken out. The recommendation was to use –create-options instead.


Apart from these we have had challenges to migrate customers from one installation to the other but that goes without saying. There are enough documentation out there that help with these.

Here are some performance benchmarks using supersmack:


The 5.0.92 Percona machine seems to have much higher wait IO and lower throughput:
bash-3.2# super-smack update-select.smack 30 10000
Query Barrel Report for client smacker
connect: max=2ms min=0ms avg= 1ms from 30 clients
Query_type num_queries max_time min_time q_per_s
select_index 300000 12 0 1740.80
update_index 300000 18 0 1740.80


MySQL 5.5.15
bash-3.2# /usr/local/bin/super-smack update-select.smack 30 10000
Query Barrel Report for client smacker
connect: max=15ms min=0ms avg= 5ms from 30 clients
Query_type num_queries max_time min_time q_per_s
select_index 300000 3 0 5209.22
update_index 300000 12 0 5209.22

bash-3.2# mysql --version
mysql Ver 14.12 Distrib 5.0.92, for unknown-linux-gnu (x86_64) using readline 5.1
bash-3.2# mysqladmin status
Uptime: 99573 Threads: 6 Questions: 973831 Slow queries: 0 Opens: 1290 Flush tables: 1 Open tables: 320 Queries per second avg: 9.780

bash-3.2# mysql --version
mysql Ver 14.14 Distrib 5.5.15, for Linux (x86_64) using EditLine wrapper
bash-3.2# mysqladmin status
Uptime: 99247 Threads: 6 Questions: 1822824 Slow queries: 0 Opens: 1686 Flush tables: 1 Open tables: 96 Queries per second avg: 18.366


Its certainly fast and much improved InnoDB comes as a relief. Performance Schema is exciting too although it will be interesting to see if it adds up to any overhead.

InnoDB and TEMPORARY tables

Test Environment
5.0.92 64 Bit – Percona Mysql Server, 4 CPU, 32 GB RAM, CentOS, VMWare.

I have been looking at ways to optimize performance our J2EE applications while running expensive tasks and stumbled upon an interesting optimization that is described below

For InnoDB users, Its a common practice for developers to use InnoDB as the engine for whatever we use. Common causes for that would be copying over create statements from some other existing tables or maybe just addicted to InnoDB. This Maybe true for even TEMPORARY tables that are used as part of JDBC transactions.

It may well be a mistake in some cases.

Here is a snippet of SQL code:

CREATE TEMPORARY TABLE temp3(user_id VARCHAR(32), total MEDIUMINT, PRIMARY KEY (user_id))ENGINE=InnoDB DEFAULT CHARSET=utf8;

We used these sort of tables heavily within the scope of a transaction and ran some heavy queries out of these, and eventually cleaned them up. Generally these sort of calls were made as part of some iterations of N.

Here is what I changed it to:

CREATE TEMPORARY TABLE temp3(user_id VARCHAR(32), total MEDIUMINT, PRIMARY KEY (user_id))ENGINE=MyISAM DEFAULT CHARSET=utf8;

Without Optimizations:
160 seconds (Avg. of 100 iterations)

With Optimizations:
50 seconds (Avg. of 100 iterations)

That's a Huge Improvement!

Although its known that MyISAM is faster in certain scenarios, and InnoDB works better in others, its a choice the developer has to make while defining tables that do not care for referential integrity, selects vs updates and so forth. Choosing MyISAM for scenarios as described above not only makes it go faster but also transfers some load out of the InnoDB engine out to MyISAM, whereas InnoDB could be better utilized for other parts of your application. Highly recommend to try for yourself and prove.

Another interesting test would be to perform similar tests on Mysql 5.5 which has got a better InnoDB engine. I will try to publish some results on these soon.

Mysql O_DIRECT and InnoDB Performance

TEST Environment

Mysql 5.0.92 - Percona Mysql Server, VMWare, CentOS, 32 GB RAM, 4 CPU, SAN.

In my tests with O_DIRECT set as the InnoDB flush method it was found to be resulting in a slow down for IO bound queries while running with SAN as the MySQL backend storage. A lot of our batch jobs or features like reports use temporary tables that are IO bound.

Some extracts from the Mysql Documentation in support of these findings

[["innodb_flush_method If set to fdatasync (the default), InnoDB uses fsync() to flush both the data and log files. If set to O_DSYNC, InnoDB uses O_SYNC to open and flush the log files, and fsync() to flush the data files. If O_DIRECT is specified (available on some GNU/Linux versions, FreeBSD, and Solaris), InnoDB uses O_DIRECT (or directio() on Solaris) to open the data files, and uses fsync() to flush both the data and log files. Note that InnoDB uses fsync() instead of fdatasync(), and it does not use O_DSYNC by default because there have been problems with it on many varieties of Unix. This variable is relevant only for Unix. On Windows, the flush method is always async_unbuffered and cannot be changed.

Different values of this variable can have a marked effect on InnoDB performance. For example, on some systems where InnoDB data and log files are located on a SAN, it has been found that setting innodb_flush_method to O_DIRECT can degrade performance of simple SELECT statements by a factor of three." ]]


It was observed in my testing that running with defaults gave me at least a 60% improvement. Here are some performance metrics:


With O_DIRECT


[root@host]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 21533436 489548 8243880 0 0 2 66 2 13 1 0 99 0 0
0 1 0 21456184 489556 8243964 0 0 13734 4272 2204 2598 6 2 67 24 0
0 1 0 21426052 489564 8243972 0 0 2598 7415 1923 1849 13 2 67 18 0
0 1 0 21403608 489564 8244136 0 0 1130 7932 1885 1739 14 2 68 16 0
1 0 0 21379676 489564 8244084 0 0 1632 8526 1979 1943 13 2 68 17 0
1 0 0 21360580 489572 8244216 0 0 1165 8336 1909 1821 13 2 68 17 0
0 1 0 21340988 489576 8244144 0 0 1114 8719 1963 1946 12 2 68 18 0
0 1 0 21323752 489580 8244336 0 0 694 8711 1940 1854 12 2 68 18 0
0 1 0 21305896 489580 8244248 0 0 778 8207 1888 1791 13 2 68 18 0
0 0 0 21287792 489580 8244444 0 0 787 8398 1916 1799 12 2 68 18 0
0 1 0 21270556 489584 8244392 0 0 611 8661 1926 1853 13 2 68 17 0
0 0 0 20929308 489652 8246276 0 0 0 393 989 91 0 0 100 0 0



Without O_DIRECT (Mysql Defaults)


[root@host]# vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 18700144 489492 8238128 0 0 2 66 2 13 1 0 99 0 0
0 1 0 18681236 489492 8238844 0 0 16 11651 2162 2086 18 3 69 10 0
1 0 0 18662140 489500 8239296 0 0 0 10927 2073 1933 18 2 69 11 0
0 0 0 18643356 489500 8239652 0 0 16 11022 2084 1963 18 3 68 11 0
0 1 0 18624508 489500 8239648 0 0 0 11665 2162 2072 17 3 69 11 0
1 0 0 18606032 489508 8240008 0 0 0 11210 2113 2006 18 2 68 12 0
1 0 0 18587308 489512 8240132 0 0 0 11120 2097 1983 18 3 68 11 0
1 0 0 18567964 489516 8240564 0 0 0 10951 2067 1944 18 2 68 11 0
1 0 0 18550604 489520 8241068 0 0 0 11098 2105 1982 17 3 68 12 0
1 0 0 18531012 489520 8241008 0 0 0 11286 2113 2005 19 3 69 9 0
0 1 0 18512412 489528 8241360 0 0 0 11349 2123 2015 17 3 68 12 0
0 0 0 18282208 489536 8245508 0 0 10 2454 1183 404 3 1 94 2 0


Some of the key differences seen in VMSTAT are highlighted above. Mainly seen in bi, bo and wa.

It is worth testing this further with other tools specifically for IO against SAN or just measuring stats for reads/writes before and after the changes, but as documented by MySQL we just need to try it out ourselves and use what works best on a given platform/environment.


Useful tools we could use to further test this and more:

1. http://googlux.com/bonnie.html

2. http://vegan.net/tony/supersmack/

Sunday, August 3, 2008

A Quick Start Guide to Hatatap

HATATAP

– has evolved from the sound of a north-european trying to pronounce 'http'. Why HTTP? HATATAP is a script based HTTP testing tool. This is built on top of the strong foundation that Metware’s HTTP Unit gives you. Many questions revolve around the head as to when would somebody make use of HTTP Unit and when HATATAP? This article aims at giving the readers an insight into this. Another goal for writing this article is that we don’t find full fledged examples over the web to get started quickly.

Why hatatap?

1. It gives us a strong test framework based on HTTP Unit and also realizes the power of ‘o:xml’ (Object Oriented XML).
2. The easy to code and understand scripts are transformed into ‘o:xml’ before they are executed.
3. The best part is that you could also mix up the hatatap scripts with ‘o:xml’ to build your ideal test script.
4. Another reason to vote for hatatap is that the scripts integrate well with Apache Ant, so that it could be built into any Enterprise level build environment.
5. One may also run the scripts from commandline and have a batch utility written to execute a set of scripts.
6. Hatatap can also be run as a web application where it provides a ready to use Servlet based test framework which just requires us to drop in hatatap scripts under the webapp and run it through your browser.
7. Last but not the least, they could unleash the power of XPATH, which is what I was looking to use in my project.

Note: Above is a generalized listing of key features. There are much more things you could do with hatatap, for which you would have to look over some articles on the web, since they are out of the scope of this document.

We’ll there are a couple of words of caution as well:
1. There are some known bugs related to Xalan and Java that makes hatatap go crazy.
2. For support, though there is a known forum I had a bad experience with it. Though I got over my issues long back, there is no response from the author to my post.
3. Watchout for the ‘.’ Operator. Avoid using ‘.’ To separate variable names or as part of a variable name. This causes the scripts to be dead scripts.

OK, lets try to do a small example and in the process try to learn as much as possible. For the example purpose we would make use of hatatap with Apache Ant.

Get set Go:

1. This example assumes that we have Ant and java installed and ready to use. If not please download and install.
2. Create a working directory to use as the base directory for the sample application. For the sample sake let’s have the directory names ‘hatatap’.
3. Create a 'lib' folder and place the following libraries (.jar)
a. hatatap.jar (The main library which includes the hatatap and object box libraries used for running hatatap tests. You can find this in the hatatap distribution)
b. xalan.jar (Hatatap uses the XSLT engine. We need to provide the TransformerFactory (org.apache.xalan.processor.TransformerFactoryImpl), since the one which comes bundled with java 1.5 and higher has compatibility issues with hatatap. Earlier versions of java were never bundled with Xalan so we have to provide the engine.
c. serializer.jar (Since we are providing the XSLT engine, we also need to provide our XML Serializer)
d. Apart from the libraries, place the hatatap XSD (script.xsd) which was bundled with the hatatap distribution. This will be good to reference in the scripts.
4. Create an ‘input’ directory under ‘hatatap’. Here we would park the hatatap scripts to be executed.
5. Create the build environment. Let’s put together a brief build.properties and a build.xml to compile and run hatatap scripts we will create.

Build.properties

server=http://www.pingdynasty.com

# directory requirements
test_input=input
test_output=output

#log directory
log_dir=log


Above, we have a brief properties file for the build which lists a few properties such as input, output, the server to test with, log directory for any logs generated etc.
Build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project name="hatatap" default="alltests" basedir=".">

<!-- build properties -->
<property file="build.properties"/>
<!--Defines the path where the Hatatap library resides. -->
<path id="lib_path">
<fileset dir="lib">
<include name="*.jar*"/>
</fileset>
</path>
<!-- Hatatap task def -->
<taskdef name="hatatap" classname="com.pingdynasty.hatatap.HatatapTask" classpathref="lib_path"/>
<!-- Clean logs and output -->
<target name="clean">
<delete dir="${log_dir}" />
<delete dir="${test_output}" />
</target>
<!-- Recreate required folders -->
<target name="prepare">
<mkdir dir="${log_dir}" />
<mkdir dir="${test_output}" />
</target>

<!-- TEST TARGETS -->
<target name="alltests" depends="clean, prepare">
<antcall target="StartMLRecord"/>
</target>
<hatatap in="input/test_server.xml" out="output/output_test_server.xml">
<param name="server" value="${server}"/>
<param name="outputdir" value="${test_output}"/>
<classpath refid="lib_path"/>
</hatatap>
<antcall target="StopMLRecord"/>
</target>

<!-- LOG TARGETS -->
<!-- Begin recording events to the Master Log file, placed in the top level of the Sessiontests folder. -->

<target name="StartMLRecord" depends="oldMLDelete">
<record name="${log_dir}\ML_Log.txt" action="start" loglevel="info"/>
<echo file= "${log_dir}\ML_Log.txt" message="Master Log has started recording"/>
</target>
<!-- Halt recording to test Log file. -->
<target name="StopMLRecord">
<record name="${log_dir}\ML_Log.txt" action="stop"/>
</target>
<target name="oldMLDelete" description="Removes old Master log files">
<delete file="${log_dir}\ML_Log.txt"/>
</target>

</project>


In the above build script we have generic build features such as clean (Cleans up the output and log folders), prepare (prepares the required directories for the test run). We have also implemented a basic logging framework to trace progress of the test run and report test results. The main thing to note here is the hatatap task definition and the call. We make use of “com.pingdynasty.hatatap.HatatapTask” to execute the hatatap scripts.

Let’s go over the hatatap ant call.

<hatatap in="input/test_server.xml" out="output/output_test_server.xml">
<param name="server" value="${server}"/>
<param name="outputdir" value="${test_output}"/>
<classpath refid="lib_path"/>
</hatatap>

• The hatatap definition takes in an ‘in’ and ‘out’ attribute which is used to define the hatatap input script. Output file is the one generated after the test run.
• Then, we have ‘param’ tags which are used to pass any parameters to the hatatap script. In a real life scenario. We might be reading several parameters’ from properties and config files and make them available to the hatatap script by means of ‘param’ elements.
• We also have the classpath referenced here since we need to override libraries that are inbuilt.


6. Let’s create the hatatap scripts now. Create the below given XML (‘test_server.xml’) under the ‘input’ folder.

<?xml version="1.0" encoding="UTF-8"?>
<script xmlns="urn:hatatap" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:hatatap ../lib/script.xsd">

<param name="server" default="http://www.pingdynasty.com"/>
<param name="outputdir" default=""/>

<!-- Check if the Server is up and running -->
<send url="{$server}"></send>

<verify response-code="200"/>

<save href="{$outputdir}/test_server_response.xml"/>

</script>


So, above we have a very basic hatatap script, which tests the server response code.
• The script element is the root node, and declares the required namespaces for hatatap
• Then we have param elements required for the test run. Eventually, they will be substituted with values coming in from the Ant script. They also could prove handy for a failovers.
• Then follows the most important thing. The ‘send’ element which has a ‘url’ attribute, which denotes the server to be tested. There are other supported attributes like ‘method’ etc. You could find more information on the web site. When executed the ‘send’ makes a connection to the given ‘url’ and reads in the response. In a real life scenario we would also send along parameters’ along with the ‘send’. We will look at this further down.
• Once we have made a request and have a response, we could make use of built in elements to test them. ‘verify’ does just that. Above we are just testing the response code to be ‘200’ (OK) assuming the website is up and running.
• Last but not the least, we save the complete response to a file under the ‘output’ directory. Again in practical sense, this would be inspected or tested against.

7. So, we are set to do the first test run here.
• Open up your command prompt and navigate to the hatatap directory
• Run ‘ant’. This will run the default task and must give you the below results:

Buildfile: build.xml

clean:
[delete] Deleting directory C:\hatatap\log
[delete] Deleting directory C:\hatatap\output

prepare:
[mkdir] Created dir: C:\hatatap\log
[mkdir] Created dir: C:\hatatap\output

alltests:

oldMLDelete:

StartMLRecord:
[hatatap] input: C:\hatatap\input\test_server.xml
[hatatap] output: C:\hatatap\output\output_test_server.xml
[hatatap] info: send url: http://www.pingdynasty.com
[hatatap] info: response: 200 OK 579ms
[hatatap] info: save file: output/test_server_response.xml
[hatatap] finished test: test_server.xml

StopMLRecord:

BUILD SUCCESSFUL
Total time: 3 seconds


Before moving ahead, run through the ‘hatatap’ folder and look at the new files/folders the scripts generated for you.
• The ‘Log’ folder which has a log file with contents identical to what you saw on the console.
• The ‘output’ folder which contains the test run details and the response you saved.

Now what? Let’s look at adding some more cool and easy to use features to your hatatap script. In the above description, I mentioned that we could use XPATH with hatatap to test parts of the response.

Update your hatatap script as given below (text in bold):


<?xml version="1.0" encoding="UTF-8"?>
<script xmlns="urn:hatatap" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:hatatap lib/script.xsd">

<param name="server" default="http://www.pingdynasty.com"/>
<param name="outputdir" default=""/>

<!-- Check if the Server is up and running -->
<send url="{$server}"></send>

<verify response-code="200"/>

<save href="{$outputdir}/test_server_response.xml"/>

<verify test="count($dom//html/head) = '1'" msg="Expected '1' but was {count($dom//html/head)}" />

<verify test="count($dom//html/body) = '1'" msg="Expected '1' but was {count($dom//html/body)}" />

<verify test="$dom//title = 'Pingdynasty Technology Culture Art Politics'" msg="Expected 'Pingdynasty Technology Culture Art Politics' but was {$dom//html/title}" />


</script>


Above, we have added a few statements highlighted in ‘bold’ text. Before going through the above added statements, I would like to tell you a bit about the DOM support that comes along with hatatap. Hatatap gives you a default ‘dom’ object to play with the response. You could use this with XPATH to inspect parts of the response. What we just did above is that we sent a request to the server and read the response. We did inspect the response code. What about other parts of the response? Don’t you think we would need to inspect the response in more detail.? The added statements above do just that:

1. The first of the ‘verify’ statements uses the inbuilt ‘dom’ object to verify if we have a valid response which has a ‘head’ (html – head element) node. The ‘test’ attribute lets you implement conditional clauses and thereby make your test case more meaningful. Try updating the value expected to be ‘2’ and run the test. You would see the assertion error which is thrown when the test fails.

2. The second ‘verify’ statement is similar to the first one, but checks for an html ‘body’ node.

3. The third of the ‘verify’ statement makes a comparison of the expected title in the response.


Exploring ‘o:xml’ and integration with Hatatap:

We could make use of the power of ‘o:xml’ with hatatap test scripts to test complex scenarios and cases. We would try to explore a bit using the above example. Imagine a scenario wherein you would like to make a request to an application server, get the authorized session_id, and then make use of the session_id to test other modules of the application. How would you do that?

Here is how we would do it:
• Using hatatap connect to the app server and make a request
• Using Hatatap validate and save the Response in a file
• Using o:xml check for the file existence
• Using o:xml parse the XML
• Using o:xml set up local variables for use in the desired module.

Let’s update the Ant script to add another hatatap script, which would parse and read in data from an XML. For simplicity, let’s parse a simple XML file since the motive is to demonstrate parsing:

Add the following XML file under ‘input’.

File Name: sample.xml

<?xml version="1.0" encoding="UTF-8"?>
<sample>
<server>http://www.pingdynasty.com</server>
</sample>


Also add the hatatap script under ‘input’

File Name: test_file_parser.xml

<?xml version="1.0" encoding="UTF-8"?>
<script xmlns="urn:hatatap" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:o="http://www.o-xml.org/lang/" xmlns:io="http://www.o-xml.org/lib/io/" xsi:schemaLocation="urn:hatatap ../lib/script.xsd">

<o:import href="lib/io/File.oml"/>

<param name="server" default="http://www.pingdynasty.com"/>
<param name="outputdir" default=""/>
<param name="inputdir" default=""/>

<o:if test="io:File($inputdir,'sample.xml').exists()">
<!-- logs -- >
<log msg="File exists. lets parse and fetch the Server URL ..."/>

<!-- Parse XML -- >
<o:set sample="io:File($inputdir,'sample.xml').parse()"/>

<!-- Set up a local variable by data being read from XML -->
<o:set url="$sample//server" />

<!-- output value -->
<log msg="Server URL from sample.xml = {$url}"/>
</o:if>

</script>


Things to note above are
• The namespace elements for ‘o:xml’ and the File API
• The import statement which imports the required library
• o:xml conditional blocks (if)
• XML parsing (Instead of a default ‘dom’ object we used in the previous example, here we have our own object by the name ‘sample’ defined, which is nothing but a DOM object.
• Setting up a local variable with data read from XML.


Next, we need to update the ‘build.xml’ so that we could run the test.

build.xml (update only the target – alltests)


<target name="alltests" depends="clean, prepare">
<antcall target="StartMLRecord"/>
<hatatap in="input/test_server.xml" out="output/output_test_server.xml">
<param name="server" value="${server}"/>
<param name="outputdir" value="${test_output}"/>
<classpath refid="lib_path"/>
</hatatap>
<hatatap in="input/test_file_parser.xml" out="output/output_test_file_parser.xml">
<param name="server" value="${server}"/>
<param name="outputdir" value="${test_output}"/>
<param name="inputdir" value="${test_input}"/>
<classpath refid="lib_path"/>
</hatatap>
<antcall target="StopMLRecord"/>
</target>

Running the above updated example should give you:

Buildfile: build.xml

clean:
[delete] Deleting directory C:\hatatap\log
[delete] Deleting directory C:\hatatap\output

prepare:
[mkdir] Created dir: C:\hatatap\log
[mkdir] Created dir: C:\hatatap\output

alltests:

oldMLDelete:

StartMLRecord:
[hatatap] input: C:\hatatap\input\test_server.xml
[hatatap] output: C:\hatatap\output\output_test_server.xml
[hatatap] info: send url: http://www.pingdynasty.com
[hatatap] info: response: 200 OK 516ms
[hatatap] info: save file: output/test_server_response.xml
[hatatap] finished test: test_server.xml
[hatatap] input: C:\hatatap\input\test_file_parser.xml
[hatatap] output: C:\hatatap\output\output_test_file_parser.xml
[hatatap] info: File exists. lets parse and fetch the Server URL ...
[hatatap] info: Server URL from sample.xml = http://www.pingdynasty.com
[hatatap] finished test: test_file_parser.xml

StopMLRecord:

BUILD SUCCESSFUL


Conclusion:

There are some cool features one could use, for example: hatatap allows you to run scripts generated by JMeter. So an automated test environment could make use of JMeter to also create ready to use hatatap scripts. Another thing to look at would be features available through o:xml. There is lot of stuff one could do, like defining functions, using loops, conditional blocks etc. Hatatap on its own has got great features and one needs to play around to get the best out of it and build a dynamic test environment. Hope this article gives you an environment to do just that.


References:

1. www.o-xml.org
2. http://hatatap.pingdynasty.com



Cheers,
Kris.

Saturday, July 12, 2008

Write Custom Ant Tasks

I know that there are several documents over the internet including those on the Apache Ant manual that describes adding custom tasks to your ant scripts. My focus in this article is to give a brief understanding and provide a simple guideline to add custom task definition with a few examples and provide references for more complex problems.

This article will talk about introducing your own ant task definitions which could be used in your Ant scripts for custom logic execution. To give you a brief overview of how this is accomplished, I would say that Apache Ant is built on top of a set of libraries which allow you to make use of task definitions in the Ant scripts to get your job done in an easier way. If you observe, your Ant installation comes bundled with a basic set of libraries (ANT_HOME/lib) which enables us to write useful Ant scripts. Like wise we could have our own library or a set of libraries defined and use them in the Ant scripts.

Framework Overview:

All Ant tasks inherit from “org.apache.tools.ant.Task”. This is an Abstract class which extends “org.apache.tools.ant.ProjectComponent”. The basic rule is that custom task definitions must inherit from “org.apache.tools.ant.Task” so that we could use them in the Ant Scripts. Alternatively, we could make use of other predefined tasks provided by Ant like “org.apache.tools.ant.taskdefs.JDBCTask” for tasks meant to talk to the database, “org.apache.tools.ant.taskdefs.MatchingTask” that should be used by all those tasks that require to include or exclude files based on pattern matching, “org.apache.tools.ant.taskdefs.Pack/Unpack” for processing ZIP archives and many more such tasks.

So a basic custom Task would look as given below:

package com.test;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;

public class PrintTask extends Task
{
private String strMessage;

public void execute() throws BuildException
{
System.out.println(strMessage);
}

public void setMessage(String strMessage)
{
this.strMessage = strMessage;
}
}

Method execute is used to execute the custom logic. This is where we would carry out custom logic to be executed by the task. In the above case, we are just trying to print text to the console. We also have a setter method “setMessage” which sets up the member variable “strMessage”.

At this point, let’s look at the Ant script and how the setter method is used to print text to the console.

build.xml

<?xml version="1.0"?>

<project name="TestProject" default="print" basedir=".">
<taskdef name="printtask" classname="com.test.PrintTask "/>
<target name="print">
<printtask message="Hello World!"/>
</target>
</project>

Before we analyze the ant script, we need to make sure that we compile the custom task written above and add the same to the classpath. An easy alternative would be to jar the task and place the jar file under “ANT_HOME/lib”.

If we analyze the ant script above we have added a “taskdef” element which has attributes “name” which is the name of the custom task and “classname” which is the qualified class name of the custom task. Once we have done this, we could use the custom task like any other pre existing ant tasks. To print to the console we make a call to the newly introduced taskdef with an attribute “message” with the value to print. Its that easy. :-)

Running “ant” on the “TestProject” should give you:

Buildfile: build.xml

print:
[printtask] Hello World!

BUILD SUCCESSFUL


Total time: 0 seconds

Lets look at adding some more functionality to the above task. Say, for example we need to have some conditional logic in the task based on which we need to setup an ant property to use further down in the ant scripts. How do we do that?

Here is how:

package com.test;


import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;

public class PrintTask extends Task
{
private String strMessage;
private String strPropertyName;

public void execute() throws BuildException
{
System.out.println(strMessage);

// set up the ant property here
getProject().setUserProperty(strPropertyName, "testval");
}

public void setMessage(String strMessage)
{
this.strMessage = strMessage;
}

public void setPropertyName(String strPropertyName)
{
this.strPropertyName = strPropertyName;
}

}

We have enhanced the basic example above, to include an additional member variable which lets the task know which property to setup. In the execute() method we have made use of getProject().setUserProperty(). getproject() returns us an instance of the current ant project being executed, in the form of “org.apache.tools.ant.Project”. There are several things we could do with the “Project” object. We use it to set up a user property by the name “test.property” with a value “test” for the current project.

Here is how the build.xml would look like:

build.xml

<?xml version="1.0"?>

<project name="TestProject" default="print" basedir=".">
<property name="test.property" value="testval" />
<echo message="test.property before assignment = ${test.property}"/>
<taskdef name="printtask" classname="com.test.PrintTask "/>
<target name="print">
<printtask message="Hello World!" propertyname=”test.property”/>
<echo message="test.property after assignment = ${test.property}"/>
</target>
</project>

We have enhanced the “build.xml” to include a property definition by the name “test.property” which defaults to “testval”. For the example purpose we print the value for the property twice, i.e before and after assignment. The custom print task sets up the value for the property as described above. This also shows that the custom ant task overrides the property’s default value.

Running the above example should give you this:

Buildfile: build.xml
[echo] test.property before assignment = testval

print:
[printtask] Hello World!
[echo] test.property after assignment = test

BUILD SUCCESSFUL
Total time: 0 seconds


Easy isn’t it ? :-)

Let’s do another variant of “org.apache.tools.ant.Task”now that we have seen some of the standard Ant custom tasks. The“org.apache.tools.ant.taskdefs.JDBCTask” is a handy Task provided by Apache Ant that we could use to talk to a RDBMS using Apache Ant. Here is how we would do it:

This example assumes that you have a MYSQL database setup, and has a database with a table as given below:

Table: Products

= = = = = = = = =
[ Id ] [ version]
= = = = = = = = =
[ 1 ] [ 1.1 ]
= = = = == = = =


Having said this, its really easy to use with any other RDBMS. We just need to change the driver and URL configurations to get it up and running.

Lets write the Ant Task now:

package com.test;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;


import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.taskdefs.JDBCTask;

public class GetProductVersion extends JDBCTask {

private String strProductId;
private String strPropertyName;

private String getProductVersion() throws SQLException

{
String retval = "0";
Connection conn = null;
Statement stmt = null;
ResultSet rs = null;
try

{
conn = getConnection();
stmt = conn.createStatement();
rs = stmt.executeQuery("select version from products where id = '" + strProductId + "'");

if(rs.first())
retval = rs.getString(1);
}
finally

{
if(rs != null) rs.close();
if(stmt != null) stmt.close();
if(conn != null) conn.close();
}
return retval;
}

public void setProductId(String strProductId)
{
this.strProductId = strProductId;
}


public void setPropertyName(String strPropertyName)
{
this.strPropertyName = strPropertyName;
}

public void execute() throws BuildException
{
if(strProductId == null)
throw new BuildException("ProductId must be specified", getLocation());

try

{
getProject().setUserProperty(strPropertyName, getProductVersion());
}
catch(SQLException sqlEx)

{
throw new BuildException(sqlEx, getLocation());
}
}

}

In the above task we have made use of the JDBCTask, which makes easier for us to talk to the RDBMS using ANT. The custom task doesn’t need to worry about the connection and the drivers to make the connection. These have been taken care by the JDBCTask. The execute method just implements custom logic passed on parameters passed in.

Here is how we would use the Task in the Ant Script:

build.xml

<?xml version="1.0"?>
<project name="TestProject" default="jdbctest" basedir=".">
<taskdef name="GetProductVersion" classname="com.test.GetProductVersion"/>
<target name="jdbctest">
<GetProductVersion userid="${db.userid}"
productid="1"
password="${db.password}"
url="${db.url}"
driver="${db.driver}"
propertyname="prod.version" />
<echo message="prod.version in the DB = ${prod.version}" />
</target>
</project>

What we have here is a “taskdef” for “GetProductVersion”. If you observe in the task above, we have passed in several other parameters like userid, password, url and driver other than the productid and the propertyname for which we have customized the task. These are utilized by the JDBCTask and thus allow us to just code in the SQL blocks without worrying about stuff like driver and connection. That’s why I mentioned that its easy to make use of any RDBMS. Assuming you have provided accurate values for the task params, this is what the output should look like:

Buildfile: build.xml

jdbctest:
[echo] prod.version in the DB = 1.1

BUILD SUCCESSFUL
Total time: 0 seconds


Hope this article gives you a heads up on Ant Task Customization. Having said this we could do a lot more customization and make use of Nested elements for custom tasks. Read more on that from the Ant developer Manual. The reason I don’t intend to write examples for SubAnt and supporting Nested Task elements is because, the Ant tutorial gives us a satisfactory description along with example for the same. Anybody more interested in Ant projects, could also read about the Ant Contrib. project and its features. I have been using it in many of my projects so far.


Cheers!
Kris