Sunday, July 17, 2016

Writing a Custom Message Builder for WSO2 ESB

In WSO2 ESB, when you write a certain mediation logic, a user can do various types of alterations to the message body, headers, etc. Also a user can just passthrough the incoming raw payload without any message processing inside ESB. In situations where the message processing is done, WSO2 ESB uses a universal payload format within ESB. This universal payload format is SOAP message format.

When ESB receives a certain payload, the raw payload is converted to SOAP message format before further processing the content. For this purpose we use Message Builders. Depending on the receiving message's content type, a particular message builder is selected at the run time, for building the message. 

By default bellow Message Builders are available in WSO2 ESB.
  • SOAPBuilder
  • MIMEBuilder
  • MTOMBuilder
  • ApplicationXMLBuilder
  • XFormURLEncodedBuilder
  • BinaryRelayBuilder
  • PlainTextBuilder
For most of the message content types, these Builders can be used to build the message. If a particular user thinks that they need to furthermore customize the Message building process, WSO2 ESB has the facility to extend by adding custom message builders as well. During the latter part of this post I'll go through how to write a custom message builder for such requirements. 

Users can configure which message builder should be used for a particular content type by configuring the message builders at axis2.xml (This can be found at <carbon_home>/repository/conf/axis2/ directory) . There you can find a section called <messageBuilders></messageBuilders> . 

Let's have a look at the bellow configuration element. 

<messageBuilder contentType="text/plain" class="org.apache.axis2.format.PlainTextBuilder"/>

We add this configuration element under the messageBuilder element to configure the message builder to be used for a certain content type. 

According to the example we have configured ESB to use PlainTextBuilder for the incoming messages having content type text/xml.

Internally in ESB selecting the message Builder is triggered from wso2-synapse Engine. When the message comes to the ESB and then inside the synapse engine You can find class RelayUtils. There you can find the method called builldMessage(). Then through the getDocument() of DeferredMessageBuilder we call the processDocument() method of the Builder which is enabled for the particular builder. You will be able to get a clear idea regarding this procedure by debugging the synapse code base on the particular points I have highlighted here.

If a certain user would like to use his own, custom message builder, instead of the default message builders, WSO2 ESB has been provided the flexibility to add such custom message builders. What the user has to do is write his own custom message builder and then enable the particular message builder for the desired content type. Bellow code snippet shows such custom message builder I have written, in order to BASE64 encode an xml entry field.

When writing the custom Builder you need to implement the Builder Interface and then override the processDocument method. Inside the process document method you can define your specific logic to process the content and accordingly convert it to the SOAP format. Bellow sample code snippet assumes the incoming payload is as <sampleElement>Sample Content</sampleElement>. Then I get the content and encode the text before converting the incoming payload to SOAP before processing the content in the mediation flow.

package org.test.builder;

import org.apache.axiom.soap.SOAPBody;
import org.apache.axiom.soap.SOAPEnvelope;
import org.apache.axiom.soap.SOAPFactory;
import org.apache.axis2.AxisFault;
import org.apache.axis2.Constants;
import org.apache.axis2.builder.Builder;
import org.apache.axis2.context.MessageContext;
import org.apache.commons.codec.binary.Base64;


 * Created by nadeeshaan on 7/14/16.

public class CustomBuilderForTextXml implements Builder{
    public OMElement processDocument(InputStream inputStream, String s, MessageContext messageContext) throws AxisFault {
        SOAPFactory soapFactory = OMAbstractFactory.getSOAP11Factory();
        SOAPEnvelope soapEnvelope = soapFactory.getDefaultEnvelope();

        PushbackInputStream pushbackInputStream = new PushbackInputStream(inputStream);

        try {
            int byteVal =;
            if (byteVal != -1) {

       xmlReader = StAXUtils.createXMLStreamReader(StAXParserConfiguration.SOAP,
                        pushbackInputStream, (String) messageContext.getProperty(Constants.Configuration.CHARACTER_SET_ENCODING));

                StAXBuilder builder = new StAXOMBuilder(xmlReader);
                OMNodeEx documentElement = (OMNodeEx) builder.getDocumentElement();
                String elementVal = ((OMElement) documentElement).getText();
                byte[]   bytesEncoded = Base64.encodeBase64(elementVal.getBytes());
                ((OMElement) documentElement).setText(new String(bytesEncoded ));
                SOAPBody body = soapEnvelope.getBody();
        } catch (IOException e) {
        } catch (XMLStreamException e) {

        return soapEnvelope;

Similarly you can write your own Message Formatters as well, in order to manipulate the out going payload from ESB. Under the Message formatters section add the bellow configuration element in order to enable the message formatter for the particular content type.

<messageFormatter contentType="text/xml" class="org.apache.axis2.transport.http.SOAPMessageFormatter"/>

Find more information about the message builders and formatters on Official Documentation

Sunday, June 26, 2016

Dynamically Selecting SOAP Message Elements in WSO2 ESB

Most recently I faced a requirement where I had to select elements dynamically from a SOAP response message which comes to WSO2 ESB. The use case is as follows.

I have a proxy service where I get a request from a client and let's say the request looks like following.

1:  <Request>  
2:      <Values>  
3:          <value>2</value>  
4:      </Values>  
5:  </Request>  

The value can be changed for each request. However when the request is sent to the backend server from ESB we get a response as following

1:  <Response>  
2:      <Events>  
3:          <Event><TestEntry>Entry Val1</TestEntry></Event>  
4:          <Event><TestEntry>Entry Val2</TestEntry></Event>  
5:          <Event><TestEntry>Entry Val3</TestEntry></Event>  
6:          <Event><TestEntry>Entry Val4</TestEntry></Event>  
7:          <Event><TestEntry>Entry Val5</TestEntry></Event>  
8:      </Events>  
9:  </Response>  

Depending on the Value specified in the Request, we need to extract the specified number of event entries from the response (If the value in the request is 2 then we will have to extract two event entries from the response).

In order to achieve the requirement I had to follow the bellow configurations.

Sequence Calling the xslt Transform

Use the xslt mediator to transform the incoming payload to ESB

1:  <sequence name="get_document_list_seq" trace="disable" mlns="">  
2:   <property expression="//value" name="limit"  
3:    scope="default" type="STRING" xmlns:ns="http://org.apache.synapse/xsd"/>  
4:   <xslt key="payload_transform" source="//Response"  
5:    xmlns:ns="http://org.apache.synapse/xsd"  
6:    xmlns:s11="" xmlns:tem="">  
7:    <property expression="get-property('limit')" name="PARAM_NAME"/>  
8:   </xslt>  
9:   <respond/>  
10:  </sequence>  

XSLT Transformation to transform the payload and extract the elements

With the Property set with the name PARAM_NAME which passed to the xslt from the sequence above, will use to determine the number of elements to be extracted.

1:  <localEntry key="payload_transform" xmlns="">  
2:   <xsl:stylesheet version="2.0" xmlns:xsl="">  
3:    <xsl:output indent="yes" method="xml" omit-xml-declaration="yes"/>  
4:    <xsl:param name="PARAM_NAME"/>  
5:    <xsl:template match="/">  
6:     <list>  
7:      <responseList>  
8:       <xsl:for-each select="//Response/Events/Event[position()&lt;=number($PARAM_NAME)]">  
9:         <entry>  
10:          <xsl:value-of select="TestEntry"/>  
11:         </entry>  
12:       </for-each>  
13:      </responseList>  
14:     </list>  
15:    </xsl:template>  
16:   </xsl:stylesheet>  
17:  </localEntry>  

Saturday, February 6, 2016

Working with WSO2 ESB Event Sink and WSO2 DAS

In this post I am going to point out how to integrate WSO2 ESB and WSO2 DAS to send events from ESB to the DAS server. In order to achieve this I am going to use WSO2 ESB v4.9.0 and WSO2 DAS v 3.0.0.
In wso2 ESB you can publish events through publish event mediator and event sinks. In order to achieve our task first we need to create an event receiver and an event stream at the DAS end. Also we create an event sink at the ESB end, and use publish event mediator to publish our event to the DAS’s event receiver.

Creating an event stream
Log on to DAS server and navigate to streams section as follows.

You can add three types of attributes for the event. Payload data Attributes, Meta Data Attributes and correlation data attributes. In this post I am just using the payload data attributes only. Next persist the event stream.

Creating an Event Receiver
Now we need to create an event receiver to receive the events we send from the ESB end. click on the Receivers tab and you will be navigated to the following page. There you can select the desired event type from the available number of event types. In our scenario we are going to use the wso2event type. Also you select your desired event stream you created earlier.

Creating an Event Sink

Click on the Event Sink tab and you will be navigated to the following page. Fill the relevant data to create the Event Sink.

Give a name, the username, password and the receiver URL (tcp//:<host name>:7611). Save the event sink.

<proxy xmlns=""
                        <attribute name="atr1" type="STRING" defaultValue="" value="testValue"/>
                        <attribute name="atr2" type="STRING" defaultValue="" value="Test Region"/>

Create a proxy as above. You can see that I have used the publishEvent mediator. You include the event sink name, stream name and the stream version. In the attributes you can see we include the metadata, correlation data and payload data.

Now hit the defined proxy and you can browse data in the DAS through the Data Explorer tab. Select the table and search. Then you can see the data we published through our event sink.

Enabling per car file Logging in WSO2 ESB 4.9.0

In WSO2 ESB you can deploy multiple ESB artifact with a car file. In WSO2 ESB 4.9.0 you can get logs identified specially for esb artifacts deployed from a car file. In order to enable this feature you can follow the following steps.

  1. Locate the configuration file at CARBON_HOME/repository/conf directory
  2. Comment log4j.appender.CARBON_CONSOLE.layout=org.apache.log4j.PatternLayout entry
  3. Add log4j.appender.CARBON_CONSOLE.layout=org.apache.synapse.AppenderWrapper entry instead

Supported artifact types

  1. MQTT Inbound Endpoints
  2. Proxy Services
  3. APIs
  4. Sequences
  5. Endpoints
  6. Inbound Endpoints
    • Vfs Inbound Endpoints
    • Http Inbound Endpoints
    • Https Inbound endpoints
    • Kafka inbound endpoints
    • HL7 Inbound Endpoints

Sunday, December 27, 2015

Using WSO2 ESB HTTP Endpoints to define Restful Endpoints

When we consider about the endpoints which has been allowed in WSO2 ESB there are several as follows

  • Address Endpoint
  • Dynamic Load-balance Endpoint
  • Failover Group
  • HTTP Endpoint
  • WSDL Endpoint
  • Load-balance Group
  • Indirect and Resolving Endpoints
  • Default Endpoint
  • Template Endpoint
  • Recipient List Endpoint

In this post I am going to take you through, how to use HTTP Endpoints to define Restful Endpoints. You can use Uri Templates to achieve this. There we can use the variables to retrieve the values of the uri at run time.

In order to demonstrate this I am going to define an api which I am going to access through an HTTP Endpoint and a proxy service.

<endpoint xmlns="" name="TestHttpEP">
   <http uri-template="http://localhost:8280/tutorial/{uri.var.param}/{uri.var.val}" method="get">

In the above defined HTTP Endpoint, uri-template attribute has been set to the resource defined in the bellow API. Also You can see the variables inside the curly Braces. These variables get the defined property values at the run time and generate the required uri. These properties have been defined in the proxy service's in sequence in which I am invoking.

<proxy xmlns=""
         <property name="uri.var.param" value="parameter" scope="default" type="STRING"/>
         <property name="uri.var.val" value="1234" scope="default" type="STRING"/>
         <property name="NO_KEEPALIVE" value="true" scope="axis2"/>
            <property name="In_Seq" value="Hit In Sequence"/>
            <endpoint key="TestHttpEP"/>
            <property name="Out_Seq" value="Hit Out Sequence"/>

<api xmlns="" name="testApi" context="/tutorial">
   <resource methods="GET" uri-template="/test/val" faultSequence="fault">
            <property name="IN_SEQ_LOG" value="HIT_IN_SEQ"/>
         <payloadFactory media-type="xml">
               <test xmlns="">TEST</test>
            <property name="OUT_SEQ_LOG" value="OUT_SEQ_HIT"/>

Now invoke the proxy service through any rest client and you will see the response sent through the API as follows.
<test xmlns="">TEST</test>

Sunday, July 12, 2015

Integrating WSO2 ESB Connectors in real world integration Scenarios

For a moment consider some of the third party services we use daily. For an example We can consider Twitter as such a service. When we consider this fact we can identify a lot of service we use in our day to day life. Usually we use APIs to connect to these services as developers. WSO2 ESB Connectors allow you to interact with such third parties’ API from ESB Message flow. With these connectors you can access the API interact with it through the exposed functionalities.

Eg: With the Twitter Connector you can access the twitter API and then retrieve tweets of a user.

At the moment you can find more than hundred of connectors at Connector Repo. And you can build one connector at a time simply with the maven build tool.

Working with one connector can be much simpler as well as less frequently used. Consider a scenario in which we are communicating with two such third parties’ APIs. Such scenarios are complex to handle and messy in configuring them. With WSO2 ESB Connectors you can easily acquire such integrations for real world scenarios as well as business scenarios.

During this blog post Let’s have a look at such real world scenario which addresses an important project management task.

Integration Scenario with RedMine, Google Spreadsheet and Gmail.

In this integration scenario my objective is to track the expired Redmine assignments to a particular user/all the users and then retrieve the information as well as sending an email notification and log them in to google spreadsheet.

In order to accomplish this task we need to have the Redmine Connector, Google Spreadsheet Connector and the Gmail Connector. After building the connectors, before you write the configuration you need to upload them to ESB. Make sure to enable the connector after uploading them.

Enable Connectors

Configuring connectors

Gmail Connector
<username>your_user_name</username> <oauthAccessToken>your_access_token</oauthAccessToken>

Redmine Connector

Google Spreadsheet Connector

Note: For the development purposes you can access Google OAuth 2.0 Playground in order to get the access tokens for gmail and google spreadsheet connectors.

<?xml version="1.0" encoding="UTF-8"?> <definitions xmlns=""> <registry provider="org.wso2.carbon.mediation.registry.WSO2Registry"> <parameter name="cachableDuration">15000</parameter> </registry> <taskManager provider="org.wso2.carbon.mediation.ntask.NTaskTaskManager"/> <import name="gmail" package="org.wso2.carbon.connector" status="enabled"/> <import name="redmine" package="org.wso2.carbon.connector" status="enabled"/> <import name="evernote" package="org.wso2.carbon.connector" status="enabled"/> <import name="googlespreadsheet" package="org.wso2.carbon.connectors" status="enabled"/> <proxy name="evernote_gtask_proxy" transports="https http" startOnLoad="true" trace="disable"> <description/> <target> <inSequence onError="faultHandlerSeq">

<redmine.listIssues> <statusId>*</statusId> <limit>2</limit> <assignedToId>me</assignedToId> </redmine.listIssues>
<property name="cur_date" expression="get-property('SYSTEM_DATE', 'yyyy-MM-dd')" scope="default"/> <iterate continueParent="true" preservePayload="true" attachPath="//issues" expression="//issues/issue"> <target> <sequence> <log level="custom"> <property name="ITERATOR" value="Iterating Over the redmine feature list"/> </log> <property name="issue-id" expression="//issues/issue/id"/> <property name="project-name" expression="//issues/issue/id/@name"/> <property name="description" expression="//issues/issue/description"/> <property name="due-date" expression="//issues/issue/due_date"/> <script language="js">var current_date = mc.getProperty("cur_date").split("-"); var due_date = mc.getProperty("due_date"); if (due_date === null){ mc.setProperty("is_due","false"); } else{ var due_date_arr = due_date.split("-"); var due_date_obj = new Date(due_date_arr[0],due_date_arr[1],due_date_arr[2]); var cur_date_obj = new Date(current_date[0],current_date[1],current_date[2]); if((cur_date_obj&gt;due_date_obj)&gt;0){ mc.setProperty("is_due","true"); } else{ mc.setProperty("is_due","true"); } } </script> <gmail.init> <username>username</username> <oauthAccessToken>access_token</oauthAccessToken> </gmail.init> <filter source="get-property('is_due')" regex="true"> <then> <gmail.sendMail> <subject>Subject</subject> <toRecipients>recipients</toRecipients> <textContent>Email_Body</textContent> </gmail.sendMail> </then> </filter> </sequence> </target> </iterate> <script language="js">var current_date = mc.getProperty("cur_date").split("-"); var issues = mc.getPayloadXML().issue; var returnCsv = "Issue_ID,Project_Name,Due_Date\n"; for(i=0;i&lt;issues.length();i++){ var id = issues[i].id; var name = issues[i].project.@name; var due_date = issues[i].due_date; if (due_date != null){ var due_date_arr = due_date.split("-"); var due_date_obj = new Date(due_date_arr[0],due_date_arr[1],due_date_arr[2]); var cur_date_obj = new Date(current_date[0],current_date[1],current_date[2]); if((cur_date_obj&gt;due_date_obj)&gt;0){ mc.setProperty("task_due","true"); returnCsv=returnCsv+id+","+name+","+due_date+"\n"; } else{ mc.setProperty("task_due","false"); } } }                mc.setPayloadXML(                    &lt;text&gt;{returnCsv}&lt;/text&gt; );</script> <log level="full"/> <googlespreadsheet.oAuth2init> <oauthConsumerKey>consumer_key</oauthConsumerKey> <oauthConsumerSecret>consumer_secret</oauthConsumerSecret> <oauthAccessToken>access_token</oauthAccessToken> <oauthRefreshToken>refresh_token</oauthRefreshToken> </googlespreadsheet.oAuth2init> <filter source="get-property('task_due')" regex="true"> <then> <googlespreadsheet.importCSV> <spreadsheetName>spread_sheet_name</spreadsheetName> <worksheetName>work_sheet_name</worksheetName> <batchEnable>true</batchEnable> <batchSize>10</batchSize> </googlespreadsheet.importCSV> </then> </filter> <respond/> </inSequence> <outSequence> <log level="full"/> <send/> </outSequence> </target> </proxy> <localEntry key="csv_transform"> <xsl:stylesheet xmlns:xsl="" version="1.0"> <xsl:output method="text" indent="no" encoding="UTF-8"/> <xsl:template match="/"> Issue_ID,Project_Name,Due_Date <xsl:value-of select="//issues/issue/id"/> <xsl:text>,</xsl:text> <xsl:value-of select="//issues/issue/project/@name"/> <xsl:text>,</xsl:text> <xsl:value-of select="//issues/issue/due_date"/> </xsl:template> </xsl:stylesheet> <description/> </localEntry> <sequence name="fault"> <log level="full"> <property name="MESSAGE" value="Executing default 'fault' sequence"/> <property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/> <property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/> </log> <drop/> </sequence> <sequence name="main"> <in> <log level="full"/> <filter source="get-property('To')" regex="http://localhost:9000.*"> <send/> </filter> </in> <out> <send/> </out> <description>The main sequence for the message mediation</description> </sequence> </definitions>

According to the above configuration, at first via the redmine connector we get the corresponding redmine issues. Then we use the iterate mediator to iterate through the result list. Depending on the result, use the script mediator to extract the required data fields and append the data to the payload accordingly. Then use the google spreadsheet connector to write them to the spreadsheet.

Get the Connectors Here

Sunday, December 28, 2014

MySQL Performance Testing with MySQLSLAP and Apache Bench

When we create a database it is always a difficult task to determine the performance of the database system under heavy data sets. In the real time, there are situations in which more than one user tries to access your application, web site, etc.. concurrently. In such situations the performance degradation might have a great affect with your database. Therefore it is always necessary to put your database under stress test before you put it to work in the real time.
In this article I am trying to give a brief explanation about how to use some effective tools to test your database system under heavy loads.


This tool always come with your MySQL installation and can be found PATH_TO_MYSQL/mysql/bin . Inside this directory you can find a script called mysqlslap.exe (In Windows). After locating the relevant script direct to the location and then you can invoke the mysqlslap as mysqlslap [options].

Above figure shows most widely used set of options to test a certain query with mysqlslap. With this query under the options parameter --query we include the location of relevant sql file in which you have mentioned your query. Under the parameter --create-schema you include the name of your database. Under the parameter --concurrency mention the number of concurrent accesses which tries to access the query concurrently. Then you can mention under the parameter -- iterations, the number of time the query runs. If you wish to include more than one query in the sql you need to mention the query delimiter to separate one query from the other and use --delimiter to specify the relevant delimiter.
According to the above figure the mentioned query runs 5 time under one concurrent access. As a total the same query runs 250 times for the configurations.
After the benchmarking you can see the results and analyze them depending on your requirements. Rather than the above mentioned options there are other options which you can find in mysqlslap official documentation.

Apache Bench

With mysqlslap you can mention the set of or individual queries to be tested under stress testing. With Apache bench it is easy to test your web application when a certain page loads. With this tool you do not need to mention all the queries and this allows to measure the performance of your application when executing a set of queries when the loads.
After installing Apache Bench in your system you  need to direct to the location where you have installed it.
Now you can simply issue the command ab –n 500 –c 100 <Enter Url here> to test the page which is mentioned by the url. With the options -n you can ensure the number of requests to perform and the parameter -c determines the number of concurrent requests.
You can find the other available set of options in Apache Bench Options.