How to setup a Cluster using wso2 Message Broker

It is possible to install multiple instances of WSO2 products in a cluster to ensure that if one instance becomes unavailable or is experiencing high traffic, another instance will seamlessly handle the requests.A cluster consists of multiple instances of a product that act as if they are a single instance and divide up the work.
WSO2 provides Hazelcast Community Edition as its default clustering engine.

For this setup I have use two WSO2 MB nodes with a mysql database. For this sample setup I have use the same server for both MB nodes and database. So host configurations needs to be changed on a real environment.

NOTE: I will be using Node 1 with default ports and Node 2 with a port offset of 3 as I'm running the both instances in the same server. So first change the port offset of the 2nd MB instance by changing the following property in carbon.xml file

<offset>0</offset

Creating and sharing the Database

Create following four databases.

• WSO2_USER_DB -     JDBC user store and authorization manager
• REGISTRY_DB     - Shared database for config and governance registry mounts in the product's nodes
• REGISTRY_LOCAL1     - Local registry space in the manager node
• REGISTRY_LOCAL2     - Local registry space in the worker node

From most production environments, it is recommended to externalize the databases to a JDBC database of your choice and split the registry space to manage registry resources in a better way.

create database WSO2_USER_DB;

use WSO2_USER_DB;

source /dbscripts/mysql.sql;
grant all on WSO2_USER_DB.* TO regadmin@localhost identified by "regadmin";

create database REGISTRY_DB;
use REGISTRY_DB;
source /dbscripts/mysql.sql;
grant all on REGISTRY_DB.* TO regadmin@localhost identified by "regadmin";

create database REGISTRY_LOCAL1;
use REGISTRY_LOCAL1;
source /dbscripts/mysql.sql;
grant all on REGISTRY_LOCAL1.* TO regadmin@localhost identified by "regadmin";

create database REGISTRY_LOCAL2;
use REGISTRY_LOCAL2;
source /dbscripts/mysql.sql;
grant all on REGISTRY_LOCAL2.* TO regadmin@localhost identified by "regadmin";

Configure the Node 1:

In axis2.xml

Enable Hazelcast clustering

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

Use well known address based multi-casting.

<parameter name="membershipScheme">wka</parameter>

Update the host name with the correct IP value of this node.

<parameter name="localMemberHost">10.100.7.118</parameter>

Add the details of the other members. In my case it is the other mb node which runs with offset of 3 in the same server.When configuring your WSO2 products for clustering, it is necessary to use a specific IP address and not localhost or host names in your configurations. So, keep this in mind when hosting WSO2 products in your production environment.

<members>
<member>
    <hostname>10.100.7.118</hostname>
    <port>4003</port>
</member>
</members>

        
In carbon.xml

Uncomment and add the  IP address of the machine hosting this server for host name and carbon management console host name.

<hostname>10.100.7.118</hostname><mgthostname>10.100.7.118</mgthostname>

In master-datasources.xml
Comment the datasource with the name "WSO2_CARBON_DB"

Add the following three datasources.

 

<datasource>
    <name>REGISTRY_LOCAL1</name>
    <description>The datasource used for registry- local</description>
    <jndiConfig>
        <name>jdbc/WSO2CarbonDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:mysql://localhost:3306/REGISTRY_LOCAL1?autoReconnect=true</url>
            <username>regadmin</username>
            <password>regadmin</password>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>
<datasource>
    <name>REGISTRY_DB</name>
    <description>The datasource used for registry- config/governance</description>
    <jndiConfig>
        <name>jdbc/WSO2RegistryDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:mysql://localhost:3306/REGISTRY_DB?autoReconnect=true</url>
            <username>regadmin</username>
            <password>regadmin</password>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>
<datasource>
    <name>WSO2_USER_DB</name>
    <description>The datasource used for registry and user manager</description>
    <jndiConfig>
        <name>jdbc/WSO2UMDB</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <url>jdbc:mysql://localhost:3306/WSO2_USER_DB</url>
            <username>regadmin</username>
            <password>regadmin</password>
            <driverClassName>com.mysql.jdbc.Driver</driverClassName>
            <maxActive>50</maxActive>
            <maxWait>60000</maxWait>
            <testOnBorrow>true</testOnBorrow>
            <validationQuery>SELECT 1</validationQuery>
            <validationInterval>30000</validationInterval>
        </configuration>
    </definition>
</datasource>

In registry.xml
Add details of the shared registry

<dbConfig name="sharedregistry">
    <dataSource>jdbc/WSO2RegistryDB</dataSource>
</dbConfig>

<remoteInstance url="https://localhost:9443/registry">
    <id>instanceid</id>
    <dbConfig>sharedregistry</dbConfig>
    <readOnly>false</readOnly>
    <enableCache>true</enableCache>
    <registryRoot>/</registryRoot>
    <cacheId>regadmin@jdbc:mysql://localhost:3306/REGISTRY_DB?autoReconnect=true</cacheId>
</remoteInstance>
 
<mount path="/_system/config" overwrite="true">
    <instanceId>instanceid</instanceId>
    <targetPath>/_system/config</targetPath>
</mount>
 
<mount path="/_system/governance" overwrite="true">
    <instanceId>instanceid</instanceId>
    <targetPath>/_system/governance</targetPath>
</mount>

In user-mgt.xml

Change the datasource to WSO2CarbonDB  as follows.

<Property name="dataSource">jdbc/WSO2CarbonDB</Property>

Configure the node 2 :

Change the Node 2 also in the same way. Note following differences in the two nodes.

In axis2.xml
localMemberPort should be offset by 3 as follows.

<parameter name="localMemberPort">4003</parameter

The members should be updated correctly

<members>
    <member>
        <hostName>10.100.7.118</hostName>
        <port>4000</port>
    </member>
</members>

In master-datasources.xml

Instead of REGISTRY_LOCAL1 , the details of REGISTRY_LOCAL2 should be added as follows.

<datasource> 
    <name>REGISTRY_LOCAL2</name> 
    <description>The datasource used for registry- local</description> 
    <jndiConfig> 
        <name>jdbc/WSO2CarbonDB</name> 
    </jndiConfig> 
    <definition type="RDBMS"> 
        <configuration> 
            <url>jdbc:mysql://localhost:3306/REGISTRY_LOCAL2?autoReconnect=true</url> 
            <username>regadmin</username> 
            <password>regadmin</password> 
            <driverClassName>com.mysql.jdbc.Driver</driverClassName> 
            <maxActive>50</maxActive> 
            <maxWait>60000</maxWait> 
            <testOnBorrow>true</testOnBorrow> 
            <validationQuery>SELECT 1</validationQuery> 
            <validationInterval>30000</validationInterval> 
        </configuration> 
    </definition> 
</datasource> 

In Carbon.xml
Required port offset is added as described earlier.

<Offset>3</Offset>

Once you have configured the broker nodes and the DBMS for storage, you can start up the Message Broker cluster. Start one MB node and wait until the startup completed. You must wait until the Management Console URL appears on your command line and then start the other node.

References :
[1] https://docs.wso2.com/display/CLUSTER44x/Sharing+Databases
[2] https://docs.wso2.com/display/CLUSTER44x/Clustering+Message+Broker