Open in app

Sign in

Write

Sign in

[WSO2] Clustering EI Analytics Profile

Dilshani Subasinghe
5 min readApr 23, 2017

Setup

  • Enterprise Integrator 6.1.0
  • Analytics profile configure with Oracle 12 c database
  • Analytics two nodes in separate instances

Data source Configuration

  1. Install Oracle database server.
  2. Create databases in Oracle.

Create

  • REGISTRY database (ei610regana),
  • USER MANAGEMENT database (),
  • WSO2_ANALYTICS_EVENT_STORE_DB (anaevesto61),
  • WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB (anaprodstor61) and
  • Two carbon databases (wso2carbonana1, wso2carbonana2) as we need 2 local databases for analytics nodes.
drop user ei610regana cascade;
Create user ei610regana identified by ei610regana account unlock;
grant connect to ei610regana;
grant resource, create session, dba to ei610regana;
grant unlimited tablespace to ei610regana;
grant create table to ei610regana;
commit;drop user ei610user cascade;
Create user ei610user identified by ei610user account unlock;
grant connect to ei610user;
grant resource, create session, dba to ei610user;
grant unlimited tablespace to ei610user;
grant create table to ei610user;
commit;drop user anaevesto61 cascade;
Create user anaevesto61 identified by anaevesto61 account unlock;
grant connect to anaevesto61;
grant resource, create session, dba to anaevesto61;
grant unlimited tablespace to anaevesto61;
grant create table to anaevesto61;
commit;drop user anaprodstor61 cascade;
Create user anaprodstor61 identified by anaprodstor61 account unlock;
grant connect to anaprodstor61;
grant resource, create session, dba to anaprodstor61;
grant unlimited tablespace to anaprodstor61;
grant create table to anaprodstor61;
commit;drop user wso2carbonana1 cascade;
Create user wso2carbonana1 identified by wso2carbonana1 account unlock;
grant connect to wso2carbonana1;
grant resource, create session, dba to wso2carbonana1;
grant unlimited tablespace to wso2carbonana1;
grant create table to wso2carbonana1;
commit;drop user wso2carbonana2 cascade;
Create user wso2carbonana2 identified by wso2carbonana2 account unlock;
grant connect to wso2carbonana2;
grant resource, create session, dba to wso2carbonana2;
grant unlimited tablespace to wso2carbonana2;
grant create table to wso2carbonana2;
commit;

Configure data source related configs in EI Analytic profile

3. Download Oracle JDBC driver. Add driver jar in to lib folder ($EI_HOME/lib)

4. Add carbon data source details in to master-datasources.xml ($EI_HOME/wso2/analytics/conf/datasources)

Replace <ip >with instance IP of which installed oracle server.

<datasources>
       <datasource>
            <name>WSO2_CARBON_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
                    <username>wso2carbonana1</username>
                    <password>wso2carbonana1</password>
                    <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <defaultAutoCommit>false</defaultAutoCommit>
               </configuration>
            </definition>
        </datasource><datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2RegDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
                    <username>ei610regana</username>
                    <password>ei610regana</password>
                    <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                    <defaultAutoCommit>false</defaultAutoCommit>
                </configuration>
            </definition>
        </datasource><datasource>
            <name>WSO2_UM_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2UMDB</name>
            </jndiConfig>
            <definition type="RDBMS">
            <configuration>
                   <url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
                   <username>ei610user</username>
                   <password>ei610user</password>
                   <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
                   <maxActive>50</maxActive>
                   <maxWait>60000</maxWait>
                   <testOnBorrow>true</testOnBorrow>
                   <validationQuery>SELECT 1 FROM DUAL</validationQuery>
                   <validationInterval>30000</validationInterval>
            </configuration>
        </definition>
</datasource>
</datasources>

5. Add WSO2_ANALYTICS_EVENT_STORE_DB and WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB data source details in to analytics-datasources.xml ($EI_HOME/wso2/analytics/conf/datasources).

Replace <ip >with instance IP of which installed oracle server.

<datasource>
            <name>WSO2_ANALYTICS_EVENT_STORE_DB</name>
            <description>The datasource used for analytics record store</description>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
                    <username>anaevesto61</username>
                    <password>anaevesto61</password>
                    <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <validationQuery>SELECT 1 FROM DUAL</validationQuery>
                    <defaultAutoCommit>false</defaultAutoCommit>
                    <initialSize>0</initialSize>
                    <testWhileIdle>true</testWhileIdle>
                    <minEvictableIdleTimeMillis>4000</minEvictableIdleTimeMillis>
                    <defaultTransactionIsolation>READ_COMMITTED</defaultTransactionIsolation>
                </configuration>
            </definition>
        </datasource><datasource>
            <name>WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB</name>
            <description>The datasource used for analytics record store</description>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:oracle:thin:@<ip>:1521/pdbora12c</url>
                    <username>anaprodstor61</username>
                    <password>anaprodstor61</password>
                    <driverClassName>oracle.jdbc.driver.OracleDriver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <validationQuery>SELECT 1 FROM DUAL</validationQuery>
                    <defaultAutoCommit>false</defaultAutoCommit>
                    <initialSize>0</initialSize>
                    <testWhileIdle>true</testWhileIdle>
                    <minEvictableIdleTimeMillis>4000</minEvictableIdleTimeMillis>
                    <defaultTransactionIsolation>READ_COMMITTED</defaultTransactionIsolation>
                </configuration>
            </definition>
        </datasource>
 </datasources>

6. To share the user store among the nodes, modify user-mgt.xml ($EI_HOME/wso2/analytics/conf) as follows:

[Change <Property name=”dataSource”> as given]

 <Property name=”dataSource”>jdbc/WSO2UMDB</Property>

7. Add or modify “govregistry” datasource attribute in registry.xml ($EI_HOME/wso2/analytics/conf) as follows:

Replace <ip >with instance IP of which installed oracle server.

<dbConfig name="wso2registry">
        <dataSource>jdbc/WSO2CarbonDB</dataSource>
    </dbConfig><dbConfig name="govregistry">
        <dataSource>jdbc/WSO2RegDB</dataSource>
    </dbConfig><remoteInstance url="https://localhost:9443/registry">
        <id>gov</id>
        <cacheId>jdbc:oracle:thin:@<ip>:1521/pdbora12c</cacheId>
        <dbConfig>govregistry</dbConfig>
        <readOnly>false</readOnly>
        <enableCache>true</enableCache>
        <registryRoot>/</registryRoot>
    </remoteInstance><mount path="/_system/governance" overwrite="true">
        <instanceId>gov</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount><mount path="/_system/config" overwrite="true">
        <instanceId>gov</instanceId>
        <targetPath>/_system/config</targetPath>
    </mount>

Configure clustering related configs in EI Analytic profile

8. Edit axis2.xml ($EI_HOME/wso2/analytics/conf/axis2/) in each node as follows.

  • Enable clustering.
<clustering class = "org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
  • Enable wka mode.
<parameter name="membershipScheme">wka</parameter>
  • Add other analytics profile nodes details under members tag.
<members><member><hostName>[node1 IP]</hostName><port>[node1 port]</port></member><member><hostName>[node2 IP]</hostName><port>[node2 port]</port></member></members>

Sample Config:

<members>
    <member>
       <hostName>192.168.**.***</hostName>
       <port>4100</port>
     </member>
</members>
  • For each node, enter the respective server IP address as the value for the localMemberHost property

<parameter name="localMemberHost">[Server_IP_Address]</parameter>

9. Configure the event-processor.xml ($EI_HOME/wso2/analytics/conf/) file as follows to cluster CEP in the Receiver.

  • Enable the HA mode

<mode name="HA" enable="true">

  • Disable the Distributed mode

<mode name="Distributed" enable="false">

  • For each node, enter the respective server IP address under the HA mode Config section.
<eventSync>
 <hostName>*.*.*.*</hostName>

10. Update spark-defaults.conf ($EI_HOME/wso2/analytics/conf/analytics /spark/) file as follows to use the Spark cluster embedded within Analytic profile.

  • Keep the carbon.spark.master configuration as local. This instructs Spark to create a Spark cluster using the Hazelcast cluster.
  • Enter 2 as the value for the carbon.spark.master.count configuration. This specifies that there should be two masters in the Spark cluster. One master serves as an active master and the other serves as a stand-by master. (Change count according to number of nodes)
carbon.spark.master local
carbon.spark.master.count 2
  • Create a symbolic link to <$AnalyticProfile_Home>in both nodes, where paths of those symbolic links are identical. This ensures that if we use the symbolic link to access the analytic profile, we can use a common path.

In this scenario we are having analytics profile in two different instances which are having two different OSes. Therefore choose a common directory as follows:

/usr/share/das/das_symlink

Use following command to create symlink folder.

ln -s <$AnalyticProfile_Home> <$Symlink_Folder>

Sample config:

ln -s /home/debian/analytics2/wso2ei-6.1.0/wso2/analytics /usr/share/das/das_symlink61

Give permission to that folder to have access while creating analytics cluster. (User group and user name should be change accordingly). Use following command for that.

sudo chown -R username:group directory

Ex: sudo chown -R cloud-user:cloud-user /user/share/das/das_symlink61

Check for updated permission for the folder using command ls -al

[cloud-user@ei6s2-analytics1 das]$ ls -la

Sample output:

lrwxrwxrwx. 1 cloud-user cloud-user 69 මාර් 17 11:35 das_symlink61 -> /home/cloud-user/EI610Analytics1/wso2ei-6.1.0-SNAPSHOT/wso2/analytics

Now you are done with creating symlink.

Add symlink in spark-defaults.conf file.

carbon.das.symbolic.link /usr/share/das/das_symlink61/

11. Add followings in to wso2server.sh ($EI_Home/wso2/analytics/bin/)

-Dcarbon.das.symlink.config.dir.path=<SYM_LINK_PATH>/conf
-Dcarbon.das.symlink.external.lib.dir.path=<SYM_LINK_PATH>/../../lib
-Dcarbon.das.symlink.dropins.dir.path=<SYM_LINK_PATH>/../../dropins
-Dcarbon.das.symlink.internal.lib.dir.path=<SYM_LINK_PATH>/../lib
-Ddas.symlink.components.repo=<SYM_LINK_PATH>/../components/plugins

Sample config:

ANA_HOME=”/usr/share/das/das_symlink61"
...-Dcarbon.das.symlink.config.dir.path=$ANA_HOME/conf \
-Dcarbon.das.symlink.external.lib.dir.path=$ANA_HOME/../../lib \
-Dcarbon.das.symlink.dropins.dir.path=$ANA_HOME/../../dropins \
-Dcarbon.das.symlink.internal.lib.dir.path=$ANA_HOME/../lib \
-Ddas.symlink.components.repo=$ANA_HOME/../components/plugins \

12. Add following in to load-spark-env-vars.sh file ($EI_Home/wso2/analytics/bin/).

export CARBON_CONFIG_DIR_PATH=/usr/share/das/das_symlink61/conf

Note: Replace path with given path.

You are done with configurations :) .

When you started, you can check for spark cluster to identify status. Default url for spark cluster : <ip_instance>:8082/

Sample URL:

http://192.168.**.**:8082/

Note: As we didn’t configure the deployment synchronizer, it is required to deploy any C-App use in the Analytics profile cluster set up to both the nodes.

Important Notes

  1. When changing datasources/clean up datasources within cluster, you should have to clean following folders too. ( rm -r * ).
  • solr/*
  • tmp/*
  • repository/data/*
  • repository/database/*

2. When you change symplinks, clear above folders and additionally followings too.

  • work/*

3. When you are going start cluster, start one node and in parallel you can start other node too. Otherwise you will get known issue as follows:

ERROR {org.wso2.carbon.event.processor.manager.core.internal.HAManager} -  CEP HA State syncing failed, No execution plans exist for tenant  -1234
org.wso2.carbon.event.processor.manager.core.exception.EventManagementException: No execution plans exist for tenant  -1234
    at org.wso2.carbon.event.processor.core.internal.CarbonEventProcessorManagementService.restoreState(CarbonEventProcessorManagementService.java:83)
    at org.wso2.carbon.event.processor.manager.core.internal.HAManager.syncState(HAManager.java:336)
    at org.wso2.carbon.event.processor.manager.core.internal.HAManager.access$100(HAManager.java:49)
    at org.wso2.carbon.event.processor.manager.core.internal.HAManager$2.run(HAManager.java:276)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Refer [1] for more details and for known issues.

References:

  1. https://docs.wso2.com/display/CLUSTER44x/Minimum+High+Availability+Deployment+-+DAS+3.1.0
Wso2
Analytics
Wso2ei
Clustering

--

--

Dilshani Subasinghe

Written by Dilshani Subasinghe

44 Followers

Techie | Traveler | Writer

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams