How to do advance level OSGI debug to resolve dependency conflicts

The reason behind writing this article was due to the intermittent issue[1] reported in WSO2 EMM server startup. In order to find the root cause I had to debug the code at OSGI level to identify the conflicts, which gave a great length of knowledge on how OSGI works. Therefore I thought of sharing the knowledge I gained through the process.

Problem:

When starting the EMM server sometimes webapps don't get deploy due to ClassNotFoundException. The webapps refer some classes in org.wso2.carbon.device.mgt.common OSGI bundle.

Reasoning (Thinking pattern):

Since this is an intermittent issue, the first thing that comes to our mind is that sometimes webapps get deployed before OSGI resolve the org.wso2.carbon.device.mgt.common bundle. But WSO2 platform is built in a way that webapp deployments always happen after the server startup. Therefore the next doubt that comes to our mind is dynamic imports. When we set the resolution "optional" for dynamic import in OSGI configurations, OSGI will resolve the bundle first and later try to resolve the dynamic imports (dependencies). At this time, for a particular dynamic import, if OSGI encounter any conflicts, it will skip resolving the dynamic import and try again later after iterating through the list. This cause might delay the class loading and result in ClassNotFoundException for webapps.

Debugging:

How can we debug this issue? 

There are several ways to debug...

1. Start the server with OSGI console and investigate
2. Enabling OSGI level debug logs
3. Debugging the Equinox OSGI source 

1) Start the server with OSGI console and investigate
You can start the server OSGI console wih below command.
           sh wso2server.sh -DosgiConsole

Inspecting the org.wso2.carbon.device.mgt.common bundle status.

osgi> ss device.mgt.common
id State       Bundle
262 ACTIVE      org.wso2.carbon.device.mgt.common_1.2.2.SNAPSHOT

osgi> b 262

org.wso2.carbon.device.mgt.common_1.2.2.SNAPSHOT [262]

  Id=262, Status=ACTIVE      Data Root=/home/daag/work/product-emm/modules/distribution/target/wso2emm-2.2.0-SNAPSHOT/repository/components/default/configuration/org.eclipse.osgi/bundles/262/data

  "No registered services."

  No services in use.

  Exported packages

    org.wso2.carbon.device.mgt.common.configuration.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.permission.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.scope.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.app.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.authorization; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.operation.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.notification.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.license.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.spi; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.device.details; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.push.notification; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.group.mgt; version="1.2.2.SNAPSHOT"[exported]

    org.wso2.carbon.device.mgt.common.search; version="1.2.2.SNAPSHOT"[exported]

  Imported packages

    com.fasterxml.jackson.annotation; version="2.6.1.wso2v1"<jackson-annotations_2.6.1.wso2v1 [72]>

    javax.xml.bind.annotation; version="0.0.0"<org.eclipse.osgi_3.9.1.v20130814-1242 [0]>

    org.wso2.carbon.apimgt.api.model; version="6.0.5"<org.wso2.carbon.apimgt.api_6.0.5 [209]>

    io.swagger.annotations; version="[1.5.0,2.0.0)"<unwired><optional>

  No fragment bundles

  Named class space

    org.wso2.carbon.device.mgt.common; bundle-version="1.2.2.SNAPSHOT"[provided]

  No required bundles

With above logs, we can ensure the org.wso2.carbon.device.mgt.common bundle is in active state. Therefore we need to go deep in debugging to investigate the root cause.

2) Enabling OSGI level debug logs in EMM server.
We can use this method to track the dynamic class loading in OSGI.

See <EMM_HOME>/repositories/conf/etc/osgi-debug.options file. You will see there are many levels of OSGI debug options. for our scenario, we can enable the below debug options.

### OSGi resolver options
# Turns on debugging for the resolver
org.eclipse.osgi/resolver/debug = true
# Prints out wiring information after the resolver has completed the resolve process
org.eclipse.osgi/resolver/wiring = true
# Prints out Import-Package information
org.eclipse.osgi/resolver/imports = true

After enabling those debug logs, start the server with bellow command.

sh wso2server.sh -DosgiDebugOptions

You will start seeing large number of debug logs printed in the console. By tracking these logs we can get an idea of how things happen.

Ex: Following line shows that the dynamic class loading for org.wso2.carbon.device.mgt.common bundle fails over and over due to dependency issue.

Failed to resolve dynamic import: org.wso2.carbon.device.mgt.common

So how to track which dynamic import cause this issue?... For this, we will have to go to 2nd level debugging with Equinox OSGI source

3) Debugging the Equinox OSGI source 

Download the corresponding version of the OSGI source jar.
Ex: http://grepcode.com/snapshot/repository.grepcode.com/java/eclipse.org/4.3.1/org.eclipse/osgi/3.9.1

Load it to IntelliJ Idea.

• CTRL+N
• Type the class name "GroupingChecker"
• Once opened click on the "Choose sources..." on the top right corner of the file and load the downloaded source jar.

Now we can set the debug point. The place we need to debug is org.eclipse.osgi.internal.module.GroupingChecker#isConsistentInternal method.



Here it checks for any dependency conflicts. If there are any dependency conflicts, those will be returned to results list object. Else it returns null. Since this method gets hit very frequent for every bundle that gets loaded, we need to add a condition to this debug point.

• Right click on the debug point (red dot) and add the following condition.

matchingExport.getBaseDescription().getName().equals("org.wso2.carbon.device.mgt.common") && results != null

Now when you start the server with remote debug enabled. If there are any conflicts for org.wso2.carbon.device.mgt.common bundle, that information will be returned to results list object and the debug point will get triggered.

In my case, after multiple server restarts, I was able to see a version conflict with the com.fasterxml.jackson.annotation dynamic import bundle.

When I check this in the server OSGI console with below command, I saw that it has loaded 2 versions (2.7.4 and 2.6.1wso2v1) of the same bundle.

sh wso2server.sh -DosgiConsole

osgi> p com.fasterxml.jackson.annotation

The org.wso2.carbon.device.mgt.common bundle is configured to use the latest version of the bundle. When OSGI framework first loads the 2.6.1wso2v1 version, the common bundle fails to do the dynamic import. This result in not loading the classes and eventually triggers the CNF Exception.

Solution: 

Updating the versions back to one single version resolved the issue.

[1] https://wso2.org/jira/browse/EMM-1646

Hope you learned something new from this investigation. 

Happy debugging !!!

Git push changes in tag to different repository branch

• The following method will push changes in the tag to different repository branch along with all the commit history details.
• If you don't have a branch in the new repo to commit updates, go to the git web UI and create a branch. 



• Now in your old repository, let's assume the tag name is v3.6.0. Change the head to tag
• git checkout v3.6.0
• Now add the external git repo URL as a remote reference. Let's name it as support.
• git remote add support https://github.com/user/project.git
• Now you can push your tag updates to the branch by using below command.
• git push support +v3.6.0~0:dev-3.6.0

Apache CXF web service implementation with dynamic WSDL

According to the requirements, I need to generate web services from dynamically loaded WSDL files. This is not practical with wsdl2java since it generates classes mapped to WSDL, which result in piling up classes for different WSDL files loaded at runtime.

So after going through Apache CXF API and after experimenting with it for some time, I was able to come up with a POC on how to achieve my requirement.

The sample code is at my Github: https://github.com/amalhub/cxf-test

Happy coding!

How to reset Ubuntu Software Update source list

Are you getting the "Failed to fetch ... " error when running the update in Ubuntu.

Enter the following commands in a Terminal to clear the source lists and reset them -

sudo rm /var/lib/apt/lists/* -vf

sudo rm /var/lib/apt/lists/partials/* -vf ##Optional

sudo apt-get update




How to find and replace text in files recursively from terminal

Search command

find . -name "*.xml" -exec grep -r "4.4.7" {} +

Ex:
Search all xml files containing the string "4.4.7"

             find . -name "pom.xml" -exec grep -r "4.4.7" {} +

Replace command

find <mydir> -name <filetype> -exec sed -i 's/<oldString>/<newString>/g' {} +

-to include all files types
      use -type f instead -name or exclude the filter.

Ex:

Old string: 4.4.7-SNAPSHOT
New string: 4.4.7

            find . -name "pom.xml" -exec sed -i 's/4\.4\.7-SNAPSHOT/4.4.7/g' {} +

Note: When using sed, search string is sensitive to regular expression. Keep in mind to use the escape characters where necessary.

How to enable debug logs for BPMN in WSO2 BPS Server

Add the below configurations to <BPS_HOME>/repository/conf/log4j.properties

log4j.logger.org.activiti=DEBUG
log4j.logger.org.activiti.engine.impl.db=INFO
log4j.logger.org.activiti.engine.impl.persistence=INFO
log4j.logger.org.activiti.engine.impl.cfg.standalone=INFO
log4j.logger.org.activiti.engine.impl.interceptor.LogInterceptor=INFO
log4j.logger.org.activiti.engine.impl.history.DefaultHistoryManager=INFO
log4j.logger.org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable=INFO

Creating a custom LDAP structure with Apache Directory Studio

The Lightweight Directory Access Protocol (LDAP) is an application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network. More info about LDAP.

• Download Apache Directory Studio (ADS) Tutorial created with 2.0.0 version. 

• Run ADS.

• Lets assume that we need to create a LDAP structure looks like below example

• First you have to create a LDAP server

• In ADS window at bottom left corner, go to LDAP servers tab right click inside the panel > New > New server.

• Select the server version, enter a name and click finish to create the server.

• You will see the new server at the LDAP servers list.

• Now if you observe the structure that we are going to create, it starts with the root c=US.

• In-order to do this we have to create a partition with the suffix c=US inside the server we created.

• Right click on the server we created in LDAP server list (caption 2) and click Open Configuration

• In the opened server.xml file, go to the partition section by clicking the partition tab at the bottom of the page.

• Click add button and give a name to ID and set the suffixc=US.

• Save the settings (ctrl+s).

• Now go to LDAP server list (caption 2). Right click on the server and click Run.

• Again right click on the server and click Create a connection.

• Go to the Connections tab (next to the LDAP servers tab , caption 2)

• You will see a new connections has been created with the server name.

• Double click on the new created connection. It will open the LDAP structure in the LDAP browser.

• Now we can start creating the LDAP structure.

• First we'll create the country inside the root.

• Right click on the Root in LDAP Browser and click New > New Entry...

• In the new window select Create entry from scratch and click Next.

• Form the Available object classes select Country and by clicking the Add button add it to the Selected object classes. Click Next.
  • Note that it will add the dependent objects automatically. Do not remove them.

• Keep the parent empty since this will be the root and fill RDN values c and US. 

• Click Next and then click Finish.

• You will now see the c=US root element in the LDAP browser.

• Now let's create the organization unit as BPS.

• Right click on the o=WSO2 element in LDAP Browser and click New > New Context Entry...

• In the new window select Create entry from scratch and click Next.

• Form the Available object classes select OrganizationalUnit and by clicking the Add button add it to the Selected object classes. Click Next.
• Note that it will add the dependent objects automatically. Do not remove them.

• For the DN pattern enter ou=BPS,o=WSO2,c=US and click Next and Finish. 

• Now we can add the admin user to BPS.

• Right click on the ou=BPS element in LDAP Browser and click New > New Context Entry...

• In the new window select Create entry from scratch and click Next.

• Form the Available object classes addOrganizationalPerson, uidObject to the Selected object classes. Click Next.
• Note that it will add the dependent objects automatically. Do not remove them.

• For the DN pattern enteruid=admin,ou=BPS,o=WSO2,c=US and click Next.

• Set the same value admin to cn and sn by right clicking on those attributes and select Edit value. 

• Click Finish after setting all the values as shown in the above caption.

• Note that we are unable to create groups without having at least one user. Now since we have at least one user created in the LDAP structure we can start creating groups.

• Right click on the ou=BPS element in LDAP Browser and click New > New Context Entry...

• In the new window select Create entry from scratch and click Next.

• Form the Available object classes add groupOfNames to the Selected object classes. Click Next.
• Note that it will add the dependent objects automatically. Do not remove them.

• For the DN pattern entercn=allUsers,ou=BPS,o=WSO2,c=US and click Next. 

• There will be a new popup window asking to enter at-least one member to the group.

• Give the admin user DN patter (uid=admin,ou=BPS,o=WSO2,c=US) and click OK.

• Click Finish after setting all the values as shown in the above caption.

• Lets create another group named as group1.

• Right click on the ou=BPS element in LDAP Browser and click New > New Context Entry...

• In the new window select use existing entry as template and make sure it has mentioned the correct DN patter for the existing group allUsers. Click Next.

• Note that the objects has been already loaded to theSelected object classes since we are using the settings from allUser group. Click Next.

• For the DN pattern entercn=group1,ou=BPS,o=WSO2,c=US and click Next.

• Keep the rest of the settings as it is and click Finish.

• Now lets create some users in allUsers group.

• Right click on the uid=admin element in LDAP Browser and click New > New Context Entry...

• In the new window select use existing entry as template and make sure it has mentioned the correct DN patter for the existing user admin. Click Next.

• Note that the objects has been already loaded to theSelected object classes since we are using the settings from admin user. Click Next.

• For the DN pattern enteruid=user1,cn=allUsers,ou=BPS,o=WSO2,c=US and click Next.

• Set the same value user1 to cn and sn by right clicking on those attributes and select Edit value.

• Click Finish after setting all the values.

• Repeat the same steps to create more users. Create user2 and user3.

• Now lets add user2, user3 to group1.
• Click on the group1 from the LDAP browser. The settings will open in the right side window.

• Right click on the empty space and select New Attribute.

• In the new window set member for the attribute type and click Finish.

• In the new popup window enter the DN pattern for user1 as   uid=user2,cn=allUsers,ou=BPS,o=WSO2,c=US.

• Repeat the above 4 steps to add user3 to group1 as well.

• Congratulations !! You have created the LDAP structure.  

• After creating an LDAP structure, if you want to backup the data, you can always right click on the Root element in LDAP browser and select Export as a ldif file. Then next time you can Import them in the same way after creating the partition (c=US in this example) in the server with the suffix.