Monitor and Manage Alfresco via JMX

WHAT’S JMX?

The Java Management Extensions (JMX) technology is a standard part of the Java Platform. It has been available from version 5 of the Java Standard Edition. The JMX specification (JSR 3 - http://www.jcp.org/en/jsr/detail?id=3) defines the architecture, design patterns, services, and APIs for managing and monitoring applications written in the Java language.
When we use the JMX technology an application can be monitored and managed by providing so called Management Beans or MBeans. The application registers these management beans with an MBean server. The MBean server acts as an agent and can be run on most devices that have been enabled for the Java language. The agent also contains a set of services for handling the MBeans. The JVM itself publishes a lot of information about its internal management of memory, threads, class-loading, garbage collection etc. in the form of MBeans.
The following figure shows an overview of the JMX technology

The above figure also shows that it is possible to connect remotely to the JMX Agent via connectors such as the RMI connector. Both the JConsole and the VisualVM products can connect locally and remotely to the JMX Agent.
JMX Connectors do not have a standard API and this is a problem for third party tool vendors as they have no standard API to code against. This has been fixed with the JMX Remoting specification (JSR 160), which extends the JSR 3 specification by providing a standard API to connect to remote JMX-enabled applications. The JMX Remoting specification also defines a mandatory connector based on RMI.
If we now look more specifically at the Tomcat/Alfresco platform we will see the following JMX configuration:

Remote connections are supported via the RMI adapter/connector but Tomcat also provides another way of talking remotely to the MBean server. It supports a JMX proxy via a Servlet available in the Tomcat Manager web application. This JMX Proxy can be very handy in production environments as we will see later on.
The figure also shows that there are a number of MBeans available in different namespaces. All the Alfresco MBeans are contained in the Alfresco namespace and the Apache Tomcat MBeans are contained in the Catalina namespace. So now when we know what JMX is it’s time to connect to an Alfresco server and have a look

MAKING A LOCAL CONNECTION TO AN MBEAN SERVER

During development and testing it is alright to connect to the Alfresco MBean server locally even if the JMX client is taking up a lot of resources. In a production environment you would not want to do this (or most likely we will not be allowed to do it) as it will not show you real statistics.
We will be using JConsole that comes with the Java Development Kit (jdk_installation_dir/bin/jconsole.exe) to connect to a local Alfresco-Tomcat instance. When the Alfresco server is running on the same machine as JConsole it picks up the Alfresco Tomcat process for us

Select the process with the name org.apache.catalina.startup.Bootstrap.start to start monitoring and managing Alfresco.
If you don’t see the Alfresco Tomcat process listed for the Local Process option as in the above figure, then that maybe because you are running Alfresco as a service. The Local Process option only works if jconsole is running with the same userid as the process you want to connect to, which is not the case if Tomcat is setup to run as a service. Then use the Remote Process option instead as follows:

Here we use a JSR-160 standard connection URL (service:jmx:rmi:///jndi/rmi://localhost:50500/alfresco/jmxrmi) and a username (controlRole)and password (change_asap).
The JMX username and password are the default once that we can find in the alfresco-jmxrmi.access and alfresco-jmxrmi.password files located in the tomcat/webapps/alfresco/WEB-INF/classes/alfresco directory.

MONITORING MEMORY AND CPU USAGE OF AN ALFRESCO INSTALLATION

The first screen we will see after we have logged in with JConcole contains JVM information about memory usage, number of active threads, number of loaded classes, and CPU utilization

I am running Alfresco 3.4c community 64bit on Windows 7 box. We can see in the above Heap Memory Usage graph that there has been a garbage collection going on. Now let’s see what happens if we upload a 3Mb Power Point file to Alfresco, we will then see the following changes in these graphs:

As we can see there are big spikes in memory and CPU usage when we login and then upload a document. This a good way of testing the Alfresco implementation after we have done lots of customizations with for example rules that process content and add extra metadata.
What you always want to happen after a while is that garbage collection (GC) kicks in as follows

If we want to have a more detailed look at the memory consumption we can click on the Memory tab in JConsole and the following screen is displayed

This screen will verify that there have been 4 garbage collections done (i.e. PS MarkSweep – 4 collections) that took 2.234 seconds. This is good to know as we can then see if the GC has an impact on availability on the system. If the time the system spends on garbage collection is reaching minutes and not seconds you will have users starting to complain that the system is unresponsive.
It will also give us the possibility to inspect how much heap is taken up by PermGen (i.e. Permanent Generation Heap). The Java Heap space is managed in different generations of objects depending on how old they are. And there are different GC algorithms for the different generations. The permanent generation is different from the other generations as it holds properties (i.e. metadata) with information about Alfresco classes and methods (actually all classes that are not part of the Java language).
Because this generation of objects are permanent they will always be in the heap until we shutdown Alfresco. If PermGen is not big enough then you will get the following kind of error in the logs:

OutOfMemoryException: PermGen space

To fix this problem set a higher value for the VM parameter -XX:MaxPermSize. To see the current PermGen allocated space select Memory Pool “PS Perm Gen” from the Chart selection box:

This shows us that the local Alfresco installation is consuming around 187MB of PermGen and that is just okay as the max PermGen size that we have started the server with is -XX:MaxPermSize=512m, which is the default in Alfresco 3.4c Community Edition.
If you are running Alfresco as Windows Service then you can find out the MaxPermSize setting by running the following command:

c:\alfresco3.4c\tomcat\bin>tomcat6w.exe //ES//alfrescoTomcat

The alfrescoTomcat name is the service name. This will display the properties for the service, and if you click on the Java tab you will see the following

On the other hand, if we see the following errors in the log file: java.lang.OutOfMemoryError: Java heap space Then this means that there is not enough space in the heap for temporary objects.
You will then have to increase the settings for the max available heap space, which you do with the VM parameter -Xmx. If we want to see exactly how the Tomcat instance was started we can click on the VM Summary tab in JConsole. It will then show us something like the following:

We can see that a 64bit Java VM version 1.6 was used to start the Alfresco server. Looking at the VM arguments line we can see the initial heap memory size setting (-Xms), max heap size (-Xmx), and max perm gen size (-XX:MaxPermSize). In this case the max heap is 768MB, the initial heap is 128MB, and the max permanent generation heap is 512MB.

MANAGING AN ALFRESCO INSTALLATION VIA MBEANS

So far we have only been looking at the Java memory utilization, CPU utilization etc. What if we wanted to look at specific Alfresco Management beans, how would we do that? If we click on the MBeans tab in JConsole we will see the following screen

We can see that it is not only Alfresco that is providing management beans. Tomcat, Apache Log4J, Apache Axis etc also provides management beans that we can inspect. Let’s look at the MBeans available from Alfresco by expanding that folder in the tree (the folder tree is organized so the top level folders correspond to the different MBean namespaces):

Okay it looks like there are loads of management beans to get to grips with. A couple of examples will make things clearer.
Note. If you only see one MBean under the Alfresco namespace then you might have a version of Alfresco where there is currently a bug so you will only see the VirtServerRegistry entry. This was the case with version 3.4c, I had to switch to 3.3.4 for it to work.
First let’s see how we can turn on JavaScript debug logging, we will assume we have a new JavaScript that we want to see the logging from, and we cannot shutdown the server to turn on debug logging.
To do this click on the Log4jHierarchy MBean, which is represented by a tree item with the same name

This basically shows us that the MBean has a configurable attribute for the JavaScript logger. Now we just need to find out where we can change the log level for it as it cannot be done here. To do this we need to go back up in the tree and select the log4j namespace top level tree item and then select the org.alfresco.repo.jscript MBean and set the priority attribute to DEBUG

We will now start to get logs like:

10:55:25,735 User:martin DEBUG [repo.jscript.RhinoScriptProcessor] Imports resolved, adding resource '_root 10:55:26,687 User:martin DEBUG [repo.jscript.RhinoScriptProcessor] Time to execute script: 817.77ms

If we now shutdown the server and then start it again the DEBUG log setting is gone. So these settings that we do via the JMX beans are not always permanently stored. If you wanted to do some debugging on JavaScripts and you needed to start and stop the server a lot then this is not the way to go. Then you should just update the log4j.properties file and set the logger to DEBUG there. Use JMX when the Alfresco server is in a live environment and stopping it to debug something is not an option
Ok, let’s look at another example where we want to disable the guest login without having to stop the server. To do this look up the authentication subsystem for which you want to disable guest login. I am using a default installation so my authentication subsystem name is going to be alfrescoNtlm1, which authenticates users with the local MySQL database All authentication subsystem configurations can be found under the /Alfresco/Configuration/Authentication/managed item as follows

Change the alfresco.authentication.allowGuestLogin attribute value to false as in above figure. Now if we try and just hit the http://localhost:8080/alfresco URL it will always display the login dialog.
This setting will be permanent and after a restart of the Alfresco server we will still not be able to see a Guest Home page.

MAKING A REMOTE CONNECTION TO AN MBEAN SERVER

When the system has gone into production it is not advisable to run JConsole on the same machine as Alfresco is running as it takes a lot of resources. So we need to setup the Sun MBean Server to accept connections over RMI. Remoting is supported by JSR 160 that extends JSR 3 by providing a standard API to connect to remote JMX-enabled applications When we set this up we need to specify the following VM properties:

• com.sun.management.jmxremote.port - RMI Port number
• com.sun.management.jmxremote.ssl - If we want secure connection or not
• com.sun.management.jmxremote.authenticate - If authentication is required or not

The other required property com.sun.management.jmxremote is already specified by default after installing Alfresco, so we can access the MBean server locally. Setting this property enables the Sun MBean Server for local access, even though it says jmxremote…
Set these properties via the Windows Service properties dialog (alfresco/tomcat/bin>tomcat6w.exe //ES//alfrescoTomcat)

If you are starting Alfresco via script files then enable remote access by updating the alfresco start script (see alfresco.bat or alfresco.sh) so it contains the following Java Options (note. the memory settings can be different) this:

JAVA_OPTS=-Xms128m -Xmx512m -Xss96k -XX:MaxPermSize=160m -server -Dalfresco.home=%ALF_HOME%  
-Dcom.sun.management.jmxremote  
-Dcom.sun.management.jmxremote.port=6060 
-Dcom.sun.management.jmxremote.ssl=false  
-Dcom.sun.management.jmxremote.authenticate=false

Choose another port than 6060 if it is already bound to some other service. Restart Alfresco. Now use JConsole to connect to the VM that runs Alfresco from a remote machine with the following connection string:

service:jmx:rmi:///jndi/rmi://<hostname>:6060/jmxrmi

Note. Do not use the …alfresco/jmxrmi string at the end as that will not work. The Tomcat MBean server does not register its URL like that in this case when we connect remotely

If you are experiencing connection problems then you might be trying to access an Alfresco system running behind a firewall. Let’s say we have the following setup

And we have told the System Administrators to open up port 6060 in the firewall but things are still not working. In this case it will not work no matter what we try because the JMX RMI Connector opens up 2 ports where one is for the registry, which is the one we specify with the com.sun.management.jmxremote.port property. The other one is used to export JMX RMI connection objects for data.
This second port is unfortunately allocated dynamically at runtime and set to a random port number. We do not need to know this second port number to connect to the Mbean server but it causes all kinds of troubles as we got a firewall between JConsole and the MBean server, and the Firewall does not have this randomly allocated port open.
So what to do? Either run JConsole inside the DMZ or use another solution such as a system monitoring tool with local agents or a tunneling solution. See also next couple of sections.

Tomcat JMX Proxy

We already talked about the Tomcat Manager web application and that it contains a JMX Proxy Servlet that can be used to access the MBean server remotely via HTTP. This is quite handy in situations when we cannot open enough ports in the company firewall to be able to connect via RMI.
First verify that you can access the manager application via the http://<hostname>:8080/manager/html URL

If you get a login dialog and you cannot login then the tomcat-users.xml file located in the alfresco/tomcat/conf directory need to be updated with the following username and password for the manager role:

<role rolename="manager"/> <user username="admin" password="admin" roles="manager"/>

Now we can start to access the tomcat instance, to list all applications execute the following request:

http://<hostname>:8080/manager/list

To list server information:

http://<hostname>:8080/manager/serverinfo

To see default session information send a query with the path to the context whose session information you are interested in:

http://<hostname>:8080/manager/sessions?path=/

To start, stop, and undeploy a web application:

http://<hostname>:8080/manager/start?path=/alfresco 
http://<hostname>:8080/manager/stop?path=/alfresco 
http://<hostname>:8080/manager/undeploy?path=/alfresco

Okay, so now let’s access some MBeans via the JMX Proxy. We will do that with another URL http://<hostname>:8080/manager/jmxproxy. By default this URL returns a list of all MBeans.
To query the Log4J priority attribute value for JavaScript logging execute the following request:

http://<hostname>:8080/manager/jmxproxy/?qry=log4j:logger=org.alfresco.repo.jscript&att=priority

To query the guest page setting we can do the following request:

http://<hostname>:8080/manager/jmxproxy/?qry=Alfresco:Type=Configuration,Category=Authentication,id1=managed,id2=alfrescoNtlm1&att=alfresco.authentication.allowGuestLogin

This is a quite nice way of accessing information in the Alfresco installation from wherever we are. We can also set attributes this way with a URL like follows:

http://<hostname>:8080/manager/jmxproxy/?set=<mbean object name>&att=<attribute name>&val=<attribute value>

For example, to enable guest login do

JMX MX4J Http Adaptor There is also the possibility to activate the JMX MX4J Http Adaptor in Tomcat so you can connect via HTTP remotely.
Add the following configuration in tomcat/conf/server.xml:

<Connector port="6061"
              handler.list="mx"
              mx.enabled="true"
              mx.httpHost="mbergljung-PC"
     mx.httpPort="6062"
     protocol="AJP/1.3" />

Then add the log4j-1.2.14.jar and mx4j-tools.jar libraries to the alfresco/tomcat/lib directory.
This will start the JMX Http Adaptor on port 6062. Specify also AJP port number (i.e. Apache JServ Protocol port number so web server can talk to Tomcat Servlet container). When the server has started successfully the log will look like:

… 16-Mar-2010 14:57:58 org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8080 
16-Mar-2010 14:57:58 org.apache.jk.common.JkMX classExists INFO: className [mx4j.adaptor.http.HttpAdaptor] does not exist 
16-Mar-2010 14:57:58 org.apache.jk.common.JkMX loadAdapter INFO: Started MX4J console on host mbergljung-PC at port 6062 log4j:WARN No appenders could be found for logger (org.apache.log4j.jmx.HierarchyDynamicMBean). log4j:WARN Please initialize the log4j system properly. 
16-Mar-2010 14:57:58 org.apache.jk.common.JkMX init INFO: Registering the JMX hierarchy for Log4J 
16-Mar-2010 14:57:58 org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:6061 
16-Mar-2010 14:57:58 org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/117  config=null 
16-Mar-2010 14:57:58 org.apache.catalina.startup.Catalina start INFO: Server startup in 101507 ms

If we would like authentication to be enabled this can be done with the mx connector parameters

mx.authMode="basic", mx.authUser="tomcat", and mx.authPassword="secret".

To connect to the JMX server remotely via this adaptor we open a browser and access the following URL:

http://mbergljung-PC:6062/

The browser should display a page looking something like this

And here we will have access to MBeans in the Alfresco, Catalina, and Java domains. The interface is not as nice as JConsole but it gives us the possibility to connect thru a firewall and have a look at the configuration and status of different components. There are certain attributes that you can set such as Booleans. You can also invoke operations such as for example a garbage collection (GC) via this interface

More Sharing ServicesShareShare on facebookShare on twitterShare on email