The dispatcher is a pretty straight-forward cache, which has some built-in rules for invalidation. The most important rule is: Whenever a page is invalidated, all pages, which are located deeper in the content hierarchy below this page, are invalidated as well.

Some background: This might be based on the assumption, that a change on any page can influence the appearance of pages higher in the hierarchy; you should think of a change of the page title, which must be reflected in the navigation items of all other pages in that area. Combined with the /statfileslevel parameter you should be able to deliver always a correctly rendered page and never outdated content. At the cost of the cache hit ratio in the dispatcher.

I already described the inner working of the dispatcher in an older article. Knowing these internals also give you some more options if you want to tune this. In the following I describe an approach, which allows you to implement custom invalidation rules. The caching and the delivery of the cached files doesn’t change, only the cache invalidation.

The basic idea is to replace the call to /dispatcher/invalidate.cache (which is the standard handler for cache invalidation) by a custom script, which runs the invalidation on the cache. This custom script is configured as target URL in the invalidation agent(s) and updates the “last modification date” of the statfile according to your requirement.

For example you can have this perl script, which should behave like the dispatcher invalidation logic:

/usr/bin/perl -w
use strict;
use CGI;
use File::Find;
use File::Touch;
my $cacheroot = "/opt/www/docroot";
my $q = CGI->new;
my $path = q->param("Handle");
my $invalidation_path = $cacheroot . $path;
File::Find::find({wanted => \&wanted}, $invalidation_path);
exit;

sub wanted {
  my $f = shift;
  if ($f =~ /\.stat/) {
    touch ($f);
  }
}

Configure this script as a CGI on “/dispatcher/custom.invalidation”, then configure your invalidation accordingly, and voila, you’re done. You run your custom invalidation script. If you don’t like the standard behaviour (obviously you don’t …), adjust the logic in the wanted procedure.

• assembling parameters
• setting environment variables
• start/stop/status of the server process (start and stop are also available a separate scripts for ease of use)

This would easily qualify this script as a initscript, but it lacks one important task: Changing the uid for the to-be-started process, it doesn’t run as root user. While from a purely functional perspective this isn’t important, every sysadmin will tell you, that you shouldn’t run processes as root, when they don’t require such permissions to perform their tasks. And the security guys won’t even let you golive with such a setting. So if you write an init script, you should add this functionality there.

And one last plea: Do not start the java process directly in the initscript and assemble the commandline yourself, but call the start&stop scripts. Especially if you are not that familiar with the serverctl script. So, a simple initscript  could look like this:

#!/bin/sh
CQ5_ROOT=/opt/cq5
CQ5_USER=cq5

########
SERVER=${CQ5_ROOT}/crx-quickstart/server
START=${SERVER}/start
STOP=${SERVER}/stop
STATUS="${SERVER}/serverctl status" 

case "$1" in
  start)
    su - ${CQ5_USER} ${START}
  stop)
    su - ${CQ5_USER} ${STOP}
  status)
    su - ${CQ5_USER} ${STATUS}
  *)
    echo "Unknown argument $1"
esac

Another intesting site I recently fell about is CQ Blueprints, which has the claim to collect best practices for CQ 5.2 till CQ 5.4; I browsed through and found some interesting ideas, but also some tips, which are not “best practices” in my opinion. Haven’t found the time to write constructive comments there.

A short outlook: I plan to write short posts about “init scripts for CQ5″ and “dispatcher invalidation”. The latter topic is rather intesting, as I recently adviced several people to bypass the regular invalidation, while in the last years there was never a need to do so. So, stay tuned.

So, the clear answer is: No, it isn’t supported. Mostly because of some issues which can cause corruptions in the Lucene index. So, of course, you give it a try and tackle the risk yourself (as you can run CQ5 on a Windows 7 box or on Debian Linux); but don’t complain, if you receive some strange behaviour.

ps: Just adding -XX:-UseLoopPredicate to your JVM parameters won’t solve the problem (according to the Lucene Website).

If you decide to instrument your code and provide some information via JMX (that’s something I would recommend to everyone, who adds non-trivial services to CQ5), have a look at Apache Aries, especially at the JMX whiteboard. Deploy this bundle to your CQ5 and then just register your mbeans as services. Voila, that’s it. You don’t need to register and unregister your mbeans, as this is handled by the JMX whiteboard.

Sadly documentation is currently rather poor, but the sourcode isn’t that hard to understand. You can start with the initial patch in the Aries issue tracking.

Further investigation using threaddumps and profiler showed, that there was indeed a I/O problem. Because they showed a lot of blocked threads within the JVM, which wanted to do I/O. So how goes this together?

Basically it can be easily explained. Your operation system is optimized to handle multiple parallel threads/processes, which want to do I/O. If there are too much of them or the I/O subsystem is too slow, the I/O wait ratio will increase. But in the CRX case it’s a bit different. Because of its internal structures CRX requires locking to synchronize its write actions to disk (reads are done in parallel). So for many operations (like updating metadata, writing journal log etc) only 1 thread can actually write to the filesystem, all others are waiting for this thread to finish its write action. For the operating system this looks like a single write action, which can be handled quite easily without the I/O wait skyrocketing.

So if you suspect that you have an I/O problem, use a profiler or threaddumps (or /crx/diagnostic/prof.jsp), the data which are displayed by top don’t tell the whole truth.

And next to these improvements a small extension to the online backup was introduced. Up to and including CRX 2.0 you backuped to a zip file (which contains the complete quickstart and the repository). In CRX 2.1 it is no longer required to zip the backuped data, but the backup procedure can simply drop it into a directory. This directory still contains all data (quickstart and the repository), so you run the zip programm yourself if you want. So, what’s the great deal here?

First of all, your backup software can do an incremental backup on this directory. Especially the tar files and the index will change quite often (think of the TarPM optimizer), but the datastore files won’t change, so they are ideal candidates for incremental backup.

Speaking of the datastore and its immutable files: Because files are only added to the datastore (and not changed), you can optimize the backup. The online backup backups the data based on its knowledge of the directory, where the quickstart is placed in. Everything in that directory and beneath is covered by the backup process. If you move the datastore to a directory outside the directory, where the quickstart.jar resides, the datastore isn’t backupped. Why is that good? Because if we always our backup in the same directory, we can use rsync to backup the datastore. Rsync is much better suited for this job, because it works incremental (CRX online backup doesn’t) and works outside of the CRX/CQ5 java process (which is always good). Remember: Files in the datastore are only added, not changed!

I recommend to change your CQ5 filesystem layout as follows:

cq5
 - backup
   - datastore
   - cq5
     - cq-wcm-quickstart.jar
     - crx-quickstart
     - ...
     - license.properties
 - datastore
 - cq5
   - cq-wcm-quickstart.jar
   - crx-quickstart
   - ...
   - license.properties

This layout ensures, that you would be able to start the CQ5 directly from the backup directory (altough you shouldn’t do it) or simply move it to the “production” location, without any configuration changes.

Moving the datastore is easy: Shutdown CQ5/CRX  and replace the line
<DataStore class="com.day.crx.core.data.ClusterDataStore"/>

in the repository.xml with the following ones:

<DataStore class="com.day.crx.core.data.ClusterDataStore">
<param name="path" value="../../../../backup/datastore" />
</DataStore>

I recommend to use a relative adress here; because it’s relative to the directory crx-quickstart/repository/repository, you should enter
"../../../../datastore” here. Adding a symlink won’t help!

Then create this directory structure and move all the datastore files there (crx-quickstart/repository/repository/datastore/*) and make sure, that the ACLs are set properly. Startup and have fun!

And now the rough skeleton of a backup script to illustrate the process; it requires curl, rsync and a little perl. For the windows-based systems I cannot give much advice; probably there is also some tool like rsync.


#!/bin/sh

HOST=localhost:4502
ADMINPW=admin # must be URL encoded!!

BACKUP_FILENAME="" # we want CRX to backup to the directory, no zip file!
BACKUP_DIR=/opt/cq5/backup
INSTANCE_DIR=/opt/cq5

COOKIE=cookie.txt
CURLPARAMETERS="-s -S"

# hacky, using a appropriate perl module would be better
ENCODED_BACKUP_DIR=`echo "${BACKUP_DIR} | perl -pe 's|\/|%2F|g'`
touch ${COOKIE}
chmod 600 ${COOKIE}
echo "Start: `date`"
curl -c ${COOKIE} ${CURLPARAMETERS} "http://${HOST}/crx/login.jsp?UserId=admin&Password=$ADMINPW&Workspace=crx.default" > /dev/null
curl -b ${COOKIE} ${CURLPARAMETERS} -o progress.txt "http://${HOST}/crx/config/backup.jsp?action=add&targetDir=${ENCODED_BACKUP_DIR}&zipFileName=${BACKUP_FILENAME}" > /dev/null
rm progress.txt
rm ${COOKIE}

echo "Syncing datastore"
DATASTORE_DIR="${INSTANCE_DIR}/datastore/"
DATASTORE_BACKUP_DIR="${BACKUP_DIR}/datastore"
rsync -a ${DATASTORE_DIR} ${DATASTORE_BACKUP_DIR}
echo "Finished: `date`"

Update:Fixed path in the repository.xml snippet. Thanks Thomas.

• CRX 2.2 supports the shared-nothing clustering.
• CRXDE light is no longer a dedicated web application, but is included into the CRX webapp.
• The CQSE serverctl script has been cleaned and simplified; so the description in “Bootstrapping the Java process” does no longer apply. Most of all the “serverctl psmon” process is gone now.
• Since CRX 2.1 it is possible to perform a backup to a directory, not only to a ZIP file. This offers some more space for optimization. Expect a follow-up article on this.
• The CRX Quickstart requires you to have a “ulimit -n” (number of open files) of at least 8192. It will probably affect your unpack run, but neither CQSE nor CQ5 itself does enforce it then …
• It is still possible to build CQ5 custom images
• CQ 5.4 offers an enhanced replication: There are these new options to a replication agent (on the “Trigger” tab): “no status update” and  “no versioning”. Let’s try to build a shadow copy of an author instance!

That’s for the moment. Have fun with CQ 5.4!

In CQ5 it is incredibly easy to install ServletFilters (thanks OSGI and Declarative Services); so I wrote a small servlet filter, which blocks requests originating from users, which are not whitelisted. That’s a nice solution, which does not require any intrusive operation such as changing ACLs or such. You just need to deploy a tiny little bundle, put “admin” on the whitelist and enable the maintenance in the Felix webconsole. That’s it.

I will submit this package (source code plus compiled bundle) to the Day package share, licensed under Apache 2.0 License. It may take a bit, but I will place it to the public area, so you can grab it and study the source (it’s essentially only the servlet class).

One way is to provide a detailled list of instructions plus the required files to the people responsible for it. It’s important to be consistent over all affected installations and environments, so you can remedy problems and issues because of missing fixes or wrong installation. But then a lot of manual work is included, which isn’t the thing IT people want to do.

I needed to provide several CQ5 installations in the last time. Because my standard installation recommendation consists of CQ 5.3 plus CRX 2.1 plus performancepack 30015 (using CQSE and the TarPM) at the moment, just deploying a CQ 5.3 the usual way isn’t sufficient. But on the other hand I don’t want to have the work of a manual installation of CRX 2.1 and the performancepack on top of a default CQ 5.3, both including restarts.

So I decided to build an image, which contains all these components, without the need for an restart, without fiddling around with the package manager, just by using some hidden features of CQ5.3 and CRX: the package installer and the flawless upgrade procedure of CRX 2.x (x={0,1}, will probably work also for later versions of CRX). You can find the documentation of the upgrade process also on the official documentation site.

1.) Unpack a plain CQ 5.3:

$ cd cq530
$ java -jar cq-wcm-quickstart-author-5.3.0.jar -unpack

2.) Get CRX 2.1 and unpack it:

$ cd crx21
$ java -jar crx-2.1.0.20100426-enterprise.jar -unpack

3.) Copy the CRX webapplication file of CRX 2.1 into the unpacked CQ 5.3 installation:

$ cp crx21/crx-quickstart/server/webapps/crx-explorer_crx.war cq530/crx-quickstart/server/webapps

4.) Remove the CRXDE webapplication of CQ 5.3, as it is no longer needed for CRX 2.1

$ cd cq530/crx-quickstart/server/webapps
$ rm crx-de_crxde.war

5.) Edit also the server.xml, and remove the crxde webapp

6.) Define an order, in which the packages are deployed to CRX; as the packages are deployed in the order, they are listed by default in a shell, I define an order by explictly naming the files like “01_cq-content-5.3.jar”, “10_cq-documentation-5.3.zip” and so. The files must be placed in the cq530/crx-quickstart/repository/install folder.

Make sure that the original “cq_content-5.3.jar” is deployed as first package, as it contains the WCM code. But then you can place there any CQ package you want: hotfixes, custom application code, initial content etc.

$ cd cq530/crx-quickstart/repository/install
$ cp cq-content-5.3.jar 01_cq-content-5.3.jar
$ cp cq-documentation-5.3.zip 10_cq-documentation-5.3.zip
$ cp .........../cq-5.3.0-featurepack-30015-1.0.zip 50_cq-5.3.0-featurepack-30015-1.0.zip

7.) If you want to use the CRXDE, you should download the file cq53-update-crxdesupport-2.1.0.zip from Day PackageShare and copy it also into the install directory:

$ cp ......../cq53-update-crxdesupport-2.1.0.zip 05_cq530/crx/repository/install

8.) For convenience you can place now your license.properties file next in the toplevel directory of your installation, the result should be something like this:

$ ls -la
-rw-r--r--   1 jorghoh  staff  233110810  6 Aug 09:38 cq-wcm-quickstart-author-5.3.0.jar
drwxr-xr-x  10 jorghoh  staff        340  7 Okt 17:55 crx-quickstart
-rw-r--r--   1 jorghoh  staff        217  6 Aug 09:39 license.properties

If you don’t want to deliver the license file with that image, you can omitt it; if the instance is started the first time, it is asked then.

9.) Now all parts are in place; so you can create an image file (tar file) and distribute it all over your environments:

$ cd cq530/..
$ tar -cf cq530-crx21-author-image.tar cq530/*

(if you rename the cq-wcm-quickstart-author-5.3.0.jar file to cq-wcm-quickstart-publish-5.3.0.jar, you have an image for a publish instance.)

Just unpack it and use your usual startup mechanisms (“start.bat” or “start”), and the framework will startup as usually, create a repository and also deploy all packages in the install folder directly to it. If you encounter problems, you may check with the Felix console, if all bundles are started.

Now you have an image, which you can copy and uncompress everywhere, even the plattform (Unix, Windows) doesn’t matter, as all is only dependent on the features of CRX Launchpad and CRX itself. A setup of a new emtpy instance, independent of the number of installed packages and hotfixes, can now be done within 2 minutes and can be fully automated.