Community Map Builder Software Developers' Guide

Abstract

A generic software developers' guide which draws upon the best open source convensions, processes and tools.

Note, this guide is still in alpha state.

This guide was built with generguide version $Name$.


Table of Contents

1. Introduction
1.1. Identification
1.2. System overview
2. Language
2.1. Javascript
3. Configuration Management
3.1. CVS
3.2. Release Components
3.3. Version numbers
3.4. Tagging releases
3.5. Release Process
4. Team Organisation
4.1. Users
4.2. Contributors
4.3. Committers
4.4. Project Management Committee (PMC)
5. Communication
5.1. Email lists
6. Bug Tracking
7. Build
7.1. Javascript Compression
8. Design
8.1. Design Document
8.2. Inline Documentation
8.2.1. Jsdoc
8.3. Unified Modeling Language (UML)
8.3.1. Modelling Web Applications
8.3.2. Poseidon UML
9. Coding Conventions
9.1. Directory Names
9.2. Class Naming Conventions
10. Refactoring
11. Code Profiling
12. Test
13. Documentation
13.1. Quick Start
13.2. Why Docbook?
13.3. Modular documentation
13.4. Publishing docbook files
13.4.1. Publishing using xsltproc
13.4.2. Publishing using ant/maven
13.4.3. Publishing using maven
13.4.4. Publishing using java tools.
13.4.4.1. Joining document parts using XSL template defined in selectivexinclude.xsl.
13.5. Docbook Editors
13.6. Docbook References

1. Introduction

1.1. Identification

Cameron Shorter

This document provides comprehensive software development processes tailored for open source projects. It covers processes, conventions, and recommended tools. The guide aims to help developers quickly get up to speed with best practises.

You are not expected to read this guide cover-to-cover. You probably should be familiar with the contents and reference the appropriate section when you need it. Documentation aims to be concise with references elsewhere on the web for details like installation instructions.

This guide is written in a modular fashion so that different projects can easily add, delete, or modify sections. It is hoped that this guide will become the de-facto standard software developers guide for java based open source projects.

For this guide to be useful it needs to be continually added to and improved as tools are developed, processes improved and projects grow. Please consider improving or adding a section if you feel it is required.

Ideally, all recommended tools would be open source, however we have included some no-cost tools where there are gaps in the open source tool set. Where applicable, widely accepted conventions and open standards are used. It has been satisfying to discover the breadth of quality open source tools which support software development. It is hoped that this document will highlight areas where tools can be improved or developed and encourage developers to focus on these areas.

1.2. System overview

This section should be replaced and explain your project.

2. Language

Cameron Shorter

This section describes the language(s) and compilers used by this project.

2.1. Javascript

This project contains Javascript code which requires XML functions. In order to run these functions you will require generation 6 browsers. Ie, Internet Explorer 6 or Netscape 6 or greater.

3. Configuration Management

Cameron Shorter

Software configuration management involves recording and retreiving all versions of software in a system.

3.1. CVS

Concurrent Versions System (CVS) is a version control system that allows multiple developers to work on the same files at the same time.

Information on how to download and edit this project's source code are provided at: http://sourceforge.net/cvs/?group_id=35246.

Note, before you can edit files, you will need to be granted write access from the Community Map Builder Project Management Committee.

An excellent CVS manual can be found at http://cvsbook.red-bean.com/.

3.2. Release Components

This section describes the terms used when building a release. This section is explained in more detail at https://sourceforge.net/docman/display_doc.php?docid=6445&group_id=1#componentoverview.

Project

A project represents a web site, and group of people working on the same functionality. For instance, http://generguide.sourceforge.net is the web site for the Generguide project.

Package

A package is a bundle of software being developed within a project. A project may develop multiple packages. For instance, the Generguide project may develop the packages: generguide-lib and developersguide.

Release

A release is a snapshot in the development of a package. It is represented by the package name and release number. For instance, generguide-0.1, or generguide-0.2.

File

Each release will probably contain multiple files. A file is expected to have multiple versions, however only one version of a file can be in a release.

3.3. Version numbers

Release version numbers are based on 3 main digits and an optional release candidate digit : <major>.<minor>.<patch>-rc<release candidate>. It looks like 2.3.4 or 2.3.4-rc5 .

major version

Major version number should be incremented to indicate that project has lost full compatibility with earlier versions. So you can safely upgrade to later versions of a module so long as the major version has not changed.

minor version

The second digit (minor) is incremented whenever new features are added. The project is forward compatible across minor versions, but usually not backward compatible.

patch version

The third digit is for patches (bug fixes). It is used to indicate fixes in bugs only. No new features were made and full compatibility is preserved.

This digit is optional.

release candidate

A project may optionally produce release candidate releases for beta testing before the main release.

For projects that use maven, the version number is stored in the <currentVersion> tag in maven's project.xml file.

To do: We need to discuss the meaning of:

  • alpha

  • beta

3.4. Tagging releases

Every time the version number is incremented, you need to tag the relevant files in the repository. This ensures that previous releases can be recreated for future debugging.

Tag names are based on the module version and look like <package_name>-<major>_<minor>_<patch>-rc<release candidate>. Eg: generguide-lib-2_3_4-rc5 .

Note that full stops from the version number are replaced with underscores. This is because CVS doesn't allow full stops in tags.

Typically, you would create a release tag with the following statements in CVS:

cd mapbuilder
cvs tag generguide-lib-2_3_4

3.5. Release Process

This section describes the steps to follow when building a release.

Notify developers that you are about to make a release

At least 24 hours before building a release, email the developers to notify them you are about to build a release. This should allow them to ensure their fixes are in the code base.

Check all code back into CVS

If you have not done so already, check all the code you have been editing into CVS.

cvs commit -m"Fixed xxx bugs" <filenames>
Get latest updates from CVS

Ensure you have the latest files from CVS.

cd generguide/
cvs update -d .
Update README file

Ensure the README file is still relevant. This file should be in the base directory.

Update the CHANGES file

Update and commit CHANGES file with updates between the last release and this one. This file should be in the base directory.

The CHANGES file should contain a heading for each release, most recent release at the top. Under each heading should be a description of the changes since the last release.

All notable outstanding issues should be included. (This might be a list of the high priority bugs of the list is too long). You should be able to copy the bug ID and title from the Bug Tracker.

All notable issues fixed since last release should be included. Again, copy the bug ID and title from the Bug Tracker.

<package> Release Information
=============================
$Id:$
This file contains information about updates to each release:

<package>_0.2 (beta quality)
============================
* Added extra gismos.
* Fixed bugs related to the crash of last release.

Outstanding Issues:
-------------------
962714 Slow load with no user feedback

Issues addressed since last release:
------------------------------------
962718 Crash of system using Mozilla

<package>_0.1 (alpha quality)
=============================
* Initial release
Tag the release

Tag the files that should be in the release. (Usually, not all files in CVS are included in a release).

cd generguide/
cvs tag <tag> dir1 dir2 dir/file1 ...
Create a temporary directory
mkdir ~/tmp
Export the release

Exporting code from CVS is the same as checking it out, except that the CVS/ directories are not checked out as well.

cd ~/tmp
export CVS_RSH=ssh
export CVSROOT=:ext:camerons@cvs.sourceforge.net:/cvsroot/generguide
cvs export -r <tag> generguide
Build/Import Non-CVS files

Many projects will include files in their release which are not contained in CVS. For instance:

  • Libraries imported from another project.

  • Automatically generated documentation (like javadocs).

  • Documentation that needs to be converted from Docbook to HTML.

These files need to be built and copied into the release directory.

Add tag name to generguide directory
cd ~/tmp
mv generguide/ generguide-0.2
Zip up release
zip -r generguide-0_2.zip generguide-0.2
tar -zcvf generguide-0_2.tar.gz generguide-0.2
Ftp release file Sourceforge

Refer to the Sourceforge File Release System for more information.

  • ftp to upload.sourceforge.net

  • login as "anonymous"

  • use email address as password

  • set ftp client to binary mode

  • cd /incoming (on the server)

  • upload release file(s).

Copy to generguide release directory

From the http://sourceforge.net/projects/generguide, select Admin, File Releases, Add Release.

Walk through the various forms.

  • For the release name, use the tag name: eg generguide-0.2

  • For release notes and change log, extract this information from the comments added to the CHANGES file above.

Download and test

Download the release from sourceforge, uncompress it, and check it. (This can be covered by the following step).

Email release message

Send an email notification to the project's announce email list.

Update release information on Freshmeat

4. Team Organisation

The Jakarta Project

Cameron Shorter

James Macgill

The roles and responsibilities that people can assume in this project are based on merit. Everybody can help no matter what their role. Those who have been long term or valuable contributors to the project obtain the right to vote and commit directly to the source repository.

4.1. Users

Users are the people who use the products of the Project. People in this role aren't contributing code, but they are using the products, reporting bugs, making feature requests, and such. This is by far the most important category of people as, without users, there is no reason for the Project.

When a user starts to contribute code or documentation patches, they become a Contributor.

4.2. Contributors

Contributors are the people who write code or documentation patches or contribute positively to the project in other ways. A volunteer's contribution is always recognized. In source code, all volunteers who contribute to a source file may add their name to the list of authors for that file.

4.3. Committers

Contributors who give frequent and valuable contributions to a subproject of the Project can have their status promoted to that of a "Committer" for that subproject. A Committer has write access to the source code repository and gains voting rights allowing them to affect the future of the subproject.

In order for a Contributor to become a Committer, another Committer can nominate that Contributor or the Contributor can ask for it.

Once a Contributor is nominated, all existing committers will vote. If there are at least 3 positive votes and no negative votes, the Contributor is converted into a Committer and given write access to the source code repository. This is an example offer letter that should be sent to the volunteer after 3 positive votes have been received:

Dear Contributor,

Our project would like to offer you commit privileges. We have been impressed with your contributions up till now, and believe that your involvement will improve the quality of the code we produce. If you are interested in having commit privileges, please set up an account with http://sourceforge.net and let us know your account name.

We all hope that you accept this invitation.

The Community Map Builder Project Management Committee.

Committers are asked to coordinate their efforts with the maintainers of the modules they wish to modify/extend.

At times, Committers may go inactive for a variety of reasons. A Committer that has been inactive for 6 months or more may lose their status as a Committer. Getting access back is as simple as re-requesting it on the project's Developer mailing list.

4.4. Project Management Committee (PMC)

Committers who frequently participate with valuable contributions may have their status promoted to that of a Project Management Committee Member. This committee is the official managing body of the Project and is responsible for setting overall project direction. In order to become a Member, someone on the PMC must nominate the Committer. The individual may then be approved with a 3/4 majority of the PMC.

5. Communication

Cameron Shorter

5.1. Email lists

Email lists provide an effective form of communication. Details about this project's email lists can be found at: http://sourceforge.net/mail/?group_id=35246.

6. Bug Tracking

Cameron Shorter

During alpha development, issues are embedded into the code using @task tags which can be read from the maven output files.

Once code has been released, bugs are tracked using this project's bug tracking system: http://sourceforge.net/tracker/?group_id=35246.

7. Build

Building involves processing source files to produce target files. For example, compiling and linking source code, or converting docbook to html documentation.

7.1. Javascript Compression

Javascript files need to be uploaded across potentially slow internet connections. Consequently, it is desirable to keep file sizes to a minimum.

Javascript files can be compressed with The Creativyst CSS and JavaScript Compressor.

To do: Find an open source compression algorithm. The above algorithm removes white spaces and comments, but does not compress file names, which could reduce file size even more.

8. Design

Cameron Shorter

A design describes how to implement a product. A design is useful during development and maintainance as it provides developers with a quick high level understanding of the product being developed.

8.1. Design Document

A design document should provide a brief overview of the design to help developers gain a quick understanding of the system. The design documentation is enhanced by using more diagrams and less text.

Text in the design document should be written in docbook format. Refer to the documentation section for more details about using docbook.

8.2. Inline Documentation

Documentation about procedures and classes should be embedded in the source code and then extracted into Detailed Design documentation using an external parser.

8.2.1. Jsdoc

Jsdoc is a tool that parses inline documentation in JavaScript source files, and produces a HTML summary. Refer to the Code Convensions section to see how comments should be written.

8.3. Unified Modeling Language (UML)

UML is a standard notation used to specify, visualise, construct and document the components of an object-oriented software-intensive system. UML diagrams are used as part of design documentation.

8.3.1. Modelling Web Applications

Web applications are modelled in accordance with Modeling Web Application Design with UML, by Jim Conallen. This white paper describes class stereo types to model web components. Class stereotypes provided are:

  • Server Page

  • Client Page

  • Form

  • Frameset

  • Target

  • Scriplet

  • XML

8.3.2. Poseidon UML

Poseidon is a UML editor which extends the open source Argo UML. It provides a no-cost version which is more polished than Argo UML.

Note that some of ArgoUML's features are disabled in Poseidon. In particular, Poseidon 2.1 disables reverse engineering of java code.

You can reverse engineer code using Argo UML, then import the project into Poseidon.

Poseidon can be installed into Netbeans as a module. Refer to Poseidon download instructions for more details.

9. Coding Conventions

Cameron Shorter

Coding conventions describe the coding styles developers should use when writing code. For example, whether you use 2, 4, or 8 space indents. Standardizing on a coding style across a project improves legibility of the code, and automatic code formatters make conforming to these standards easy.

9.1. Directory Names

Directory names shall be all lower case with no spaces. Some versions of windows do not distinguish between upper and lower case, and in unix, writing spaces in filenames is painful.

9.2. Class Naming Conventions

to do: need to fill out this section.

10. Refactoring

Cameron Shorter

Refactoring is the process of restructuring/renaming your code to ensure your design remains clean as your requirements and functionality change and grow with time.

11. Code Profiling

Andrea Aime

Code profilers are tools which analyse performance and memory of applications.

12. Test

As you code, you should write a unit test for each class to test the functionality and robustness of the class.

13. Documentation

Cameron Shorter

Artur Hefczyc

This section describes how to create modular simple docbook files.

13.1. Quick Start

This quick start section will get you editing simple docbook sections quickly and easilly. For more detailed information, refer to the rest of the documentation section.

Simple steps for docbook editing

  1. Download and install a Java Virtual Machine if you have not already done so.

  2. Download and install XXE, a What You See is What You Mean (WYSIWYM) docbook editor.

  3. If you creating a new section, then copy an existing section and modify it. If you are updating an existing section - well, that is even easier.

    Example 1. Edit a docbook section file

    cp doclicense.xml yoursection.xml
    xxe yoursection.xml
  4. Figure 1. Editing with XXE

    Editing with XXE

    XXE is similar to any GUI editor. You type in the main panel and most of the list/table/paragraph/section/figure formatting options are selectable from the toolbar. Remember that simple docbook is structured XML. You will often need to select an XML node so you can specify whether the next paragraph should be part of this section or a new section. Select XML nodes with arrow icons:

  5. Commit your changes back into the source repository.

  6. To publish your document after editing, either tell the main document author and get them to do it, or read the rest of the documentation section.

13.2. Why Docbook?

Documentation for this project is written using Simplified Docbook format. Simplified Docbook is a subset of Docbook XML, a versatile format used by the Linux Documentation Project (among others).

Docbook can be easily converted into numerous output formats, like HTML, PDF, etc.

Using Docbook ensures documentation content is kept separate from presentation so content can written once and published in numerous formats and styles.

There are a few GUI editors for docbook now, with more editors promising to support docbook in the future.

According to The Software Release's Howto, docbook is the open source format of the future, and most of the high profile open source projects have moved or are moving to docbook as their coding standards. So we are saving ourselves pain in the future by embracing Docbook now.

13.3. Modular documentation

Modular DocBook means content is broken up into smaller file modules that are recombined for publication. The advantages of modular documentation include:

  • Reusable content units.

  • Smaller file units to load into an editing program.

  • Distributed authoring.

  • Finer grain version control.

This project uses modular documents for it's documentation.

Example 2. Including sections into a master document

design.xml
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN"
"http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd">
<article
id="index">
  <articleinfo>
    <title>Software Design Description</title>
  </articleinfo>

  <section id="scope">
    <title>Scope</title>

    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
    href="doclicense.xml"/>
  </section>
</article>
doclicense.xml
<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE section PUBLIC "-//OASIS//DTD Simplified DocBook XML V1.0//EN"
"http://www.oasis-open.org/docbook/xml/simple/1.0/sdocbook.dtd">
<section id="licence">
  <title>License.</title>

  <para>Copyright (c) 2002 Cameron Shorter. Permission is granted to copy,
  distribute and/or modify this document under the terms of the <ulink
  url="http://www.fsf.org/copyleft/fdl.html">GNU Free Documentation License</ulink>,
  Version 1.1 or any later version published by the Free Software Foundation;
  with the Invariant Sections being with no Invariant Sections, with the
  Front-Cover Texts being no Front-Cover Texts, and with the Back-Cover Texts
  being no Back-Cover Texts.</para>
</section>

If you want to document a new module, you need to write a new module.xml section file, and then add an <xi:include> tag into design.xml.

Notice in the above example that <section> tags contain an id attribute. This means the URL of published sections will be human readable, something like http://net.sourceforge.generguide/docs/design.html#licence instead of http://net.sourceforge.generguide/docs/design.html#id2754172.

13.4. Publishing docbook files

13.4.1. Publishing using xsltproc

Publishing modular documents is possible with xsltproc. At the time of writing, other XSLT engines don't process modular documents yet because the <xi:include> tags cause the XML to be invalid. (To do, check to see if Saxon and Xalan support <xi:include> yet.)

Instructions on installing and using xsltproc can be found at: http://www.sagehill.net/xml/docbookxsl.

Example 3. Using xsltproc to publish modular docbook files

xsltproc --xinclude --novalid -o index.html
          docbook-xsl/html/docbook.xsl  main.xml

13.4.2. Publishing using ant/maven

ant can publish standard simple docbook files but cannot publish modular docbook files because they are not valid XML. Hopefully this will change in future.

In version 1.5, ant has a bug which prevents it from publishing more than one docbook document at a time. The workaround is to run ant a number of times. Earlier versions of ant have more bugs and less functionality.

13.4.3. Publishing using maven

Maven allows uses to process docbook to (arnika/velocity?) and then to html. Since maven is based on ant, it also cannot process modular docbook files yet and at the time of writing, this functionality was still very bug ridden. Hopefully it will improve soon.

13.4.4. Publishing using java tools.

13.4.4.1. Joining document parts using XSL template defined in selectivexinclude.xsl.

To process XML files containing xinclude elements you need only 2 things:

  1. XSL template containing transformation including external documents parts. Good implementation is in selectivexinclude.xsl file available at address on dev.w3.org site.

  2. Any XSL processor. Described above xsltproc available from xmlsoft or saxon available from saxon site or xalan which is one of projects on xml.apache.org. All these tools are good for simple cases. However each of them offers different extensions so you choice depends of your unusual needs. However the use of xsltproc to join parts of modular documentation with selectivexinclude.xsl seems to not make great sense because of built-in support for XInclude into xsltproc.

    Java users often ask a question: saxon or xalan? As I said above, if you need only standard XSLT processing they both are the same, they both (current stable versions: xalan-2.5.1, saxon-6.5.3) implement full XSLT 1.0 version and XPath 1.0 version. Current saxon development version: 7.6.5 implements XSLT-2.0 and XPath-2.0 and it is worth to note that the author of saxon library - Michael Kay is editor of the XSLT-2.0 spec. On the other side xalan is accepted library by SUN and is included with their JDK-1.4 and above what, on the third side, causes some problems when you want to use more recent xalan version than the one included in JDK.

Below are presented sample commands for each of 2 java tools for generating target, one XML document from modular documentation. These samples are tested with the most recent versions: saxon-7.6.5 and xalan-2.5.1 and some parameters may not work with earlier versions.

  • masterxmlfile.xml is source file containing XInclude elements.

  • targetalldoc.xml is target file containing included content of external files in place of XInclude elements.

  • selectivexinclude.xsl is XSL template containing transformation definition for including content from external files.

  • LIB is an environment variable pointing to directory on your disk where you keep JAR files.

Below are 2 kinds of commands. The simpler showing XSL inclusion only and a little bit more complex presenting also how to use CATALOGS with these tools. There are some additional elements in these commands: resolver.jar - library resolving URL to DTDs stored on your local file system and $PROJECTS_HOME/sgml - a directory where is placed CatalogManager.properties - file providing the same function as SGML_CATALOG_FILES environment variable. More information about using CATALOGS you can find in section Tools and methods for XML processing.

Example 4. Sample bash command for saxon.

export SAXON_CLASSPATH=$LIBS/saxon.jar
java -cp $SAXON_CLASSPATH net.sf.saxon.Transform \
    -w0 -u \
    -o targetalldoc.xml \
    masterxmlfile.xml selectivexinclude.xsl

Example 5. Sample bash command for saxon using CATALOGS.

export SAXON_CLASSPATH=$SAXON_CLASSPATH:$LIBS/resolver.jar:$PROJECTS_HOME/sgml
java -cp $SAXON_CLASSPATH net.sf.saxon.Transform \
    -x org.apache.xml.resolver.tools.ResolvingXMLReader \
    -y org.apache.xml.resolver.tools.ResolvingXMLReader \
    -r org.apache.xml.resolver.tools.CatalogResolver \
    -w0 -u \
    -o targetalldoc.xml \
     masterxmlfile.xml selectivexinclude.xsl

Please remark below examples. If you work with JDK-1.4 or later you can not use CLASSPATH to point to your xalan jar file. There is some xalan version included with JDK-1.4, unfortunately outdated, and it is loaded first before classes given in CLASSPATH. To enforce the use of your classes you must put them in BOOTCLASSPATH as presented below.

Example 6. Sample bash command for xalan.

export XALAN_CLASSPATH=$LIBS/xalan.jar:$LIBS/xercesImpl.jar:$LIBS/xsml-apis.jar
java -Xbootclasspath/p:$XALAN_CLASSPATH org.apache.xalan.xslt.Process \
    -in masterxmlfile.xml -xsl selectivexinclude.xsl \
    -out targetalldoc.xml

Example 7. Sample bash command for xalan using CATALOGS.

export XALAN_CLASSPATH=$XALAN_CLASSPATH:$LIBS/resolver.jar:$PROJECTS_HOME/sgml
java -Xbootclasspath/p:$XALAN_CLASSPATH org.apache.xalan.xslt.Process \
    -ENTITYRESOLVER org.apache.xml.resolver.tools.CatalogResolver \
    -URIRESOLVER org.apache.xml.resolver.tools.CatalogResolver \
    -in masterxmlfile.xml -xsl selectivexinclude.xsl \
    -out targetalldoc.xml

13.5. Docbook Editors

You can edit docbook with the following free tools:

  • XXE is a java based, What You See Is What You Mean (WYSIWYM) editor. It provides a slightly clunky but workable GUI interface for editing Docbook documents. It is well worth trying.

  • Netbeans: Simple Docbook is XML, so XML editors work well. Netbeans has a nice XML plugin which I use.

  • vim and emacs, or any text editor can be used.

  • There are a number of WYSIWYM editors developing Docbook export/import functionality at the time of writing. Of note, Open Office and Lyx look promising.

  • Viewing Docbook: Tomas has written a useful web page for viewing Docbook pages at http://www.cartesia.org/modules.php?op=modload&name=NS-Docbook&file=index.

13.6. Docbook References

There are a few references worth knowing: