clarify package directory vs. work directory

[ikiwiki.git] / docs / howtos / howtoporttodragonfly / index.mdwn

# How to port to DragonFly

[[!toc levels=4 ]]

This page is intended to provide some useful information to help you port applications to your DragonFly operating system. This system uses the NetBSD Packages Collection (pkgsrc) to manage application ports. A basic knowledge of this collection, of GNU developer tools, and possibly of imake, is required to successfully complete this task. For more information, references to these tools are available in the References section below.

This paper will focus on applications made for the pkgsrc collection. But many information will also be useful to those who simply wants to port applications to DragonFly without specific context.

## Preparing the workspace

To port software with pkgsrc, two approaches are possible:

* Add new packages in pkgsrc.
* Modify an existing package.

In both cases, the source code of the package must be extracted into a work directory before starting the porting work. When a source file of the original application is modified, its original will be stored with the extension “. orig”. The utility mkpatches, that we'll see later in this document, will use the original files as a basis to create patches. Please note that *the version with the extension .orig must always be that of the original distribution package*.

### Add a new package

The page [Creating a new pkgsrc package from scratch](http://www.netbsd.org/docs/pkgsrc/creating.html)
 gives the main steps required to add a new application to the pkgsrc collection.This tutorial is very well done and summarizes the main steps required to create a new package.

### Modify an existing package

Suppose you would want to modify the application foo/bar from the pkgsrc collection. You then start by visiting its dedicated directory in pkgsrc, and by running the following command:

        cd /usr/pkgsrc/foo/bar
        bmake patch

This will restore the changes previously made, which will be a good starting point for the porting. The bmake command could also be used without options to attempt an immediate first compilation. But you might have to change some files first, like the GNU scripts for example.

Suppose that the foo application of the collection is version 1.0. Let's go into the newly extracted source directory located right here<a name="AEN1" href="#FTN.AEN1">[1]</a>:

        cd /var/obj/pkgsrc/foo/bar/work/foo-1.0

Voilà, your ready! The porting of the source code can now begin.

## Working with GNU developper tools

If the application to port uses GNU tools like autoconf, GNU scripts shall be the first thing to look at. We'll first make a brief introduction to these tools. Then we will cover some modifications required to adapt their scripts to DragonFly. 

### Introduction to GNU tools

GNU provides a series of very popular tools which aims at helping C and C++ cross-platform application development. These tools are Autoconf, Automake and Libtool. They can operate independently, but they can also be highly integrated.

Their use in common might not be necessarily easy to master. The GNU Autobook (see the References section), available online for free, can help you find your way with it.

Briefly, the following files are defined by the developers:

* Makefile.am and config.h (automake), to generate the Makefile.in files.
* configure.ac (autoconf), to generate the configure script.

The aclocal tool generates the file aclocal.m4, which contains dependencies for the configure script. It is usually run before other tools. The autoconf generated configure script will in turn generate the makefiles from other generated files, like Makefile.in. So, there are several levels of file generation. 

The first level generation is called bootstrapping (see the Bootstrap section). This process generates a set of files and directories right in the current build directory. For example, in DragonFly 2.8, the files generated by default are:

        autom4te.cache/
        .deps/
        configure
        missing
        Makefile.in
        install-sh
        depcomp
        config.sub
        config.guess
        aclocal.m4

When porting an application, it is usually not necessary to regenerate these files and directories. You simply need to adjust the Makefile and configure files. But occasionally, you'll have no other choice. The following sections attempt to give you some leads to follow.

### Editing configure, config.guess and config.sub

When the script configure is run, it creates settings suitable for the host platform. In cases where DragonFly is not on the list of supported systems, we'll need to add it. 

By default, the command uname -m is used to detect the host platform. In our case, it will return the DragonFly label. The host_os variable will be defined after this value and will take the form “dragonflyX” (where X is the version of the host system, ex. 2.8). The configure script then uses this information to define OSARCH. A good place to start the port would be to add the following definition:

        case "${host_os}" in
             […]
             dragonfly*)
             OSARCH=DragonFly
             ;;

The host_os variable is normally generated by running config.guess. It can also be explicitly defined by using the “build” option of the configure script.

Then you can continue by searching through the script to locate:

* Other occurrences of case "${host_os}".
* Occurrences of the “BSD” string.
* Changes made to other BSD systems, like NetBSD for example. If the software has already been brought to one of these systems, and it will often be the case, this will allow you to quickly find many of the modifications required for DragonFly. Of course this is not to take for granted.

Ideally, you would have to review the entire configure script to make sure that all necessary changes were made. Some will prefer to go by trial and error, by first changing what's obvious and then actively testing the results.

The config.guess file may also need to be reviewed. But the latest versions of this script already include DragonFly in its supported systems. Running it on DragonFly 2.8, for example, returns *i386-unknown-dragonfly2.8*.

The config.sub file contains a set of subroutines used by the configure script. If this file is not a symlink, it may also need to be changed. But the DragonFly system is already included by default in it, but possibly lightly implemented.

One special thing to consider in the review of GNU scripts for DragonFly is that some configuration scripts tries to collect all BSD like systems like this:

        case "$uname" in
                *BSD*)
        […]
        esac

In our case, uname will contain the string DragonFly, without the “BSD” suffix. This pattern should be searched to explicitly add DragonFly after the sequence:

        case "$uname" in
                *BSD*|DragonFly)
        […]
        esac

You might want to solve this problem by forcing variables like host_os and OSARCH to “DragonflyBSD”. But it may be more difficult than simply searching for occurrences of *BSD and applying the necessary changes.

### Regenerate GNU scripts

There may be cases where you would want to regenerate the GNU scripts rather than just port them. Some basic knowledge are required to perform this task. For those who do not have this knowledge, the GNU Autobook is an excellent reference for it. We'll briefly cover the topic here, by giving some basic information and clarifying some difficulties.

#### Autoconf compatibility

As mentioned before, autoconf is used to regenerate the configure script. Operating systems are not bundled with the same version of it. We must therefore see to use the correct version of GNU tools if we want to regenerate completelly an application's build scripts.

#### The makefile.am file

This file contains macros needed to generate the Makefile.in files, which in turn generates the makefiles. The automake tool defines a syntax of its own. It can generate complex and portable makefiles from a small makefile.am script.

#### The configure.ac or configure.in file

This file contains macros necessary to generate the configure script. These are m4 macros, which generates a shell script when you run autoconf. Required changes to this file are similar to those proposed for the configure script itself.

#### Bootstrap

The procedure aiming at regenerating all the GNU build files is called *bootstrap*. Some ports are bundled with a script of this name. It runs all the required tools in the correct order. The typical case will define the following sequence:

        aclocal \
        && automake --gnu --add-missing \
        && autoconf

### Compiling with FreeBSD as build target

As DragonFly is a FreeBSD fork, many are building it's ports by using the FreeBSD build target. Although it sometimes works well, like for PostgreSQL, this is risky because compile compatibility with FreeBSD is not among DragonFly's goals. You must therefore expect that this shortcut is less and less usable over time, or can possibly generate badly optimized binaries.

In cases where this method still works, they may still be some changes required in GNU scripts. As mentioned earlier, scripts sometime tries to catch all the BSD variants by using the “*BSD” string. In these cases, without minimal changes, the final results might not entirely be what was anticipated for FreeBSD.

## Working with imake and xmkmf

Arguably, imake and xmkmf could be seen as the old building tools for those wishing to develop portable applications. These tools were designed for the venerable X Window System and its applications, themselves brought to many operating systems, including UNIX®, the BSD family, GNU/Linux, AIX®, Solaris® and Windows®. This section will cover briefly these build tools, because they have been gradually discarded in favor of the GNU build tools.

X Window normally comes with a lib/X11/config directory, which contains macro files dedicated to xmkmf. This directory contains files with extensions *.cf, *.tmpl and *.rules.

* *.cf files contains settings specific to it's hosts. X Window comes normally with a DragonFly.cf file.
* *. rules files defines macros that implement the rules that can be used in Imakefile files.
* *. tmpl files contains templates that defines compile options.

As for GNU automake, these macros are used to generate makefiles for the operating system that hosts the X Window System. The normal xmkmf's procedure is to define Imakefile files, then to build the makefiles by running xmkmf. This tool will in turn call imake, which is a C preprocessor interface for the make tool. The tool xmkmf will invoke imake with the appropriate arguments for DragonFly.

When xmkmf is executed with the -a option, it automatically runs make Makefiles, make includes and make depends.

Here's a simple Imakefile example:

        BINDIR = /usr/local/bin
        
        .SUFFIXES: cpp
        
        CXXEXTRA_INCLUDES = -I. -I.. -I/usr/pkg/include
        LOCAL_LIBRARIES = -L/usr/pkg/lib
        EXTRA_LIBRARIES = $(XMLIB) $(XTOOLLIB) $(XLIB) $(XMULIB) $(XPMLIB)
        CXXDEBUGFLAGS = -O2 -g
        
        HEADERS = bar.h
        SRCS = bar.cpp
        OBJS = bar.o
        
        ComplexCplusplusProgramTarget(bar)

Note the similarity of Imakefile with makefiles. You can see the Imakefile as makefiles with macros.

In the example above, the ComplexCplusplusProgramTarget macro defines a rule to generate the C++ "bar" binary, according to the parameters that precedes it. Notably, the parameter .SUFFIXES indicates that the source files will have the extension “.cpp”.

In DragonFly, imake and xmkmf are available by installing the package devel/imake. For more information, links are given in the References section.

## Editing the code

To tailor a program to DragonFly, you can either use the definition used to identify the DragonFly system or use the definitions generated by the GNU tools.

### The DragonFly definition

Idealy, an application's code would be reviewed in full to ensure that it uses the right functions and algorithms for DragonFly.

In fact, the method mostly used is probably to try to compile, to correct the code when it does not, and to test the final results. When the code is adapted to DragonFly, it may look like this: 

        #if defined(__DragonFly__)
                /* code specific to DragonFly */
        #elif
                /* code for other systems */
        #endif

An initial change might be to scan the definitions for the other BSD variants, as FreeBSD or NetBSD. We often see this kind of syntax in the code tree:

        #if defined(__OpenBSD__) || defined(__NetBSD__) || defined(__FreeBSD__) || defined(__Darwin__) || defined(__DragonFly__)
        #include <fcntl.h>
        #include <net/route.h>
        #endif

### GNU Tools definition

Porting code can focus on identifying available functions rather than on identifying host systems. A typical case would be the bcopy function, normally used instead of strcpy on BSD derivatives. Rather than referring to the host system, the code might look like this (from the GNU Autobook<a name="AEN2" href="#FTN.AEN2">[2]</a>):

        #if !HAVE_STRCPY
        #  if HAVE_BCOPY
        #    define strcpy(dest, src)   bcopy (src, dest, 1 + strlen (src))
        #  else /* !HAVE_BCOPY */
             error no strcpy or bcopy
        #  endif /* HAVE_BCOPY */
        #endif /* HAVE_STRCPY */

This guarantees a form of compatibility between systems at the code level. Unfortunately, this approach is moderately used, which generates more work for porters.

### The BSD4.4 Heritage

Many developers are unaware that a BSD definition was created for BSD4.4 and it's derivatives, as DragonFly, FreeBSD and NetBSD. This definition is used like this (from pkgsrc developer's guide<a name="AEN3" href="#FTN.AEN3">[3]</a>):

        #include <sys/param.h>
        #if (defined(BSD) && BSD >= 199306)
        /* BSD-specific code goes here */
        #else
        /* non-BSD-specific code goes here */
        #endif

The NetBSD group, which maintains pkgsrc, recommends the use of this definition. In fact, it is rarely used. In the specific case of DragonFly, this definition no longer exists. Therefore it must not be used.

## Patching for pkgsrc

Ideally, a port to DragonFly shall always be at the project level. But realistically, it is faster and easier to modify applications at the pkgsrc level.

pkgsrc contains the patches and some basic information needed to build a package in the *package directory* (by default `/usr/pkgsrc/<category>/<package>`).

The `pkgtools/pkgdiff` suite of tools helps with creating and updating patches to a package. Extract the source code into the *work directory* (by default `/usr/pkgobj/bootstrap/work/pkgsrc/<category>/<package>/work/`) by invoking

        bmake patch

from the package directory. This fetches the source code if necessary, extracts it and applies any existing pkgsrc patch, saving the unpatched files with a `.orig` extension in the work directory.

To create a new patch, save a copy of the original file with that same `.orig` extension. If it exists, just keep it – do not overwrite or change `.orig` files or your patches will not apply later on! You may choose to use `pkgvi` from the `pkgdiff` suite to automate this.

You can preview the patches using `pkgdiff <file>`. To generate all patches invoke

        mkpatches

from the package directory (not the work directory!) The new patches will be saved in the `patches/` directory along with backups of the previous patchset. When you are content with the generated patches commit them and update the patch checksums:

        mkpatches -c
        bmake makepatchsum

You may also revert to the old patches by calling `mkpatches -r`.

Now clean up and try to rebuild your package:

        bmake clean
        bmake

If you have any other changes to add, you can remove the package again and repeat these steps.

## Submitting a package in pkgsrc

If you plan to port a program to DragonFly and wish to share your work, pkgsrc is ideal for you. The first thing to do is to inquire whether the application is already in the collection. If not, you're free to add your work and be registered as a port maintainer.

This section will attempt to give you some minimal guidance on submitting changes in the pkgsrc collection. This information is incomplete, please consult the page [Submitting and Commiting](http://www.netbsd.org/docs/pkgsrc/submit.html) carefully before submitting anything for real.

A source code package can be submitted with the gtk-send-pr package (pr=Problem Report), or by visiting the page [NetBSD Problem Report](http://www.netbsd.org/support/send-pr.html). The indications given by the pkgsrc developer's guide in connection with this tool are summarized here:

“In the form of the problem report, the category should be “pkg”, the synopsis should include the package name and version number, and the description field should contain a short description of your package (contents of the COMMENT variable or DESCR file are OK). The uuencoded package data should go into the “fix” field.” ̶  http://www.netbsd.org/docs/pkgsrc/submit.html#submitting-your-package

It is also possible to import new packages in pkgsrc-wip. See [http://pkgsrc-wip.sourceforge.net/](http://pkgsrc-wip.sourceforge.net/) for more information.

## References

[The pkgsrc guide](http://www.netbsd.org/docs/pkgsrc/)

[The pkgsrc developer's guide](http://www.netbsd.org/docs/pkgsrc/developers-guide.html)

[[pkgsrc on DragonFly|/docs/howtos/HowToPkgsrc/]]

[GNU autoconf](http://www.gnu.org/software/autoconf/)

[GNU automake](http://www.gnu.org/software/automake/)

[GNU libtool](http://www.gnu.org/software/libtool/)

[GNU “autobook”, or “GNU AUTOCONF, AUTOMAKE, AND LIBTOOLS”, from Gary V. Vaughan](http://sources.redhat.com/autobook/)

[imake](http://www.x.org/archive/X11R6.7.0/doc/imake.1.html)

[xmkmf](http://www.x.org/archive/X11R6.7.0/doc/xmkmf.1.html)

## Notes

<table border="0">

  <tr>
      <td align="LEFT" valign="TOP" width="5%">
        <a name="FTN.AEN1" 
            href="#AEN1">[1]
        </a>
      </td>
      <td align="LEFT" valign="TOP">
        <p>
          Previously, the work directory was created directly in the directory dedicated to the ported application, under the pkgsrc directory. But to preserve the integrity of these directories and to be able to compile on another volume, the working files have been moved under /usr/obj. The source code located in /usr/src followed the same path.
        </p>
      </td>
  </tr>

  <tr>
      <td align="LEFT" valign="TOP" width="5%">
        <a name="FTN.AEN2" 
            href="#AEN2">[2]
        </a>
      </td>
      <td align="LEFT" valign="TOP">
        <p>
          Source: <a href="http://sources.redhat.com/autobook/autobook/autobook_50.html#SEC50">http://sources.redhat.com/autobook/autobook/autobook_50.html#SEC50</a>. “Copyright (C) 2000, 2006 Gary V. Vaughan. Verbatim copying and distribution of this entire article is permitted in any medium, provided this notice is preserved.”
        </p>
      </td>
  </tr>

  <tr>
      <td align="LEFT" valign="TOP" width="5%">
        <a name="FTN.AEN3" 
            href="#AEN3">[3]
        </a>
      </td>
      <td align="LEFT" valign="TOP">
        <p>
          Source: <a href="http://www.netbsd.org/docs/pkgsrc/fixes.html#fixes.build.cpp">http://www.netbsd.org/docs/pkgsrc/fixes.html#fixes.build.cpp</a>. Copyright © 1994-2007 The NetBSD Foundation, Inc.
        </p>
      </td>
  </tr>

</table>