State of GSoC Wed May 26

[ikiwiki.git] / docs / developer / GEMdrmKMS / index.mdwn

# Port of GEM and KMS

## Status

The latest drm code from FreeBSD 9.x current has been successfully ported to DragonFly BSD, [patch](http://leaf.dragonflybsd.org/~davshao/r600fbsd.diff) and git branch **r600fbsd** git://leaf.dragonflybsd.org/~davshao/dragonfly.git, and tested with a Radeon HD 4550 on an x86_64 machine (Shuttle SG45H7).  However it must be remembered that testing has only been done with this one graphics card and it is completely unknown whether say an Intel machine will lock up solid.  Also nothing was done to port the extensive Via drivers FreeBSD has already ported.

## Introduction or what the heck is being talked about

We are talking about modern graphics card drivers that have already or are in the process of being written for Linux.  Fortunately for the BSDs the source code that is the basis of these drivers, part hosted in git repositories accessible from [freedesktop.org](http://cgit.freedesktop.org/), and even the part that is now residing in the Linux kernel, is mostly licensed under terms compatible with the [MIT X License](http://www.opensource.org/licenses/mit-license.php), and therefore can be directly ported to DragonFly.

When we refer to DRM, we are referring to the [Direct Render Manager](http://dri.freedesktop.org/wiki/DRM) that is a kernel module arbitrating requests for various graphics related services.  Because of this acronym, the source code for DragonFly's DRM module can be found in directory **sys/dev/drm**, while the source code for Linux'd DRM module has been split into **include/drm** hosting common header files and **drivers/gpu/drm**.  Furthermore the Linux files have had vendor specific code split off into their own separate directories, a change the BSDs should consider since having all files in one directory is becoming rather unwieldy.

## Previous work makes the port possible

Simply take a diff of the FreeBSD and DragonFlyBSD versions of DRM and one can readily see that the hard work of translating the semantics of the locking for the Linux drivers has mostly been done, and better still can be done almost automatically.  The FreeBSD port represents the limits of what can be ported having worked out exclusive access mechanisms, but not having completely worked out equivalents of other Linux APIs such as **idr**, small integer ID management.

## What has been done and should be done

The software used for graphics whether the kernel or Mesa is surprisingly resilient.  As long as it compiles it will try and find a way with dealing with other mismatched components by falling back on default behavior.  Working on a fast x86_64 machine, it is easy to be lulled into a false sense that things are working properly unless one checks the logs.

### **dmesg* output

Here the proper kernel module appears to be loaded.  I wasted quite a bit of time earlier this week not realizing an error that led to the kernel module not being loaded at all that appeared from the messages to be a device non-existence problem.

    drm0: <ATI Radeon HD 4550> on vgapci0
    vgapci0: child drm0 requested pci_enable_busmaster
    info: [drm] Initialized radeon 1.31.0 20080613
    info: [drm] Setting GART location based on new memory map
    info: [drm] Loading RV710 Microcode
    info: [drm] Resetting GPU
    info: [drm] writeback test succeeded in 1 usecs

### **var/log/Xorg.0.log** output

Somewhere on one's system are the logs of the latest startup of the X server and the modules it finds.  There can be an alternate place if one installs a second X.org, say at */opt/xtest*, in which case instead of **/var/log/Xorg.0.log** one will look in **/opt/xtest/var/log/Xorg.0.log**.

Here's a case of a failure where things are fine: I am composing this wiki from within Firefox 3.6.3 on a system where the proper module fails to load and yet where because of the machine's speed there is no discernible problem.

    drmOpenDevice: node name is /dev/dri/card0
    drmOpenDevice: open result is 10, (OK)
    drmOpenByBusid: Searching for BusID pci:0000:01:00.0
    drmOpenDevice: node name is /dev/dri/card0
    drmOpenDevice: open result is 10, (OK)
    drmOpenByBusid: drmOpenMinor returns 10
    drmOpenByBusid: drmGetBusid reports pci:0000:01:00.0
    (EE) AIGLX error: dlopen of /usr/pkg/lib/dri/r600_dri.so failed (Cannot open "/usr/pkg/lib/dri/r600_dri.so")
    (EE) AIGLX: reverting to software rendering
    (II) AIGLX: Loaded and initialized /usr/pkg/lib/dri/swrast_dri.so

The Mesa 7.4.x series simply didn't *have* an r600_drv.so driver so of course it can't be found.  Mesa happily falls back to using software rendering and nothing seems greatly wrong.

Now here is an example using latest Mesa and everything else from git where the driver is found:

    [    95.404] drmOpenDevice: node name is /dev/dri/card0
    [    95.405] drmOpenDevice: open result is 11, (OK)
    [    95.405] drmOpenByBusid: Searching for BusID pci:0000:01:00.0
    [    95.405] drmOpenDevice: node name is /dev/dri/card0
    [    95.405] drmOpenDevice: open result is 11, (OK)
    [    95.405] drmOpenByBusid: drmOpenMinor returns 11
    [    95.405] drmOpenByBusid: drmGetBusid reports pci:0000:01:00.0
    [    95.507] (II) AIGLX: enabled GLX_MESA_copy_sub_buffer
    [    95.507] (II) AIGLX: enabled GLX_SGI_make_current_read
    [    95.507] (II) AIGLX: enabled GLX_texture_from_pixmap with driver support
    [    95.507] (II) AIGLX: Loaded and initialized /opt/xtest/lib/dri/r600_dri.so
    [    95.507] (II) GLX: Initialized DRI GL provider for screen 0
    [    95.508] (II) RADEON(0): Setting screen physical size to 508 x 285

Ironically I knew I was on the right track porting the latest git versions of the X.org stack when I succeeded in locking up hard my machine.  That meant a real hardware acceleration module was being loaded by that the previous kernel could not handle it, which was why the latest drm from FreeBSD had to be imported for Radeon r600.

## Disclaimer

Code that is uploaded to the experimental git branches has been verified to work for one graphics card only
and one x86_64 system only.  It is highly likely that systems exist where using the experimental branches
will leave one's system unable to use software based on X.org.  Use at one's own risk.

I have experienced from an ill-fated experiment forced rebooting with corruption of a ufs2 root filesystem;
therefore, the same could well happen to you.  Only use this code on a system with no valuable or
irreplaceable data.  I disclaim any warranty or fitness of code.

## Current Progress for Google Summer of Code 2010

Current work on porting the Linux drm to DragonFly BSD can be found in
git branch **gsocdrmalpha** git://leaf.dragonflybsd.org/~davshao/dragonfly.git

As of Wednesday, May 26, 2010, the first goal is to port the API from *drm.h* and *drmP.h*.

I have ported the easy 90% that does not require changing any working code.  The next harder step
will be to fully use the already existing implementation of *list_head* to replace the custom list
and locking for agp memory.

## Installing from pkgsrc

### GNU m4 1.4.14 and bison 2.4.2

Patch both *devel/m4* and *devel/bison* using the patch idea from:
<http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=43098>

Unfortunately the problems are in the *work* directories so that one needs the knowledge from
<http://www.netbsd.org/docs/pkgsrc/components.html>
to even get the patches to be applied.

Install *pkgtools/pkgdiff*.

### gstreamer 0.10

As of 2010-05-09, I find it necessary on x86_64 to use the following patch for
pkgsrc gstreamer0.10, an eventual dependency of the full xfce desktop:

    --- pkgsrc/multimedia/gstreamer0.10/Makefile.orig       2010-03-29 16:04:23 -0700
    +++ pkgsrc/multimedia/gstreamer0.10/Makefile    2010-03-29 16:09:04 -0700
    @@ -44,6 +44,11 @@
 
     .include "../../mk/bsd.prefs.mk"
 
    +# __udivti3 error otherwise
    +.if ${OPSYS} == "DragonFly" && ${MACHINE_ARCH} == "x86_64"
    +MAKEFLAGS+=    CCVER=gcc44
    +.endif
    +
     .if ${OPSYS} == "NetBSD"
     # We must have a glib2 compiled with the RTLD_GLOBAL fix; if not, plugins
     # won't work at all.

See for example the following bug report for more details:
<http://mail-index.netbsd.org/pkgsrc-bugs/2010/03/30/msg037304.html>

## Building a separate version of X.org

We try something **ill-advised**, we build a separate version of X.org in another directory, say */opt/xbeta*,
on the same machine where we have installed the latext pkgsrc.  We do this because we have no choice but to
use the graphics hardware on this machine, but the risk is that it is very easy to link the wrong libraries
from pkgsrc.

### expat and gettext

We start with installing into */opt/xbeta* [expat 2.0.1](http://sourceforge.net/projects/expat/)
and [gettext 0.17](http://www.gnu.org/software/gettext/).

### Git repository for most of X.org

We use the following listing:

[freedesktop.org git repository browser](http://cgit.freedesktop.org/)

One can easily script getting, updating, and building the projects listed on that page.

### Example of using ldd

    $ ldd ./libXau.so.6          
    ./libXau.so.6:
        libc.so.7 => /usr/lib/libc.so.7 (0x800640000)
    $ ldd ./libXdmcp.so.6        
    ./libXdmcp.so.6:
        libc.so.7 => /usr/lib/libc.so.7 (0x800640000)

### libxcb problem with Python 2.6 cElementTree

Library *xcb/libxcb* unfortunately needs to be built early and there is a catch for
using pkgsrc Python 2.6.  Unfortunately at the current time not all of
*cElementTree* appears to be functional; therefore, one needs a patch similar to

    diff --git a/src/c_client.py b/src/c_client.py
    index d86d05e..36a7039 100755
    --- a/src/c_client.py
    +++ b/src/c_client.py
    @@ -1,5 +1,5 @@
     #!/usr/bin/env python
    -from xml.etree.cElementTree import *
    +from xml.etree.ElementTree import *
     from os.path import basename
     import getopt
     import sys

As of May 20, 2010, the situation for libxcb seems to be more complicated.  The latest *xcb/proto* needs
to be installed.  If one does not wish to install on top of one's current Python modules,
define

    export PYTHONPATH=${PREFIX}/lib/python2.6/site-packages

and apply a patch similar to below to change from using Python 2.6's *cElementTree* to *ElementTree*

    diff --git a/xcbgen/matcher.py b/xcbgen/matcher.py
    index e7958fa..16e8273 100644
    --- a/xcbgen/matcher.py
    +++ b/xcbgen/matcher.py
    @@ -7,7 +7,7 @@ we do not create a new type object, we just record the existing one under a new
     '''

     from os.path import join
    -from xml.etree.cElementTree import parse
    +from xml.etree.ElementTree import parse

     import state
     from xtypes import *
    diff --git a/xcbgen/state.py b/xcbgen/state.py
    index 51efc94..e72dc3e 100644
    --- a/xcbgen/state.py
    +++ b/xcbgen/state.py
    @@ -2,7 +2,7 @@
     This module contains the namespace class and the singleton module class.
     '''
     from os.path import dirname, basename
    -from xml.etree.cElementTree import parse
    +from xml.etree.ElementTree import parse

     import matcher
     from error import *

### libXext

      CCLD   libXext.la
    /usr/libexec/binutils217/elf/ld: .libs/extutil.o: relocation R_X86_64_PC32 against `xgeExtRegister' can not be used when making a shared   object; recompile with -fPIC
    /usr/libexec/binutils217/elf/ld: final link failed: Bad value

Use the following patch from <http://lists.x.org/archives/xorg-devel/2009-November/003724.html>

    diff --git a/src/Xge.c b/src/Xge.c
    index 7a583e5..2ea5d27 100644
    --- a/src/Xge.c
    +++ b/src/Xge.c
    @@ -294,7 +294,7 @@ _xgeEventToWire(Display* dpy, XEvent* re, xEvent* event)
     /*
      * Extensions need to register callbacks for their events.
      */
    -Bool
    +_X_HIDDEN Bool
     xgeExtRegister(Display* dpy, int offset, XExtensionHooks* callbacks)
     {
         XGEExtNode* newExt;

### FreeType 2

We install [FreeType 2.3.12](http://www.freetype.org/).  It appears that *GNUMAKE=gmake* is required.
Most of the X.org projects use *autogen.sh*, but here we use a standard call to *configure* somewhat
similar to:

        export ACLOCAL="aclocal -I ${PREFIX}/share/aclocal"
        export PKG_CONFIG_PATH="${PREFIX}/lib/pkgconfig"
        GNUMAKE=gmake ./configure --prefix=${PREFIX} CPPFLAGS="-I${PREFIX}/include -I${PREFIX}/X11/include" LDFLAGS="-L${PREFIX}/lib -Wl,-rpath -Wl,${PREFIX}/lib -L${PREFIX}/X11/lib -Wl,-rpath -Wl,${PREFIX}/lib/X11"

where *$PREFIX* is where we install X.org, say */opt/xbeta*,
the first two exports of *ACLOCAL* and *PKG_CONFIG_PATH* occur for most X.org projects,
*CPPFLAGS* and *LDFLAGS* ensure that the *$PREFIX* include directories and library paths
are used, and *-Wl,-rpath ...* is an incantation that ensures that previous built
*libtool* libraries can be used.

### libXfont

*autogen.sh* script seems to build easier with
    --without-xmlto
option.

### Mesa

As of 2010-05-09, known problem with GNU m4 pkgsrc on DragonFly

    /usr/pkg/bin/gm4: m4_esyscmd subprocess failed: Operation not permitted
    /usr/pkg/bin/gm4:configure.ac:8: cannot run command `${MAKE-make} -s -f bin/version.mk version | tr -d '\n'': Operation not permitted
    configure.ac:8: error: Failed to get the Mesa version from `make -f bin/version.mk version`

For now just manually run command
    $ make -f bin/version.mk version
    7.9.0
and substitute it using a patch similar to

    diff --git a/configure.ac b/configure.ac
    index c40c842..224a858 100644
    --- a/configure.ac
    +++ b/configure.ac
    @@ -4,7 +4,7 @@ AC_PREREQ([2.59])
 
     dnl Versioning - scrape the version from configs/default
     m4_define([mesa_version],
    -    [m4_esyscmd([${MAKE-make} -s -f bin/version.mk version | tr -d '\n'])])
    +    [7.9.0])])
     m4_ifval(mesa_version,,
         [m4_fatal([Failed to get the Mesa version from `make -f bin/version.mk version`])])

#### C99 fpclassify()

    diff --git a/src/mesa/main/querymatrix.c b/src/mesa/main/querymatrix.c
    index ca292aa..a0969f6 100644
    --- a/src/mesa/main/querymatrix.c
    +++ b/src/mesa/main/querymatrix.c
    @@ -71,7 +71,7 @@ fpclassify(double x)
     }

     #elif defined(__APPLE__) || defined(__CYGWIN__) || defined(__FreeBSD__) || \
    -     (defined(__sun) && defined(__C99FEATURES__))
    +     defined(__DragonFly__) || (defined(__sun) && defined(__C99FEATURES__))

     /* fpclassify is available. */

### xserver

If one uses the patches alluded to in the pkgsrc section, one can avoid the following build error:

      YACC   parser.c
    bison: m4 subprocess failed: Operation not permitted
    gmake[3]: *** [parser.c] Error 1

Developer documentation appears to now be enabled by default; therefore, if one is bootstrapping
a new tree, one might want to use the option to *autogen.sh* to not build developer documentation:

I experienced the following build error:

      GEN    Xserver-spec.txt
    No way to convert HTML to text found.

Corrected using the *autogen.sh* flag:

    --disable-devel-docs

### xinit

    diff --git a/xinit.c b/xinit.c
    index 313806e..0d31637 100644
    --- a/xinit.c
    +++ b/xinit.c
    @@ -48,6 +48,12 @@ in this Software without prior written authorization from The Open Group.
     #endif
     #endif
 
    +/* For PRIO_PROCESS and setpriority() */
    +#ifdef __DragonFly__
    +#include <sys/time.h>
    +#include <sys/resource.h>
    +#endif /* __DragonFly__ */
    +
     #include <stdlib.h>

     #ifndef SHELL

## Building a Separate Version of GNOME

### glib

See the bug report at <https://bugzilla.gnome.org/show_bug.cgi?id=618754> for an error similar to

    /usr/libexec/binutils217/elf/ld: .libs/gthread.o: relocation R_X86_64_PC32
    against `_g_mem_thread_init_noprivate_nomessage' can not be used when making a
    shared object; recompile with -fPIC
    /usr/libexec/binutils217/elf/ld: final link failed: Bad value

This bug is especially a problem because it may be related to some interaction with gcc 4.1.2 only on DragonFly.

The following patch is a complete hack:

    diff --git a/configure.in b/configure.in
    index 38288c3..51a1c07 100644
    --- a/configure.in
    +++ b/configure.in
    @@ -3033,7 +3033,7 @@ _______EOF
     #define G_GNUC_INTERNAL __attribute__((visibility("hidden")))
     #elif defined(__SUNPRO_C) && (__SUNPRO_C >= 0x550)
     #define G_GNUC_INTERNAL __hidden
    -#elif defined (__GNUC__) && defined (G_HAVE_GNUC_VISIBILITY)
    +#elif defined (__GNUC__) && defined (G_HAVE_GNUC_VISIBILITY) && !defined(__DragonFly__)
     #define G_GNUC_INTERNAL __attribute__((visibility("hidden")))
     #else
     #define G_GNUC_INTERNAL

### Graphics Image Formats

*libpng* <http://www.libpng.org/pub/png/libpng.html> version *1.4.2*
can be installed using *autogen.sh* and then *configure*, in contrast to
the X.org projects that pass parameters from *autogen.sh* to run *configure* automatically.

Independent JPEG Group's library <http://www.ijg.org/> version *8b* can be installed
using *configure* alone (there is no autogen.sh).

LibTIFF <http://www.remotesensing.org/libtiff/> has a recommended dependency on jpeg.
LibTIFF version 3.9.2 can be installed using *autogen.sh* and then *configure*.

Running
    gmake check
after building any of the above three projects appears to produce passes tests.

### XML

#### libxml2

*autogen.sh* calls *configure*

11 tests fail after *gmake check*

### Cairo

Seems to require

    export LDFLAGS="-L${PREFIX}/lib -Wl,-rpath -Wl,${PREFIX}/lib -L${PREFIX}/lib/X11 -Wl,-rpath -Wl,${PREFIX}/lib/X11"
    export CPPFLAGS="-I${PREFIX}/include -I${PREFIX}/X11/include"

When using gcc 4.1.2, *gmake check* gives:

      LINK  check-link
    ./.libs/libcairo.so: undefined reference to `__umodti3'
    ./.libs/libcairo.so: undefined reference to `__udivti3'

When using *CCVER=gcc44* to force gcc 4.4, one obtains the worse:

      CHECK cairo.h
    cc1: error: unrecognized command line option "-Wlogical-op"
      CHECK cairo-deprecated.h
    cc1: error: unrecognized command line option "-Wlogical-op"
    ...

The problem is 128-bit division using gcc 4.1.2 as discussed earlier.  One can examine
*src/cairo-wideint.c* and see the telltale code enclosed in an if test for *HAVE_UINT128_T*
with 128-bit division if defined.

### Pango

Install *gnome-common* first, required.

If Cairo is not built with *CCVER=gcc44*, obtain build error similar to

      CCLD   pango-view
    /opt/xcatch/lib/libcairo.so: undefined reference to `__umodti3'
    /opt/xcatch/lib/libcairo.so: undefined reference to `__udivti3'
    gmake[3]: *** [pango-view] Error 1

Following patch corrects an API change not propagated to the test for *gmake check*
<https://bugzilla.gnome.org/show_bug.cgi?id=619456>

Not including patch itself because there are tabs.  But edit file *pango/pangoft2.def*
and change
    pango_fc_font_create_metrics_for_context
to
    pango_fc_font_create_base_metrics_for_context

### GObject Introspection

*cElementTree* is changed to *ElementTree* in files
*giscanner/girparser.py*, *giscanner/glibtransformer.py*, and *giscanner/scannermain.py*.

Using *pkg_alternatives* for a python wrapper fails as a script is generated with

    #! /usr/pkg/bin/python

    import os
    import sys

A shell script cannot use another shell script as its interpreter.
What works for now is to manually create */usr/pkg/bin/python* as a symlink to */usr/pkg/bin/python2.6*.

### gtk+

*gobject-introspection* is listed as a dependency but can be avoided.
Use option *--disable-introspection* to avoid an error

    checking for gobject-introspection... yes
    ./configure: ${INTROSPECTION_GIRDIR/...}: Bad substitution