(no commit message)

[ikiwiki.git] / docs / howtos / HowToUpdateChromium / index.mdwn

**NOTE: This page is work in progress**

## Introduction

[[Chromium|https://www.chromium.org/]] is a massive piece of software that includes a number of third-party programs and libraries at a specific version, probably to have a controlled environments for builds and runtime.

The following "channels" are supported currently, among which there is no BSD so custom patches are needed. See [[Chromium channels|https://www.chromium.org/getting-involved/dev-channel]]

The chromium port is called "www/chromium/" in FreeBSD ports and so it is in DPorts but, as of the time of this writing, the DPorts version is slightly different since we require changes to the FreeBSD Makefile we cannot do in our overlay (Makefile.DragonFly)

## Third-party software

### Compiler, linker and standard C++ library

Chromium uses LLVM's clang to build and lld for the linkage phase. It also uses [[libc++|https://libcxx.llvm.org/]] as standard C++ library. They use a very specific version of these tools which can be obtained with their own scripts and utilities.

For more information on how to build Chromium from source for other platforms see [[here|https://chromium.googlesource.com/chromium/src/+/lkcr/docs/clang.md]].

### List of third-party software

The list of third-party software is a long one, you can check it [[here|https://cs.chromium.org/chromium/src/third_party/]].

## Current update procedure

The outline is as follows:

1. Import the FreeBSD ports' specific Chromium version to a new empty branch.
2. Create a new branch called _'xx-freebsd'_ based on the branch created in step 1.
3. Import all patches from FreeBSD port _www/chromium_ to the _'xx-freebsd'_ branch and commit it one commit per file.
4. Create a new branch called _'xx-dfly'_ based on the branch _'xx-freebsd'_ after step 3, that is, with all patches included.
5. Cherry-pick the commit(s) needed from the previous 'xx-dfly' branch, resolve conflicts.
6. Adapt the FreeBSD port www/chromium, starting from the changes in DPorts' port _www/chromium_.
7. Make sure your custom _www/chromium_ builds.
8. Once ready, copy the files to the right place and send a PR to [[DeltaPorts|https://github.com/DragonFlyBSD/DeltaPorts/pulls]].

This is how we are performing the Chromium updates right now, nothing is written in stone, contributions/ideas are very welcome :-)

### Repository location and contributions

The "DragonFly Chromium" repository is located in server **crater**, where all the source repositories are located. There is also a clone of them in server **leaf**. You can access the [[dragonfly's gitweb page|https://gitweb.dragonflybsd.org/]] to browse the code without the need of cloning it. The repository is called _chromium-dfly.git_.

We recommend cloning from crater just in case the mirroring on leaf fails for some reason:

    
    # git clone git://crater.dragonflybsd.org/chromium-dfly.git
    Cloning into 'chromium-dfly'...
    remote: Counting objects: 469609, done.
    remote: Compressing objects: 100% (254827/254827), done.
    [....]
    

Only DragonFly committers can push to this repository in crater, you can specify a different url for pushing so that the SSH keys is only used for _git-push_, see below an example of _.git/config_:

    
    [remote "crater"]                                                                                                    
            url = git://crater.dragonflybsd.org/chromium-dfly.git
            pushurl = ssh://crater.dragonflybsd.org/repository/git/chromium-dfly.git
            fetch = +refs/heads/*:refs/remotes/crater/*
    

Anyone can submit patches, just go to [[our bug tracker and send an issue for the tracker "Submit"|https://bugs.dragonflybsd.org/projects/dragonfly/issues/new]] (category 'chromium').

### Repository layout

Vanilla Chromium versions are imported from tarballs to branches named like "chromium-[exact-version]", see below the current ones:


    # git branch -a | fgrep crater/chromium
      remotes/crater/chromium-60.0.3112.113
      remotes/crater/chromium-60.0.3112.116
      remotes/crater/chromium-67.0.3396.87
      remotes/crater/chromium-68.0.3440.106

The FreeBSD branches are named like: _[major chromium version]-freebsd_  . In case there are several Chromium branches with the same major version, then the last part of the version is used, see below the current ones:


    # git branch -a | grep freebsd |grep crater
      remotes/crater/60.113-freebsd
      remotes/crater/67-freebsd

In the above the branch _"60.113-freebsd"_ is based on the vanilla chromium branch _"chromium-60.0.3112.113"_ . As for _"chromium-67.0.3396.87"_ there is only one chromium-67 so the FreeBSD branch is called "67-freebsd".

The DragonFly branches are named like: _[major chromium version]-dfly_ . In case there are several Chromium branches with the same major version the same as for FreeBSD applies. For DragonFly there might be user-specific branches. See below the current ones:


    # git branch -a | grep dfly |grep crater
      remotes/crater/60-dfly
      remotes/crater/67-dfly

### Importing a new Chromium version

Normally we would import the next Chromium version that is in FreeBSD ports. In this example we're going to use version _68.0.3440.106_. The download URL is: [[https://commondatastorage.googleapis.com/chromium-browser-official/chromium-68.0.3440.106.tar.xz|https://commondatastorage.googleapis.com/chromium-browser-official/chromium-68.0.3440.106.tar.xz]].

* Create a new branch based on a previous one for the import, remove all its contents and amend the commit message:

        # cd chromium-dfly
        # git checkout -b chromium-68.0.3440.106 crater/chromium-60.0.3112.113
        # rm -fr *
        # rm .*
        # git commit -a --amend --allow-empty -m 'Initial import chromium-68.0.3440.106'

* Import the tarball to your newly created branch:

        # tar --strip-components 1 --xzf /usr/distfiles/chromium-68.0.3440.106.tar.xz
        # git add -f .
        # git commit --amend -m 'Initial import chromium-68.0.3440.106'

* Push the new branch to crater (if you are a DragonFly developer) and maybe prune old branches if they are older than 4 releases:

        # git push crater chromium-68.0.3440.106

### Creating the '-freebsd' branch

This requires either a checkout of FreeBSD ports or at least the www/chromium directory from it. **Make sure you are working with the correct Chromium version!**

    # fgrep PORTVERSION ~/s/freebsd-ports/www/chromium/Makefile
    PORTVERSION=    68.0.3440.106

Now create a branch for FreeBSD:

    # cd chromium-dfly
    # git checkout -b 68-freebsd crater/chromium-68.0.3440.106

Apply all FreeBSD patches to your new branch. Make sure all patches apply cleanly, there should be no .rej files and all .orig files should be removed. If there are any .orig files originally in the Chromium repo you can either decide to remove them or restore them:

    # git rev-parse --abbrev-ref HEAD
    68-freebsd
    # sh -c 'for p in ~/s/freebsd-ports/www/chromium/patch*; do patch -p0 < $p; done'
    # find . -iname "*.rej"
    # find . -iname "*.orig" -delete

Now all FreeBSD patches should be in, make sure the 'git status' command shows a sane list, that there is no files left behind, etc. Also 'git status --ignored' and 'git ls-files -mo' should have the same amount of iles:

    # git status --ignored | less
    # git ls-files -mo |wc -l                                               
     609
    # git status --ignored --porcelain | wc -l
     609

All changes should be committed with one rule: one file per commit, since it makes merges more granular and there are potentially a lot of merges to be done. Please note that the command below will take a while since the repo is quite big:

    # sh -c 'for f in $(git ls-files -mo); do git add $f ; git commit -m "$f"; done'
    #

Once you're sure all patches have been applied correctly and that the commits contain exactly one file (they should if you ran the command above) you can push the freebsd branch to crater:

    # git push --set-upstream crater 68-freebsd
    Enumerating objects: 4672, done.
    Counting objects: 100% (4672/4672), done.
    [...]
     * [new branch]            68-freebsd -> 68-freebsd
    Branch '68-freebsd' set up to track remote branch '68-freebsd' from 'crater'.

### Creating the 'dfly' branch

Once the _'-freebsd'_ branch is created with all the FreeBSD patches for the specific Chromium version on it, you can create the _'-dfly'_ branch from it.

        
    # git checkout -b 68-dfly 68-freebsd
    

### Cherry-picking changes from the previous version

At this point you'll have a _'-dfly'_ branch which is exactly the same as the _'-freebsd'_ one. Next step is to cherry pick the changes from the latest working Chromium, in our case _'67-dfly'_, commit:

    commit 317fe2dbcdc3946787ed8de610e66f2829fc4ffb (67-dfly)
    Author: Antonio Huete Jimenez <tuxillo@quantumachine.net>
    Date:   Mon Aug 20 05:04:23 2018 -0700
    
        Changes to be applied on top of 67-freebsd

Just cherry-pick it:

    # git cherry-pick 317fe2dbcdc3946787ed8de610e66f2829fc4ffb

Most likely there'll be conflicts to resolve. All of them need to be resolved before issuing a 'git cherry-pick --continue'. For now we are keeping all the changes to be applied on top of the '-freebsd' branch in one commit so there shouldn't be more commits to apply.

### Building

Here it is described how I have personally done the test build, you don't need to do it the same way but there are some things definitely needed.

The port definition for the latest Dports' Chromium version is [[here|https://github.com/DragonFlyBSD/DPorts/tree/master/www/chromium]]. The port definition for the latest FreeBDS ports' Chromium version is [[here|https://github.com/freebsd/freebsd-ports/tree/master/www/chromium]]. The preference is to always go with the FreeBSD latest changes since most of the work is done based on FreeBSD's patches but there must be things kept from the Dports port definition because they'll be DragonFly specific.

For example, there is a conditional around the compiler selection which allows the FreeBSD port to be build with either base clang or the port clang.

For the FreeBSD port it looks like:

    .if ${OPSYS} == FreeBSD && ${OSVERSION} < 1101513
    BUILD_DEPENDS+= clang60:devel/llvm60
    CC=             clang60
    CXX=            clang++60
    EXTRA_PATCHES+= ${FILESDIR}/extra-patch-llvm-port
    .else
    BUILD_DEPENDS+= ${LOCALBASE}/bin/ar:devel/binutils
    EXTRA_PATCHES+= ${FILESDIR}/extra-patch-llvm-base
    .endif

Since the DragonFly port is always built with DPort's clang it should look something like:

    BUILD_DEPENDS+= ${LOCALBASE}/lib/c++/libstdc++.so:devel/libc++
    BUILD_DEPENDS+= clang60:devel/llvm60
    CC=             clang60
    CXX=            clang++60
    EXTRA_PATCHES+= ${FILESDIR}/extra-patch-llvm-port

Note that there is an extra dependency from DPorts: _devel/libc++_ , which will be used as the standard C++ library instead of the system base which is libstdc++.

As of now there are two important differences from the FreeBSD port:

Allow 'tools/gn' to be compiled and linked against libc++:

    do-configure:
            # GN generator bootstrapping and generating ninja files
            cd ${WRKSRC} && ${SETENV} ${CONFIGURE_ENV} CC=${CC} CXX=${CXX} LD=${CXX} \
                    READELF=${READELF} AR=${AR} NM=${NM} CXXFLAGS="-I${LOCALBASE}/include/c++/v1" \
                    LDFLAGS="-L${LOCALBASE}/lib/c++" ${PYTHON_CMD} \
                    ./tools/gn/bootstrap/bootstrap.py ${GN_BOOTSTRAP_FLAGS}

Pass extra_cxxflags and extra_ldflags for Chromium to be compiled/linked against devel/libc++:

    ## Appendix# TODO bz@ : install libwidevinecdm.so (see third_party/widevine/cdm/BUILD.gn)
    #
    # Run "./out/${BUILDTYPE}/gn args out/${BUILDTYPE} --list" for all variables.
    # Some parts don't have use_system_* flag, and can be turned on/off by using
    # replace_gn_files.py script, some parts just turned on/off for target host
    # OS "target_os == is_bsd", like libusb, libpci.
    GN_ARGS+=       clang_use_chrome_plugins=false \
                    enable_nacl=false \
                    enable_one_click_signin=true \
                    enable_remoting=false \
                    fieldtrial_testing_like_official_build=true \
                    is_clang=true \
                    toolkit_views=true \
                    treat_warnings_as_errors=false \
                    use_allocator="none" \
                    use_allocator_shim=false \
                    use_aura=true \
                    use_bundled_fontconfig=false \
                    use_custom_libcxx=false \
                    use_gnome_keyring=false \
                    use_gtk3=true \
                    use_lld=true \
                    use_sysroot=false \
                    use_system_freetype=true \
                    use_system_harfbuzz=true \
                    use_system_libjpeg=true \
                    extra_cxxflags="-stdlib=libc++ -cxx-isystem ${LOCALBASE}/include/c++/v1 -I${LOCALBASE}/include" \
                    extra_ldflags="-L${LOCALBASE}/lib/c++ -L${LOCALBASE}/lib -L /usr/lib -L /lib"


Once the DPorts port definition, what I usually do is to copy it to: /usr/dports/www/chromiumXX (in this case XX = 68) and build from there.

There are two little scripts you can use to make generating the patches easier but make sure you adjust the paths. For running those scripts the current working directory must be the git repository where you are working on.

The patch below removes all the patch-* from the port specified and generates a patch-all file which is enough for the build:

    #!/bin/csh
    #
    
    set nonomatch
    rm -f /usr/dports/www/chromium68/files/patch-*
    git diff --no-prefix crater/chromium-68.0.3440.106 . > /usr/dports/www/chromium68/files/patch-all

Once you've extracted chromium and kicked off the build, you'll likely find errors very early. To avoid having to extract over and over to test your patches the script below simplifies the task by detecting what files need to be changed in the chromium's OBJDIR and copies them over, so you can restart the build directly:

    #! /bin/sh
    
    OBJDIR=/usr/obj/dports/www/chromium68/chromium-68.0.3440.106/
    
    for f in $(git ls-files -m)
    do
        if diff -q $f $OBJDIR/$f >/dev/null; then
            echo $f does not need repatch
        else
            cp -fv $f $OBJDIR/$f
        fi
    done

To kick-off the build, it can be done as for any other port:

    # cd /usr/dports/www/chromium68
    # make | tee /tmp/build.out

I recommend sending the output to a log file where you can inspect it more easily.

Once the build finishes successfully you can check the _chrome_ binary in 'out/Release/' in the OBJDIR and see if it runs properly. If it does, please check point above *"Repository location and contributions"* on how to submit your work. Thanks!

## Appendix

### Tools

The Chromium project offers a cross-reference tool which is extremely useful when working on Chromium source code: [[Chromium Search Code|https://cs.chromium.org/]]


### Support

You can contact me (tuxillo) and others in our mailing list (users@) or by IRC in EFnet #dragonflybsd for questions.