X-Git-Url: https://gitweb.dragonflybsd.org/ikiwiki.git/blobdiff_plain/7666fbf29cd5e95f4ba0faed200c5f901d1db0ad..HEAD:/docs/howtos/HowToDPorts/index.mdwn diff --git a/docs/howtos/HowToDPorts/index.mdwn b/docs/howtos/HowToDPorts/index.mdwn index cd051f7b..cbc5a134 100644 --- a/docs/howtos/HowToDPorts/index.mdwn +++ b/docs/howtos/HowToDPorts/index.mdwn @@ -1,95 +1,676 @@ -## DPorts background: +[[!toc levels=3]] -Dports is an effort to use FreeBSD’s ports system as a base for DragonFly, and the pkg tool as a way to manage binary packages built from DPorts. This is complicated, so I’ll explain each part in order. +# DPorts and pkg(8) -FreeBSD ports are a FreeBSD-specific collection of software installation files that automate building 3rd-party software on FreeBSD. You’ve probably already heard of them. (Note there’s no mention of DragonFly.) +DPorts is DragonFly's own third-party software build system. It is based on FreeBSD's Ports Collection. Differences between ports and DPorts are intentionally kept to a minimum, both to maintain familiarity for mutual users of both operating systems and also to leverage the tremendous amount of work the FreeBSD contributors put into ports. DPorts can and does feature ports unique to DragonFly, so it's truly a native system. -DPorts is a collection of files that map to existing FreeBSD ports, and contain any changes necessary to make that port also build on DragonFly. Many of those programs build without changes on DragonFly. DPorts builds from source. +The __pkg(8)__ tool called "pkg" is a modern and fast binary package manager. It was developed for FreeBSD, but PC-BSD used it in production first, followed soon after by DragonFly. FreeBSD has since removed their legacy tools so __pkg(8)__ is exclusively used by FreeBSD as well. -pkg is used for package management, and is usable on FreeBSD, PC-BSD, and on DragonFly. The binary packages produced from building with DPorts can be installed from remote locations and managed separately using pkg, so that software upgrades and installation can be performed with binaries only. (It’s much faster that way.) +__pkg(8)__ is not a replacement for port management tools like `ports-mgmt/portmaster` or `ports-mgmt/portupgrade`. While `ports-mgmt/portmaster` and `ports-mgmt/portupgrade` can install third-party software from both binary packages and DPorts, __pkg(8)__ installs only binary packages. On the other hand, many people use __pkg(8)__ by itself and never install the optional portupgrade or portmaster tools. -Every port seen in DPorts is known to build on DragonFly. A port is added only after it builds successfully, using poudriere as a bulk software tool. Ports are only updated to a newer version when that newer version builds, too, so once something arrives in DPorts, it should never break from being updated at some point in the future. +## Getting started with pkg(8) -## Most people want pre-built binary packages rather than building from source. +DragonFly daily snapshots and Releases (starting with 3.4) come with __pkg(8)__ already installed. Upgrades from earlier releases, however, will not have it. If the "pkg" program is missing on the system for any reason, it can be quickly bootstrapped without having to build it from source or even having dports installed: -The 3.4 Release of DragonFly is the first to provide official repositories. It also provides pkg and pkg.conf preinstalled with the release repository pre-selected. + # cd /usr + # make pkg-bootstrap + # rehash + # pkg-static install -y pkg + # rehash -The /usr/local/etc/pkg.conf is predefined as: +After any installation or reinstallation of the __pkg(8)__ package, you may want to open review the configuration files to customize it, e.g. select a different mirror. - PACKAGESITE: http://avalon.dragonflybsd.org/dports/${ABI}/RELEASE +## Configuring pkg(8) -The user would first run "pkg update" to download the inventory from the PACKAGES. From there the user can browse the repository, install packages, etc. "pkg help" and "man pkg" has a wealth of information and much of the FreeBSD handbook is pertinent: [5.5 Using pkgng for Binary Package Management](http://www.freebsd.org/doc/en/books/handbook/pkgng-intro.html). +The __pkg(8)__ program has a configuration file at `/usr/local/etc/pkg.conf` (the default installation is typically fine and requires no alteration). It uses a a separate location for repository configuration at `/usr/local/etc/pkg/repos/` directory. -The RELEASE repository is static, but the LATEST repository is periodically updated. If bleeding edge is what you want, change pkg.conf to reflect that: +For fresh installations, the file `/usr/local/etc/pkg/repos/df-latest.conf.sample` is copied to `/usr/local/etc/pkg/repos/df-latest.conf` so that __pkg(8)__ works out of the box. The files ending in the `.sample` extension are ignored; __pkg(8)__ only reads files that end in `.conf` and it will read as many as it finds. - PACKAGESITE: http://avalon.dragonflybsd.org/dports/${ABI}/LATEST +The default is to use the California-based __Avalon__, which is also the master. Avalon will have new packages first and they'll be updated as a set. The others are mirrors and may be much, much faster depending on the user's location, but there will be a period between set updates where the mirror is mix of old and new packages. -DragonFly 3.5 users can also use pre-build binary packages, but they will need to override the ABI in pkg.conf: +The simple case is to just use the **AUTO** repository (see also the [Mirrors](/mirrors/) page): - ABI: dragonfly:3.4:x86:32 # for i386 platform - ABI: dragonfly:3.4:x86:64 # for x86-64 platform + AUTO: { + url : https://pkg.dragonflybsd.org/pkg/${ABI}/LATEST + mirror_type : HTTP + enabled : yes + } -There are already a few mirrors available which can be set in pkg.conf + Avalon: { + [...] + enabled : no + } -* North America: http://avalon.dragonflybsd.org/dports/${ABI}/LATEST -* Europe: http://pkg.wolfpond.org/${ABI}/LATEST -* Europe: http://mirrors.ircam.fr/pub/DragonFlyBSD-dports/${ABI}/LATEST -* Europe: http://dfly.schlundtech.de/dports/${ABI}/LATEST +(The default **Avalon** rep can be optionally disabled.) -## Building DPorts from source +The provided `df-latest.conf` may not be up to date. There may be defunct repositories listed and there may be active mirrors that are not listed as options. You may edit or add new repo configuration files as necessary. + +It is possible to use multiple repositories. The normal use case is that someone builds specific packages (e.g. using __dsynth(8)__) which should have a higher priority over the official packages. In other words, whenever a package has been built locally, __pkg(8)__ installs the local version and only uses the external repository to install packages not available locally. + +In the above case, a custom repository configuration file should be created, and it should include the `priority` field, e.g. `priority: 10`. The higher the number, the higher the priority. The priority when unspecified is 0. + +The last output of `pkg -vv` lists all enable repositories and their priority values. + +Before using, consult the man page (`man pkg`) and then try these examples: + + # pkg search editors + # pkg install vim + +## Basic pkg(8) Operations + +Usage information for __pkg(8)__ is available in the pkg(8) manual page, or by running `pkg` without additional arguments. + +Each __pkg(8)__ command argument is documented in a command-specific manual page. To read the manual page for `pkg install`, for example, run either: + + # pkg help install + # man pkg-install + +## Obtaining Information About Installed Packages with pkg(8) + +Information about the packages installed on a system can be viewed by running `pkg info`. Similar to pkg_info(1), the package version and description for all packages will be listed. Information about a specific package is available by running: + + # pkg info packagename + +For example, to see which version of __pkg(8)__ is installed on the system, run: + + # pkg info pkg + pkg-1.0.12 New generation package manager + +## Installing and Removing Packages with pkg(8) + +In general, most DragonFly users will install binary packages by typing: + + # pkg install + +For example, to install curl: + + # pkg install curl + + Updating repository catalogue + Repository catalogue is up-to-date, no need to fetch fresh copy + The following packages will be installed: + + Installing ca_root_nss: 3.13.5 + Installing curl: 7.24.0 + + The installation will require 4 MB more space + + 1 MB to be downloaded + + Proceed with installing packages [y/N]: y + ca_root_nss-3.13.5.txz 100% 255KB 255.1KB/s 255.1KB/s 00:00 + curl-7.24.0.txz 100% 1108KB 1.1MB/s 1.1MB/s 00:00 + Checking integrity... done + Installing ca_root_nss-3.13.5... done + Installing curl-7.24.0... done + +The new package and any additional packages that were installed as dependencies can be seen in the installed packages list: + + # pkg info + ca_root_nss-3.13.5 The root certificate bundle from the Mozilla Project + curl-7.24.0 Non-interactive tool to get files from FTP, GOPHER, HTTP(S) servers + pkg-1.0.12 New generation package manager + +Packages that are no longer needed can be removed with `pkg delete`. For example, if it turns out that curl is not needed after all: + + # pkg delete curl + The following packages will be deleted: + + curl-7.24.0_1 + + The deletion will free 3 MB + + Proceed with deleting packages [y/N]: y + Deleting curl-7.24.0_1... done + +## Upgrading Installed Packages with pkg(8) + +Packages that are outdated can be found with `pkg version`. If a local ports tree does not exist, pkg-version(8) will use the remote repository catalogue, otherwise the local ports tree will be used to identify package versions. + +Packages can be upgraded to newer versions with __pkg(8)__. Suppose a new version of curl has been released. The local package can be upgraded to the new version: + + # pkg upgrade + Updating repository catalogue + repo.txz 100% 297KB 296.5KB/s 296.5KB/s 00:00 + The following packages will be upgraded: + + Upgrading curl: 7.24.0 -> 7.24.0_1 + + 1 MB to be downloaded + + Proceed with upgrading packages [y/N]: y + curl-7.24.0_1.txz 100% 1108KB 1.1MB/s 1.1MB/s 00:00 + Checking integrity... done + Upgrading curl from 7.24.0 to 7.24.0_1... done + +## Auditing Installed Packages with pkg(8) + +Occasionally, software vulnerabilities may be discovered in software within DPorts. __pkg(8)__ includes built-in auditing. To audit the software installed on the system, type: + + # pkg audit -F + +# Advanced pkg(8) Operations + +## Automatically Removing Leaf Dependencies with pkg(8) + +Removing a package may leave behind unnecessary dependencies, like `security/ca_root_nss` in the example above. Such packages are still installed, but nothing depends on them any more. Unneeded packages that were installed as dependencies can be automatically detected and removed: + + # pkg autoremove + Packages to be autoremoved: + ca_root_nss-3.13.5 + + The autoremoval will free 723 kB + + Proceed with autoremoval of packages [y/N]: y + Deinstalling ca_root_nss-3.13.5... done + +## Backing Up the pkg(8) Package Database + +__pkg(8)__ includes its own package database backup mechanism. To manually back up the package database contents, type: + + # pkg backup -d + +Additionally, __pkg(8)__ includes a periodic(8) script to automatically back up the package database daily if `daily_backup_pkgng_enable` is set to `YES` in periodic.conf(5). To prevent the `pkg_install` periodic script from also backing up the package database, set `daily_backup_pkgdb_enable` to `NO` in periodic.conf(5). + +To restore the contents of a previous package database backup, run: + + # pkg backup -r + +## Removing Stale pkg(8) Packages + +By default, __pkg(8)__ stores binary packages in a cache directory as defined by `PKG_CACHEDIR` in pkg.conf(5). When upgrading packages with pkg upgrade, old versions of the upgraded packages are not automatically removed. + +To remove the outdated binary packages, type: + + # pkg clean + +##Modifying pkg(8) Package Metadata + +__pkg(8)__ has a couple of built-in commands to update package origins. + +_Note_: Previously it was recommended to use the -o option, e.g. "-o < category / oldport >:< category / newport >" but this option is deprecated and will be removed in the future. + +_Note_: The --change-name (-n) option requires the package name. Normally it's intuitive based on the name of the port, but not always. To get the definitive name, the following command can be used: + + # make -V PKGBASE -C /usr/dports// + +For example, if `lang/php5` was originally at version 5.3, but has been renamed to lang/php53 for the inclusion of version 5.4, the package database can be updated to deal with this. For __pkg(8)__, the syntax is: + + # pkg set -n : + +For example, to change the package origin for the above example, type: + + # pkg set -n php5:php53 + +As another example, to update lang/ruby20 to lang/ruby21 (the current default), type: + + # pkg set -n ruby20:ruby + +As a final example, to change the origin of the libglut shared libraries from graphics/libglut to graphics/freeglut, type: + + # pkg set -n libglut:freeglut + +_Note_: When changing package origins, in most cases it is important to reinstall packages that are dependent on the package that has had the origin changed. To force a reinstallation of dependent packages, type: + + # pkg install -Rf graphics/freeglut + + +# Bullet-proof (conflict-proof) upgrade technique + +Several people have run into problem upgrading binary packages when default versions change (e.g. PHP default changes from version 5.4 to 5.6, or PostgreSQL default changes from 9.0 to 9.2). Usually there are upgrading hints at /usr/dports/UPDATING, but there are issues with this method of communication + + 1. You must have /usr/dports installed and current, which kind of defeats the purpose of using binary packages + 2. You usually don't go looking there until after you hit a failure + 3. Proactively reading /usr/dports/UPDATING perhaps from [DragonFly's Gitweb](http://gitweb.dragonflybsd.org/dports.git/blob/HEAD:/UPDATING) would be the best approach, but honestly nobody is going to do that. + +Additionally, over time Administrators install packages to evaluate them or otherwise don't need them any longer, but often the Administrators forget to remove the package and its dependencies later. These extra packages still get upgraded unnecessarily. + +## Technique Instructions + +The following technique will guarantee conflict-proof, integral updates and at the same time allow manually pruning of unwanted packages. + +### Step 1: Get list of primary packages + +For __pkg(8)__ version 1.5.1 or later, the "prime-list" alias is predefined in the pre-installed pkg.conf, so you can run the following command: + + > pkg prime-list > ~/prime.list + +If you are using an older version of package, or already had pkg.conf which was not overwritten, you can use the equivalent of the alias like so: + + > pkg query -e "%a = 0" "%n" > ~/prime.list + +The file ~/prime.list contains the list of packages that were manually installed. For example, if an Administrator only requested one package to be installed, Libreoffice, __pkg(8)__ might pull in 200 dependencies, but ~/prime.list would only contain the name of a single package. + +### Step 2: Review and edit primary package list + +Using your favorite editor, load ~/prime.list, which indicates which packages were manually installed. If there are any packages that are no longer desired, simple delete the line that contains it. + +### Step 3: Wipe everything out + +Now everything except package needs to be removed off the system. + + > pkg delete -ay + +If you are using an HTTPS mirror you'll get an error similar to this one: + + > Certificate verification failed for /C=US/O=Let's Encrypt/CN=Let's + > Encrypt Authority X3 + > 34371318292:error:14007086:SSL routines:CONNECT_CR_CERT:certificate verify + > failed:/usr/src/lib/libressl/../../crypto/libressl/ssl/ssl_clnt.c:1121: + +To workaround it, simply add the following line to your /usr/local/etc/pkg.conf: + + > PKG_ENV { + > SSL_NO_VERIFY_PEER=1 + > } + +Don't forget to remove or comment them after the upgrade. + +### Step 4: Upgrade the repository category + +What we really want to do here is update the repository catalog, but if there's a newer version of __pkg(8)__, we can upgrade it at the same time. + + > pkg upgrade + +Say yes if __pkg(8)__ wants permission to update itself on the system. + +### Step 5: Cleanse the package cache + +To avoid any possible reinstallation of a stale package, we want to remove obsolete packages from the cache. + + > pkg clean -y + +### Step 6: Reinstall the primary packages + +Now we reinstall the packages that we really want on the system and let __pkg(8)__ pull in the dependencies that are needed without fear of conflict. + + > pkg install `cat ~/prime.list` + +Provide affirmative answers to any confirmations __pkg(8)__ asks. + +## How to create the _prime-list_ alias for future use + +Rather than try to remember that hard-to-type query, it's more convenient to define this command as an alias. Note this is not necessary for new installations since around 22 April 2015 (The alias is predefined for __pkg(8)__ version 1.5.1 and later). To defined the alias, create or append the */usr/local/etc/pkg.conf* file with the following: + + ALIAS : { + prime-list: query -e "%a = 0" "%n", + } + +If ALIAS is already defined, just add the prime-list line to the list. You may review */usr/local/etc/pkg.conf.sample* for other ALIAS suggestions. There are several useful ones offered there. + +# Building DPorts from source The average user will probably not build packages from source. However, it's easy to do and it can be done even when packages have already been pre-installed on the system. Common reasons to build from source are: * The port is new and there's no pre-binary available yet * The pre-built binaries use the default options and the user needs a package built with a different set of options -* Testing FreeBSD port in order to patch them and submit to dports +* Testing FreeBSD port in order to patch them and submit to DPorts * The user just prefers building from source -### Installing DPorts tree +## Installing DPorts tree + +DragonFly 3.4 or later is the minimum version that can build DPorts from source. + +It's probably that pkgsrc binaries are already installed because it comes bootstrapped with new systems. It is necessary to rename `/usr/pkg` directory so that the existing pkgsrc binary tools and libraries don’t get accidentally used while building DPorts, causing breakage. For the installation of the DPorts tree, type: + + # cd /usr + # make dports-create-shallow + +If the `/usr/pkg directory` has already been renamed, `git` won’t be in the search path any more. One option is to download a tarball of DPorts and unpack it. To do this, type: + + # cd /usr + # make dports-download + +For future updates, pull delta changes via `git` is fastest, so it is suggested to convert the static tree to a git repository by typing: + + # cd /usr/dports/devel/git + # make install + # cd /usr + # rm -rf /usr/dports + # make dports-create-shallow + +The git repository is hosted on the [github account of DragonFly](https://github.com/DragonFlyBSD/DPorts/#readme). + +## Final thoughts + +Building from source works similar to ports and pkgsrc: cd into the appropriate program's directory, and type 'make'. 'make install' to install the software, 'make clean' to clean up work files, and so on. Use 'make config-recursive' if you want to set all the port's options, and the options of its dependencies, immediately instead of during the build. + +To take all the default build options and avoid getting the pop-up dialog box, set `NO_DIALOG=yes` on either the command line or the make.conf file. + +If you just want to set the options for one package, and accept the default for all of its dependencies, do 'make config' in the package in you want non-default options, and then 'make NO_DIALOG=yes'. Note that this is only necessary if you want to build from source with a non-default set of options, or if no pre-built binary package is available yet. + +## More reading +* How fix/add broken ports: [[docs/howtos/fixdports]] +* [Trick: How to get i386-only software via dports](http://leaf.dragonflybsd.org/mailarchive/users/2013-06/msg00023.html) + +# DPorts contributor notes + +## Repositories + +* FreeBSD Ports: This is the FreeBSD Ports collection git repository which is generated from the SVN repository. +* DeltaPorts: This is an overlay which, when applied on top of the FreeBSD Ports collection with the correct scripts, creates the DPorts collection. +* DPorts: This is the DragonFly BSD Ports collection which is just a modified version of FreeBSD Ports. + +## Steps of every sync with FreeBSD Ports + +1. Around the 15th of each month a monthly branch is created in a local clone of the freebsd ports repository. This is taken as the base for the sync between freebsd ports and dports. +1. The transformation scripts are applied on each freebsd ports' port and a temporary dports tree (called potential tree) is generated. +1. In the staged build box (right now it's sting) a full build of the potential tree is built. Every package that builds will go to ports/staged branch and will also leave a trace of the build outcome in deltaports master branch. +1. Once that build is done, a snapshot of the current failures is taken (snap-failures) and the port build fixing process starts. + + +## Recommended way to fix DPorts failures + +### Setup the environment + +In this guide the base directory will be `/build/chroot`. + +Create a vanilla chroot installation in /build/chroot , that is from a DragonFly BSD source tree without modifications. If you need to build for RELEASE, use the release branch then. + + # export BASEDIR=/build/chroot + # cd /usr/src + # make buildworld + # make installworld DESTDIR=$BASEDIR + # cd etc && make distribution DESTDIR=$BASEDIR + +Clone the repos for freebsd ports, deltaports and dports. This 3 repositories should not be modified except for updates and such. + + # git clone https://github.com/freebsd/freebsd-ports.git $BASEDIR/root/s/freebsd-ports + # git clone https://github.com/DragonFlyBSD/DPorts.git $BASEDIR/root/s/DPorts + # git clone https://github.com/DragonFlyBSD/DeltaPorts.git $BASEDIR/root/s/DeltaPorts + +Copy /etc/resolv.conf to $BASEDIR/etc so that the chroot can have internet access. + +Clone the DPorts repo in $BASEDIR/usr/dports and switch to staged branch. The port fixes will be done here: + + # chroot $BASEDIR + # cd /usr && make dports create + # cd dports + # git checkout -b staged origin/staged + +Install your favourite editor and other utils in the chroot. You only have to run 'make pkg-bootstrap' the first time. + + # chroot $BASEDIR + # cd /usr && make pkg-bootstrap + # pkg install -y vim + # pkg install -y genpatch + +Setup dsynth according your computer parameters (cpu, mem, disk) and needs. + +For helping with the dports, one has to setup dsynth to use OpenSSL and not LibreSSL. For reference the following is the configuration used on sting.dragonflybsd.org. + + DPORTS_BUILDER= yes + DISABLE_LICENSES= yes + DEFAULT_VERSIONS= ssl=openssl + FORCE_PACKAGE= yes + DEVELOPER= yes + +Save the adapted dports configuration in $BASEDIR/usr/local/etc/dports.conf, it will be used by the sync scripts.Example contents: + + FPORTS=/build/chroot/root/s/freebsd-ports + MERGED=/tmp/dports + DPORTS=/build/chroot/root/s/DPorts + DELTA=/build/chroot/root/s/DeltaPorts + POTENTIAL=/build/chroot/root/s/potential + INDEX=${FPORTS}/INDEX-3 + COMQUEUE=/tmp/crondports + +Create script $BASEDIR/root/bin/getdport.sh + + #!/bin/sh + + . /usr/local/etc/dports.conf + + if [ -z "$1" ]; then + echo category/port to handle + exit 1 + fi + + [ -d /root/s/freebsd-ports/$1 ] || (echo $1 does not exist in ports && exit 1) + ${DELTA}/scripts/generator/sync1.sh $1 + [ -d /usr/dports/$1 ] && rm -fr /usr/dports/$1 + cp -r ${POTENTIAL}/$1 /usr/dports/$1 + cd /usr/dports/$1 + +### Check latest failures + +Latest failures can be checked in here: + + https://sting.dragonflybsd.org/dports/snap-failures/Report/ + +Normally you would sort the report data by Skip. Skip means the number of ports that were skipped during the build because of that port. + +### Fixing port failures + +Please note all the commands here are run while chrooted. + +Make sure FPORTS is at commit of the current sync. See the timestamp and commitid in this document: https://github.com/DragonFlyBSD/DeltaPorts/blob/master/docs/synclog.json + +For this guide we're going to pick `www/node12` which currently shows 7 skips. All actions will take place inside the chroot. If you have enough RAM+swap, you can probably mount tmpfs in /usr/obj within the chroot. + + # chroot $BASEDIR tcsh + # mount_tmpfs tmpfs /usr/obj + # /root/bin/getdport.sh www/node12 + # cd /usr/dports/www/node12 + +At this point you might want to verify that the changes from the getport.sh script were done correctly, for example with 'git diff .' where you should see changes already but only when the freebsd ports version and the dports/staged version differ. + +`www/node12` failed in the patch phase, here's the failure text so you want to get to that point, locally. + + # make -DBATCH -DDEVELOPER patch + [...] + ===> Patching for node12-12.18.0 + ===> Applying ports patches for node12-12.18.0 from /xports/www/node12/files + ===> Applying dragonfly patches for node12-12.18.0 from /xports/www/node12/dragonfly + 2 out of 2 hunks failed--saving rejects to deps/uv/src/unix/udp.c.rej + ===> FAILED Applying dragonfly patch-deps_uv_src_unix_udp.c + ===> FAILED to apply cleanly dragonfly patch(es) patch-deps_uv_src_unix_udp.c + *** Error code 1 + + Stop. + make[1]: stopped in /xports/www/node12 + *** Error code 1 + + Stop. + make: stopped in /xports/www/node12 + +The source directory is specified in the WRKSRC make variable, you can easily print it: + + # make -V WRKSRC + /usr/obj/dports/www/node12/node-v12.18.0 + +In the work source directory you will usually find the .rej files that were generated as a result of the patch phase failure: + + # cd /usr/obj/dports/www/node12/node-v12.18.0 + sting-chroot# find . -iname "*.rej" + ./deps/uv/src/unix/udp.c.rej + +Fix the patch and regenerate it again. In this case you move it to the dragonfly directory overwriting the previous patch, which failed to apply + + # $EDITOR deps/uv/src/unix/udp.c + # genpatch deps/uv/src/unix/udp.c + generated patch-deps_uv_src_unix_udp.c + # mv patch-deps_uv_src_unix_udp.c /usr/dports/www/node12/dragonfly/ + +The patch phase should now successfully complete: + + # make -DBATCH -DDEVELOPER clean + ===> Cleaning for node12-12.18.0 + # make -DBATCH -DDEVELOPER patch + [...] + ===> Patching for node12-12.18.0 + ===> Applying ports patches for node12-12.18.0 from /usr/dports/www/node12/files + ===> Applying dragonfly patches for node12-12.18.0 from /usr/dports/www/node12/dragonfly + +More problems will likely arise, you have to keep adding fixes until the port fully builds and packages: + + # make -DBATCH -DDEVELOPER package |& tee /tmp/node12.log + [...] + ===> Building package for node12-12.18.0 + +Make sure it passes a 'dsynth test www/node12', this may take a while. + + # dsynth test www/node12 + Processing 1 ports + 100.00% + + Total 2 + Resolving dependencies...done + Total Returned 54 + Scanning 1345 packages + 100.00% + + Total 1345 + all done + Check Template: Good + [000] START www/node12 idep=137 depi=00/00 dep=0.23G + [000] SUCCESS www/node12 00:07:23 + + Initial queue size: 1 + packages built: 1 + ignored: 0 + skipped: 0 + failed: 0 + missing: 0 + + Duration: 00:07:23 + +Now it should be ready for submit. + +### Submitting your work + +#### Layout +See below the key elements that might need to be included in the PR. Many examples exists already in the DeltaPorts repository, you can get most of the things you'll need to do from there. + +#### Makefile.DragonFly + +This is evaluated for every port build. As much stuff as possible should be included here, like variable override or modification, sed commands, etc. + +#### diffs/*.diff + +A way of patching directly the files within the port directory. For example, diffs/Makefile.diff will be applied to the port Makefile. + +#### diffs/TOREMOVE + +File which contains the file to be removed from the port, one file per line. + +#### dragonfly/ + +This directory holds the Dragonfly specific patches. Same rules as for FreeBSD files apply, i.e. just one patch per file, etc. + +#### How to submit + +Create a PR with the appropriate changes here: https://github.com/DragonFlyBSD/DeltaPorts/pulls + +### Local repository for the chroot + +There are some packages that will have quite some dependencies and since you are going to be building with dsynth a lot, it is a good practice to keep a local binary repository in the chroot. That way you can quickly install need packages for a specific build. You can just add the repository as a local in the pkg repos configuration. + + # fgrep -i packages= /etc/dsynth/dsynth.ini + Directory_packages= /build/dsynth/live_packages + # cat /usr/local/etc/pkg/repos/00_synth.conf + # Automatically generated. + + Synth: { + url : file:///build/dsynth/live_packages + priority : 0, + enabled : yes, + } + +# Internal: Procedure to sync freebsd ports + +This documentation is not intended for anyone other than the person(s) responsible for generating dports from the freebsd tree using the DragonFly BSD project's muscle server. All of these commands are done as root. + +## Update the world / kernel of muscles + +Ideally the host machine and the jail's world should match. In reality, if changes are close, the jail's world can be newer but it's best that they always match. + + # cd /usr/src + # make -j12 buildworld > /tmp/bw.log + # make -j12 kernel > /tmp/kernel.log + # make installworld > /tmp/iw.log + # make upgrade + # reboot + +## Update the jail's world + + # cd /root/adm/exec-dports-builder + # make update-world + +Note that after a new release of DragonFly, a new world has to be create rather than updated. This will appear in documentation later. + +## Find out what's the current ports checkout version + +You can run: + + # date -r `cat /usr/local/portsnap/tag | cut -f 2 -d '|'` + Thu Apr 25 05:33:17 PDT 2019 + +## Bring in latest freebsd ports + + # make update-ports + +## Purge the previous dports tree + + # make merge-purge + +This can take several minutes. -DragonFly 3.3 or later is the minimum version that can build dports from source. +## Merge the framework -It's probable that pkgsrc binaries are already installed. It is necessary to -rename /usr/pkg so that the existing pkgsrc binary programs don’t get accidentally used while building DPorts, causing breakage. If anything goes wrong with DPorts when you are installing it, you can go back by removing all the DPorts packages with "pkg delete -ay" and renaming /usr/pkg back to normal. (Don’t confuse pkg, the management tool, with /usr/pkg, the normal installation directory for pkgsrc. ) For the installation of the base port files: + # make merge-mk - cd /usr - make dports-create-shallow +It is very common that one or more of the patches will not apply. If that happens: -If you’ve already renamed your /usr/pkg directory, git won’t be in your path any more. You can instead download a tarball and unpack it, which also happens to be possible automatically via that same Makefile. + 1. su automaton + 2. cd ~/DeltaPorts/special/Mk/diffs + 3. Regenerate the diff patch or patches as necessary + 4. exit + 5. make merge-purge + 6. make merge-mk - cd /usr - make dports-download +Repeat cycle until every diff patch applies. When it's clear, issue this comand -Downloading via git is fastest, so if you do need to use the tarball via make dports-download, build devel/git, delete /usr/dports, and then pull it again with make dports-create-shallow. This all comes from [John Marino’s Github site for DPorts](https://github.com/jrmarino/DPorts/#readme). + # make sync-mk -### Managing DPorts +## Merge everything -DPorts doesn’t use pkg_info, pkg_add, and the other tools traditionally seen on DragonFly for pkgsrc. Instead, package management is done with pkg. Use pkg info, pkg install, pkg remove, and pkg update to list, install, delete, and upgrade various packages on your system. Packages built from source or downloaded as prebuilt binaries are managed the same way, using these tools. + # make merge-purge + # make merge -See some of the [other](https://mebsd.com/make-build-your-freebsd-word/pkgng-first-look-at-freebsds-new-package-manager.html) [writing](http://ivoras.net/blog/tree/2012-07-26.pkgng---best-thing-since-sliced-bread!.html) about pkg for FreeBSD for details on how it works. +This takes several minutes. During the merge, you will almost certainly see ports fail to sync due to that port utilizing a "diff" against the freebsd which has changed since the last sync. Take note of all failed ports. If you lost that information, you can execute "make reject-list" command to see which ports failed to merge. You can see the reject files at /usr/xports/[category]/[portname] for more information. -Since DPorts doesn’t update a package until it gets a successful build, and installations are of successfully built binary packages, upgrades with prebuilt packages should always succeed. Since they’re binary, they should be fast. There’s a lot of ‘shoulds’ in this sentence, but these are reasonable suppositions. +Each port has to be fixed before continuing. The general process for doing this is: -### Building + # su automaton + # cd ~/Deltaports/ports/[category]/[portname] + (assuming that the Makefile diff failed to apply in this example): + # cp /usr/ports/[category]/[portname]/Makefile . + # dupe Makefile + # modify Makefile accordingly (see diffs/Makefile.diff) + # genpatch Makefile > diffs/Makefile.diff + # exit + # ./sync [category]/[portname] + (if the sync command has no output, then the fix was good) -Building from source works similar to ports and pkgsrc: cd into the appropriate program's directory, and type 'make'. 'make install' to install the software, 'make clean' to clean up work files, and so on. Use 'make config-recursive' if you want to set all the port's options, and the options of its dependencies, immediately instead of during the build. +## Finish merge -*You will eventually want to do this:* Set BATCH=yes to take all default package options without having the build stop to ask questions. + # make rsync-tree -If you just want to set the options for one package, and accept the default for all of its dependencies, do 'make config' in the package in you want non-default options, and then 'make BATCH=yes'. +## Prune old ports -Note that this is only necessary if you want to build from source with a non-default config, or if you don't have a binary repository available to pull files from with pkg. +Remove dports where the freebsd ports have been removed. -## What about pkgsrc? + # make prune-confirm -Pkgsrc and DPorts shouldn’t be used at the same time, since one system’s packages may be at different versions but still get picked up during building for the other system. That’s about it for restrictions. +## Reclaim hard disk space -DPorts is brand new so the normal caveats apply; don’t install it on a mission-critical machine, and be ready to deal with any surprises from using it if you do try it out. + # make dist-clean -## Which ports are available? +This removes all distfiles that are no longer needed. It takes a few minutes, but it can reclaim gigabytes of space. You can usually go a few months without running this command (so it's optional; skip it if you're in a hurry to get started). -Browsing the Github repo will show you all listed packages. It changes as more packages are built. Changes from DragonFly builds are sometimes relevant to the original FreeBSD port, so there’s benefits for everyone here. The pkg tool can show which packages were actually built and available for your platform (e.g. pkg search, pkg rquery, etc.) +## Kick off full build -## What next? + # make synth-full-run -This is experimental; this will not necessarily replace pkgsrc. It's available in this release for further testing and feedback. Remember, this is a new project, so a willingness to deal with problems and contribute to fixes is necessary. +This command takes 3-4 days to execute.