(no commit message)
[ikiwiki.git] / docs / howtos / HowToDPorts / index.mdwn
55aaaf94 1[[!toc levels=3]]
ad964fe1 3# DPorts and pkg(8)
80be6ca4 4
6c9b25cf 5DPorts is DragonFly's own third-party software build system. It is based on FreeBSD's Ports Collection. Differences between ports and DPorts are intentionally kept to a minimum, both to maintain familiarity for mutual users of both operating systems and also to leverage the tremendous amount of work the FreeBSD contributors put into ports. DPorts can and does feature ports unique to DragonFly, so it's truly a native system.
80be6ca4 6
ff956e39 7The __pkg(8)__ tool called "pkg" is a modern and fast binary package manager. It was developed for FreeBSD, but PC-BSD used it in production first, followed soon after by DragonFly. FreeBSD has since removed their legacy tools so __pkg(8)__ is exclusively used by FreeBSD as well.
80be6ca4 8
ff956e39 9__pkg(8)__ is not a replacement for port management tools like `ports-mgmt/portmaster` or `ports-mgmt/portupgrade`. While `ports-mgmt/portmaster` and `ports-mgmt/portupgrade` can install third-party software from both binary packages and DPorts, __pkg(8)__ installs only binary packages. On the other hand, many people use __pkg(8)__ by itself and never install the optional portupgrade or portmaster tools.
80be6ca4 10
ad964fe1 11## Getting started with pkg(8)
80be6ca4 12
a2ff8c27 13DragonFly daily snapshots and Releases (starting with 3.4) come with __pkg(8)__ already installed. Upgrades from earlier releases, however, will not have it. If the "pkg" program is missing on the system for any reason, it can be quickly bootstrapped without having to build it from source or even having dports installed:
3bcc74c7 14
a61e5e08 15 # cd /usr
676f236e 16 # make pkg-bootstrap
17 # rehash
18 # pkg-static install -y pkg
19 # rehash
5232b755 20
11f8ebe3 21After any installation or reinstallation of the __pkg(8)__ package, you may want to open review the configuration files to customize it, e.g. select a different mirror.
676f236e 22
ad964fe1 23## Configuring pkg(8)
5232b755 24
11f8ebe3 25The __pkg(8)__ program has a configuration file at /usr/local/etc/pkg.conf (the default installation is typically fine and requires no alteration). It uses a a separate location for repository configuration at /usr/local/etc/pkg/repos/ directory.
a2ff8c27 26
11f8ebe3 27For fresh installations, the file /usr/local/etc/pkg/repos/df-latest.conf.sample is copied to /usr/local/etc/pkg/repos/df-latest so that __pkg(8)__ works out of the box. The files ending in the ".sample" extension are ignored; __pkg(8)__ only reads files that end in ".conf" and it will read as many as it finds.
1297c263 28
11f8ebe3 29The default is to use the California-based Avalon, which is also the master. Avalon will have new packages first and they'll be updated as a set. The others are mirrors and may be much, much faster depending on the user's location, but there will be a period between set updates where the mirror is mix of old and new packages.
a2ff8c27 30
11f8ebe3 31The simple case is to use a single repository. If you want to switch off the overloaded Avalon, simply change the *enabled* value of Avalon to "no" and enable a new single repository, e.g.:
1297c263 32
33 Avalon: {
a2ff8c27 34 url : pkg+http://mirror-master.dragonflybsd.org/dports/${ABI}/LATEST,
11f8ebe3 35 mirror_type : NONE
a2ff8c27 36 [...]
37 enabled : no
38 }
11f8ebe3 39
40 Wolfpond: {
41 url : http://pkg.wolfpond.org/${ABI}/LATEST,
581c2549 42 enabled : yes
1297c263 43 }
ff963ca1 44
11f8ebe3 45The provided df-latest.conf may not be up to date. There may be defunct repositories listed and there may be active mirrors that are not listed as options. You may edit or add new repo configuration files as necessary.
47It is possible to use multiple repositories. The normal use case is that someone builds specific packages (e.g. using poudriere) which should have a higher priority over the official packages. In other words, whenever a package has been built locally, __pkg(8)__ installs the local version and only uses the external repository to install packages not available locally.
49In the above case, a custom repository configuration file should be created, and it should include the "priority" field, e.g. "priority: 10". The higher the number, the higher the priority. The priority when unspecified is 0.
51The last output of "pkg -vv" lists all enable repositories and their priority values.
a2ff8c27 53Before using, consult the man page (`man pkg`) and then try these examples:
55 # pkg search editors
56 # pkg install vim
8fd01ee6 57
ad964fe1 58## Basic pkg(8) Operations
f60756b0 59
ff956e39 60Usage information for __pkg(8)__ is available in the pkg(8) manual page, or by running `pkg` without additional arguments.
5232b755 61
ff956e39 62Each __pkg(8)__ command argument is documented in a command-specific manual page. To read the manual page for `pkg install`, for example, run either:
676f236e 63
64 # pkg help install
65 # man pkg-install
ad964fe1 67## Obtaining Information About Installed Packages with pkg(8)
676f236e 68
69Information about the packages installed on a system can be viewed by running `pkg info`. Similar to pkg_info(1), the package version and description for all packages will be listed. Information about a specific package is available by running:
71 # pkg info packagename
ff956e39 73For example, to see which version of __pkg(8)__ is installed on the system, run:
676f236e 74
75 # pkg info pkg
76 pkg-1.0.12 New generation package manager
ad964fe1 78## Installing and Removing Packages with pkg(8)
676f236e 79
80In general, most DragonFly users will install binary packages by typing:
82 # pkg install <packagename>
84For example, to install curl:
86 # pkg install curl
88 Updating repository catalogue
89 Repository catalogue is up-to-date, no need to fetch fresh copy
90 The following packages will be installed:
92 Installing ca_root_nss: 3.13.5
93 Installing curl: 7.24.0
95 The installation will require 4 MB more space
97 1 MB to be downloaded
99 Proceed with installing packages [y/N]: y
100 ca_root_nss-3.13.5.txz 100% 255KB 255.1KB/s 255.1KB/s 00:00
8a3d16f3 101 curl-7.24.0.txz 100% 1108KB 1.1MB/s 1.1MB/s 00:00
676f236e 102 Checking integrity... done
103 Installing ca_root_nss-3.13.5... done
104 Installing curl-7.24.0... done
106The new package and any additional packages that were installed as dependencies can be seen in the installed packages list:
108 # pkg info
109 ca_root_nss-3.13.5 The root certificate bundle from the Mozilla Project
110 curl-7.24.0 Non-interactive tool to get files from FTP, GOPHER, HTTP(S) servers
111 pkg-1.0.12 New generation package manager
113Packages that are no longer needed can be removed with `pkg delete`. For example, if it turns out that curl is not needed after all:
115 # pkg delete curl
116 The following packages will be deleted:
118 curl-7.24.0_1
120 The deletion will free 3 MB
122 Proceed with deleting packages [y/N]: y
123 Deleting curl-7.24.0_1... done
ad964fe1 125## Upgrading Installed Packages with pkg(8)
676f236e 126
127Packages that are outdated can be found with `pkg version`. If a local ports tree does not exist, pkg-version(8) will use the remote repository catalogue, otherwise the local ports tree will be used to identify package versions.
80be6ca4 128
ff956e39 129Packages can be upgraded to newer versions with __pkg(8)__. Suppose a new version of curl has been released. The local package can be upgraded to the new version:
80be6ca4 130
676f236e 131 # pkg upgrade
132 Updating repository catalogue
133 repo.txz 100% 297KB 296.5KB/s 296.5KB/s 00:00
134 The following packages will be upgraded:
136 Upgrading curl: 7.24.0 -> 7.24.0_1
138 1 MB to be downloaded
140 Proceed with upgrading packages [y/N]: y
141 curl-7.24.0_1.txz 100% 1108KB 1.1MB/s 1.1MB/s 00:00
142 Checking integrity... done
143 Upgrading curl from 7.24.0 to 7.24.0_1... done
80be6ca4 144
ad964fe1 145## Auditing Installed Packages with pkg(8)
80be6ca4 146
ff956e39 147Occasionally, software vulnerabilities may be discovered in software within DPorts. __pkg(8)__ includes built-in auditing. To audit the software installed on the system, type:
80be6ca4 148
676f236e 149 # pkg audit -F
80be6ca4 150
ad964fe1 151# Advanced pkg(8) Operations
80be6ca4 152
ad964fe1 153## Automatically Removing Leaf Dependencies with pkg(8)
80be6ca4 154
676f236e 155Removing a package may leave behind unnecessary dependencies, like `security/ca_root_nss` in the example above. Such packages are still installed, but nothing depends on them any more. Unneeded packages that were installed as dependencies can be automatically detected and removed:
80be6ca4 156
676f236e 157 # pkg autoremove
158 Packages to be autoremoved:
159 ca_root_nss-3.13.5
161 The autoremoval will free 723 kB
163 Proceed with autoremoval of packages [y/N]: y
164 Deinstalling ca_root_nss-3.13.5... done
80be6ca4 165
ad964fe1 166## Backing Up the pkg(8) Package Database
676f236e 167
ff956e39 168__pkg(8)__ includes its own package database backup mechanism. To manually back up the package database contents, type:
676f236e 169
ad964fe1 170 # pkg backup -d <pkg.db>
676f236e 171
ff956e39 172Additionally, __pkg(8)__ includes a periodic(8) script to automatically back up the package database daily if `daily_backup_pkgng_enable` is set to `YES` in periodic.conf(5). To prevent the `pkg_install` periodic script from also backing up the package database, set `daily_backup_pkgdb_enable` to `NO` in periodic.conf(5).
676f236e 173
174To restore the contents of a previous package database backup, run:
ad964fe1 176 # pkg backup -r </path/to/pkg.db>
676f236e 177
ad964fe1 178## Removing Stale pkg(8) Packages
676f236e 179
ff956e39 180By default, __pkg(8)__ stores binary packages in a cache directory as defined by `PKG_CACHEDIR` in pkg.conf(5). When upgrading packages with pkg upgrade, old versions of the upgraded packages are not automatically removed.
676f236e 181
182To remove the outdated binary packages, type:
184 # pkg clean
ad964fe1 186##Modifying pkg(8) Package Metadata
676f236e 187
1fee31d7 188__pkg(8)__ has a couple of built-in commands to update package origins.
676f236e 189
a7f35c8e 190_Note_: Previously it was recommended to use the -o option, e.g. "-o < category / oldport >:< category / newport >" but this option is deprecated and will be removed in the future.
1fee31d7 191
192_Note_: The --change-name (-n) option requires the package name. Normally it's intuitive based on the name of the port, but not always. To get the definitive name, the following command can be used:
194 # make -V PKGBASE -C /usr/dports/<category>/<portname>
196For example, if `lang/php5` was originally at version 5.3, but has been renamed to lang/php53 for the inclusion of version 5.4, the package database can be updated to deal with this. For __pkg(8)__, the syntax is:
198 # pkg set -n <oldname>:<newname>
676f236e 199
200For example, to change the package origin for the above example, type:
1fee31d7 202 # pkg set -n php5:php53
676f236e 203
1fee31d7 204As another example, to update lang/ruby20 to lang/ruby21 (the current default), type:
676f236e 205
1fee31d7 206 # pkg set -n ruby20:ruby
676f236e 207
208As a final example, to change the origin of the libglut shared libraries from graphics/libglut to graphics/freeglut, type:
1fee31d7 210 # pkg set -n libglut:freeglut
676f236e 211
212_Note_: When changing package origins, in most cases it is important to reinstall packages that are dependent on the package that has had the origin changed. To force a reinstallation of dependent packages, type:
214 # pkg install -Rf graphics/freeglut
35ed6c39 216
217# Bullet-proof (conflict-proof) upgrade technique
219Several people have run into problem upgrading binary packages when default versions change (e.g. PHP default changes from version 5.4 to 5.6, or PostgreSQL default changes from 9.0 to 9.2). Usually there are upgrading hints at /usr/dports/UPDATING, but there are issues with this method of communication
221 1. You must have /usr/dports installed and current, which kind of defeats the purpose of using binary packages
222 2. You usually don't go looking there until after you hit a failure
223 3. Proactively reading /usr/dports/UPDATING perhaps from [DragonFly's Gitweb](http://gitweb.dragonflybsd.org/dports.git/blob/HEAD:/UPDATING) would be the best approach, but honestly nobody is going to do that.
225Additionally, over time Administrators install packages to evaluate them or otherwise don't need them any longer, but often the Administrators forget to remove the package and its dependencies later. These extra packages still get upgraded unnecessarily.
227## Technique Instructions
229The following technique will guarantee conflict-proof, integral updates and at the same time allow manually pruning of unwanted packages.
231### Step 1: Get list of primary packages
002cae5a 233For __pkg(8)__ version 1.5.1 or later, the "prime-list" alias is predefined in the pre-installed pkg.conf, so you can run the following command:
35ed6c39 234
235 > pkg prime-list > ~/prime.list
002cae5a 236
237If you are using an older version of package, or already had pkg.conf which was not overwritten, you can use the equivalent of the alias like so:
239 > pkg query -e "%a = 0" "%n" > ~/prime.list
35ed6c39 240
ad964fe1 241The file ~/prime.list contains the list of packages that were manually installed. For example, if an Administrator only requested on package to be installed, Libreoffice, __pkg(8)__ might pull in 200 dependencies, but ~/prime.list would only contain the name of a single package.
35ed6c39 242
243### Step 2: Review and edit primary package list
245Using your favorite editor, load ~/prime.list, which indicates which packages were manually installed. If there are any packages that are no longer desired, simple delete the line that contains it.
247### Step 3: Wipe everything out
249Now everything except package needs to be removed off the system.
251 > pkg delete -ay
253### Step 4: Upgrade the repository category
ad964fe1 255What we really want to do here is update the repository catalog, but if there's a newer version of __pkg(8)__, we can upgrade it at the same time.
35ed6c39 256
257 > pkg upgrade
ff956e39 259Say yes if __pkg(8)__ wants permission to update itself on the system.
35ed6c39 260
261### Step 5: Cleanse the package cache
263To avoid any possible reinstallation of a stale package, we want to remove obsolete packages from the cache.
265 > pkg clean -y
267### Step 6: Reinstall the primary packages
ff956e39 269Now we reinstall the packages that we really want on the system and let __pkg(8)__ pull in the dependencies that are needed without fear of conflict.
35ed6c39 270
271 > pkg install `cat ~/prime.list`
ff956e39 273Provide affirmative answers to any confirmations __pkg(8)__ asks.
35ed6c39 274
275## How to create the _prime-list_ alias for future use
002cae5a 277Rather than try to remember that hard-to-type query, it's more convenient to define this command as an alias. Note this is not necessary for new installations since around 22 April 2015 (The alias is predefined for __pkg(8)__ version 1.5.1 and later). To defined the alias, create or append the */usr/local/etc/pkg.conf* file with the following:
35ed6c39 278
279 ALIAS : {
280 prime-list: query -e "%a = 0" "%n",
281 }
283If ALIAS is already defined, just add the prime-list line to the list. You may review */usr/local/etc/pkg.conf.sample* for other ALIAS suggestions. There are several useful ones offered there.
676f236e 285# Building DPorts from source
287The average user will probably not build packages from source. However, it's easy to do and it can be done even when packages have already been pre-installed on the system. Common reasons to build from source are:
289* The port is new and there's no pre-binary available yet
290* The pre-built binaries use the default options and the user needs a package built with a different set of options
291* Testing FreeBSD port in order to patch them and submit to DPorts
292* The user just prefers building from source
80be6ca4 293
676f236e 294## Installing DPorts tree
80be6ca4 295
676f236e 296DragonFly 3.4 or later is the minimum version that can build DPorts from source.
80be6ca4 297
676f236e 298It's probably that pkgsrc binaries are already installed because it comes bootstrapped with new systems. It is necessary to rename `/usr/pkg` directory so that the existing pkgsrc binary tools and libraries don’t get accidentally used while building DPorts, causing breakage. For the installation of the DPorts tree, type:
0fcf3e89 299
676f236e 300 # cd /usr
301 # make dports-create-shallow
0fcf3e89 302
676f236e 303If the `/usr/pkg directory` has already been renamed, `git` won’t be in the search path any more. One option is to download a tarball of DPorts and unpack it. To do this, type:
80be6ca4 304
676f236e 305 # cd /usr
306 # make dports-download
80be6ca4 307
676f236e 308For future updates, pull delta changes via `git` is fastest, so it is suggested to convert the static tree to a git repository by typing:
80be6ca4 309
676f236e 310 # cd /usr/dports/devel/git
311 # make install
312 # cd /usr
313 # rm -rf /usr/dports
314 # make dports-create-shallow
80be6ca4 315
8eaa203d 316The git repository is hosted on the [github account of DragonFly](https://github.com/DragonFlyBSD/DPorts/#readme).
80be6ca4 317
676f236e 318## Final thoughts
80be6ca4 319
676f236e 320Building from source works similar to ports and pkgsrc: cd into the appropriate program's directory, and type 'make'. 'make install' to install the software, 'make clean' to clean up work files, and so on. Use 'make config-recursive' if you want to set all the port's options, and the options of its dependencies, immediately instead of during the build.
80be6ca4 321
676f236e 322To take all the default build options and avoid getting the pop-up dialog box, set `NO_DIALOG=yes` on either the command line or the make.conf file.
80be6ca4 323
676f236e 324If you just want to set the options for one package, and accept the default for all of its dependencies, do 'make config' in the package in you want non-default options, and then 'make NO_DIALOG=yes'. Note that this is only necessary if you want to build from source with a non-default set of options, or if no pre-built binary package is available yet.
44526695 325
326## More reading
327* How fix/add broken ports: [[docs/howtos/fixdports]]
328* [Trick: How to get i386-only software via dports](http://leaf.dragonflybsd.org/mailarchive/users/2013-06/msg00023.html)
447ce4e2 329
330# Internal: Procedure to sync freebsd ports
1050d96b 332This documentation is not intended for anyone other than the person(s) responsible for generating dports from the freebsd tree using the DragonFly BSD project's muscle server. All of these commands are done as root.
447ce4e2 333
334## Update the world / kernel of muscles
336Ideally the host machine and the jail's world should match. In reality, if changes are close, the jail's world can be newer but it's best that they always match.
338 # cd /usr/src
339 # make -j12 buildworld > /tmp/bw.log
340 # make -j12 kernel > /tmp/kernel.log
341 # make installworld > /tmp/iw.log
342 # make upgrade
343 # reboot
1050d96b 345## Update the jail's world
347 # cd /root/adm/exec-dports-builder
348 # make update-world
350Note that after a new release of DragonFly, a new world has to be create rather than updated. This will appear in documentation later.
30d48b3d 352## Bring in latest freebsd ports
354 # make update-ports
182be2d0 356## Purge the previous dports tree
358 # make merge-purge
907cad6d 360This can take several minutes.
a2fc4a70 361
362## Merge the framework
364 # make merge-mk
366It is very common that one or more of the patches will not apply. If that happens:
368 1. su automaton
369 2. cd ~/DeltaPorts/special/Mk/diffs
370 3. Regenerate the diff patch or patches as necessary
371 4. exit
907cad6d 372 5. make merge-purge
373 6. make merge-mk
a2fc4a70 374
b69cb84b 375Repeat cycle until every diff patch applies. When it's clear, issue this comand
377 # make sync-mk
a2fc4a70 378
907cad6d 379## Merge everything
381 # make merge-purge
382 # make merge
682500f7 384This takes several minutes. During the merge, you will almost certainly see ports fail to sync due to that port utilizing a "diff" against the freebsd which has changed since the last sync. Take note of all failed ports. If you lost that information, you can execute "make reject-list" command to see which ports failed to merge. You can see the reject files at /usr/xports/[category]/[portname] for more information.
8f38efc6 385
386Each port has to be fixed before continuing. The general process for doing this is:
14b2a454 388 # su automaton
682500f7 389 # cd ~/Deltaports/ports/[category]/[portname]
14b2a454 390 (assuming that the Makefile diff failed to apply in this example):
682500f7 391 # cp /usr/ports/[category]/[portname]/Makefile .
14b2a454 392 # dupe Makefile
393 # modify Makefile accordingly (see diffs/Makefile.diff)
394 # genpatch Makefile > diffs/Makefile.diff
395 # exit
682500f7 396 # ./sync [category]/[portname]
14b2a454 397 (if the sync command has no output, then the fix was good)
399## Finish merge
401 # make rsync-tree
c7c16010 403## Prune old ports
405Remove dports where the freebsd ports have been removed.
14b2a454 406
407 # make prune-confirm
b69cb84b 409## Reclaim hard disk space
411 # make purge-distfiles
1893c574 413This removes all distfiles that are no longer needed. It takes a few minutes, but it can reclaim gigabytes of space. You can usually go a few months without running this command (so it's optional; skip it if you're in a hurry to get started).
b69cb84b 414
f3c34b84 415## Kick off full build
417 # make synth-full-run
419This command takes 3-4 days to execute.