[ikiwiki.git] / release54 / index.mdwn

# DragonFly BSD 5.4

* Version 5.4.0 released 03 December 2018
* Version 5.4.1 released 24 December 2018

DragonFly version 5.4 brings a new system compiler in GCC 8, improved NUMA support, a large number of network and virtual machine driver updates, and updates to video support.  This release is 64-bit only, as with previous releases.  5.4.1 adds more support for dhcpcd, though it is not yet default, and [better on-disk topology for HAMMER2](http://lists.dragonflybsd.org/pipermail/users/2018-December/358016.html).

The details of all commits between the 5.2 and 5.4 branches are available in the associated commit messages for [5.4.0rc](http://lists.dragonflybsd.org/pipermail/commits/2018-November/718131.html), [5.4.0](http://lists.dragonflybsd.org/pipermail/commits/2018-December/718171.html), and [5.4.1](http://lists.dragonflybsd.org/pipermail/commits/2018-December/718276.html).
 
## Big-ticket items

* Much better support for asymmetric NUMA (Non-Uniform Memory Access) configurations.  In particular, both the memory subsystem and the scheduler now understand the Threadripper 2990WX's architecture.  The scheduler will prioritize CPU nodes with direct-attached memory and the memory subsystem will normalize memory queues for CPU nodes without direct-attached memory (which improves cache locality on those CPUs).

* Incremental performance work.  DragonFly as a whole is very SMP friendly.  The type of performance work we are doing now mostly revolves around improving fairness for shared-vs-exclusive lock clashes, reducing cache ping-ponging due to non-contending SMP locks (i.e. massive use of shared locks on shared resources), and so forth.

* Major updates to dports brings us to within a week or two of FreeBSD's ports as of this writing, in particular major updates to chromium, and making the whole mess work with gcc-8.

* Major rewriting of the tty clist code and the tty locking code, significantly improving concurrency across multiple ttys and ptys.

### GCC 8 

* DragonFly now ships with GCC 8.0, and runs as the default compiler. It is also now used for building dports.

* GCC 4.7.4 and GCC 5.4.1 are still installed.  4.7.4 is our backup compiler, and 5.4.1 is still there to ensure a smooth transition, but should generally not be used.  buildworld builds all three by default to ensure maximum compatibility.

* Many passes through world sources were made to address various warnings and errors the new GCC brought with it.

### HAMMER2

* HAMMER2 is recommended as the default root filesystem in non-clustered mode.
* Clustered support is not yet available.
* Increased bulkfree cache to reduce the number of iterations required.
* Fixed numerous bugs.
* Improved support on low-memory machines.
* Significant pre-work on the XOP API to help support future networked operations.
* (in 5.4.1) HAMMER2 filesystem meta-consistency protection for
snapshots and crashes has been improved, as has speed of snapshot generation.  See full notes at the end of the document

## Details

### Checksums

MD5 (dfly-x86_64-5.4.1_REL.img) = 5819175d148e8cfa08c384483b0b25e2
MD5 (dfly-x86_64-5.4.1_REL.iso) = 8030f4ff31c3308b20ffbeeeb89351ba
MD5 (dfly-x86_64-5.4.1_REL.img.bz2) = 271c0e5c247479791a35404973e71fcc
MD5 (dfly-x86_64-5.4.1_REL.iso.bz2) = 9ddb7eabf4aed34a39e188849cd4f36c

### Upgrading

If you have an existing 5.2.x system and are running a generic kernel, the normal upgrade process, described below, will work.  

Change your local /usr/src to 5.4:

    cd /usr/src
    git fetch origin
    git branch DragonFly_RELEASE_5_4 origin/DragonFly_RELEASE_5_4
    git checkout DragonFly_RELEASE_5_4
    git pull

And then rebuild: (in /usr/src )

    make buildworld
    make buildkernel
    make installkernel
    make installworld
    make upgrade

    (reboot your system)

    make initrd 

'make initrd' is optional for systems with unencrypted disks, and has been possible for several releases; it's mentioned here for completeness. Look at the [initrd(7)](https://leaf.dragonflybsd.org/cgi/web-man?command=initrd&section=ANY) man page for details on what it does.

Don't forget to upgrade your existing packages.  5.4 packages have already been built and are immediately available.

    pkg update
    pkg upgrade

## All changes since DragonFly 5.2

### Security Issues

* Fix CVE-2018-8897, debug register issue.
* Complete the full Spectre mitigation support.  Default disabled due to extreme performance loss.  The machdep.spectre_support sysctl can be used to probe support, and machdep.spectre_mitigation sysctl can be used to enable/disable support (can be done at run time).
* Change the default /root perms from 755 to 700 in the build template.
* OpenBSD collab - completely remove delayed FP state to avoid known side-channel attack.
* OpenBSD collab - proactively clean FP state on switch to avoid known side-channel attack.
* OpenBSD collab - proactively zero user registers on entry into kernel (syscall, interrupt, or exception) to avoid certain speculative side-channel attacks.

### Kernel

* drm has been updated to match Linux kernel 4.7.10 in a number of locations.
* The radeon driver has been updated; currently matches Linux 3.18.
* CVE-2018-8897 mitigated.
* x2apic timer support added.
* Add a private_data field to struct file to improve application support.
* Fix pageout/sbusy race.
* Refactor AcpiOs*Lock() operation to use lockmgr locks to reduce freezes and improve debugability.
* Improve SPINLOCK and acpi_timer performance when under emulation.
* nvme - Improve the likelihood of dump success by removing certain blocking locks from the dump path.
* Numerous spinlock performance and edge-case improvements.  Deal with shared starvation in the face of a long series of exclusive locks, and vise-versa.  Add TSC-based windowing as a fall-back when excessive contention is present.
* Add a dirty vnode management facility.
* Remove bottlenecks from the rlimit handling code by reflecting p_limit into td_limit.
* Increase the size of the vm_object hash table by 4x to reduce collisions.
* Fix duplicate da* reporting at boot time and fix the related lockup.
* Implement a per-thread (fd,fp) cache to reduce cache ping-ponging in heavily-threaded processes.
* Refactor the lwkt_token pool hash algorithm to improve concurrency.
* Refactor uidinfo
* Refactor lockf operation to remove SMP bottlenecks.
* Increase copyin item batching from 8 to 32 for select() and poll().
* Improve concurrent tmpfs use by removing SMP bottlenecks.
* Improve allocvnode() by removing SMP bottlenecks.
* Improve common file descriptor internals for better SMP operation.
* Improve namecache performance.
* Improve scheduler concurrency by addressing some of the more blatent cache ping-ponging.
* Remove numerous kernel global debugging and (unused) statistics counters that were causing cache ping-ponging.
* Carefully refactored contended tokens and spinlocks for better SMP operation on many-cores systems.
* Carefully refactored the mutex contention code.
* Fix a kfree() statistics accounting bug.
* Improves kfree() SMP operation.
* Fix legacy IRQ bug by ensuring that legacy IRQs only appear on one CPU.
* Fix a multitude of usched issues that could cause unrelated processes to be misscheduled.
* Refactor the nice priority calculations to improve expected relative behavior when multiple cpu-bound processes are NICEd to different values.
* Refactor machdep.cpu_idle_hlt for better AMD handling.
* Rearrange pmap PV caching a bit to improve performance.
* Optimize the syscall path to improve performance.
* Rearrange a few structures to localize fields for cache line optimization.
* Remove vm_zeroidle.c.  Remove the asynchronous page zeroing code.  Instead, we now zero pages on demand, just before they are used, to reduce unnecessary cache interactions and to reduce a long-time debuggability issue with pre-zero'd pages.
* X2APIC support added and tested.
* ACPI updated from Intel base.
* Fix a swapcache cleaning issue that could cause swapcache to behave in a non-useful manner.
* Change closefrom() EINTR behavior.  closefrom() no longer breaks out of its loop when EINTR is received from close() (which could occur with NFS descriptors).  EINTR will still be rolled up and returned (callers usually ignore this).
* Fix a rare pmap_scan_callback() panic.
* Fix a umtx race in the kernel.
* Fix a NULL pointer indirection on lost socket bug.
* Enable NX for PROT_READ-only mappings by default.  NX cannot be enabled for PROT_WRITE (without PROT_EXEC) due to assumptions made by applications.  Suggested by OpenBSD.
* Refactor low-memory handling in the objcache subsystem.
* Bring in some jail work from FreeBSD.
* Rewrite the TSC probe code several times to improve boot times and reduce SMP issues on many-cores cpus and under emulation.
* Refactor the if_clone code and improve stability when used with TAP.
* Increase MAXCPUFIFO from 32 to 256 to fix deadlocks which can occur on ncpus > 32 systems.
* AMD topology detection rewritten.
* Add a heuristic to the scheduler to prioritize the 'better' cores in assymetric NUMA configurations (aka threadripper 2990WX).
* Add memory-on-node weighting to the schedule which detects assymetric NUMA configurations and assigns memory to nodes with too little actual memory.  This gives us better memory isolation on nodes without direct-attached memory, significantly improving their performance.
* Fix missing wakeup()s in kern_lock.c that could (rarely) lead to unkillable blocked processes.
* Fix a startup race in usched_dfly which sometimes caused early boot panics.
* Fix a missing brelse() in nvtruncbuf()'s error path which could deadlock the system on certain filesystem errors.
* Fix an early-boot interrupt race that sometimes caused a null-pointer deref panic.
* Optimize bcopy, bzero, memset, part 1.  part 2 didn't quite make it into this release.
* Improve kernel performance by macroing trivial mem*() and bcopy/bzero*() operations with known, small, static sizes.
* Improve kernel performance
* Add wait6(), waitid(), and si_pid/si_uid siginfo support.  This fixes the 'lightdm' X display manager.
* Move nearly all global tty_token uses to per-tty or per-pty tokens, significantly improving concurrency when writing to multiple ttys or ptys.  Remove unnecessary critical sections.
* Remove use of tty_token in the session management code, removing an annoying bottleneck from the fork/exec/wait/exit path.
* The keyboard subsystem now uses its own kbd_token instead of using tty_token.
* Rewrite the tty clist code.  Get rid of clists.  Instead, use a single linear buffer and a FIFO mechanism.  The original clist code was obsolete 20 years ago.
* Fix a kernel exit race in killalllwps() that could sometimes prevent a heavily threaded process from completing its exit().
*

### Filesystems

* autofs - Various updates to autofs
* ufs - Fix an old UFS bug that would panic on certain filesystem full conditions.
* ufs - Rename UFS constants throughout to improve separation.
* ufs - Remove duplicate TRIMs and significantly improve 'rm -rf' performance.
* tmpfs - Fix NFS exports of tmpfs filesystems.
* tmpfs - Significantly improve tmpfs concurrency.
* tmpfs - Fix a rare deadlock.
* Fix minor bugs in HAMMER1 that could result in an assertion under heavy loads.
* HAMMER2 is now the recommended default filesystem for root.
* Fix ENOTDIR/EISDIR logic in certain situations.
* uid, gid, and (fake) inum added to the fstat() information for pipe()s and socket()s.

### Networking

* The network tunnel driver, [tun(4)](https://leaf.dragonflybsd.org/cgi/web-man?command=tun&section=4), has been cleaned up and updated.  It's now clonable for anyone building VPN links.
* The [tap(4)](https://leaf.dragonflybsd.org/cgi/web-man?command=tap&section=4) and interface cloning have also received related updates and fixes.
* Fix an arp issue in the bridge code, plus add a few bug fixes.
* Interface groups are now supported in the kernel and [pf(4)](https://leaf.dragonflybsd.org/cgi/web-man?command=pf&section=4).  They can be configured with [ifconfig(8)](https://leaf.dragonflybsd.org/cgi/web-man?command=ifconfig&section=8)
* DragonFly now can use [dhcpcd(8)](https://leaf.dragonflybsd.org/cgi/web-man?command=dhcpcd&section=8) for DHCP, for both IPv4 and IPv6.  See [the announcement](http://lists.dragonflybsd.org/pipermail/users/2018-November/357989.html) for details.  The existing utilities dhclient(8) and rtsold(8) are still supported, so no configuration change is required yet.
* The ena (Elastic Network Adapter) network driver has been added.
* if_iwm has had multiple updates.
* if_vtnet, the virtual I/O Ethernet driver, has been updated.
* Significant work to ipfw3.
* Add BPF_MOD and BPF_XOR support to bpf, and update libpcap.
* Fix a user-after-free mbuf issue - obtained from NetBSD via FreeBSD.
* IPSEC has been removed from the system, for many reasons.  Basically, VPNs are far superior and more deterministic in terms of maintainability.
* ifconfig(8) - synchronized from FreeBSD.
* Reduce confusion by giving SEQPACKET sockets their own sysctls for limit setting.
* Increase all default limits for unix domain sockets to 65536 (fixes issues with wpa_supplicant and provides better defaults against application expectations).
* Add ip6addrctl(8) from FreeBSD.
* Fix a SACK related NULL pointer dereference.
* dhcpcd has been added to the base system (for IPV6), and will replace dhclient and rtsold - though not in this release.
* A number of network device drivers have been added.
* A couple of edge-case panics have been fixed.
* wpa_supplicant is installed via dports, by default, to override the default version in base.  The base version remains so that a mass pkg deletion doesn't accidentally remove the ability to get online.
* Fix getsockname() on unnamed AF_LOCAL sockets to properly initialize the sockaddr.
* Set net.inet6.icmp6.nd6_onlink_ns_rfc4861 to 1 by default, contemporary implementations pretty much require it.
* Fix a misunderstanding in how igb(4) and ix(4) calculate the packet hash for encapsulated packets.

### Driver updates

* A number of drivers of use when running DragonFly as a guest VM have been added or updated, including ena (for AWS support), if_vtnet, virtio_balloon, and virtio_pci.
* Improvements for DRM and radeon.
* Serial-output-only installs are now possible.
* The virtio_balloon memory driver has been added.
* /dev/sndstat can now be opened multiple times by the same device.
* mpr(4) driver for LSI Fusion-MPT 3/3.5 SAS controllers added.
* MosChip PCIe serial communications now supported.
* Add missing descriptions for usb4bsd C610/X99 controllers
* igb(4) bring in WOL (wake-on-lan) support from FreeBSD.
* Add support for PCIe serial com, and console support.
* Remove numerous old PCI and ISA serial drivers.
* nvme(4) fixes for AWS.  Handle interfaces without attached volumes.
* corepower(4) - Add support for platform energy counter.
* ig4(4) Numerous changes to improve low-power support and stability.  Handle situations where ig4(4) is listed but is non-operational (probably because the BIOS has stolen it).
* dsp(4) fixes for mmap() handling.
* sound(4) now allows /dev/sndstat to be open multiple times from one process.

### Userland

* Refactor /usr/Makefile (for src-create and other targets).
* Refactor buildworld, in particular moving the initrd build from installworld to buildworld
* Add rc support for ipfw3
* Major refactor of the initrd infrastructure.
* Update pwcache(3) from NetBSD
* Getty - minor sync from FreeBSD
* Update vis(3) and unvis(3) from NetBSD
* Update pciconf database.
* Fix -Walloc-size-large-than bug in gcc-4.7 and 5.0.
* Remove rhosts from the default template (the rcmds, like rcp, rlogin, etc. were removed from the tree in the previous release).
* Fix a signed overflow in mktime() that blew up GCC-8's -O2 constant folding code.
* Add strsuftoll[,x]() to libc in preparation for makefs(8).  This helps various dports as well.  Taken from NetBSD.
* Add tcsetsid() to libc.
* Adjust symlink() prototype to match standards.
* Improve buildworld concurrency when building the compilers.
* Significant crunchgen updates.  Add 70 more tools to the crunch.
* Add -N option to install(1).
* The resolver now allows underscores in the name, fixing numerous domains and domain forwarding issues.
* mtree - Replace with NetBSD's version.
* dumpon(8) - Improve error message clarity.
* Cache 4K, 8K, and other larger allocations in posix_memalign(), significantly improving its performance.

### Boot

* The initrd has had significant improvement and now forms a much more complete rescue system, with over 70 tools added via [crunchgen(1)](https://leaf.dragonflybsd.org/cgi/web-man?command=crunchgen&section=1).  See the [initrd(7)](https://leaf.dragonflybsd.org/cgi/web-man?command=initrd&section=7) man page for details.
* Crypto bootstrap, fix vfs.root.real_root to be vfs.root.realroot.
* Rescue boot mode does not try to mount the real root.
* Handle tftp block overflows properly, allowing the kernel image to exceed 65536 blocks when tftp pxe-booting is used instead of NFS pxe-booting.
* Slow down the NFS twiddle a bit.

### Various tools have been upgraded in the base system:

* [dhcpcd](http://roy.marples.name/projects/dhcpcd) 7.0.8 imported.
* [openresolv](https://roy.marples.name/projects/openresolv) 3.9.0 imported, installed as [resolvconf(8)](https://leaf.dragonflybsd.org/cgi/web-man?command=resolvconf&section=8).
* [ip6addrctl(8)](https://leaf.dragonflybsd.org/cgi/web-man?command=ip6addrctl&section=8) together with a rc script imported from FreeBSD.
* DHCP server for network installs updated to net/isc-dhcp44-server.
* OpenSSH updated to 7.6p1.
* ACPICA updated to 20181031.
* Time zone data updated to tzdata2018g.
* Compiler - see next section
* diff utils updated.

### Compiler status

* gcc 8.0 is the default compiler, for kernel, world, and dports.  GCC versions 4 and 5 are still installed and built if needed.

### Package updates

* A full set of new binary packages has been built for 5.4, available through the pkg tool.
* There's a number of options now for running a web browser on DragonFly; check the [browser documentation page](https://www.dragonflybsd.org/docs/docs/howtos/WebBrowsers/) for a full list.
* wpa_supplicant is installed as a package by default; delete if your system does not use wireless.  The version in base remains as a safety measure in case the dports version is deleted.

### HAMMER2 changes in 5.4.1

* H2 filesystem now has meta-consistency protection for
snapshots and crashes.  This is on top of the lower-level radix tree
protections it already had.  The coherency protection handles
directory-entry-vs-inode dependencies which prior to this patch could
sometimes get broken by a crash, causing the nlinks count on the inode to
not match the number of directory entries referencing it, sometimes putting
directory entry flushes and the related inode flushes in different sync
groups, and often causing snapshot operations to 'cut' the filesystem while
it is in an inconsistent state, when made in the midst of modifying
activity.  Some people have noticed these issues recently, and while they
can be cleaned up with manual hammer2 directives after the fact, it was
really annoying to have to deal with.
* The update now under test in master fixes all of these issues and is also
able to guarantee buffer cache consistency... hence also write() atomicy
boundaries on snapshot or crash.
* As an added bonus, most of the front-end stalls that currently occur during a filesystem sync have also been cleaned up.  The frontend.. that is, the filesystem system calls made by programs, is now able to operate concurrently with a filesystem sync.   Previously the front-end stalled on
any modifying operation made to any inode during a filesystem sync (something I did to try to 'fix' the prior consistency issues, but which
didn't sufficiently fix the problems).  With the new work, the front-end
now only stalls on the specific inodes which are in the sync queue, and
will reorder any inodes it stalls on to the front of the sync queue in
order to retire their flushes as quickly as possible.  Once the inode is no
longer on the sync queue, the front-end can proceed with its modifying
operation concurrent with the continuing filesystem sync.  The result is
significantly better concurrency and far shorter-duration stalls for those
situations where a stall is mandatory.   The new code keeps track of and
handles any dependencies between inodes on the syncq and inodes not on the
syncq which arise during this operation.
* The new patch also significantly improves snapshot operation, and adds a
new directive called 'snapshot-debug' (which I will probably rename) which
foregoes the standard sync-before-snapshot that the primary 'snapshot'
directive implements.  Both directives will snapshot a fully consistent
filesystem, the only difference is that snapshot-debug might not get
changes made just prior to issuing the command (it uses a recorded blockmap
from the most recent sync instead of forcing a new sync).