pass 4

[ikiwiki.git] / release52 / index.mdwn

# DragonFly BSD 5.2

* Version 5.2.0 released xx April 2018

** Release 5.2 is not yet out.  This page is a work in progress.  Please do not link to it until the actual release. **

DragonFly version 5.2 brings etc etc etc  

The details of all commits between the 4.8 and 5.0 branches are available in the associated commit messages for [5.2.0rc]() and [5.2.0]().
 
## Big-ticket items

### Meltdown and Spectre mitigation support

* Meltdown isolation and spectre mitigation support added.  Meltdown mitigation is automatically enabled for all Intel cpus.  Spectre mitigation must be enabled manually via sysctl if desired.

* machdep.spectre_mitigation

* machdep.meltdown_mitigation

### HAMMER2

* H2 has received a very large number of bug fixes and performance improvements.  We can now recommend H2 as the default root filesystem in non-clustered mode.  

* Clustered support is not yet available.

### ipfw Updates

* Implement state based "redirect", i.e. without using libalias.

* ipfw now supports all possible ICMP types.

* Fix ICMP_MAXTYPE assumptions (now 40 as of this release).

### Improved graphics support

* The drm/i915 kernel driver has been updated to support Intel Coffeelake GPUs

* Add 24-bit pixel format support to the EFI frame buffer code.

* Significantly improve fbio support for the "scfb" XOrg driver.  This allows EFI frame buffers to be used by X in situations where we do not otherwise support the GPU.

* Partly implement the FBIO_BLANK ioctl for display powersaving.

### Other user-affecting changes


## Details

### Checksums


### Upgrading

If you have an existing 5.0.x system and are running a generic kernel, the normal upgrade process, described below, will work.  

* Note that DSA OpenSSH keys were deprecated in the 4.6 to 4.8 release.  If you are upgrading from a release older than 4.8 and use DSA OpenSSH keys, please update your private/public key pair or risk locking yourself out.

Change your local /usr/src to 5.2:

    cd /usr/src
    git fetch origin
    git branch DragonFly_RELEASE_5_2 origin/DragonFly_RELEASE_5_2
    git checkout DragonFly_RELEASE_5_2
    git pull

And then rebuild: (in /usr/src )

    make buildworld
    make buildkernel
    make installkernel
    make installworld
    make upgrade

Don't forget to upgrade your existing packages.  5.2 packages have already been built and are immediately available.

    pkg upgrade

## All changes since DragonFly 5.0

### Kernel

* The kernel-based if_ppp nic has been replaced with user ppp, i.e. ppp(8) and tun(4).  if_ppp has not been part of the base kernel config for a while.  It has now been removed.

* faith(4), also not part of the base kernel config for a while, has also been removed.  It has been determined that this IPV4<->IPV6 shim is no longer needed.

* tmpfs performance significantly improved when a large number of files are present.

* Fix a rare lockmgr() state transition, improving stability.

* Improve tsleep()/wakeup() queue collisions, significantly reducing spurious wakeup-related IPIs and improving performance.

* Fix GCC reordering issues with td_critcount that could lead to misordering and instability.

* Refuse to swapoff under certain conditions (presence of swapped tmpfs or vn pages), instead of panicing.

* cache-line separation for many things, reducing cache line collisions and improving performance on many-core systems.

* Essentially rewrite lockmgr(), significantly increasing shared lock performance and improving most other code paths.

* Refactor the buffer cache subsystem, removing B_MALLOC support entirely.  Also remove repurposebuf() (This was a shim prior to KVABIO support that is no longer needed).  Remove geteblk().

* Add KVABIO support to the buffer cache subsystem.  This allows buffer cache buffers to avoid smp_invltlb() calls on its buffer mappings when the related filesystems and storage devices in the chain also support KVABIO.  In particular, HAMMER2, the disk layer, and the NVME controller now support KVABIO.  tmpfs also supports KVABIO.

* Remove vm_hold_load_pages() and vm_hold_free_pages().  The only remaining use case was in CAM and we can change CAM to use getpbuf_mem() instead.

* Refactor SMP collisions statistics somewhat.

* Improve mountlist_scan() performance by using a shared token whenever possible.  Also fix a rare race, improving stability during periods of heavy mounting and unmounting (such as synth's use of tmpfs mounts).

* Refactor the pipe() code to significantly reduce IPIs and improve performance on many-cpu systems.  Performance is improved 75-100% (almost double in some cases).

* Increase the ncmount_cache array, improving the hit ratio to almost 100%.  We were hitting the slow path too often in synth runs.  The slow path is really horrible.

* Simplify umtx_sleep() and umtx_wakeup() support to improve pthreads performance and reduce bottlenecks.  Also rip-out vm_page_action() and vm_page_event() which this API used.  The vm_page_action/event code was pretty unfriendly to SMP.  Use a shared spinlock in places where it was exclusive before, significantly improving concurrent threaded umtx operations (pthreads again).

* Increase the pmap placemarks hash from 16 to 64 entries.  This improves concurrent fault performance a bit for heavily threaded programs.

* Refactor uidinfo.  Again improves SMP performance on many-cpus systems by removing a memory bottleneck.  Hits can now function locklessly.

* Partition large anonymous mmap()s, improving concurrent VM fault operation for threaded programs.  Note that the pmap itself is still a bottleneck.

* Add atomic_fcmpset_*() (suggested by mjg_), and use it all over the place to reduce unnecessary cache line ping-ponging during contended locking operations.

* Clean up ucred and plimit cache line locality, reducing unnecessary cache-line ping-ponging and improving performance.

* Refactor vm_page_busy() to add support for acquiring a soft-busy count without requiring ah ard-busy.  This allows us to 'lock' a vm_page in a shared manner via the soft-busy for situations where we only intend to read from it, aka sendfile() and vop_helper_read_shortcut().  Improves read() performance through filesystems.

* Add a D_QUICK device flag which allows certain static devices such as /dev/urandom, /dev/zero, and /dev/null to handle open() and close() in a more SMP-friendly manner.

* Improve SMP performance for devfs VOPs.  Ipmroves getattr(), read(), and readlink().  Mainly reduces contention during concurrent exec()s which almost always do the same bit of manipulation on specific devices, such as obtaining a random seed.

* Improve pmap hinting and performance.

* Refactor sysctl locking, significantly reducing SMP contention in heavily loaded environments.  Also allow certain sysctl()s to execute completely unlocked.

* Fix ACPI confusion that could arise due to ACPI_SEMAPHORES_MAX_PENDING being too small.  Redefine to a very large number.

* Adjust the proc structure a little so getppid() can run lockless.

* Implement exclusive priority for tokens under certain circumstances, improving performance when an exclusive token is needed on a heavily shared token.

* Add a second pageout daemon which only operates on anonymous pages.  This reduces low memory deadlocks that can arise due to the pageout daemon getting stuck on a filesystem-backed vm_page.

* Rewrite sendfile() to fix panics and corruption and to improve performance.  sendfile() now uses a UIO_NOCOPY/VMIO based VOP_READ() in the blind and loops to obtain file data instead of messing with individual pages.  We retain the synchronous VOP_READ() operation we had before because there isn't much point asynchronizing it for a web server which is already very heavily threaded.

* Fix a cluster_awrite() race, improving stability.

* Fix bugs in the namecache cleaning code and refactor the code.  This fixes a cpu live-lock situation that can arise when the namecache is dealing with a large number of hardlinked files.

* Fix boot and kernel issues when running with greater than 512GB of physical ram.  There were numerous scaling issues, including a bad DMAP initialization loop.  Tested to 1TB of ram.  DragonFly can now theoretically support up to 64TB of physical ram.

* Expand all page-count fields to 64 bits.  32-bit page-count fields limit us to 8TB of physical ram.  Expand to allow up to the DMAP limit of 64TB.

* Fix a 32-bit overflow in the pv_entry init code related to systems with large amounts of physical memory.

* Significantly increase the amount of KVA (Kernel Virtual Memory) appropriated in order to properly scale to larger amounts of physical ram.

* Improve magazine recycling performance in objcache by increasing MAGAZINE_CAPACITY_MAX.

* Fix an overflow in the vm.max_proc_mmap() calculation which could cause unexpected mmap() failures.

* Remove tons of debugging statistics that are no longer needed.  This also cleans up a ton of unnecessary cache-line ping-ponging that was occurring in the VFS subsystem.

* Fix bugs in vm_fault and vm_map related to concurrent faults in heavily threaded programs.  Also fix a bug in vm_fault_page().

* Attempt to fix a CAM unit numbering race that could result in two devices being assigned the same unit number.

* Increase the syscons history buffer from 4 screens to 10 screens worth.

* Adjust nbuf, maxvnodes, and max_softdeps counts to accomodate systems with a lot of physical memory.  Cap some of the fields to avoid wiring too much physical memory in our default configuration, as large-memory configurations are most typically intended to be used by userland, not kernel caching.

* Change several sysctls from 32-bits to 64-bits (e.g. hw.usermem, etc, which were in units of pages).  Implement a backwards compatibility feature for applications which still use 32-bit ints.

* Fix a crash which can occur when a cdev is destroyed.

* Fix an arbitrary maximum file size limitation in tmpfs.

* EFI code refactored to use TianoCore EDK II headers.

* Misc i386 cruft removed.

* Refactor and clean up a large chunk of the nata driver so it can be modularized.

* Increase the microcode size limit from 32KB to 4MB so we can support newer microcodes.

* Handle ig4 startup errors for situations where the BIOS makes it inaccessible.  Also implement driver priortization for ig4 and gpio_intel to manage probe order because we don't handle ACPI _DEP methods properly.

* lock xhci during attach to avoid spurious interrupts from executing too early.

* Add Kabylake support to corepower.ko

* Fix kernel minidumps.  These were broken when we extended physical memory support to multiple DMAP pages.  The minidump now dumps the PDP array for the entire system VA space.  Minidumps will actually be a bit smaller on systems with lots of memory due to the page-table optimizations made in the dump format.

* Significantly reduce kernel boot times due to various timer calibration functions.

* Fix CAM peripheral error handling, primarily for AHCI.  This fixes instabilities which could develop when a storage device throws lots of errors, ultimately resulting in a kernel panic.

* Fix a dataloss issue that can occur with tmpfs vnodes when the kernel devices to recycle the vnodes.

* Remove kprintf %r, %b, %y support, as well as from libstand.

### Networking

* Limit the number of accepted sockets that kevent() reports.  This improves the performance for heavily optimized userland networking workloads.  Specifically, nginx.

* Remove the power-of-2 cpu threads limitation for netisrs.  Netisrs now run on all cpu threads, improving load balancing and performance.

* In if_vtnet, disable rx csum offload due to unuspported ipv6 rx csum offloading.

* In if_vtnet, clean up the IFCAP flags.

* Make non-prefix and directly reachable inet6 routes work properly.

* Properly zero out syncache_percpu, fixing a rare panic.

* Synchronize ix to Intel's ix-3.2.17.

* Synchronize igb with Intel's igb-2.5.3.

* Synchronize em/emx to Intel's em-7.6.2.

* Change how the TCP local ephermal port is selected to reduce the chance of connection forwarding between cpus.  This change significantly reduces the IPI rate for connection establishment.

* Increase max stage packet count to 16.  This increases the IP forwarding rate in tests by 200Kpps to 300Kpps.  Normal forwarding achieves 11.3Mpps while fastforwarding achieves 14.2Mpps.  Values higher than 16 do not improve the rate further.

* Set ip_porthash_trycount based on the number of netisrs.  This reduces connect IPIs on many-cores machines and improves the connection rate a bit in tests.

* Use a fast close if SO_LINGER and linger == 0.  This is widely used to reset a TCP connection by userland applications.

### Other drivers

* Bring in vmx(4) (VMWare virtual network driver, aka vmxnet3)

* nvme driver now supports KVABIO for significantly higher performance and concurrency on many-core systems.

* Fix the capacity calculation in virtio_blk

* Upgrade arcmsr(4) to Areca's Revision 1.40.00.00, which adds support for ARC1203, ARC1216, ARC1226, and ARC1884.

* Fix mtime updates for deferred writes from shared R+W mmap()s.  This bug could cause make and other programs to get confused, often triggered by ld.gold's use of shared R+W mmap().

* Fix a vclean() assertion due to a short-cut taken by TMPFS.

* Improve compatibility for kqueue(2)'s EVFILT_FS support.

* Bring in autofs.

* Restore floppy driver functionality.  Floppies work again!  REJOICE!

* Bring in evdev from FreeBSD (a generic input event interface).  Add evdev support to kbdmux, ukbd, and ums.

### Userland

* Removed many 'r' commands and daemons.  rcp, rlogin, rlogind, rsh, and rshd have been removed.  People that still need them can get them from the net/bsdrcmds port.

* Remove mailaddr(7).

* Kernel configuration cleanup, numerous options that have been NOPs for a long time have been removed outright.

* NOTE!  Due to base compiler updates, upgrading to 5.1 and later versions now require at least a 4.4 (or later) system.

* Bring s_ceill() into libc from OpenBSD.

* Fix a few issues in libthread_xu that we believe were confusing certain heavily-threaded user applications, improving the stability of those applications.

* Refactor rtld locks as well as add an atfork() facility.  Refactor many libthread_xu modules (pthreads).  All this work improves the stability for fork/exec operations in heavily threaded environments (particularly chrome threads).

* Numerous pthreads fixes for chrome, thunderbird, and other heavily threaded programs.  Also use the new atfork() facility in our malloc implementation to fix deadlocks which could sometimes arise when heavily threaded programs fork().

* Fix locale support for cal(1).

* Fix symbol conflicts between libcrypt and LIBRECRYPTO (buildworld issues).

* Add static PAM modules support.

* Remove inherited helpers left over from the initial FreeBSD fork 10+ years ago.  Also remove helpers for upgrading directly from pre-4.4, we no longer support upgrading directly from pre-4.4 (its basically impossible due to compiler differences).

* Really support a static toolchain.

* Jailbreak games from Dungeon Master.  Games are no longer group-shared, improving security.  Only allow users in the games group to write to the /var/games directory.

* Remove dm(8).

* Patch and cleanup a few games.  battlestar(6), hack(6), rogue(6).

* Move-out non-games from /usr/games.  Move strfile(8) and unstr(8) to /usr/bin.

* Move banner(6) into games/ where it belongs.  Bring in OpenBSD's banner(1).  Bring in OpenBSD's boggle(6), bring in OpenBSD's tetris(6).

* Add hostprog variant of tic and link against the wide version of ncurses.

* Update strfile() to big-endian and produce architecture-independent .dat files.

* Fix the ownership of several directories in /usr/share.

* Add a few more fortunes to fortune(6).

* Use O_CLOEXEC in many more places to reduce descriptor leakages when heavily threaded programs fork().

* Fix seg-faults in top(1) on many-core systems.

* Preemptive censor certain unnamed quotes in fortune because honestly we're a laid back BSD and don't want to deal with the flying crap.

* Add a helper script, efisetup(8), which will set up a blank storage device with appropriate gpt partitions for EFI and a DFly disklabel in s1.

* Add cfmakesane() to libc/termios.

* Add support for sha512 to md5(1).

* Handle size_t overflows in posix_memalign() and malloc().

* Add decoding functionality to morse(6).  You can use -r to supply a sequence of dots and dashes and morse(6) will decode it.

* Improve application compatibility by relaxing the return code on fflush() when used on a read-only stream.  Return success for this case.  Needed to support the c++ runtime library.  Taken from FreeBSD.

* Many manual page cleanups.

* Misc libc/locale cleanups and a few adjustments.

* libc/nls synchronized with FreeBSD (Jan 4 2018, roughly).

* Fix an old bug in newsyslog related to log file retention.  One extra file was being retained.

### Various tools have been upgraded in the base system:

* cpdup has a few new features to make .cpignore files more useful.

* cpdup is now able to rmdir chflagged directories (it already handles this properly for regular files).

* cpdup now ignores the UF_ARCHIVE file flag when deciding whether to copy a file.

* ls command has been enhanced with new options -S and -D, see manual page.

* Enhance nologin to log nologin attempts to syslog.

* Fix set-faults on crypt(3) failures.

* OpenSSH upgraded to 7.6p1.

* Update the pciconf(8) database to the Dec 20 2017 snapshot.

* Fix vmstat formatting to deal with large numbers and not run numbers into each other any more.

* Zoneinfo synchronized to tzdata2018c.

* Minor fixes to pr(1).

* newlocale(3) add compat support for c++ runtime libs.  Add support for native complex locale.

* Amd10h support added to cpucontrol.  Allows AMD microcodes to be installed from their packed format, instead of having to unpack them first (from FreeBSD).

* lvm(8) Add line editing support via libedit.

* Add a new modifier 'R' to newsyslog, alternative to sending a signal to the process.  Also, -n now implies -r.  mtime is updated after archiving.  Fix a few incorrect messages.  Fix interval-based rotations when the -t flag is used.  And a fix for NFS.  And many other adjustments. Mostly from FreeBSD.

* fish(6) no longer allows users to request cards they have made books for.

### Hammer1 Changes

* Numerous cleanups.

### Hammer2 Changes

* Many, many bug fixes, improving stability.

* Numerous performance fixes, including KVABIO support which improves performance on many-core systems.

* Add support for hammer2 cleaning in the periodic cron, enabled by default.

* Work around a bug in LZ4_compress_limitedOutput(), the code could sometimes overrun the supplied buffer.

* Add hammer2 utilities to initrd for hammer2 crypto bootstrapping and rescue.

### Compiler status

* A port of gcc-80 is in-progress but not yet ready.

* Clang currently available via a dport, but there may still be issues trying to compile the world or kernel with it.

### Package updates

* We can now build over 29000 packages.

* Note that firefox does not currently build due to the rust language being out of date.  We are working on that and also working on getting a more recent version of chrome into the tree.

### 64-bit status

* Note that DragonFly is a 64-bit-only operating system as of 4.6, and will not run on 32-bit hardware.