10 months agodsynth - Change default compression from .tgz to .txz
Matthew Dillon [Wed, 6 Nov 2019 17:31:26 +0000 (09:31 -0800)]
dsynth - Change default compression from .tgz to .txz

* Change default compression from .tgz to .txz, just so the
  results are space-efficient, even though this will slow down
  bulk builds.

* Anyone can change the compression algo via Package_suffix,
  see 'man dsynth'.  .tgz is recommended if reducing bulk build
  time is critical for you.

Requested-by: zrj
10 months agosys/vfs/msdosfs: Remove seemingly unneeded unlock/relock of dvp
Tomohiro Kusumi [Wed, 6 Nov 2019 16:35:02 +0000 (01:35 +0900)]
sys/vfs/msdosfs: Remove seemingly unneeded unlock/relock of dvp

from freebsd/freebsd@21f579c3fb6b782ee610255d7f5bbb22e326b66c

10 months agosys/vfs/msdosfs: Remove unrelated comment on rmdir
Tomohiro Kusumi [Wed, 6 Nov 2019 13:34:04 +0000 (22:34 +0900)]
sys/vfs/msdosfs: Remove unrelated comment on rmdir

FreeBSD explicitly calls cache_purge() here.

10 months agodsynth - Add ARCH make var to bulk scan
Matthew Dillon [Tue, 5 Nov 2019 21:47:48 +0000 (13:47 -0800)]
dsynth - Add ARCH make var to bulk scan

* Add the ARCH make variable and env variable to bulk scan operations.

Reported-by: tuxillo
10 months agosys/vfs/msdosfs: Sync debug prints with FreeBSD
Tomohiro Kusumi [Tue, 5 Nov 2019 13:18:35 +0000 (22:18 +0900)]
sys/vfs/msdosfs: Sync debug prints with FreeBSD

from freebsd/freebsd@2531d8deb3ae1fe51c43a46f528fda3fdf1806e2

10 months agodsynth - Conditionalize use of check-plist stage
Matthew Dillon [Mon, 4 Nov 2019 19:48:21 +0000 (11:48 -0800)]
dsynth - Conditionalize use of check-plist stage

* Only run the check-plist stage for:

  dsynth -P <anything>
  dsynth -D everything
  dsynth test ...

10 months agodevfs: Add a vnode pointer argument to dev_dopen()
François Tigeot [Sun, 3 Nov 2019 21:00:19 +0000 (22:00 +0100)]
devfs: Add a vnode pointer argument to dev_dopen()

* Initialize fp->f_data with it when possible

* drm drivers expect to be able to access vnode information in
  their open() function

Reviewed-by: dillon
10 months agodrm/linux: Rework wait_event_xxx() macros
François Tigeot [Sun, 3 Nov 2019 20:31:43 +0000 (21:31 +0100)]
drm/linux: Rework wait_event_xxx() macros

The wait queue lock must not be held permanently.

This commit should fix issue #3200, reported-by Peeter.

10 months agodsynth - Enhance FAILURE log message
Matthew Dillon [Sun, 3 Nov 2019 20:06:54 +0000 (12:06 -0800)]
dsynth - Enhance FAILURE log message

* Enhance the 02_failure_list.log messages to also print the number of
  other packages being skipped due to the failure.

Requested-by: zrj
10 months agofile(1): Fix CVE-2019-18218.
Sascha Wildner [Sun, 3 Nov 2019 12:31:12 +0000 (13:31 +0100)]
file(1): Fix CVE-2019-18218.

For further details see:


Reported-by: zrj
10 months agosbin/hammer2: Add /dev/ad* and /dev/vn* to "info"/"mountall" targets
Tomohiro Kusumi [Sun, 3 Nov 2019 09:12:05 +0000 (18:12 +0900)]
sbin/hammer2: Add /dev/ad* and /dev/vn* to "info"/"mountall" targets

10 months agodrm/linux: Only define GFP_KERNEL once
François Tigeot [Sun, 3 Nov 2019 09:50:21 +0000 (10:50 +0100)]
drm/linux: Only define GFP_KERNEL once

10 months agodrm: Remove most system headers from linux/list.h
François Tigeot [Sun, 3 Nov 2019 09:35:50 +0000 (10:35 +0100)]
drm: Remove most system headers from linux/list.h

They do not serve any useful purpose there.

Suggested-by: zrj
10 months ago<wchar.h>: Remove limits pollution.
zrj [Sat, 2 Nov 2019 20:16:51 +0000 (22:16 +0200)]
<wchar.h>: Remove limits pollution.

 The <wchar.h> now should properly only define WCHAR_MIN/WCHAR_MAX as per
 opengroups.  Fix fallout in col(1) that makes use of INT_MIN/INT_MAX.

 No binary differences for buildworld.

 While there, add comment why <ctype.h> is in <wchar.h>.

10 months ago<stdint.h>: Remove C99 constant/limits guards for c++.
zrj [Sat, 2 Nov 2019 18:39:19 +0000 (20:39 +0200)]
<stdint.h>: Remove C99 constant/limits guards for c++.

 It seems that in C11 problematic clauses where removed.  This removes
 the inconsistency for -std=c++98 between base and DPorts c++ compilers
 (that have #include_next <stdint.h> wrapper for this particular issue).
 For now keep guard defines in <sys/cdefs.h> because of those wrappers.

 No functional change in buildworld.

10 months ago<wchar.h>: Implement correct limits for different wchar_t flavors.
zrj [Sat, 2 Nov 2019 17:02:41 +0000 (19:02 +0200)]
<wchar.h>: Implement correct limits for different wchar_t flavors.

 Try best to select correct limits while always using handpicked standard
 C wchar_t bounds that match internal rune_t and standard wint_t types.
 It might be worth to switch to using compiler provided limits for all.
 The <stdint.h> header should provide limits for all types in standard
 headers, so provide wint_t (<wchar.h>, <wctype.h>) limits too.
 The <machine/wchar_limits.h> idea taken from NetBSD.

10 months ago<ctype.h>: Internalize __ct_rune_t type.
zrj [Sat, 2 Nov 2019 14:47:58 +0000 (16:47 +0200)]
<ctype.h>: Internalize __ct_rune_t type.

 Kernel does not use rune_t types, this change essentially removes rune_t
 types from kernel namespace (rune_t types were intended to be used only
 internally in libc/locale/ scope).

 ANSI requires that ctype types should be of type 'int' so it is
 debatable if such custom type is needed at all.  Provide declaration in
 <runetype.h> as well to allow header be self sufficient.

10 months ago<wchar.h>: Refactor wchar_t/wint_t/mb_state setup.
zrj [Sat, 2 Nov 2019 11:26:14 +0000 (13:26 +0200)]
<wchar.h>: Refactor wchar_t/wint_t/mb_state setup.

 Kernel has no wchar support functions (except for VKERNEL64).

 * Extract wide-character types setup to common <machine/wchar.h>.
 * Avoid using __wchar_t for clang -fms-extensions issue.
 * Add more handling for c++ and -fshort-wchar (not supported in libc).

 This essentially removes these types from kernel namespace and paves
 the way for further both userland/kernelland visibility cleanup by
 making <sys/stdint.h> clean.
 The WCHAR_MAX/WCHAR_MIN limits will be addressed in separate commit.

10 months agopthread: Fix simple hello world c++ statically linked use cases.
zrj [Sat, 2 Nov 2019 10:39:06 +0000 (12:39 +0200)]
pthread: Fix simple hello world c++ statically linked use cases.

 Can happen if locale setup is called before pthread_create() and c++
 program is linked with -static -pthread.  Check initialization status
 in mutex and once related calls.

10 months agovkernel64: Reduce <pthread.h> exposure to generic kernel sources.
zrj [Sat, 2 Nov 2019 09:35:26 +0000 (11:35 +0200)]
vkernel64: Reduce <pthread.h> exposure to generic kernel sources.

 Implement vkernel_yield() wrapper and use it where needed.

10 months agokernel/drm: Yeet drm_free() inline.
zrj [Sat, 2 Nov 2019 06:41:56 +0000 (08:41 +0200)]
kernel/drm: Yeet drm_free() inline.

 Last consumers removed in 6af927c246477a611cb86e64d0fc460000a39284

10 months ago<sys/malloc.h>: Avoid self referencing macros.
zrj [Fri, 25 Oct 2019 18:05:10 +0000 (21:05 +0300)]
<sys/malloc.h>: Avoid self referencing macros.

 It was reported that previous kmalloc() implementation at header level
 is breaking tools like Coccinelle and likely others.  Since this header
 now has implicit requirement on other headers only for using the
 MALLOC_DEFINE() macro (<sys/kernel.h> must be included before), it is
 possible to safely use inline functions.  Inline function does not have
 flexibility limitations like preprocessor macro and expands to much
 nicer line in compiler intermediates while allowing to easily make
 future adjustments when needed.  Also we should be able to no longer
 avoid using kmalloc*() calls directly in other kernel headers.

 The LINT64 builds with SLAB_DEBUG option enabled are not affected by
 this change, but for consistency add alias for _kmalloc() too.

 Move in the "#if 1" into the _kmalloc() body, this allows compiler to
 perform additional optimizations for non __builtin_constant_p() cases.
 As a benefit if during compilation the -O0 is used, compiler disables
 optimized paths in inline functions.

 Compared with -save-temps on GENERIC without debug enabled.

10 months agolibthread_xu: Remove local versions of 3 parameter timespec{add,sub}().
Sascha Wildner [Sat, 2 Nov 2019 19:07:49 +0000 (20:07 +0100)]
libthread_xu: Remove local versions of 3 parameter timespec{add,sub}().

Adjust the parameter order. Comparison of preprocessed code before and
after this commit looks good.

10 months agosbin/hammer2: Test /dev/mampper for "info"/"mountall" targets
Tomohiro Kusumi [Fri, 1 Nov 2019 16:36:34 +0000 (01:36 +0900)]
sbin/hammer2: Test /dev/mampper for "info"/"mountall" targets

Try dirents under /dev/mapper in addition to /dev/serno.
Also ignore ".", ".." for both serno/dm, and "control" for dm.

10 months agotermios.4: Add a bit more info about TABDLY, TAB0, and TAB3.
Sascha Wildner [Fri, 1 Nov 2019 22:35:29 +0000 (23:35 +0100)]
termios.4: Add a bit more info about TABDLY, TAB0, and TAB3.

Also add a SEE ALSO section.

Taken-from: FreeBSD

10 months ago<sys/malloc.h>: Reduce kmalloc() implicit requirements on <sys/systm.h>.
zrj [Fri, 25 Oct 2019 10:54:32 +0000 (13:54 +0300)]
<sys/malloc.h>: Reduce kmalloc() implicit requirements on <sys/systm.h>.

 Use __builtin_memset() directly.  Use of __DEQUALIFY() not needed here.

 The "#if 1" case already makes use of __builtin_constant_p() compiler
 extension that we do not have feature check in <sys/cdefs.h> (using
 "#if __GNUC_PREREQ__(2, 0)" should be good enough).

 This finally decouples <sys/malloc.h> from <sys/systm.h> bzero() macro
 visibility (also in "#if 1" block).

10 months ago<sys/malloc.h>: Reduce kmalloc() implicit requirements for NULL.
zrj [Fri, 25 Oct 2019 09:57:33 +0000 (12:57 +0300)]
<sys/malloc.h>: Reduce kmalloc() implicit requirements for NULL.

 Make sure it is available for ease of use.  This potentially allows to
 not to add <sys/param.h> in kernel sources just for NULL when kmalloc()
 is used in source code.

10 months agokernel/hammer2: Make C_HASHTABLE static.
zrj [Fri, 25 Oct 2019 09:56:00 +0000 (12:56 +0300)]
kernel/hammer2: Make C_HASHTABLE static.

 Only used in lz4.c locally.  Include hammer2.h first to include common
 headers first.

10 months agokernel/dm: Include <sys/param.h> first.
zrj [Fri, 25 Oct 2019 09:54:48 +0000 (12:54 +0300)]
kernel/dm: Include <sys/param.h> first.

 Allow the <dev/disk/dm/dm.h> to include common headers first.

10 months agokernel/pc64: Move up include order in vmx.c.
zrj [Fri, 25 Oct 2019 08:35:06 +0000 (11:35 +0300)]
kernel/pc64: Move up include order in vmx.c.

 Include <sys/param.h> first and remove not necessary <sys/types.h>.

 No functional change for generated code.

10 months agovmstat(8): Remove no longer needed <sys/malloc.h> include.
zrj [Wed, 23 Oct 2019 13:52:13 +0000 (16:52 +0300)]
vmstat(8): Remove no longer needed <sys/malloc.h> include.

 The <sys/objcache.h> now provides the "struct malloc_type" definition.
 This leaves only two utilities in base userland that reference kernel
 kmalloc() header:
 * sbin/vinum through <dev/raid/vinum/vinumhdr.h>
 * usr.bin/fstat through various <vfs/FS/FS.h> headers.

10 months agofwcontrol(8): Remove <sys/malloc.h> include.
zrj [Tue, 22 Oct 2019 14:59:33 +0000 (17:59 +0300)]
fwcontrol(8): Remove <sys/malloc.h> include.

 After previous patches there should not be a case where this header in
 userland visibility would be needed anymore.

10 months ago<net/if_var.h>: Remove last explicit dependency on <sys/malloc.h>.
zrj [Tue, 22 Oct 2019 16:42:52 +0000 (19:42 +0300)]
<net/if_var.h>: Remove last explicit dependency on <sys/malloc.h>.

 These kernel sources pass M_NOWAIT flag to m_copym() and friends.
 Mark that it was for M_NOWAIT visibility.

10 months ago<sys/malloc.h>: Add comment about assumption in drm layer.
zrj [Tue, 22 Oct 2019 16:31:21 +0000 (19:31 +0300)]
<sys/malloc.h>: Add comment about assumption in drm layer.

10 months agokernel/drm: Decouple from network headers.
zrj [Tue, 22 Oct 2019 16:18:56 +0000 (19:18 +0300)]
kernel/drm: Decouple from network headers.

 This removes all <net*/*.h> headers (including <sys/mbuf.h>) from
 dependency chain in all drm devices.  There is no need to use network
 headers in drm compat layer.  Previously drm was using <net/if_var.h>
 as a fallback to get <sys/malloc.h>.  Use kmalloc/kfree and M_* flags
 in linux/gfp.h.  At this point no system headers should be including
 <sys/malloc.h> except for disk and vfs headers for obvious reasons.

 Later on even M_* flags could be moved into <sys/_malloc.h> for better
 separation to linux/slab.h.

10 months ago<sys/idr.h>: Move implementation details to Linux compat layer.
zrj [Tue, 22 Oct 2019 14:27:16 +0000 (17:27 +0300)]
<sys/idr.h>: Move implementation details to Linux compat layer.

 Nothing is needed here, linux_idr.c already includes <sys/malloc.h>,
 while drm layer still needs malloc prototypes.  So move it there.

10 months agoRemove nextboot remains.
Sascha Wildner [Fri, 1 Nov 2019 11:39:29 +0000 (12:39 +0100)]
Remove nextboot remains.

While here, staticize reboot(8) a bit.

10 months agoRemove checknr(1) which stopped being useful a long time ago.
Sascha Wildner [Fri, 1 Nov 2019 10:24:18 +0000 (11:24 +0100)]
Remove checknr(1) which stopped being useful a long time ago.

It reports a ton of errors on pretty much any manual page, regardless
of macro dialect.

mandoc(1) (and groff) have much better checking capabilities.

10 months agolibtelnet: Use NULL for a pointer.
Sascha Wildner [Fri, 1 Nov 2019 10:06:41 +0000 (11:06 +0100)]
libtelnet: Use NULL for a pointer.

10 months agolibipfw3: Use NULL for a pointer.
Sascha Wildner [Fri, 1 Nov 2019 10:06:32 +0000 (11:06 +0100)]
libipfw3: Use NULL for a pointer.

10 months agorouted(8): Use NULL for a pointer.
Sascha Wildner [Fri, 1 Nov 2019 10:06:28 +0000 (11:06 +0100)]
routed(8): Use NULL for a pointer.

10 months agousr.sbin/fstyp: Fix warning on Linux distros
Tomohiro Kusumi [Thu, 31 Oct 2019 23:47:02 +0000 (08:47 +0900)]
usr.sbin/fstyp: Fix warning on Linux distros

hammer2.c:134: warning: pointer targets in passing argument 1 of 'strcmp' differ in signedness
/usr/include/string.h:143: note: expected 'const char *' but argument is of type 'unsigned char *'

10 months agonrelease: Support to use binary packages
Aaron LI [Sun, 3 Mar 2019 13:19:11 +0000 (21:19 +0800)]
nrelease: Support to use binary packages

Add the new optional target "binpkgs" to allow to use binary packages,
i.e., installing them with pkg(8), instead of building them from
dports(7).  This can be handy for end users who want to quickly build
a custom image with custom packages.

As pointed out by Sascha Wildner, one port may have multiple binary
packages (e.g., Python flavors) or have a different name that may not be
found by pkg(8).  Therefore, allow to specify the exact binary package
name for a port by setting 'PKG_<port>=<pkg-name>'.

This feature is disabled by default, as suggested by Sascha Wildner,
because the release process should be self-contained as much as possible.

Update the help text and man page accordingly.

10 months ago<termios.h>: #define MDMBUF as an alias for CCAR_OFLOW.
Sascha Wildner [Thu, 31 Oct 2019 21:38:41 +0000 (22:38 +0100)]
<termios.h>: #define MDMBUF as an alias for CCAR_OFLOW.

10 months ago<sys/posix4.h>: Remove single use macros.
zrj [Tue, 22 Oct 2019 14:44:26 +0000 (17:44 +0300)]
<sys/posix4.h>: Remove single use macros.

 Use kmalloc()/kfree() directly.

10 months agokernel/vfs: Consistently check if MALLOC_DECLARE was defined.
zrj [Sun, 20 Oct 2019 19:12:48 +0000 (22:12 +0300)]
kernel/vfs: Consistently check if MALLOC_DECLARE was defined.

 Keep these checks even in _KERNEL context and even if <sys/malloc.h>
 is included before.  Helps a lot for userland testing.

10 months agokernel: Remove explicit dependencies on <sys/malloc.h> in headers.
zrj [Sun, 20 Oct 2019 18:15:20 +0000 (21:15 +0300)]
kernel: Remove explicit dependencies on <sys/malloc.h> in headers.

 All except <net/if_var.h> for now, it needs decoupling in drm first.
 * Include <sys/malloc.h> in foo.c if they have kmalloc()/kfree() calls.
 * Consistently check if MALLOC_DECLARE was declared before.
 * <sys/mountctl.h>: include <sys/thread.h> for _KERNEL_STRUCTURES too
   since the "struct journal" embeds "struct thread".
 * <sys/tty.h>: Only two kernel sources makes use of M_TTYS.
 * <sys/socketvar2.h>: Make it kernel only header.

10 months agokernel: Minor whitespace cleanup in few sources (part 2).
zrj [Sun, 20 Oct 2019 18:13:22 +0000 (21:13 +0300)]
kernel: Minor whitespace cleanup in few sources (part 2).

 Separated from next.

10 months ago<sys/sysref.h>: Switch to lighter <sys/_malloc.h> header.
zrj [Sun, 20 Oct 2019 18:05:43 +0000 (21:05 +0300)]
<sys/sysref.h>: Switch to lighter <sys/_malloc.h> header.

 * Make <sys/sysref2.h> a kernel only header.
 * Remove sys/types.h includes that follow <sys/param.h> in devfs(5).
 * Add sys/malloc.h includes where it is actually used in sources.

 While there, minor whitespace cleanup.

10 months ago<sys/objcache.h>: Switch to lighter <sys/_malloc.h> header.
zrj [Sun, 20 Oct 2019 17:45:00 +0000 (20:45 +0300)]
<sys/objcache.h>: Switch to lighter <sys/_malloc.h> header.

 Exposing "struct malloc_type" internals is useful even under kernel
 structures namespace. In vmstat(8) this will allow to drop dependency
 on the <sys/malloc.h> header.

10 months agokernel: Remove not needed header includes in AIO stub.
zrj [Fri, 18 Oct 2019 14:43:47 +0000 (17:43 +0300)]
kernel: Remove not needed header includes in AIO stub.

 AIO not available since 91bcbcd6bef430dfa09526e0af9e24fa8319ae50.

10 months ago<sys/slaballoc.h>: Switch to lighter <sys/_malloc.h> header.
zrj [Fri, 18 Oct 2019 14:40:32 +0000 (17:40 +0300)]
<sys/slaballoc.h>: Switch to lighter <sys/_malloc.h> header.

 The <sys/globaldata.h> embeds SLGlobalData that in turn embeds the
 "struct malloc_type".  Adjust several kernel sources for missing
 includes where memory allocation is performed.  Try to use alphabetical
 include order.

 Now (in most cases) <sys/malloc.h> is included after <sys/objcache.h>.
 Once it gets cleaned up, the <sys/malloc.h> inclusion could be moved
 out of <sys/idr.h> to drm Linux compat layer linux/slab.h without side

10 months agokernel: Minor whitespace cleanup in few sources.
zrj [Fri, 18 Oct 2019 14:36:39 +0000 (17:36 +0300)]
kernel: Minor whitespace cleanup in few sources.

10 months ago<sys/malloc.h>: Separate basic typedefs to _malloc.h hearder.
zrj [Fri, 18 Oct 2019 13:43:12 +0000 (16:43 +0300)]
<sys/malloc.h>: Separate basic typedefs to _malloc.h hearder.

 This will be used to reduce <sys/globaldata.h> pollution through the
 <sys/slaballoc.h> and will allow not to include <sys/malloc.h> for
 almost every kernel source even if no memory allocations are done.

 While there move MALLOC_DECLARE() macro too, it would help with malloc
 type visibility from headers that define it and will allow finally to
 sort most of the header includes alphabetically without side effects.

10 months ago<termios.h>: Add TABDLY, TAB0 and TAB3 to satisfy POSIX a bit better.
Sascha Wildner [Thu, 31 Oct 2019 21:01:09 +0000 (22:01 +0100)]
<termios.h>: Add TABDLY, TAB0 and TAB3 to satisfy POSIX a bit better.

* TAB3 is what we already have as OXTABS. Make the latter an alias of
  the former in <sys/_termios.h>.

* Add 'tab0' and 'tab3' operands to stty(1) too.

Most other output flags from the POSIX spec deal with actual time delays
'to allow for mechanical or other movement when certain characters are
sent to the terminal'. Blast from the past.

Taken-from: FreeBSD (with some adjustments)

10 months ago<sys/mpipe.h>: Mark as kernel only header.
zrj [Fri, 18 Oct 2019 13:24:43 +0000 (16:24 +0300)]
<sys/mpipe.h>: Mark as kernel only header.

 Not used in any of the kernel headers and should not be included by the
 userland and especially DPorts.  Adjust kern_mpipe.c too.

 While there, minor whitespace cleanup.

10 months ago<sys/event.h>: Make M_KQUEUE static.
zrj [Fri, 18 Oct 2019 12:54:18 +0000 (15:54 +0300)]
<sys/event.h>: Make M_KQUEUE static.

 Used in a single kernel source only.

 While there, minor whitespace cleanup.

10 months ago<sys/dmsg.h>: Reduce userspace pollution a bit.
zrj [Fri, 18 Oct 2019 12:39:40 +0000 (15:39 +0300)]
<sys/dmsg.h>: Reduce userspace pollution a bit.

 Kernel only prototypes where exposed after buildworld fixup in
 The 'struct kdmsg_iocom' is embedded in 'struct disk' (sys/disk.h).
 Also forward declare few structs to reduce dependencies.

 While there, remove kdmsg_icrc32*() prototypes that where added in:

10 months ago<pthread.h>: Fix typo: __restict -> __restrict.
Sascha Wildner [Thu, 31 Oct 2019 20:24:42 +0000 (21:24 +0100)]
<pthread.h>: Fix typo: __restict -> __restrict.

Reported-by: zrj
10 months agosbin/fsck_hammer2: Remove redundant hammer2_blockref_t in delta stats
Tomohiro Kusumi [Thu, 31 Oct 2019 15:14:08 +0000 (00:14 +0900)]
sbin/fsck_hammer2: Remove redundant hammer2_blockref_t in delta stats

10 months ago<sys/_termios.h>: FWIW, add 460800 and 921600 baud rate defines.
Sascha Wildner [Wed, 30 Oct 2019 21:18:59 +0000 (22:18 +0100)]
<sys/_termios.h>: FWIW, add 460800 and 921600 baud rate defines.

10 months agosbin/fsck_hammer2: Add -c to cache subtree blockref stats for speed
Tomohiro Kusumi [Tue, 29 Oct 2019 17:30:31 +0000 (02:30 +0900)]
sbin/fsck_hammer2: Add -c to cache subtree blockref stats for speed

If blockref scan succeeds and subtree blockref count exceeds a -c
value, cache subtree blockref stats and blockref itself.

This greatly improves speed when multiple PFSes share large parts
of blockrefs via snapshots. If blockrefs were somehow corrupted in
a weird way that exact same copies of blockref appear more than once,
this doesn't work, but should generally be useful.

Note that using a too small -c value may eat memory without much
performance improvement. Also note that using a too large -c value
saves memory but blockrefs may not be cached or used.

A suitable -c value depends on blockref tree structure since it's
a threshold for blockref count. A fs with more than a few GB of
consumption, should use a few thousand or more.

10 months agopthreads - Bypass third-party allocators for internal allocations
Matthew Dillon [Tue, 29 Oct 2019 23:09:33 +0000 (16:09 -0700)]
pthreads - Bypass third-party allocators for internal allocations

* Adjust libthread_xu to always use libc's malloc/free (__malloc,
  __free), bypassing third party allocators.

* Fixes interactions between third party alocators and pthreads
  which often cause deadlocks or other chicken-and-egg issues,
  or require serious hacks to work around.

10 months agotest/cocci: Improve a bit.
Sascha Wildner [Tue, 29 Oct 2019 19:46:08 +0000 (20:46 +0100)]
test/cocci: Improve a bit.

* Remove NELEM.cocci and put its contents into useful_macros.cocci, along
  with more macros (so far only from <sys/param.h).

* Mention in comma.cocci that the devel/ocaml-findlib port is needed for
  that patch.

10 months agoUse NULL for pointers in tools/usbtest too (forgotten in last commit).
Sascha Wildner [Tue, 29 Oct 2019 19:28:34 +0000 (20:28 +0100)]
Use NULL for pointers in tools/usbtest too (forgotten in last commit).

10 months agousb: Use NULL for pointers.
Sascha Wildner [Tue, 29 Oct 2019 19:26:18 +0000 (20:26 +0100)]
usb: Use NULL for pointers.

10 months agoboot/efi: Use NULL for a pointer.
Sascha Wildner [Tue, 29 Oct 2019 19:13:49 +0000 (20:13 +0100)]
boot/efi: Use NULL for a pointer.

10 months agodsynth - Allow empty env variable assignment
Matthew Dillon [Tue, 29 Oct 2019 17:27:27 +0000 (10:27 -0700)]
dsynth - Allow empty env variable assignment

* The parser was improperly complaining about empty env variable
  assignments.  Allow the case.

* Allows e.g. STRIP=

Reported-by: yellowrabbit2010
10 months agocpdup(1): Check the return value of asprintf() and exit if fails
Aaron LI [Tue, 29 Oct 2019 14:52:48 +0000 (22:52 +0800)]
cpdup(1): Check the return value of asprintf() and exit if fails

See also: https://github.com/DragonFlyBSD/cpdup/commit/90aee43e060104e974f623924c3dfc0a4359ceb5

Reported by GCC 9.x on Arch Linux.

10 months agocpdup(1): Add '-h' option and sort options in help and code
Aaron LI [Tue, 29 Oct 2019 14:49:41 +0000 (22:49 +0800)]
cpdup(1): Add '-h' option and sort options in help and code

* Add the '-h' option to show the help text
* Sort options in both code and help text
* Remove the redundant '-p' option from the source file
* The '-n' and '-S' options are missing from the help text, so add them.

See also: https://github.com/DragonFlyBSD/cpdup/commit/d87b2a60a95d0382135609bb17a014c8a0b61031

10 months agocpdup(1): Improve the help text a bit
Aaron LI [Tue, 29 Oct 2019 14:43:47 +0000 (22:43 +0800)]
cpdup(1): Improve the help text a bit

* Better format the help text to be clearer.
* The 'dest' argument is mandatory.

See also: https://github.com/DragonFlyBSD/cpdup/commit/22ef5685c8dd2ec770f7f2ee62f54b6e2b5f5f23

10 months agocpdup(1): Minor cleanups
Aaron LI [Tue, 29 Oct 2019 14:39:22 +0000 (22:39 +0800)]
cpdup(1): Minor cleanups

* In 'cpdup.h', remove an inaccurate comment about <strings.h>
* Change predefined macro '__linux' to '__linux__', meanwhile add an
  empty define of '__dead2' under '__linux__' scope.
* In 'hcproto.c', put static variable 'chflags_warning' inside
  '_ST_FLAGS_PRESENT_' macro scrope.

Referred to the changes I made in the standalone 'cpdup' repo:

10 months agonrelease: Fix 'make restartpkgs'
Aaron LI [Mon, 28 Oct 2019 15:39:29 +0000 (23:39 +0800)]
nrelease: Fix 'make restartpkgs'

'make restartpkgs' is broken because 'make install' would fail if the
port is alreay installed.  Fix this issue by executing 'make deinstall'
first, which would exit cleanly even if the port is not installed yet.

10 months agorelease.7: Sync environment variables with nrelease/Makefile
Aaron LI [Sun, 3 Mar 2019 13:37:28 +0000 (21:37 +0800)]
release.7: Sync environment variables with nrelease/Makefile

Remove obsolete "PKGBIN_MKISOFS" and "IMGUSBDEV".


Order these variables.

10 months agonrelease: Minor style improvements to the Makefile
Aaron LI [Sun, 3 Mar 2019 13:45:28 +0000 (21:45 +0800)]
nrelease: Minor style improvements to the Makefile

10 months agonrelease: Split "pkgs" target out from "customizeiso" target
Aaron LI [Sun, 3 Mar 2019 13:08:23 +0000 (21:08 +0800)]
nrelease: Split "pkgs" target out from "customizeiso" target

Separate the package installation part from "customizeiso" target out
into the "pkgs" target, in preparation for using binary packages in the
following commit.

Use "rcrestart ldconfig" to replace the ad-hoc "ldconfig" command.

Execute "/etc/periodic/weekly/320.whatis" to build the whatis(1)
database, similar to the locate(8) database with the 310.locate periodic

Some cleanups.

10 months agonrelease: Reorder and clean up the "srcs" target
Aaron LI [Sun, 3 Mar 2019 12:39:35 +0000 (20:39 +0800)]
nrelease: Reorder and clean up the "srcs" target

In the "customizeiso" target, ${ISOROOT}/usr/dports will be removed
after installing packages, this conflicts with its creation in the
"srcs" target.  So reorder the "srcs" target to be after the
"customizeiso" target.

Simplify and clean up the "srcs" target by using the the
"{dports,src}-create-shadow" targets in usr/Makefile.  This greatly
saves space and "git gc --aggressive" has little effects in this case.

10 months agonrelease: Remove obsolete targets and update the help
Aaron LI [Sun, 3 Mar 2019 12:27:17 +0000 (20:27 +0800)]
nrelease: Remove obsolete targets and update the help

The optional targets "installer" and "img" are obsolete and have no
effects, so just remove them.

Update the help text by adding the "nopkgs" optional target.

10 months agonrelease: Simplify by not handling multiple kernel configs
Aaron LI [Sun, 3 Mar 2019 07:32:08 +0000 (15:32 +0800)]
nrelease: Simplify by not handling multiple kernel configs

We don't ship multiple kernels for years, so we can get rid of handling
multiple kernel configs in nrelease/Makefile.  This simplifies the
Makefile and also get rid of the hack of 'SUBDIR += nrelease' in

Note that users can still override 'KERNCONF' the same as 'buildkernel'.

10 months agonrelease: Replace ${NRLOBJDIR}/nrelease with a temp directory
Aaron LI [Sat, 2 Mar 2019 14:14:06 +0000 (22:14 +0800)]
nrelease: Replace ${NRLOBJDIR}/nrelease with a temp directory

The "${NRLOBJDIR}/nrelease" is only used once in the "buildiso" target,
and this directory doesn't really hold any files.  So use a temporary
directory instead, which also makes the code cleaner.

10 months agousr.sbin/fstyp: Fix comparison against unterminated ondisk string
Tomohiro Kusumi [Mon, 28 Oct 2019 15:24:05 +0000 (00:24 +0900)]
usr.sbin/fstyp: Fix comparison against unterminated ondisk string

10 months agosbin/fsck_hammer2: Fix build procedure comments
Tomohiro Kusumi [Sun, 27 Oct 2019 16:12:17 +0000 (01:12 +0900)]
sbin/fsck_hammer2: Fix build procedure comments

10 months agodrm/linux: Add DEFINE_RATELIMIT_STATE()
François Tigeot [Sun, 27 Oct 2019 14:25:12 +0000 (15:25 +0100)]

Obtained-from: OpenBSD

10 months agodrm: Switch ktime_t to its post-Linux 4.10 definition
François Tigeot [Sun, 27 Oct 2019 14:08:46 +0000 (15:08 +0100)]
drm: Switch ktime_t to its post-Linux 4.10 definition

10 months agokernel/syscons: Add missing lwkt_reltoken() in some error paths.
Sascha Wildner [Sat, 26 Oct 2019 22:10:42 +0000 (00:10 +0200)]
kernel/syscons: Add missing lwkt_reltoken() in some error paths.

10 months agodrm/linux: Make fence contexts 64 bit
François Tigeot [Sat, 26 Oct 2019 20:33:06 +0000 (22:33 +0200)]
drm/linux: Make fence contexts 64 bit

10 months agosail(6): Use NULL for pointers.
Sascha Wildner [Sat, 26 Oct 2019 20:16:40 +0000 (22:16 +0200)]
sail(6): Use NULL for pointers.

10 months agousr.bin/usr.sbin: Use NULL for pointers.
Sascha Wildner [Sat, 26 Oct 2019 20:02:00 +0000 (22:02 +0200)]
usr.bin/usr.sbin: Use NULL for pointers.

10 months agolibc: Use NULL for pointers.
Sascha Wildner [Sat, 26 Oct 2019 19:31:24 +0000 (21:31 +0200)]
libc: Use NULL for pointers.

10 months agopf - Increase default state table limit
Matthew Dillon [Fri, 25 Oct 2019 23:12:06 +0000 (16:12 -0700)]
pf - Increase default state table limit

* Increase the default state table from 10000 entries to 50000
  entries.  Because 10000 is a crazy low value in modern times.

10 months agopf - Fix a few edge cases when the state table gets big
Matthew Dillon [Fri, 25 Oct 2019 22:57:16 +0000 (15:57 -0700)]
pf - Fix a few edge cases when the state table gets big

* Currently when the state table gets big the state timeout can
  be reduced all the way to 0.  This can totally mess up legitimate

  Change the algorithm.  First calculate a reduction in the timeout
  from 0% to 100%, then claw-back up to 50% of the reduction based on
  the number of packets impacting the state.  This gives the system
  the chance to reject bad state over good state or otherwise requires
  an attacker to DOS the state table based on packet rate, which is
  much harder to do.

* When sloppy state tracking is specified use a timeout of
  state that has only received SYN or SYN+ACK packets.  That is, do not
  use the full PFTM_TCP_ESTABLISHED timeout until some data actually flows.

  This reduces state bloat from redirect traffic where PF might see SYN
  or SYN+ACK and then never sees a packet again while in SLOPPY mode.

10 months agosys/vfs/msdosfs: Drop redundant ->uio_offset limit check
Tomohiro Kusumi [Fri, 25 Oct 2019 15:44:29 +0000 (00:44 +0900)]
sys/vfs/msdosfs: Drop redundant ->uio_offset limit check

10 months agosys/vfs/msdosfs: Remove DragonFly specific macro de_off2cn()
Tomohiro Kusumi [Fri, 25 Oct 2019 15:44:14 +0000 (00:44 +0900)]
sys/vfs/msdosfs: Remove DragonFly specific macro de_off2cn()

4afd80f1a2 contains several fixes, but from the way de_cluster()
is used, an explicitly casted version of de_cluster() shouldn't
be needed.

Note that maximum cluster# (and file relative cluster# too) in
FAT12/16/32 is much smaller than maximum int32_t value. Note that
maximum file size in FAT12/16/32 is < 4GiB.

10 months agosys/vfs/msdosfs: Remove DragonFly specific macro de_off2bn()
Tomohiro Kusumi [Fri, 25 Oct 2019 15:43:55 +0000 (00:43 +0900)]
sys/vfs/msdosfs: Remove DragonFly specific macro de_off2bn()

which is only used once.
Using ->pm_bnshift is clear enough, and more readable in this case.

10 months agosys/vfs/msdosfs: Remove confusing code on cluster#
Tomohiro Kusumi [Fri, 25 Oct 2019 15:43:41 +0000 (00:43 +0900)]
sys/vfs/msdosfs: Remove confusing code on cluster#

Sync comments with FreeBSD.

10 months agodrm/linux: Align Linux memory allocations to cache lines
François Tigeot [Fri, 25 Oct 2019 17:18:15 +0000 (19:18 +0200)]
drm/linux: Align Linux memory allocations to cache lines

10 months agoFix wrong dates in several manual pages.
Sascha Wildner [Thu, 24 Oct 2019 19:20:09 +0000 (21:20 +0200)]
Fix wrong dates in several manual pages.

10 months agogetifaddrs.3: Fix macro (.PP -> .Pp).
Sascha Wildner [Thu, 24 Oct 2019 19:13:18 +0000 (21:13 +0200)]
getifaddrs.3: Fix macro (.PP -> .Pp).

10 months agodsynth.1: Fix wording (reported by noob237).
Sascha Wildner [Thu, 24 Oct 2019 18:27:52 +0000 (20:27 +0200)]
dsynth.1: Fix wording (reported by noob237).

10 months agosys/vfs/msdosfs: Remove obsolete comment
Tomohiro Kusumi [Thu, 24 Oct 2019 17:35:56 +0000 (02:35 +0900)]
sys/vfs/msdosfs: Remove obsolete comment

from freebsd/freebsd@860493e2052cc33a6abb2a55500f908beeffbe85