Tomohiro Kusumi [Thu, 5 Sep 2019 14:08:04 +0000 (23:08 +0900)]
sbin/hammer2: Remove unused if0'd malloc/free wrappers
Unused since
0c3a8cd0eb169fa13895a5691f9a1c298c489721 in 2012.
Prototypes have also been removed from header.
Tomohiro Kusumi [Thu, 5 Sep 2019 13:15:03 +0000 (22:15 +0900)]
lib/libc: Whitespace fix
Antonio Huete Jimenez [Wed, 4 Sep 2019 23:10:35 +0000 (01:10 +0200)]
dhcpcd: Fix buildworld
- Fix based on
31f23bc71ced3b. Also pointed out by
kworr.
Roy Marples [Wed, 4 Sep 2019 19:43:28 +0000 (20:43 +0100)]
dhcpcd: Remove left over patches.
François Tigeot [Wed, 4 Sep 2019 19:28:09 +0000 (21:28 +0200)]
drm/linux: Add atomic64_inc_return()
Obtained-from: OpenBSD
François Tigeot [Wed, 4 Sep 2019 19:23:55 +0000 (21:23 +0200)]
drm/radeon: Correct FIRMWARE_BONAIRE definition
Submitted-by: zrj
Roy Marples [Wed, 4 Sep 2019 18:32:37 +0000 (19:32 +0100)]
dhcpcd: Adjust Makefile for dhcpcd-8.0.4
Roy Marples [Wed, 4 Sep 2019 18:31:07 +0000 (19:31 +0100)]
dhcpcd: Import new config.h
Roy Marples [Wed, 4 Sep 2019 18:30:39 +0000 (19:30 +0100)]
dhcpcd: Add README.DRAGONFLY
Roy Marples [Wed, 4 Sep 2019 18:29:08 +0000 (19:29 +0100)]
Merge dhcpcd-8.0.4 from branch 'vendor/DHCPCD'
Sascha Wildner [Wed, 4 Sep 2019 17:51:54 +0000 (19:51 +0200)]
sbin/Makefile: Sort SUBDIR.
Roy Marples [Wed, 4 Sep 2019 16:55:23 +0000 (17:55 +0100)]
Import dhcpcd-8.0.4 to vendor branch.
Matthew Dillon [Tue, 3 Sep 2019 19:44:54 +0000 (12:44 -0700)]
dsynth - Add hook_* support (2)
* Fix bug in call to doHook()
Tomohiro Kusumi [Tue, 3 Sep 2019 07:58:42 +0000 (16:58 +0900)]
lib/libdmsg: Fix compile warnings on Linux distros
--
crypto.c:210: warning: pointer targets in passing argument 5 of 'EVP_EncryptInit_ex' differ in signedness
/usr/include/openssl/evp.h:586: note: expected 'const unsigned char *' but argument is of type 'char *'
crypto.c:216: warning: pointer targets in passing argument 2 of 'EVP_EncryptUpdate' differ in signedness
/usr/include/openssl/evp.h:588: note: expected 'unsigned char *' but argument is of type 'char *'
crypto.c:216: warning: pointer targets in passing argument 4 of 'EVP_EncryptUpdate' differ in signedness
/usr/include/openssl/evp.h:588: note: expected 'const unsigned char *' but argument is of type 'char *'
crypto.c:221: warning: pointer targets in passing argument 2 of 'EVP_EncryptFinal' differ in signedness
/usr/include/openssl/evp.h:591: note: expected 'unsigned char *' but argument is of type 'char *'
...
Tomohiro Kusumi [Mon, 2 Sep 2019 23:00:07 +0000 (08:00 +0900)]
lib/libdmsg: Fix compile warnings on Linux distros
--
crypto.c: In function 'dmsg_crypto_negotiate':
crypto.c:552: warning: pointer targets in passing argument 2 of 'RSA_private_encrypt' differ in signedness
/usr/include/openssl/rsa.h:305: note: expected 'const unsigned char *' but argument is of type 'char *'
crypto.c:552: warning: pointer targets in passing argument 3 of 'RSA_private_encrypt' differ in signedness
/usr/include/openssl/rsa.h:305: note: expected 'unsigned char *' but argument is of type 'char *'
Matthew Dillon [Tue, 3 Sep 2019 03:28:51 +0000 (20:28 -0700)]
dsynth - Add hook_* support
* Add hook_* support. Note that hook_run_start and hook_run_end are
serialized, but all package-building hooks can run concurrently with
each other.
The package-building hooks are: success, failure, ignored, and
skipped.
Tomohiro Kusumi [Sun, 1 Sep 2019 23:22:11 +0000 (08:22 +0900)]
sys/vfs/msdosfs: Cleanups
Tomohiro Kusumi [Sun, 1 Sep 2019 19:44:01 +0000 (04:44 +0900)]
sys/vfs/msdosfs: Add mprintf()
Tomohiro Kusumi [Sun, 1 Sep 2019 16:51:01 +0000 (01:51 +0900)]
sys/vfs/msdosfs: Cleanups
Tomohiro Kusumi [Sun, 1 Sep 2019 12:30:12 +0000 (21:30 +0900)]
sys/vfs/msdosfs: Cleanups
Tomohiro Kusumi [Sat, 31 Aug 2019 16:22:17 +0000 (01:22 +0900)]
sys/vfs/msdosfs: Whitespace cleanups
Mostly trailing whitespace and >80 chars per line.
Also remove redundant static function prototypes.
Sascha Wildner [Sat, 31 Aug 2019 19:12:42 +0000 (21:12 +0200)]
cdefs.h/libm: Add __sym_{compat,default}() helper macros and use in libm.
While here, change the "past<function>" naming scheme to "past_<function>"
like it is already for "weak_" and "imprecise_" cases.
Macros-taken-from: FreeBSD
Tomohiro Kusumi [Sat, 31 Aug 2019 13:09:33 +0000 (22:09 +0900)]
sbin/hammer2: Introduce hammer2_media_config::thread_started
Don't test ->thread address itself to test if a pthread is running.
Having assumption on actual type breaks portability.
Tomohiro Kusumi [Sat, 31 Aug 2019 12:47:45 +0000 (21:47 +0900)]
sbin/hammer2: Don't assume pthread_t is a pointer
NULL initialization isn't necessary.
Also having assumption on actual type breaks portability.
Sascha Wildner [Fri, 30 Aug 2019 17:39:38 +0000 (19:39 +0200)]
who/utmpentry: Clean up a bit.
* Use _Static_assert() for assertions.
* Remove #if 1.
Sascha Wildner [Sat, 17 Aug 2019 20:06:57 +0000 (22:06 +0200)]
env(1)/printenv(1): Bump .Dd, remove (void) casts.
Sascha Wildner [Fri, 30 Aug 2019 17:20:39 +0000 (19:20 +0200)]
Sync utmp_update(8) with NetBSD.
Not much to see here.
Sascha Wildner [Fri, 30 Aug 2019 17:03:30 +0000 (19:03 +0200)]
setutxdb(3): Constify the file name argument.
Taken-from: FreeBSD
Sascha Wildner [Fri, 30 Aug 2019 17:01:23 +0000 (19:01 +0200)]
libc/utmpx: Some updates from NetBSD.
* Open some various descriptors as close-on-exec.
* Remove dead code.
* Use VIS_NOLOCALE in the strvisx().
Sascha Wildner [Fri, 30 Aug 2019 16:43:41 +0000 (18:43 +0200)]
<utmpx.h>: Some small updates.
* Add comments to the standard ut_type constants. Taken from FreeBSD.
* Put a number of non-standard structs, enums and constants under
__BSD_VISIBLE. Add some explaining comments. Taken from NetBSD.
Sascha Wildner [Thu, 29 Aug 2019 15:30:46 +0000 (17:30 +0200)]
init(8): Pass waitpid() status to clear_session_logs() and log to wtmpx.
Taken-from: NetBSD
François Tigeot [Wed, 28 Aug 2019 20:04:26 +0000 (22:04 +0200)]
drm/linux: Add kstrtol()
Obtained-from: FreeBSD
François Tigeot [Wed, 28 Aug 2019 19:57:25 +0000 (21:57 +0200)]
drm: Make it clear CONFIG_GENERIC_ATOMIC64 is not set on x86
Antonio Huete Jimenez [Tue, 27 Aug 2019 17:21:09 +0000 (19:21 +0200)]
jail.8 - Adjust manpage for the new changes
Antonio Huete Jimenez [Sat, 24 Aug 2019 19:14:47 +0000 (21:14 +0200)]
rc: Adjust rc.d/jail for the new changes
Antonio Huete Jimenez [Sat, 24 Aug 2019 18:03:36 +0000 (20:03 +0200)]
jail - Remove MPLOCK
- Replace with a lockmgr global lock.
Antonio Huete Jimenez [Wed, 21 Aug 2019 00:35:42 +0000 (02:35 +0200)]
jail - Rework sysctl configuration variables
- Jail sysctls are now jail-specific so that different jails
can have different settings.
Each jail will have its own subtree which can be operated
directly with sysctl(8).
Naming convention:
jail.<n>.<setting>
- All previous sysctls are now moved to 'jail.defaults' and they
are used as a template for any newly created jail.
Example:
# jls
JID Hostname Path IPs
2 t02.local /jails/02 10.0.0.3
1 t01.local /jails/01 10.0.0.2
# sysctl jail
jail.jailed: 0
jail.list: 2 t02.local /jails/02 10.0.0.3
1 t01.local /jails/01 10.0.0.2
jail.defaults.allow_raw_sockets: 0
jail.defaults.chflags_allowed: 0
jail.defaults.sysvipc_allowed: 0
jail.defaults.socket_unixiproute_only: 1
jail.defaults.set_hostname_allowed: 1
jail.1.set_hostname_allowed: 1
jail.1.socket_unixiproute_only: 1
jail.1.sysvipc_allowed: 0
jail.1.chflags_allowed: 0
jail.1.allow_raw_sockets: 0
jail.2.set_hostname_allowed: 1
jail.2.socket_unixiproute_only: 1
jail.2.sysvipc_allowed: 0
jail.2.chflags_allowed: 0
jail.2.allow_raw_sockets: 0
# sysctl jail.2.allow_raw_sockets=1
jail.2.allow_raw_sockets: 0 -> 1
# jexec 2 ping -q -c 1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.766/0.766/0.766/0.000 ms
# jexec 1 ping -q -c 1 10.0.0.1
ping: socket: Operation not permitted
# service jail stop
Stopping jails: t01.local t02.local.
# sysctl jail
jail.jailed: 0
jail.defaults.allow_raw_sockets: 0
jail.defaults.chflags_allowed: 0
jail.defaults.sysvipc_allowed: 0
jail.defaults.socket_unixiproute_only: 1
jail.defaults.set_hostname_allowed: 1
François Tigeot [Tue, 27 Aug 2019 21:51:51 +0000 (23:51 +0200)]
drm: Enable drm_pcie_get_max_link_width()
Partially obtained from FreeBSD.
Matthew Dillon [Tue, 27 Aug 2019 05:31:41 +0000 (22:31 -0700)]
dsynth - Abstract the run-status infrastructure
* Abstract the run-status infrastructure and add skeletons for html
and a 'dsynth monitor' style monitor.
Neither is implemented yet but this moves most of the hard work
out of gui.c (and renames it to ncurses.c) so it doesn't have to
be replicated for the other two.
Matthew Dillon [Tue, 27 Aug 2019 04:12:55 +0000 (21:12 -0700)]
dsynth - Make rebuild-repository work again
* Provide a meta file for the package format since it is
different from the default.
* Fixes the rebuild-repository directive.
Matthew Dillon [Mon, 26 Aug 2019 18:42:47 +0000 (11:42 -0700)]
dsynth - Fix bug in pty interlock handshake
* login_tty() closes slavefd so the handshake wasn't working.
Read from descriptor 0 instead, which login_tty() has set
up with the slave tty.
Matthew Dillon [Mon, 26 Aug 2019 18:37:46 +0000 (11:37 -0700)]
dsynth - Cleanup
* Fix a minor issue where some Ignored packages were being put in
the Skip list instead.
* Adjust setproctitle() calls to include the flavor, if any for
the workers.
* Minor cleanup and code safety.
Matthew Dillon [Mon, 26 Aug 2019 06:40:02 +0000 (23:40 -0700)]
dsynth - Implement 'test' and 'force', conditional /usr/src, etc
* Implement the 'test' and 'force' directives. Implement
removal of manually specified packages. If a generic flavor
or a meta-port is specified, its direct dependency list will
also be removed.
* If /usr/src exists under the system directory, mount it in the
chroot (to support USES=kmod modules).
* Wire up UseCCache but leave disabled and require it to be
disabled for now.
* Print a build summary at the end.
Matthew Dillon [Mon, 26 Aug 2019 01:26:39 +0000 (18:26 -0700)]
dsynth - Add missing env vars for dports scan
* PACKAGE_BUILDING was being set in the builder but not in the initial
dports scan.
* Refactor part of the BuildEnv API to allow the code to set variables
for the scan vs generally.
Matthew Dillon [Sun, 25 Aug 2019 19:23:37 +0000 (12:23 -0700)]
dsynth - Fix master/slave pty race on fork()
* If we try to re-open the slave in the forked child of the WORKER monitor,
the worker monitor may race the child when it tries to read from the
master side fd before the child has opened the slave, causing an error.
* If we re-open the slave in the WORKER monitor then the worker monitor
must not block when it close()s the slave descriptor after forking a
child. But it will if the child has written anything.
The solution is to re-open the slave in the master but to have the
master write 1 char to the slave after close()ing the slavefd. The
forked slave will read this character before proceeding, thus ensuring
that there is no output in the buffer when the master close()es
its slavefd.
Matthew Dillon [Sun, 25 Aug 2019 18:31:40 +0000 (11:31 -0700)]
kernel - Fix devfs bitmap races for pty and other devices (2)
* Fix bug in
1991e949fd5da due to misinterpretation of how
devfs_clone_bitmap_get() worked in the vn code.
François Tigeot [Sun, 25 Aug 2019 09:45:33 +0000 (11:45 +0200)]
Partial revert of "drm: Partially sync drm_ioctl() with Linux 4.7.10"
This reverts commit
c22b03749c861bc699168784bdafe2e43317465f.
It was causing application hangs with 100% cpu usage.
Matthew Dillon [Sat, 24 Aug 2019 17:16:01 +0000 (10:16 -0700)]
kcollect - Remove system calls from graph
* System calls make the graph too scratchy. Remove them for now.
Matthew Dillon [Sat, 24 Aug 2019 16:56:40 +0000 (09:56 -0700)]
dsynth - Stabilization, improve compatibility
* Disable /etc/make.conf for the bulk scan (__MAKE_CONF=/dev/null)
* Add -D to turn on developer mode (from zrj)
* Add a little infrastructure to cache ports variables, starting
with _PERL5_FROM_BIN.
Matthew Dillon [Sat, 24 Aug 2019 15:40:49 +0000 (08:40 -0700)]
dsynth - Do not count DUMMY nodes in the total
* dsynth uses DUMMY nodes to wrap all the flavors for a particular
package when other packages have a generic (rather than flavor-specific)
dependency to it.
* Do not count these nodes in the total for display purposes.
François Tigeot [Sat, 24 Aug 2019 12:17:17 +0000 (14:17 +0200)]
drm/ttm: Partially sync the vm fault code with Linux
Prevent crashes with Mesa-using applications.
François Tigeot [Sat, 24 Aug 2019 12:11:31 +0000 (14:11 +0200)]
drm/linux: Move vm_area_struct definition to the right header
François Tigeot [Sat, 24 Aug 2019 12:10:12 +0000 (14:10 +0200)]
drm/radeon: Use mm->mmap_sem
Matthew Dillon [Wed, 7 Aug 2019 05:34:17 +0000 (22:34 -0700)]
drm - Refactor task_struct and implement mm_struct
* Change td->td_linux_task from an embedded structure to a pointer.
* Add p->p_linux_mm to support tracking mm_struct's.
* Change the 'current' macro to test td->td_linux_task and call
a support function, linux_task_alloc(), if it is NULL.
* Implement callbacks from the main kernel for thread exit and
process exit to support functions that drop the td_linux_task and
p_linux_mm pointers.
Initialize and clear these callbacks in the module load/unload
in drm_drv.c
* Implement required support functions in linux_sched.c
François Tigeot [Sat, 24 Aug 2019 12:06:54 +0000 (14:06 +0200)]
drm/radeon: Update to Linux 4.7.10
* Various bugfixes, especially related to Displayport and
vblank handling
* Sea Islands and Southern Islands GPUs support improvements
* Various quirks to fix problems for specific cards, mostly
Radeon R7 370 and Radeon R9 270X models
François Tigeot [Sat, 24 Aug 2019 11:21:28 +0000 (13:21 +0200)]
drm/linux: down_read() acquiress a shared lock
Matthew Dillon [Sat, 24 Aug 2019 02:47:40 +0000 (19:47 -0700)]
dsynth - Account for extraction size
* Account for the initial size of /construction after the extract
phase.
Note that we explicitly are not accounting for space used after this
phase because they files (typically .o's and such) are added incrementally
and often not accessed continuously until the ending link phase. Including
this space thus might cause dsynth to retard the worker count too much.
Matthew Dillon [Sat, 24 Aug 2019 00:57:41 +0000 (17:57 -0700)]
dsynth - More pkg dependency fixes
* Fix the depency check based on the depth when recursing through a
dummy node. Dummy nodes do not increment the depth.
* In addition, recursing through a dummy node means that the package
only depends on the default flavor, which is only the first element
in the idepon list under that dummy node. Inform the next layer
that its scan should only iterate one element.
* In addition, when build_find_leaves() recurses through a dummy node
also use the same mechanism where only the first sub-node is
checked, because the dependency is (again) against just the default
flavor.
Note that when doing a 'dsynth everything', all the dependencies
wind up in the root list, so all the flavors get built. But when
building a specific subset of packages, this fix will reduce the
number of flavors that have to be built as dependencies.
Matthew Dillon [Sat, 24 Aug 2019 00:25:32 +0000 (17:25 -0700)]
dsynth - Unbreak ncurses coloring
* Fixup the log filter for the ncurses display.
Matthew Dillon [Sat, 24 Aug 2019 00:17:44 +0000 (17:17 -0700)]
dsynth - Report phase on failure
* Report the phase a failure occurs at.
* When logging each phase, use 79 dashes as a separator.
Matthew Dillon [Sat, 24 Aug 2019 00:06:16 +0000 (17:06 -0700)]
dsynth - Use foreground color instead of background color
* Set the foreground color instead of the background color,
just to not be too invasive.
Matthew Dillon [Fri, 23 Aug 2019 23:06:59 +0000 (16:06 -0700)]
dsynth - pty CRs, readelf, better log formatting
* Remove CR's from build logs (set tty modes on the pty).
* Parse 'readelf -x .note.tag /bin/sh' to get the operating
system and __DragonFly_version... basically the release name.
Compare the operating system name against what we get back
from the system as a validator.
* Improve the log format to make it more readable. Certain elements such
as statistics and times are right-justified.
* Filter certain log lines from stdout when curses mode is turned off.
* Highlight stdout lines for SUCCESS in green and FAILURE and red.
Matthew Dillon [Fri, 23 Aug 2019 19:45:23 +0000 (12:45 -0700)]
dsynth - Fix VersionName (UNAME_v)
* Fix VersionName (UNAME_v) to use the format e.g.
"DragonFly 5.7-SYNTH".
* Not yet pulled from elf info, currently using sysctl.
Matthew Dillon [Fri, 23 Aug 2019 19:33:52 +0000 (12:33 -0700)]
dsynth - Reduce number of pre-installed pkgs in worker
* The top-level scan installs all dependencies, but the deeper recursion
only needs to install LIB and RUN dependencies.
Matthew Dillon [Fri, 23 Aug 2019 18:51:11 +0000 (11:51 -0700)]
dsynth - Fix templating bugs, flesh out environment and make.conf
* Fix a few templating bugs
* Generate a make.conf in each chroot/jail based on the configured
environment.
* Implement the make environment in the pkglist commands as make
variables on the command line.
* Use vfork/exec in the bulk code... generate the argument vector
and environment ourselves.
* Ignore '/nonexistent:BLAH' dependencies.
François Tigeot [Fri, 23 Aug 2019 18:42:53 +0000 (20:42 +0200)]
drm/radeon: Disable acceleration on evergreen
This is a workaround for https://bugs.dragonflybsd.org/issues/3198
François Tigeot [Fri, 23 Aug 2019 11:40:12 +0000 (13:40 +0200)]
drm: Partially sync drm_ioctl() with Linux 4.7.10
Matthew Dillon [Fri, 23 Aug 2019 04:22:01 +0000 (21:22 -0700)]
dsynth - Add a pkgi depth calculation
* Calculate the largest tree-depth of packages that depend on PKG
and use it in the sort as weighting to try to prioritize building
PKG.
The intent is to try to knock out these deep chains as quickly as
possible so as not to get stuck on a linear chain of dependencies
later on in the build.
Matthew Dillon [Fri, 23 Aug 2019 00:32:51 +0000 (17:32 -0700)]
dsynth - Stabilization
* Rejigger the dynamic load management calculations and fix bugs in
how the restrictions based on pkgdep memory use are calculated.
The pkgdep memory mechanism now supports both slow-start and
slow-approach. The slow-approach mechanism begins when pkgdep
memory use goes about 1/2 the target (typically 1/4 physical memory).
In slow-approach mode, the number of workers is increased at a
rate of 1 every 30 seconds instead of 1 every 5 seconds in order
to give workers time to install the packages and fill in the estimated
memory use. The idea is that this then allows paging to ramp up
more slowly instead of getting 'shocked' by a dozen slots all
issuing install-pkgs at the same time.
* Add -m gb (in gigabytes), to override the default pkgdep memory
target.
* For now stop mounting /usr/src in the workers.
* NCurses disablement now works as expected.
* Calculate 1 gigabyte as 1GiB.
* Fix a bug in bulk accounting that could sometimes cause the bulk
scans to block indefinitely.
Matthew Dillon [Thu, 22 Aug 2019 21:16:37 +0000 (14:16 -0700)]
dsynth - Add Dynamic memory load management
* In addition to using the load and swap, also use the total
estimated installed package dependencies for each worker verses
PhysMem and reduce the number of worker slots such that this
total is less than 1/2 physical memory.
* Fix a ^C handling bug where dsynth would try to unmount while
still holding a file descriptor open on the filesystem it is
trying to unmount.
* Clean up the logs a bit more.
Matthew Dillon [Thu, 22 Aug 2019 20:25:07 +0000 (13:25 -0700)]
dsynth - Add color
* Color successful builds green and failed builds red.
* Refactor some of the log mesages.
Matthew Dillon [Thu, 22 Aug 2019 18:51:20 +0000 (11:51 -0700)]
dsynth - Add logfile tracking on the ncurses display
* Track the contents of the 00*.log file below the list of
workers.
* Handle window resize events.
* Reduce 00*.log spam by default.
* NOTE: CTRL-Z is still not properly handled, it works once or twice
and then stops working. Not sure why.
Matthew Dillon [Thu, 22 Aug 2019 17:29:09 +0000 (10:29 -0700)]
dsynth - Add IGNORE handling
* Add IGNORE handling (this also handles BROKEN)
* Do one final ncurses update when the build ends.
* Don't allocate "" strings, just leave the fields NULL for most of
the strings in the pkg structure.
Matthew Dillon [Thu, 22 Aug 2019 17:09:55 +0000 (10:09 -0700)]
dsynth - Add IGNORE/BROKEN
* Implement IgnoreCount
* Only clear the 0*.log files for build directives. Don't clear them
for things like 'dsynth cleanup'.
Matthew Dillon [Thu, 22 Aug 2019 16:46:13 +0000 (09:46 -0700)]
dphase - Stabilization
* Fix formatting bug for 'Left', 'Failed', and 'Skipped'. Use
%-5d instead of %-4d. Doh!
Matthew Dillon [Thu, 22 Aug 2019 15:35:32 +0000 (08:35 -0700)]
dphase - Stabilization
* Fix a memory leak in scanit()
* When fork()/forkpty() fails, provide a reason in log 00.
Matthew Dillon [Thu, 22 Aug 2019 15:31:29 +0000 (08:31 -0700)]
kernel - Fix devfs bitmap races for pty and other devices
* Use an internal lock to protect the integrity of bitmap
operations for devfs_clone_bitmap_*() functions, allowing
devices to use the functions without having to lock themselves.
* Devices which use devfs_clone_bitmap_chk() + devfs_clone_bitmap_set()
sequences have to be more careful, either having their own covering
lock or checking the return code from devfs_clone_bitmap_set() and
looping.
* This fixes serious /dev/ptmx pty allocation races which become obvious
when pty's are allocated concurrently at a high rate, such as by the
dsynth code.
Matthew Dillon [Thu, 22 Aug 2019 07:26:49 +0000 (00:26 -0700)]
dsynth - Stabilization
* Fix a double dot caused by the last commit.
* Fix missing free()'s and a few double-free()s.
* Fix failure cases in the Template copy code.
Matthew Dillon [Thu, 22 Aug 2019 06:26:10 +0000 (23:26 -0700)]
dsynth - Add #define for compression algo, default to .tgz
* Add a #define in dsynth.h to support .tar, .tgz, .txz, or .tbz.
It is currently hardwired to .tgz.
* Switch to using .tgz. While .txz is highly compressed, .tgz
will decompress 10x more quickly and this seriously improves
bulk build times.
Matthew Dillon [Thu, 22 Aug 2019 03:50:21 +0000 (20:50 -0700)]
dsynth - Fix waitpid() bug, revamp phase poll loop
* Fix an issue where the wrong wait*() status is processed as
the error code. This could occur especially on slow machines.
* Be sure to clean out any log data on the pty while and after
processing the wait*() on the primary pid as there might be
some as-yet unprocessed data.
* Use non-blocking mode on the masterpty to ensure that we
don't accidently block on it.
Matthew Dillon [Thu, 22 Aug 2019 01:41:33 +0000 (18:41 -0700)]
dsynth - Add manual page, change configuration directory, more
* Change the configuration location from /usr/local/etc/synth.ini
to /etc/dsynth/dsynth.ini and (as an alternate)
/usr/local/etc/dsynth/dsynth.ini
* Add 'dsynth debug <port>' directive. This will build all
dependencies and then build the specified port and freeze
its worker slot with mounts still intact.
* Add the 'dsynth init' directive. This will initialize the
/etc/dsynth directory and default configuration files.
This directive will refuse to do anything if /etc/dsynth
or /usr/local/etc/dsynth already exists.
* Allow 'Number_of_builders' and/or 'Max_jobs_per_builder' to
be specified as 0. If 0, these parameters will be
auto-configured.
* Add the dsynth(1) manual page.
* Include a procfs (/proc) mount in the worker chroot.
* Change a 'cpdup' to a 'cp' to improve portability. Also change
how the Template is generated a bit.
Matthew Dillon [Wed, 21 Aug 2019 17:46:44 +0000 (10:46 -0700)]
dsynth - Improve DynamicMaxWorkers
* Allow the slow-start mode to be overriden with an
options (-sN)
* Do not count swapcache swap in the swap calculation
* When dynamic worker limiting is active, display the
limit in the upper-right-hand corner of the ncurses
display. When not, clear that field for brevity.
* The packaging step now pre-scans the package directory
and deletes and '.new' files that might have been left
over from an interrupt.
Matthew Dillon [Wed, 21 Aug 2019 17:17:50 +0000 (10:17 -0700)]
kernel - Fix dangling ref on fifo vnode
* Fix dangling refs on the fifo vnode which prevent unmount from
succeeding.
* Fix open/close races which can occur during fifo open and close
due to blocking conditions. Use a lockmgr lock instead.
* Retain use of vp->v_token to help interlock read, write, and
ioctl operations.
* This situation occurs when a filesystem fifo (aka via mkfifo)
blocks on open() and is then interrupted.
Matthew Dillon [Wed, 21 Aug 2019 01:50:59 +0000 (18:50 -0700)]
kernel - Rejigger midistat functions to close a race
* Make sure lock has full coverage across midistat_open() and
midistat_read(). The temporary drop of the lock in midistat_read()
lead to a race which allows one to read kernel memory beyond the
end of the sbuf buffer.
* Rejigger the code to remove the global offset and just use
uio_offset, which also fixes the same race (but leave the
lock coverage in place regardless).
Taken-From: FreeBSD
Security: CVE-2019-5612
Matthew Dillon [Wed, 21 Aug 2019 01:36:46 +0000 (18:36 -0700)]
kernel - merge m_pulldown() bug fix
In m_pulldown(), before trying to prepend bytes to the subsequent mbuf,
ensure that the subsequent mbuf contains the remainder of the bytes
the caller sought. If this is not the case, fall through to the code
which gathers the bytes in a new mbuf.
This fixes a bug where m_pulldown() could fail to gather all the desired
bytes into consecutive memory.
Patch-by: sephe
FreeBSD-PR: 238787
Reported by: A reddit user
Discussed with: emaste
Obtained from: NetBSD
Obtained from: FreeBSD
Matthew Dillon [Tue, 20 Aug 2019 19:36:14 +0000 (12:36 -0700)]
dsynth - Stabilization
* Set the PORTSDIR and BATCH environment variables for bulk pkglist
operations (these were already being set properly for chroot'd
operations).
This will fix the PORTSDIR prefix test and removal for dependencies
done by resolveDepString().
Matthew Dillon [Tue, 20 Aug 2019 19:08:08 +0000 (12:08 -0700)]
dsynth - Optimize pty use
* Have the WORKER process allocate the pty once and use it in all
phases, instead of allocating and closing a new pty for each phase.
* Make N copies of /bin, /lib, /usr/bin, and /libexec into the base
directory and mount those in the chroots instead of the system
directory, essentially unsharing these directories. Each worker
gets an independent copy.
This should reduce contention and cache-line ping-ponging overhead
on /bin/sh, libc, make, etc, significantly.
* Create the template conditionally whether 'pkg' has been built or
not.
Matthew Dillon [Tue, 20 Aug 2019 17:23:29 +0000 (10:23 -0700)]
dsynth - Get skips working, etc
* Get 'Skipped' working, and print a nice depdency chain for the reason
why a packge got skipped.
* Added swap% to the dynamic worker reduction algorithm. It startsr at
10% swap used and maxes out at a 75% reduction with 40% of swap used.
* Add a slow-start feature so dsynth does not try to load all workers
at startup simultaneously. This gives the swap% and loadavg worker
reduction code a chance to activate before the system becomes totally
smashed.
* Cache the 0*.log logfile descirptors instead of open/write/close.
Roy Marples [Tue, 20 Aug 2019 13:18:51 +0000 (14:18 +0100)]
route: Remove ref counting around rt_newaddrmsg().
Doesn't seem to be needed anymore.
As proposed by sephe.
Sascha Wildner [Tue, 20 Aug 2019 09:44:00 +0000 (11:44 +0200)]
Merge remote-tracking branch 'remotes/crater/vendor/BYACC'
Tomohiro Kusumi [Mon, 19 Aug 2019 20:24:40 +0000 (05:24 +0900)]
sbin/hammer2: Use hammer2_uuid_to_str() instead of uuid_to_string()
Tomohiro Kusumi [Mon, 19 Aug 2019 19:51:28 +0000 (04:51 +0900)]
sbin/hammer2: Use hammer2_time64_to_str() in "printinode"
Matthew Dillon [Tue, 20 Aug 2019 05:56:50 +0000 (22:56 -0700)]
dsynth - Misc work
* Try to get Skipped working, propagate the NOBUILD flag upward
and process it as a Skipped in startbuild().
* Change descriptor 0 in the forkpty()'d sub-process to /dev/null
to try to shortcut ports which sometimes ask questions.
(We do set BATCH as well, but some ports still ask questions).
* Change the dynamic MaxWorkers adjustment to reduce MaxWorkers
by up to 75% at (5 * ncpus) of load. The adjustment begins
at 0% at (1.5 * ncpus) load.
Tomohiro Kusumi [Mon, 19 Aug 2019 18:12:20 +0000 (03:12 +0900)]
sbin/hammer2: Fix remaining type warnings on Linux distros
Bring in local changes existed in https://github.com/kusumi/lh1.
These ondisk fields are not of char*.
Matthew Dillon [Tue, 20 Aug 2019 03:19:45 +0000 (20:19 -0700)]
dsynth - Reap processes more generally in the poll loop
* Adjust the dsynth worker process to reap any pid needing
reaping within the poll loop instead of just checking the
primary exec'd make pid.
* This catches any orphaned children that would otherwise
accumulate during the build.
Matthew Dillon [Tue, 20 Aug 2019 02:30:30 +0000 (19:30 -0700)]
dsynth - Misc/stabilization
* Add missing environment variables PORT_DBDIR, PACKAGE_BUILDING,
and PKG_CREATE_VERBOSE.
* Increase field width for Impulse and Pkg/hour to 5 digits.
* Fix bugs in the dynamic MaxWorkers reduction code (based on load).
Matthew Dillon [Tue, 20 Aug 2019 00:12:44 +0000 (17:12 -0700)]
dsynth - Fleshout functions
* Add dynamic MaxWorker reduction based on the load. The number of
workers will be reduced by (load * 2 / ncpus) when the load
exceeds (ncpus * 2.0), with a floor of min(4, MaxWorkers).
The idea here is to reduce unnecessary swap use when building
larger packages that push the load up greatly.
* Clean-up terminal spam after ncurses has been turned on.
* Generate fatal error if required configured directories are
missing.
* Add -y (yes to all) option.
* Implement 'status' and 'status-everything'
* Implement 'purge-distfiles' (initial work)
* In scans that dump percentages, print 100% at the end before
moving onto the next step.
Matthew Dillon [Mon, 19 Aug 2019 20:36:33 +0000 (13:36 -0700)]
dsynth - Initial commit (unhooked from buildworld)
* DSynth is basically synth written in C, from scratch. It is designed
to give us a bulk builder in base and be friendly to porting and jails
down the line (for now its uses chroot's).
The original synth was written by John R. Marino and its basic flow
was used in writing this program, but as it was written in ada no
code was directly copied.
* The intent is to make dsynth compatible with synth's configuration
files and directory structure.
* This is a work in progress and not yet ready for prime-time. Pushing
so we can get some more eyeballs. Most of the directives do not yet
work (everything, and build works, and 'cleanup' can be used to clean
up any dangling mounts).
* Not connected to the build yet. A great deal more work is needed.
As-of this commit, these areas need work:
- Uname/architecture/release-versioning stuff.
- Possibly additional environment variables or make variables
for certain cases and situations.
- The repo building step.
- Web interface.
- Dynamic load management (reduce the number of workers dynamically
based on load/ncpus).
- curses cleanup and window resizing.
Tomohiro Kusumi [Mon, 19 Aug 2019 14:59:59 +0000 (23:59 +0900)]
sbin/hammer2: Minor fix in man
Correct
d371ccd28ada846c26eb398d4f4798fd4a74d47d.
Tomohiro Kusumi [Mon, 19 Aug 2019 14:13:38 +0000 (23:13 +0900)]
lib/libdmsg: Fix compile warning on Linux distros
Appeared since
030e3428e0c23c47cef4dfab51f988a8fa665d19.
--
msg.c:202: warning: pointer targets in passing argument 1 of 'atomic_set_int' differ in signedness
atomic.h:129: note: expected 'volatile u_int *' but argument is of type 'int *'