12 years agoHAMMER 39/Many: Parallel operations optimizations
Matthew Dillon [Tue, 29 Apr 2008 01:10:37 +0000 (01:10 +0000)]
HAMMER 39/Many: Parallel operations optimizations

* Implement a per-direct cache of new object IDs.  Up to 128 directories
  will be managed in LRU fashion.  The cached provides a pool of object
  IDs to better localize the object ids of files created in a directory,
  so parallel operations on the filesystem do not create a fragmented
  object id space.

* Cache numerous fields in the root volume's header to avoid creating
  undo records for them, creatly improving

  (ultimately we can sync an undo space representing the volume header
  using a direct comparison mechanic but for now we assume the write of
  the volume header to be atomic).

* Implement a zone limit for the blockmap which newfs_hammer can install.
  The blockmap zones have an ultimate limit of 2^60 bytes, or around
  one million terrabytes.  If you create a 100G filesystem there is no
  reason to let the blockmap iterate over its entire range as that would
  result in a lot of fragmentation and blockmap overhead.  By default
  newfs_hammer sets the zone limit to 100x the size of the filesystem.

* Fix a bug in the crash recovery code.  Do not sync newly added inodes
  once the flusher is running, otherwise the volume header can get out
  of sync.  Just create a dummy marker structure and move it to the tail
  of the inode flush_list when the flush starts, and stop when we hit it.

* Adjust hammer_vfs_sync() to sync twice.  The second sync is needed to
  update the volume header's undo fifo indices, otherwise HAMMER will
  believe that it must undo the last fully synchronized flush.

12 years agoPaging and swapping system fixes.
Matthew Dillon [Mon, 28 Apr 2008 21:16:27 +0000 (21:16 +0000)]
Paging and swapping system fixes.

* Do not try to free a VM page after a failed IO read from swap.  It is
  illegal to free a VM page from an interrupt.  Just deactivate it instead.

* Do not attempt to move a VM page into the cache queue after a successful
  pageout from the vnode or swap pagers, and do not try to adjust page
  protections to read-only (they should already be read-only).  Both
  operations require making serious pmap calls which we really do not
  want to do from an interrupt.

  Instead, leave the page on its current queue or, if the system is low
  on pages, deactivate the page.

The pmap protection code is supposed to be runnable from an interrupt but
testing with vkernels shows program corruption occuring under severe paging
loads.  Pmap protection changes were only being made from pageout interrupts.
brelse() itself, which can also be called from an interrupt via biodone(),
does not make such changes for asynchronous I/O.

With these changes in place the program corruption stopped or has been
greatly reduced.  Further testing in a 64MB vkernel environment is ongoing.

In addition, trying to move the page after a completed pageout/swappout
to the cache queue was improperly depressing the priority of read-heavy
pages.  Under severe paging loads we now only deactivate the page.  Plus
moving a page to the cache queue causes pmap operations to be run which
we again do not want to run from an interrupt.

12 years agoFix various names of defined values.
Sascha Wildner [Mon, 28 Apr 2008 21:16:17 +0000 (21:16 +0000)]
Fix various names of defined values.

12 years agoKTR_TESTLOG is a valid kernel option (it enables the KTR ipi performance
Matthew Dillon [Mon, 28 Apr 2008 20:00:18 +0000 (20:00 +0000)]
KTR_TESTLOG is a valid kernel option (it enables the KTR ipi performance
testing sysctls).

12 years agoFix a NULL poiner dereference in the statistics collecting code as
Matthew Dillon [Mon, 28 Apr 2008 18:04:08 +0000 (18:04 +0000)]
Fix a NULL poiner dereference in the statistics collecting code as
used by 'systat -vm 1'.  p->p_vmspace can be NULL.

12 years agoFix typo: spam -> span
Sascha Wildner [Mon, 28 Apr 2008 09:38:38 +0000 (09:38 +0000)]
Fix typo: spam -> span

Reported-by: sjg (on #dragonflybsd)
12 years agoMinor code reordering and documentation adjustments.
Matthew Dillon [Mon, 28 Apr 2008 07:07:02 +0000 (07:07 +0000)]
Minor code reordering and documentation adjustments.

12 years agoFix some pmap races in pc32 and vkernel, and other vkernel issues.
Matthew Dillon [Mon, 28 Apr 2008 07:05:09 +0000 (07:05 +0000)]
Fix some pmap races in pc32 and vkernel, and other vkernel issues.

* Fix a case where a vm_page_sleep_busy() loop can cause a page's hold_count
  to go stale, potentially resulting in a double action being taken on
  the page.  This can only occur during very heavy paging loads.  Fix
  the issue by doing the vm_page_unhold() after the blocking condition
  instead of before.  This fix applies to both the vkernel and pc32.

* The pmap_allocpte() call in in pmap_copy() can block, causing the cached
  page directory page to become stale.  Detect the case and take appropriate
  action.  This fix applies to both the vkernel and pc32.

* Add numerous KKASSERT()s to assert that the pmap tracking counters are
  correct, to try to detect races in the future.

* Fix a bug in the vkernel's signal handling. We have to catch SIGBUS as
  well as SIGSEGV.  We also have to tell the signal handler to not
  block the signal on entry or a page fault in the vkernel itself can
  cause the signal to be masked, the vkernel may then block and switch
  threads, and another page fault in the vkernel itself would wind up
  being masked.  The result is an endless loop in the vkernel retrying the
  same faulting instruction forever (until you kill the vkernel).

12 years agoHAMMER 38F/Many: Undo/Synchronization and crash recovery, stabilization pass
Matthew Dillon [Sun, 27 Apr 2008 21:07:15 +0000 (21:07 +0000)]
HAMMER 38F/Many: Undo/Synchronization and crash recovery, stabilization pass

* Fix a bug in the front-end's cached truncation off (ip->trunc_off).

* Fix a bug in the memory record visibility check when called from the backend.

12 years agoAdd basic support for 8111C; hardware checksum offload does not seems to work
Sepherosa Ziehau [Sun, 27 Apr 2008 15:10:37 +0000 (15:10 +0000)]
Add basic support for 8111C; hardware checksum offload does not seems to work
on 8111C yet.

12 years agoPrint unknown hardware version.
Sepherosa Ziehau [Sun, 27 Apr 2008 14:18:16 +0000 (14:18 +0000)]
Print unknown hardware version.

12 years agoHAMMER 38E/Many: Undo/Synchronization and crash recovery
Matthew Dillon [Sun, 27 Apr 2008 00:45:37 +0000 (00:45 +0000)]
HAMMER 38E/Many: Undo/Synchronization and crash recovery

* Fix a couple of deadlocks.

* Fix a kernel buffer cache exhaustion issue.

* Get the 'hammer prune' and 'hammer reblock' command working again.  The
  commands are now properly synchronized for crash recovery.

12 years agoHAMMER utilities: Misc documentation and new options.
Matthew Dillon [Sun, 27 Apr 2008 00:43:57 +0000 (00:43 +0000)]
HAMMER utilities: Misc documentation and new options.

* Add the -u <undoareasize> option.  This option allows the size of the
  undo FIFO buffer to be specified.

* Document missing newfs_hammer's options.

12 years agoRevert rev 1.40, which will cause deadlock, if task's function tries to
Sepherosa Ziehau [Sat, 26 Apr 2008 23:09:40 +0000 (23:09 +0000)]
Revert rev 1.40, which will cause deadlock, if task's function tries to
enqueue itself.

Approved-by: dillon@
12 years agoHAMMER 38E/Many: Undo/Synchronization and crash recovery
Matthew Dillon [Sat, 26 Apr 2008 19:08:14 +0000 (19:08 +0000)]
HAMMER 38E/Many: Undo/Synchronization and crash recovery

* Clean up interlocks between the frontend and backend.

* Deal with the case where the backend needs to sync a record to disk that
  the frontend wishes to delete.  This basically just involves converting
  the record from a deleted in-memory record to a delete-on-disk record,
  so the synced record does not become visible to userland.

* Deal with the case when an inode is being destroyed where the backend
  wishes to delete an in-memory record without syncing it to disk.

* Document numerous special cases for future attention.

12 years agowi(4) depends on wlan(4)
Sepherosa Ziehau [Sat, 26 Apr 2008 14:11:06 +0000 (14:11 +0000)]
wi(4) depends on wlan(4)

12 years agoDon't do following optimization in udp_disconnect():
Sepherosa Ziehau [Sat, 26 Apr 2008 14:08:52 +0000 (14:08 +0000)]
Don't do following optimization in udp_disconnect():
Conditionally free cached pcb route entry by predicting new laddr.

During soclose() on a connected UDP socket, this optimization will cause
cached pcb route entry being freed on wrong CPU, since f{port,addr} have
been changed.

Fix comment in udp_connect().

12 years agoRemove old FreeBSD upgrade tools.
Sascha Wildner [Sat, 26 Apr 2008 09:49:50 +0000 (09:49 +0000)]
Remove old FreeBSD upgrade tools.

12 years agoFix compilation in tools/tools/crypto and fix some minor issues.
Sascha Wildner [Sat, 26 Apr 2008 09:19:10 +0000 (09:19 +0000)]
Fix compilation in tools/tools/crypto and fix some minor issues.

12 years agoHAMMER 38E/Many: Undo/Synchronization and crash recovery
Matthew Dillon [Sat, 26 Apr 2008 08:02:17 +0000 (08:02 +0000)]
HAMMER 38E/Many: Undo/Synchronization and crash recovery

* Add record<->inode dependancies for file creation an deletion.  If a
  directory entry representing a new file is synced out, the file is also
  synced out at the same time, and vise-versa.

* Dirty reclaimed inodes are now forwarded to the flusher, which should
  prevent leaks of hammer_inode structures.  (Still needs work).

* Force finalization if the undo fifo becomes more then half full.
  This can currently break dependancies.  (Still needs work).

* Misc stabilization fixes to recent commits.

12 years agoHAMMER 38D/Many: Undo/Synchronization and crash recovery
Matthew Dillon [Sat, 26 Apr 2008 02:54:00 +0000 (02:54 +0000)]
HAMMER 38D/Many: Undo/Synchronization and crash recovery

* The flusher now waits for I/O to complete at the appropriate points.

* Implement instant crash recovery.  The UNDO FIFO is scanned backwards
  and reapplied to the filesystem on mount.  There is still more work
  to do here, inode<->inode associations (e.g. directory entry vs file)
  are not yet bound together.

* Clean up I/O sequencing a lot and get rid of a ton of unnecessary flusher

12 years agoHAMMER 38C/Many: Undo/Synchronization and crash recovery
Matthew Dillon [Fri, 25 Apr 2008 21:49:49 +0000 (21:49 +0000)]
HAMMER 38C/Many: Undo/Synchronization and crash recovery

* Classify buffers as meta, undo, or data buffers, and collect them
  into separate lists so they can be flushed in the proper order.

* Make the META buffers and volume header flushed under HAMMERs direct
  control only, as part of the UNDO sequencing.

* Major work on the flusher thread.  Flush the various buffer classes in
  the correct order (the synchronization points are not yet coded, however).

* Update the volume header's UNDO fifo indices.

* Add a ton of sanity checks on buffer modifications and narrow the size
  of some of the UNDO records.

* Clean-up after loose IOs.  An IO can be loose when its ref count drops
  to zero and the kernel attempts to reclaim its bp.  We can't garbage
  collect the IO in the kernel bioops callback so we have to remember
  that the IO is now loose and do it later (in the flusher).

* Temporarily comment out an allocator iterator feature which we cannot
  do right now because it may result in new data allocations overwriting
  old deletions which are still subject to UNDO.

12 years agoHAMMER 38B/Many: Undo/Synchronization and crash recovery
Matthew Dillon [Thu, 24 Apr 2008 22:05:13 +0000 (22:05 +0000)]
HAMMER 38B/Many: Undo/Synchronization and crash recovery

* Properly requeue an inode synchronization when BIOs are present on
  ip->bio_alt_list.  Fixes a sync stall.

12 years agoHAMMER 38A/Many: Undo/Synchronization and crash recovery
Matthew Dillon [Thu, 24 Apr 2008 21:20:33 +0000 (21:20 +0000)]
HAMMER 38A/Many: Undo/Synchronization and crash recovery

* Separate all frontend operations from all backend media synchronization.
  The frontend VNOPs make all changes in-memory and in the frontend
  buffer cache.  The backend buffer cache used to manage meta-data is
  not touched.

  - In-memory inode contains two copies of critical meta-data structures
  - In-memory record tree distinguishes between records undergoing
    synchronization and records not undergoing synchronization.
  - Frontend buffer cache buffers are tracked to determine which ones
    to synchronize and which ones not to.
  - Deletions are cached in-memory.  Any number of file truncations
    simply caches the lowest truncation offset and on-media records
    beyond that point are ignored.  Record deletions are cached as
    a negative entry in the in-memory record tree until the backend
    can execute the operation on the media.
  - Frontend operations continue to have full, direct read access to
    the media.

* Backend synchronization to the disk media is able to take place
  simultaniously with frontend operations on the same inodes.  This
  will need some tuning but it basically works.

* In-memory records are no longer removed from the B-Tree when deleted.
  They are marked for deletion and removed when the last reference goes

* An Inode whos last reference is being released is handed over to the
  backend flusher for its final disposition.

* There are some bad hacks and debugging tests in this commit.  In particular
  when the backend needs to do a truncation it special-cases any
  negative entries it finds in the in-memory record tree.  Also, if a
  rename operation hits a deadlock it currently breaks atomicy.

* The transaction API has been simplified.  The frontend no longer allocates
  transaction ids.  Instead the backend does a full flush with a single
  transaction id (since that is the granularity the crash recovery code will
  have anyway).

12 years agoFix panics which can occur when killing a threaded program. lwp_exit()
Matthew Dillon [Thu, 24 Apr 2008 08:53:02 +0000 (08:53 +0000)]
Fix panics which can occur when killing a threaded program.  lwp_exit()
was being called without the BGL from userret().  It needs the BGL.

12 years agoMake description of -C option more clear: describe what option does first.
Thomas Nikolajsen [Wed, 23 Apr 2008 22:09:07 +0000 (22:09 +0000)]
Make description of -C option more clear: describe what option does first.
Update usage() to version in fdisk.8.

12 years agoAdd HAMMER to disklabel.8
Thomas Nikolajsen [Wed, 23 Apr 2008 21:59:22 +0000 (21:59 +0000)]
Add HAMMER to disklabel.8
Add cross references to mount_hammer.8 and hammer.8

When here add 'B' (Byte) to comment part of disklabel output, to make unit explicit.

12 years agoAdd hammer_flusher.c, to make kernels with options HAMMER build.
Thomas Nikolajsen [Wed, 23 Apr 2008 21:06:19 +0000 (21:06 +0000)]
Add hammer_flusher.c, to make kernels with options HAMMER build.

12 years agoFix two A-list corruption cases.
Matthew Dillon [Wed, 23 Apr 2008 17:21:08 +0000 (17:21 +0000)]
Fix two A-list corruption cases.

When a meta-element indicates the next layer is all-allocated or all-free,
the next layer is allowed to contain garbage.  The allocator must skip an
all-allocated layer.

Similarly when a free recurses through an all-allocated layer the next layer
must be initialized so we can recurse down into it.

Reported-by: Jonathan Lemon <jlemon@cisco.com>
12 years agoAdd support for cmx(4) devices.
Hasso Tepper [Wed, 23 Apr 2008 09:44:07 +0000 (09:44 +0000)]
Add support for cmx(4) devices.

12 years agoAdd a driver for Omnikey CardMan 4040 smartcard reader - cmx(4).
Hasso Tepper [Wed, 23 Apr 2008 08:57:10 +0000 (08:57 +0000)]
Add a driver for Omnikey CardMan 4040 smartcard reader - cmx(4).

Obtained-from: FreeBSD

12 years agoRegenerate.
Hasso Tepper [Wed, 23 Apr 2008 08:45:05 +0000 (08:45 +0000)]

12 years agoAdd OMNIKEY CardMan 4040 smartcard reader.
Hasso Tepper [Wed, 23 Apr 2008 08:34:30 +0000 (08:34 +0000)]
Add OMNIKEY CardMan 4040 smartcard reader.

12 years ago* Remove the SINGLEUSE feature for telldir(), it does not conform to the
Matthew Dillon [Tue, 22 Apr 2008 21:29:42 +0000 (21:29 +0000)]
* Remove the SINGLEUSE feature for telldir(), it does not conform to the
  Open Group specification.

* Add a mutex around the ddloc hash table.  Note that NetBSD implemented
  a per-dirp list but telldir/seekdir performance just isn't an issue
  for these functions so stick with the global hash table.

* Reuse the ddloc for a previous seekdir() when calling telldir() just
  after a seekdir(), instead of allocating a new one.

* rewinddir() now cleans up any ddloc's for the dirp.

Reported-by: Gary Stanley <sinknull@crater.dragonflybsd.org>
12 years agoFix a free() race due to a misplaced mutex unlock.
Matthew Dillon [Tue, 22 Apr 2008 19:45:05 +0000 (19:45 +0000)]
Fix a free() race due to a misplaced mutex unlock.

12 years agoHAMMER 37/Many: Add a flush helper thread, clean up some inconsistencies.
Matthew Dillon [Tue, 22 Apr 2008 19:00:15 +0000 (19:00 +0000)]
HAMMER 37/Many: Add a flush helper thread, clean up some inconsistencies.

This patch generally cleans up transaction id inconsistencies betwen data
and meta-data by syncing the data and inode in a single transaction.

* Change the I/O path for write strategy calls.  Queue the BIO's to the
  flush helper thread instead of directly updating the media.

* We no longer try to sync individual buffers.  If the kernel wants to
  sync a dirty buffer, we sync the whole file to minimize inode updates.

12 years agoFix some IO sequencing performance issues and reformulate the strategy
Matthew Dillon [Tue, 22 Apr 2008 18:46:54 +0000 (18:46 +0000)]
Fix some IO sequencing performance issues and reformulate the strategy
we use to deal with potential buffer cache deadlocks.  Generally speaking
try to remove roadblocks in the vn_strategy() path.

* Remove buf->b_tid (HAMMER no longer needs it)

* Replace IO_NOWDRAIN with IO_NOBWILL, requesting that bwillwrite() not
  be called.  Used by VN to try to avoid deadlocking.  Remove B_NOWDRAIN.

* No longer block in bwrite() or getblk() when we have a lot of dirty
  buffers.   getblk() in particular needs to be callable by filesystems
  to drain dirty buffers and we don't want to deadlock.

* Improve bwillwrite() by having it wake up the buffer flusher at 1/2 the
  dirty buffer limit but not block, and then block if the limit is reached.
  This should smooth out flushes during heavy filesystem activity.

12 years agoPass the current LWP to sigexit() instead of the current process. Remove
Matthew Dillon [Mon, 21 Apr 2008 15:47:58 +0000 (15:47 +0000)]
Pass the current LWP to sigexit() instead of the current process.  Remove
the left-over hack in sigexit() that was using the first LWP instead of
the one that actually caused the core dump.

This way a GDB of the core will properly observe the thread that actually
caused the core-dump.

12 years agoDive the scheduler to implement the yield function. For the moment it just
Matthew Dillon [Mon, 21 Apr 2008 15:24:47 +0000 (15:24 +0000)]
Dive the scheduler to implement the yield function.  For the moment it just
does the same thing it did before.

12 years agoAdd an interlock for certain usb task operations.
Matthew Dillon [Mon, 21 Apr 2008 15:23:22 +0000 (15:23 +0000)]
Add an interlock for certain usb task operations.

12 years agoAdd missing newline.
Sascha Wildner [Sun, 20 Apr 2008 22:24:53 +0000 (22:24 +0000)]
Add missing newline.

12 years agoIt's syslogd which calls fsync(), not syslog.conf.
Sascha Wildner [Sun, 20 Apr 2008 21:14:39 +0000 (21:14 +0000)]
It's syslogd which calls fsync(), not syslog.conf.

Noticed-by: hasso
12 years agoFix function name in message.
Sascha Wildner [Sun, 20 Apr 2008 19:26:53 +0000 (19:26 +0000)]
Fix function name in message.

12 years agoNeither rand48 nor _rand48 are functions.
Sascha Wildner [Sun, 20 Apr 2008 19:21:42 +0000 (19:21 +0000)]
Neither rand48 nor _rand48 are functions.

12 years agoAnother round of typo fixes (mostly in messages).
Sascha Wildner [Sun, 20 Apr 2008 13:44:26 +0000 (13:44 +0000)]
Another round of typo fixes (mostly in messages).

12 years agoThe historic and deprecated setgrfile() function never existed in
Sascha Wildner [Sat, 19 Apr 2008 10:08:05 +0000 (10:08 +0000)]
The historic and deprecated setgrfile() function never existed in

Remove prototype and adjust manual page (now only mentioned in
COMPATIBILITY as deprecated).

12 years agoAdd missing 'be'.
Sascha Wildner [Sat, 19 Apr 2008 09:33:32 +0000 (09:33 +0000)]
Add missing 'be'.

Reported-by: Robin Carey <robin_carey5@yahoo.co.uk>
Dragonfly-bug: <http://bugs.dragonflybsd.org/issue991>

12 years agoa -> an
Sascha Wildner [Wed, 16 Apr 2008 23:22:48 +0000 (23:22 +0000)]
a -> an

12 years agoFinish up cpdup. Bump the protocol version to 2 and refuse to talk to
Matthew Dillon [Wed, 16 Apr 2008 17:38:19 +0000 (17:38 +0000)]
Finish up cpdup.  Bump the protocol version to 2 and refuse to talk to
older versions of cpdup run on the remote.

12 years agoRefuse to talk with the remote cpdup if it's version is not compatible.
Matthew Dillon [Wed, 16 Apr 2008 17:30:56 +0000 (17:30 +0000)]
Refuse to talk with the remote cpdup if it's version is not compatible.

12 years agoRemove whitespace after parenthesis.
Sascha Wildner [Wed, 16 Apr 2008 08:29:21 +0000 (08:29 +0000)]
Remove whitespace after parenthesis.

12 years agoRemove extra whitespace after parenthesis.
Sascha Wildner [Wed, 16 Apr 2008 07:21:57 +0000 (07:21 +0000)]
Remove extra whitespace after parenthesis.

12 years agoOops, I typed .Op where it should have been .Fl
Sascha Wildner [Wed, 16 Apr 2008 07:14:11 +0000 (07:14 +0000)]
Oops, I typed .Op where it should have been .Fl

12 years agoUpdate STANDARDS. While here, fix some mdoc issues.
Sascha Wildner [Tue, 15 Apr 2008 23:51:00 +0000 (23:51 +0000)]
Update STANDARDS. While here, fix some mdoc issues.

12 years agoAdd libc support for gcc stack protector. Compatibility with gcc34 propolice
Hasso Tepper [Tue, 15 Apr 2008 23:41:28 +0000 (23:41 +0000)]
Add libc support for gcc stack protector. Compatibility with gcc34 propolice
is preserved.

12 years agoFix reference. While here, remove trailing whitespace.
Sascha Wildner [Tue, 15 Apr 2008 23:00:52 +0000 (23:00 +0000)]
Fix reference. While here, remove trailing whitespace.

12 years agoVOP_ABORTOP(9) was removed before we forked.
Sascha Wildner [Tue, 15 Apr 2008 22:22:24 +0000 (22:22 +0000)]
VOP_ABORTOP(9) was removed before we forked.

12 years agoAdd section.
Sascha Wildner [Tue, 15 Apr 2008 22:05:38 +0000 (22:05 +0000)]
Add section.

12 years agoAdd a device_set_driver(9) manual page.
Sascha Wildner [Tue, 15 Apr 2008 22:02:59 +0000 (22:02 +0000)]
Add a device_set_driver(9) manual page.

Taken-from: FreeBSD

12 years agoAdd a vflush(9) manual page.
Sascha Wildner [Tue, 15 Apr 2008 21:00:56 +0000 (21:00 +0000)]
Add a vflush(9) manual page.

Taken-from: FreeBSD

12 years agoImprove markup.
Sascha Wildner [Tue, 15 Apr 2008 19:38:26 +0000 (19:38 +0000)]
Improve markup.

12 years agoa -> an
Sascha Wildner [Tue, 15 Apr 2008 19:19:49 +0000 (19:19 +0000)]
a -> an

12 years agoSync with FreeBSD (documents -P).
Sascha Wildner [Tue, 15 Apr 2008 18:11:32 +0000 (18:11 +0000)]
Sync with FreeBSD (documents -P).

12 years agoSync with FreeBSD (adds reload).
Sascha Wildner [Tue, 15 Apr 2008 18:11:00 +0000 (18:11 +0000)]
Sync with FreeBSD (adds reload).

12 years agovfs_export_lookup() doesn't have a manual page.
Sascha Wildner [Tue, 15 Apr 2008 16:26:02 +0000 (16:26 +0000)]
vfs_export_lookup() doesn't have a manual page.

12 years agoRemove hardcoded paths from etc/rc.d/ldconfig and move them to
Sascha Wildner [Tue, 15 Apr 2008 10:11:35 +0000 (10:11 +0000)]
Remove hardcoded paths from etc/rc.d/ldconfig and move them to

Submitted-by: Andreas Hauser <andy@splashground.de>
Dragonfly-bug: <https://bugs.dragonflybsd.org/issue209>

12 years agoAdd <sys/types.h> for size_t.
Sascha Wildner [Tue, 15 Apr 2008 08:11:50 +0000 (08:11 +0000)]
Add <sys/types.h> for size_t.

12 years agoRemove debugging assertion.
Matthew Dillon [Tue, 15 Apr 2008 01:45:22 +0000 (01:45 +0000)]
Remove debugging assertion.

12 years agoAdd __sreadahead() to help with pkgsrc's devel/m4.
Matthew Dillon [Tue, 15 Apr 2008 01:16:14 +0000 (01:16 +0000)]
Add __sreadahead() to help with pkgsrc's devel/m4.

12 years agos/.Nm/.Xr/
Sascha Wildner [Mon, 14 Apr 2008 22:18:14 +0000 (22:18 +0000)]

12 years agoUpdate the documentation for umtx_sleep() and umtx_wakeup().
Matthew Dillon [Mon, 14 Apr 2008 20:17:41 +0000 (20:17 +0000)]
Update the documentation for umtx_sleep() and umtx_wakeup().

12 years agoFix some issues in libthread_xu's condvar implementation.
Matthew Dillon [Mon, 14 Apr 2008 20:12:41 +0000 (20:12 +0000)]
Fix some issues in libthread_xu's condvar implementation.

* Non-broadcast mode is not guaranteed to signal just one waiter, loosen
  it up a bit to close race conditions and signal more if necessary.

* Clean up the condition structure.  Do not try to track non-broadcast
  wakeups.  Do not try to block waiting for individual wakeups... the
  spec does not require it and, in fact, doing so can create more
  problems then it solves.

* Load oldseq from cv->c_seqno *BEFORE* releasing the passed mutex to
  close a race.  The mutex is there precisely so that userland can
  guarantee that no race will occur between waiter and signaler.

Reported-by: Jordan Gordeev <jgordeev@dir.bg>,
             "Simon 'corecode' Schubert" <corecode@xxxxxxxxxxxx>

12 years agoFix a bug in umtx_sleep(). This function sleeps on the mutex's physical
Matthew Dillon [Mon, 14 Apr 2008 20:00:29 +0000 (20:00 +0000)]
Fix a bug in umtx_sleep().  This function sleeps on the mutex's physical
address and will get lost if the physical page underlying the VM address is
copied on write.  This case can occur when a threaded program fork()'s.

Introduce a VM page event notification mechanism and use it to wake-up
the umtx_sleep() if the underlying page takes a COW fault.

Reported-by: Jordan Gordeev <jgordeev@dir.bg>,
     "Simon 'corecode' Schubert" <corecode@xxxxxxxxxxxx>

12 years agoMinor optimization to LIST_FOREACH_MUTABLE taken from FreeBSD.
Matthew Dillon [Mon, 14 Apr 2008 19:44:18 +0000 (19:44 +0000)]
Minor optimization to LIST_FOREACH_MUTABLE taken from FreeBSD.

12 years agoDon't free held clean pages when asked to clean.
Matthew Dillon [Mon, 14 Apr 2008 19:43:02 +0000 (19:43 +0000)]
Don't free held clean pages when asked to clean.

12 years agoFix ktrace for threaded processes. Move the KTRFAC_ACTIVE flag to the LWP
Matthew Dillon [Mon, 14 Apr 2008 12:01:53 +0000 (12:01 +0000)]
Fix ktrace for threaded processes.  Move the KTRFAC_ACTIVE flag to the LWP
so a ktrace occuring on one LWP does not cause another LWP to fail to log
a trace entry.

12 years agoFix various typos in our manual pages.
Sascha Wildner [Mon, 14 Apr 2008 08:17:09 +0000 (08:17 +0000)]
Fix various typos in our manual pages.

12 years agoFix multiple issues with -p<parallel>, including several data corruption
Matthew Dillon [Mon, 14 Apr 2008 05:40:51 +0000 (05:40 +0000)]
Fix multiple issues with -p<parallel>, including several data corruption

* Use malloc'd buffers instead of static buffers
* Fix memory leaks
* Fix races with hardlink tracking structures
* Fix bug in remote mknod (it wasn't creating the node on the remote host)
* Fix stack blowouts from deep recursions by starting a new thread.
* Postpend the pid for tmp file creation.

12 years agopfsync_state doesn't have or need a hash field, the state will be hashed
Matthew Dillon [Sat, 12 Apr 2008 17:39:41 +0000 (17:39 +0000)]
pfsync_state doesn't have or need a hash field, the state will be hashed
when it is moved into a real pf_state structure.

Reported-by: Sascha Wildner <saw@online.de>
12 years agoAdd MLINKS for MD[245]Pad.3.
Sascha Wildner [Sat, 12 Apr 2008 08:38:40 +0000 (08:38 +0000)]
Add MLINKS for MD[245]Pad.3.

12 years agoStart sentence on a new line (and remove trailing space).
Sascha Wildner [Fri, 11 Apr 2008 20:22:18 +0000 (20:22 +0000)]
Start sentence on a new line (and remove trailing space).

12 years ago* Bump date for recent new features.
Sascha Wildner [Fri, 11 Apr 2008 18:27:44 +0000 (18:27 +0000)]
* Bump date for recent new features.

* Add -l to the synopsis.

* Some little mdoc fixes.

12 years agoImplement a number of major new features to PF.
Matthew Dillon [Fri, 11 Apr 2008 18:21:49 +0000 (18:21 +0000)]
Implement a number of major new features to PF.

* Implement several new options to keep/modulate/synproxy state.
  'pickups', 'no-pickups', and 'hash-only'.  Example:

    pass on $ext_if inet from any to any keep state (pickups)

* Implement a SET directive to set the default keep policy.  Example:

    set keep-policy keep state (pickups)
    pass on $ext_if inet from any to any

* Implement the 'no state' phrase for rules, disabling any previously
  specified default state policy for that rule (ala OpenBSD).

'pickups' and 'hash-only' imply no flag restrictions, no-pickups implies
flags S/SA.  When pickups are enabled the state code will re-establish state
for existing TCP connections.  Because the tcp options passed in the SYNs
are not known when this occurs, sequence space checking will be disabled
for those connections.  The 'hash-only' mode allows pickups but disables
sequence space checking regardless of whether the SYN packets were observed
or not.

The state code no longer attempts to do sequence space checking if it has
no information on TCP options passed in SYN packets.  The SYN from both
ends must be observed for sequence space checking to occur (which is what
happens by default due to flag restrictions).

It should be noted that OpenBSD defaults to 'keep state' with the equivalent
of no-pickups.  The problem with this is that rebooting you router or
otherwise losing state will cause existing TCP connections to be reset,
and I do not think this is an appropriate default for PF.  DragonFly does
not default to using keep state but introduces the global SET directive
to allow it to be enabled by default.

Finally, note that when using pickups or hash-only, if you desire PF to
not drop TCP connections on loss of state you need to have a few more
rules then usual to pickup on packets going the opposite direction.
For example, if you allow outgoing connections then you ALSO have to keep
state on INCOMING packets with 'flags /S' (i.e. 'established') in order
to ensure that active connections can be re-established on loss of state
by packets in either direction.

This isn't too bad since anyone using queueing has to use multiple rule
sets anyway to ensure that state is established on the proper queue.

In anycase, DragonFly now allows you to pick your poison.  One global SET
directive and you can use OpenBSD's policy, or you can make your own.

In-consultation-with: Max Laier <max@love2party.net>

12 years agoUpdate PORTING instructions for linux to support pthreads.
Matthew Dillon [Fri, 11 Apr 2008 17:33:11 +0000 (17:33 +0000)]
Update PORTING instructions for linux to support pthreads.

12 years agoProperly mark a transaction has being completed so the slave side of
Matthew Dillon [Fri, 11 Apr 2008 17:18:21 +0000 (17:18 +0000)]
Properly mark a transaction has being completed so the slave side of
the connection does not loop forever on its data.

12 years agoClean up remains after the Citrus update.
Sascha Wildner [Fri, 11 Apr 2008 11:18:16 +0000 (11:18 +0000)]
Clean up remains after the Citrus update.

12 years agoProperly detach children so we dont have to pthread_join() them. Fixes
Matthew Dillon [Fri, 11 Apr 2008 08:44:07 +0000 (08:44 +0000)]
Properly detach children so we dont have to pthread_join() them.  Fixes
a resource exhaustion issue which could stall cpdup when used with -pN.

12 years agoMore cpdup work.
Matthew Dillon [Fri, 11 Apr 2008 07:31:05 +0000 (07:31 +0000)]
More cpdup work.

* Add -C which is passed down to ssh to turn on compression.
* Fix a race condition in the hardlink handling code that could occur w/ -pN
* Fix a deadlock in the socket read code.  The connection's read mutex
  must be released before the master mutex can be re-acquired.
* Reduce the I/O block size when running parallel threads to try to avoid
  blocking on write().

12 years agoAdd parallel transaction support for remote source or target specifications.
Matthew Dillon [Thu, 10 Apr 2008 22:09:08 +0000 (22:09 +0000)]
Add parallel transaction support for remote source or target specifications.
The implementation is a bit crude because I don't want to take too many
chances on a codebase that wasn't originally designed to be multi-threaded,
so the master mutex is only released when a thread is waiting for input
on a socket.

* Add the -p<threads> option and compile -pthread by default.  This is only
  useful when the source and/or destination is a remote host.  Note that
  parallel transaction mode will not work with older cpdup binaries on the
  remote end.

  This greatly improves cpdup's performance when operating on a remote
  source and/or target.

* Add -l to force stdout and stderr to be line-buffered.

12 years agoSync Citrus iconv support with NetBSD.
Hasso Tepper [Thu, 10 Apr 2008 10:21:13 +0000 (10:21 +0000)]
Sync Citrus iconv support with NetBSD.

Obtained-from: NetBSD

12 years agoBring the list of variables and function docs in time(9) closer to
Sascha Wildner [Wed, 9 Apr 2008 21:41:59 +0000 (21:41 +0000)]
Bring the list of variables and function docs in time(9) closer to

Derived-from: FreeBSD

12 years agoFix ifdefs to make it possible to use time.h in standards compilant code.
Hasso Tepper [Wed, 9 Apr 2008 07:05:54 +0000 (07:05 +0000)]
Fix ifdefs to make it possible to use time.h in standards compilant code.

Obtained-from: FreeBSD

12 years agoAdd double_t and float_t typedefs for both i386 and amd64 as required by C99.
Hasso Tepper [Wed, 9 Apr 2008 06:44:37 +0000 (06:44 +0000)]
Add double_t and float_t typedefs for both i386 and amd64 as required by C99.

Obtained-from: FreeBSD with modifications

12 years agoSync with FreeBSD:
Sascha Wildner [Tue, 8 Apr 2008 13:23:38 +0000 (13:23 +0000)]
Sync with FreeBSD:

* Change semantics of -i (in-place editing) so that it treats each file
  independently from other files (desired in most practical cases). Keep
  the previous semantics (use a single continuous address space covering
  all files to edit in-place) under -I.

* Add -l option (make output line buffered).

* Make the 'y' command (translate) aware of multibyte characters.

* Add case-insensitive matching, using the 'I' flag, a la GNU sed.

* Remove 3rd clause.

* Raise WARNS to 6.

* Miscellaneous fixes and style(9) issues.

12 years agoRemove references to patches that no longer exist.
Sascha Wildner [Tue, 8 Apr 2008 12:12:53 +0000 (12:12 +0000)]
Remove references to patches that no longer exist.

12 years agoUpdate build for OpenSSH 5.0p1.
Peter Avalos [Mon, 7 Apr 2008 01:20:18 +0000 (01:20 +0000)]
Update build for OpenSSH 5.0p1.

12 years agoMFC 1.33/pf.c from NetBSD. Don't apply a window scale to the window
Matthew Dillon [Mon, 7 Apr 2008 00:43:44 +0000 (00:43 +0000)]
MFC 1.33/pf.c from NetBSD.   Don't apply a window scale to the window
size in a SYN packet.

Partial MFC 1.25/pf.c from NetBSD (specifically 1.487 from OpenBSD).
Fix a bug in the setting of sequence windows in the synproxy case.

12 years agoBring the 'probability' keyword into PF from NetBSD. This feature allows
Matthew Dillon [Sun, 6 Apr 2008 21:12:42 +0000 (21:12 +0000)]
Bring the 'probability' keyword into PF from NetBSD.  This feature allows
a rule to have a probability associated with it which governs whether the
rule is run or not for any given packet.

Suggested-by: =?ISO-8859-1?Q?C=E9dric_Berger?= <cedric@berger.to>
Obtained-from: NetBSD-current

12 years agoMention ALTQ_FAIRQ.
Sascha Wildner [Sun, 6 Apr 2008 19:29:47 +0000 (19:29 +0000)]

12 years agoStart sentences on a new line and bump date for fair queueing.
Sascha Wildner [Sun, 6 Apr 2008 19:29:23 +0000 (19:29 +0000)]
Start sentences on a new line and bump date for fair queueing.