12 years agoHAMMER 45/Many: Stabilization pass, undo sequencing.
Matthew Dillon [Thu, 15 May 2008 03:36:40 +0000 (03:36 +0000)]
HAMMER 45/Many: Stabilization pass, undo sequencing.

* The flusher was improperly requesting a reflush on buffers.  The flush
  request was being defered for any buffers with active front-end references
  and then wound up being flushed by the front-end, breaking ordering

  Remove the reflush flag entirely.  This fixes numerous crash recovery

* Add a missing unlock in the reblocking ioctl code which was responsible
  for a number of process lockups.

* Enhance the undo recovery kprintf.

* Validate the CRC in UNDO records

12 years agoHAMMER Utilities: Features
Matthew Dillon [Thu, 15 May 2008 03:23:41 +0000 (03:23 +0000)]
HAMMER Utilities: Features

* Display a 'B' flag for b-tree nodes whos CRCs fail.

12 years agoMake use of interrupt endpoint to increase responsiveness.
Hasso Tepper [Wed, 14 May 2008 20:21:22 +0000 (20:21 +0000)]
Make use of interrupt endpoint to increase responsiveness.

Obtained-from: OpenBSD

12 years agoRemove fortran from base.
Hasso Tepper [Wed, 14 May 2008 15:02:48 +0000 (15:02 +0000)]
Remove fortran from base.

12 years agoMention KTR_IFQ and KTR_IF_START
Sepherosa Ziehau [Wed, 14 May 2008 13:02:04 +0000 (13:02 +0000)]

Reminded-by: swildner@
12 years agoReduce ifnet.if_serializer contention on output path:
Sepherosa Ziehau [Wed, 14 May 2008 11:59:24 +0000 (11:59 +0000)]
Reduce ifnet.if_serializer contention on output path:
- Push ifnet.if_serializer holding down into each ifnet.if_output implementation
- Add a serializer into ifaltq, which is used to protect send queue instead of
  its parent's if_serializer.  This change has following implication:
  o  On output path, enqueueing packets and calling ifnet.if_start are decoupled
  o  In device drivers, poll->dev_encap_ok->dequeue operation sequence is no
     longer safe, instead dequeue->dev_encap_fail->prepend should be used
  This serializer will be held by using lwkt_serialize_adaptive_enter()
- Add altq_started field into ifaltq, which is used to interlock the calling
  of its parent's if_start, to reduce ifnet.if_serializer contention.
  if_devstart(), a helper function which utilizes ifaltq.altq_started, is added
  to reduce code duplication in ethernet device drivers.
- Add if_cpuid into ifnet.  This field indicates on which CPU device driver's
  interrupt will happen.
- Add ifq_dispatch().  This function will try to hold ifnet.if_serializer in
  order to call ifnet.if_start.  If this attempt fails, this function will
  schedule ifnet.if_start to be called on CPU located by ifnet.if_start_cpuid
  if_start_nmsg, which is per-CPU netmsg, is added to ifnet to facilitate
  ifnet.if_start scheduling.  ifq_dispatch() is called by ether_output_frame()
- Use ifq_classic_ functions, if altq is not enabled
- Fix various device drivers bugs in their if_start implementation
- Add ktr for ifq classic enqueue and dequeue
- Add ktr for ifnet.if_start

12 years agoHAMMER Utilities: Feature add
Matthew Dillon [Tue, 13 May 2008 20:49:34 +0000 (20:49 +0000)]
HAMMER Utilities: Feature add

Add a new command 'synctid <filesystem>' which provides a formal transaction
id representing the state of the filesystem as of when it is called.  The
filesystem will be properly synced by the command and an interlocked
timestamp will be returned.

Remove the -x option, remove the calls to sync() by the 'now' and 'now64'
directives.  No longer sleep for one second in 'now'.  Document that these
directives do not return a guaranteed transaction id.

12 years agoHAMMER 44/Many: Stabilization pass, user-guaranteed transaction ids
Matthew Dillon [Tue, 13 May 2008 20:46:55 +0000 (20:46 +0000)]
HAMMER 44/Many: Stabilization pass, user-guaranteed transaction ids

* B-Tree changes:  Allow leaves to be empty.  Do not leave internal nodes
  with subtree_offsets of 0 when deleting a B-Tree element.  Do not try to
  clean up internal nodes with subtree_offsets of 0 while scanning the B-Tree.

  The pruner will be made responsible for such cleanups.  This way the
  front-end does not modify the B-Tree at all.

* Add a new ioctl to support the hammer utility 'synctid' command.

12 years agoRemove udbp(4) form tree. It was never connected to the build and supports
Hasso Tepper [Tue, 13 May 2008 08:46:36 +0000 (08:46 +0000)]
Remove udbp(4) form tree. It was never connected to the build and supports
only ancient devices.

12 years agoAssorted fixes to ugen(4) from FreeBSD.
Hasso Tepper [Tue, 13 May 2008 08:35:12 +0000 (08:35 +0000)]
Assorted fixes to ugen(4) from FreeBSD.

Obtained-from: FreeBSD

12 years agoHAMMER 43C/Many: Performance cleanup
Matthew Dillon [Tue, 13 May 2008 05:04:39 +0000 (05:04 +0000)]
HAMMER 43C/Many: Performance cleanup

* Do not put an inode on the sync list when it becomes inactive, unless
  it no longer has filesystem visibility.  This greatly reduces the amount
  of cycling which occurs on the inode B-Tree element, and greatly reduces
  spurious backend flushes.

  This change results in a major performance improvement for both reads
  and writes in a massively parallel I/O test (blogbench).  Write
  performance more then doubles and read performance increases by 20%.

* Lock the freemap free code.

* Remove some of the debug code.

12 years agoHAMMER Utilities: Misc features and adjustments.
Matthew Dillon [Tue, 13 May 2008 04:58:10 +0000 (04:58 +0000)]
HAMMER Utilities: Misc features and adjustments.

* Directory and symlink names were not being properly clipped.

* Additional -v options will display more of the contents of the inode
  data structure.

12 years agoHAMMER 43B/Many: Correct delete-on-disk record bug.
Matthew Dillon [Tue, 13 May 2008 00:15:28 +0000 (00:15 +0000)]
HAMMER 43B/Many: Correct delete-on-disk record bug.

* When an inode is being deleted any delete-on-disk records must be flushed,
  instead of freed.

12 years agoHAMMER 43A/Many: Cleanup, bug fixes.
Matthew Dillon [Mon, 12 May 2008 23:15:46 +0000 (23:15 +0000)]
HAMMER 43A/Many: Cleanup, bug fixes.

* Fix an assertion that can occur when a deleted softlink is racing a
  sync of the inode.

* Set HAMER_INODE_DONDISK for general records as well as for data records.
  This flag is set when a record might be laid down on media before its
  inode.  (This fixes an assertion).

* Set HAMMER_INODE_DDIRTY for all mtime updates.

* Clean up hammer_ip_check_directory_empty().

12 years agoHAMMER 43/Many: Remove records from the media format, plus other stuff
Matthew Dillon [Mon, 12 May 2008 21:17:18 +0000 (21:17 +0000)]
HAMMER 43/Many: Remove records from the media format, plus other stuff

* Get rid of hammer_record_ondisk.  As HAMMER has evolved the need for
  a separate record structure has devolved into trivialities.  Originally
  the idea was to have B-Tree nodes referencing records and data.  The
  B-Tree elements were originally intended to be throw-away and the on-media
  records were originally intended to be the official representation of
  the data and contained additional meta-information such as the obj_id
  of a directory entry and a few additional fields related to the inode.

  But once the UNDO code went in and it became obvious that the B-Tree needed
  to be tracked (undo-wise) along with everything else, the need for an
  official representation of the record as a separate media structure
  essentially disappeared.

  Move the directory-record meta-data into the directory-entry data and move
  the inode-record meta-data into the inode-record data.  As a single
  exception move the atime field to the B-Tree element itself (it replaces
  what used to be the record offset), in order to continue to allow atime
  updates to occur without requiring record rewrites.  With these changes
  records are no longer needed at all, so remove the on-media record structure
  and all the related code.

* The removal of the on-media record structure also greatly improves

* B-Tree elements are now the official on-media record.

* Fix a race in the extraction of the root of the B-Tree.

* Clean up the in-memory record handling API.  Instead of having to
  construct B-Tree leaf elements we can simply embed one in the in-memory
  record structure (struct hammer_record), and in the inode.

12 years agoReword slightly and point out that DESTDIR has to exist.
Sascha Wildner [Mon, 12 May 2008 16:26:20 +0000 (16:26 +0000)]
Reword slightly and point out that DESTDIR has to exist.

12 years agoHAMMER Utilities: Cleanup.
Matthew Dillon [Mon, 12 May 2008 05:13:48 +0000 (05:13 +0000)]
HAMMER Utilities: Cleanup.

* Finish cleaning up the pruning and reblocking code.

* Add reblocking commands to reblock B-Tree, records, and data separately.

12 years agoHAMMER 42E/Many: Cleanup.
Matthew Dillon [Mon, 12 May 2008 05:13:11 +0000 (05:13 +0000)]
HAMMER 42E/Many: Cleanup.

* Finish cleaning up the pruning and reblocking code.  The reblocking
  code still does not handle internal B-Tree nodes, though.

* Add options to allow the B-Tree, records, and data to be reblocked
  independantly of each other.

12 years agoHAMMER 42D/Many: Stabilization.
Matthew Dillon [Sun, 11 May 2008 21:45:44 +0000 (21:45 +0000)]
HAMMER 42D/Many: Stabilization.

* Fix a bug in the pruning code.  When iterating backwards and we delete
  an entry, the next iteration must skip the current cursor position which,
  due to the deletion, may be pointing to a record we have already scanned
  or may be pointing beyond our range.  If it is pointing beyond our range
  an assertion will occur.

12 years agoHAMMER Utilities: Features
Matthew Dillon [Sun, 11 May 2008 20:44:44 +0000 (20:44 +0000)]
HAMMER Utilities: Features

* Add a new option '-c cyclefile' which when coupled with '-t timeout'
  allows successive invocations of a pruning or reblocking operation to
  pick up where it left off.

12 years agoFix UP real kernel crash, a vkernel define was being used iproperly
Matthew Dillon [Sun, 11 May 2008 17:20:53 +0000 (17:20 +0000)]
Fix UP real kernel crash, a vkernel define was being used iproperly

12 years agoBring ifnet(9) a bit closer to reality.
Sascha Wildner [Sun, 11 May 2008 07:11:47 +0000 (07:11 +0000)]
Bring ifnet(9) a bit closer to reality.

12 years agoAdd an ffs(5) manual page.
Sascha Wildner [Sun, 11 May 2008 06:46:20 +0000 (06:46 +0000)]
Add an ffs(5) manual page.

Adapted-from: FreeBSD's ffs(7)

12 years agoAdd useconds_t and suseconds_t used for time in microseconds.
Hasso Tepper [Sun, 11 May 2008 03:39:49 +0000 (03:39 +0000)]
Add useconds_t and suseconds_t used for time in microseconds.

12 years agoHAMMER Utilities: scan feedback
Matthew Dillon [Sat, 10 May 2008 22:56:37 +0000 (22:56 +0000)]
HAMMER Utilities: scan feedback

* Add support for and report the number of records scanned by the pruning

12 years agoComment out some unimplemented syscalls.
Sascha Wildner [Sat, 10 May 2008 20:39:51 +0000 (20:39 +0000)]
Comment out some unimplemented syscalls.

12 years agoHAMMER 42C/Many: Stabilization.
Matthew Dillon [Sat, 10 May 2008 20:07:14 +0000 (20:07 +0000)]
HAMMER 42C/Many: Stabilization.

* Fix a deadlock.  Do not flush synchronously while holding a cursor (this
  time in the prune code).

12 years agoHAMMER 42B/Many: Stabilization.
Matthew Dillon [Sat, 10 May 2008 19:52:38 +0000 (19:52 +0000)]
HAMMER 42B/Many: Stabilization.

* Fix a CRC panic when reblocking.  Properly regenerate the CRC for new
  records and B-Tree nodes.

* Fix a deadlock.  Do not flush synchronously while holding a cursor.  Doh!

12 years agoFix function name.
Sascha Wildner [Sat, 10 May 2008 18:22:41 +0000 (18:22 +0000)]
Fix function name.

12 years agoHAMMER Utilities: Feature add
Matthew Dillon [Sat, 10 May 2008 17:54:59 +0000 (17:54 +0000)]
HAMMER Utilities: Feature add

* Add another shortcut to the prune command, telling HAMMER to prune everything
  past a certain point.  For example, this will recover anything deleted over
  a year ago: 'hammer prune /mnt from 365d everything'

12 years agoFix function name.
Sascha Wildner [Sat, 10 May 2008 17:52:10 +0000 (17:52 +0000)]
Fix function name.

12 years agoFinish moving the kernel from tsc_freq (32 bits) to tsc_frequency (64 bits).
Matthew Dillon [Sat, 10 May 2008 17:24:12 +0000 (17:24 +0000)]
Finish moving the kernel from tsc_freq (32 bits) to tsc_frequency (64 bits).

Add sysctls to access hw.tsc_present and hw.tsc_frequency

Implement TSC support in the vkernel.

12 years agoiwi(4) does not support hardware based WEP encryption/descryption.
Sepherosa Ziehau [Sat, 10 May 2008 07:11:28 +0000 (07:11 +0000)]
iwi(4) does not support hardware based WEP encryption/descryption.

Reported-by: Johannes Hofmann <Johannes.Hofmann@gmx.de>
12 years agoFix feature logic so changing kern.pipe.dwrite_enable on the fly works
Matthew Dillon [Sat, 10 May 2008 01:25:55 +0000 (01:25 +0000)]
Fix feature logic so changing kern.pipe.dwrite_enable on the fly works
properly.  Before it could cause processes to block forever.

12 years agoSync member functions with our struct ifnet.
Sascha Wildner [Fri, 9 May 2008 22:38:35 +0000 (22:38 +0000)]
Sync member functions with our struct ifnet.

12 years agoHAMMER 42A/Many: Stabilization.
Matthew Dillon [Fri, 9 May 2008 22:17:43 +0000 (22:17 +0000)]
HAMMER 42A/Many: Stabilization.

* Fix a double call to biodone() when attempting to flush front-end buffer
  cache buffers associated with a deleted file.  A return statement was

12 years agoUse .Dv for signal names.
Sascha Wildner [Fri, 9 May 2008 20:31:04 +0000 (20:31 +0000)]
Use .Dv for signal names.

12 years agoFix a HAMMER assertion which turned out to be a bug in VOP_N*(). Sometimes
Matthew Dillon [Fri, 9 May 2008 17:52:18 +0000 (17:52 +0000)]
Fix a HAMMER assertion which turned out to be a bug in VOP_N*().  Sometimes
the dvp passed to these functions can be reclaimed.  The locked leaf
namecache node is not sufficient to prevent its parent directory from
being reclaimed under heavy loads.

Instead of trying to play cute tricks, actually do a formal reference of
the dvp.  We don't have to lock it, though.

12 years agoReturn EINVAL if a NULL pointer is passed to the mutex routines, instead
Matthew Dillon [Fri, 9 May 2008 16:03:27 +0000 (16:03 +0000)]
Return EINVAL if a NULL pointer is passed to the mutex routines, instead
of crashing.  This appears to be what the standard intended and what libc_r

12 years agoSync sysperf with some random stuff, and add a cld instruction tester.
Matthew Dillon [Fri, 9 May 2008 15:49:42 +0000 (15:49 +0000)]
Sync sysperf with some random stuff, and add a cld instruction tester.

12 years agoMake the code we're running under total signal mask as short as possible.
Simon Schubert [Fri, 9 May 2008 11:24:08 +0000 (11:24 +0000)]
Make the code we're running under total signal mask as short as possible.

This fixes a hang when calling pthread_create(NULL, ...), which was due
to the fact that we hit a SIGSEGV while under total signal mask, leading
to infinite page faults.

12 years agoHAMMER 42/Many: Cleanup.
Matthew Dillon [Fri, 9 May 2008 07:26:51 +0000 (07:26 +0000)]
HAMMER 42/Many: Cleanup.

* Finish up the general code to associate an inode with a cursor.

12 years agoFix many bugs and issues in the VM system, particularly related to
Matthew Dillon [Fri, 9 May 2008 07:24:48 +0000 (07:24 +0000)]
Fix many bugs and issues in the VM system, particularly related to
heavy paging.

* (cleanup) PG_WRITEABLE is now set by the low level pmap code and not by
  high level code.  It means 'This page may contain a managed page table
  mapping which is writeable', meaning that hardware can dirty the page
  at any time.  The page must be tested via appropriate pmap calls before
  being disposed of.

* (cleanup) PG_MAPPED is now handled by the low level pmap code and only
  applies to managed mappings.  There is still a bit of cruft left over
  related to the pmap code's page table pages but the high level code is now

* (bug) Various XIO, SFBUF, and MSFBUF routines which bypass normal paging
  operations were not properly dirtying pages when the caller intended
  to write to them.

* (bug) vfs_busy_pages in kern/vfs_bio.c had a busy race.  Separate the code
  out to ensure that we have marked all the pages as undergoing IO before we
  call vm_page_protect().  vm_page_protect(... VM_PROT_NONE) can block
  under very heavy paging conditions and if the pages haven't been marked
  for IO that could blow up the code.

* (optimization) Make a minor optimization.  When busying pages for write
  IO, downgrade the page table mappings to read-only instead of removing
  them entirely.

* (bug) In platform/pc32/i386/pmap.c fix various places where
  pmap_inval_add() was being called at the wrong point.  Only one was
  critical, in pmap_enter(), where pmap_inval_add() was being called so far
  away from the pmap entry being modified that it could wind up being flushed
  out prior to the modification, breaking the cpusync required.

  pmap.c also contains most of the work involved in the PG_MAPPED and
  PG_WRITEABLE changes.

* (bug) Close numerous pte updating races with hardware setting the
  modified bit.  There is still one race left (in pmap_enter()).

* (bug) Disable pmap_copy() entirely.   Fix most of the bugs anyway, but
  there is still one left in the handling of the srcmpte variable.

* (cleanup) Change vm_page_dirty() from an inline to a real procedure, and
  move the code which set the object to writeable/maybedirty into

* (bug) Calls to vm_page_protect(... VM_PROT_NONE) can block.  Fix all cases
  where this call was made with a non-busied page.  All such calls are
  now made with a busied page, preventing blocking races from re-dirtying
  or remapping the page unexpectedly.

  (Such blockages could only occur during heavy paging activity where the
  underlying page table pages are being actively recycled).

* (bug) Fix the pageout code to properly mark pages as undergoing I/O before
  changing their protection bits.

* (bug) Busy pages undergoing zeroing or partial zeroing in the vnode pager
  (vm/vnode_pager.c) to avoid unexpected effects.

12 years agoFix fork/vfork statistics. forks and vforks were being improperly counted
Matthew Dillon [Fri, 9 May 2008 06:38:19 +0000 (06:38 +0000)]
Fix fork/vfork statistics.  forks and vforks were being improperly counted
as rforks.

12 years agoFix a nasty memory corruption issue which can occur due to the kernel bcopy's
Matthew Dillon [Fri, 9 May 2008 06:35:12 +0000 (06:35 +0000)]
Fix a nasty memory corruption issue which can occur due to the kernel bcopy's
use of the FP unit.  If the destination address faults the NPX code can
lose track of the fact that the kernel was using the FP unit.  When the
fault is resolved the kernel bcopy resumes with corrupted FP registers.

The most common situation where this could occur is with pipes, and generally
only when the system is paging heavily and causing multiple processes to
fault in the kernel FP bcopy code.

12 years agoFix a race between the namecache and the vnode recycler. A vnode cannot be
Matthew Dillon [Thu, 8 May 2008 01:41:07 +0000 (01:41 +0000)]
Fix a race between the namecache and the vnode recycler.  A vnode cannot be
recycled if it's namecache entry represents a directory with locked children.
The various VOP_N*() functions require the parent dvp to be stable.

The main fix is in vrecycle() (kern/vfs_subr.c).  Do not vgone() the vnode
if we can't clean out the children.

Also create an API to assert that the parent dvp is stable, and make it
vhold/vdrop the dvp.

The race primarily effected HAMMER which uses the VOP_N*() API.

12 years agoFix some lock ordering issues in the pipe code.
Matthew Dillon [Thu, 8 May 2008 01:31:01 +0000 (01:31 +0000)]
Fix some lock ordering issues in the pipe code.

In particular fix a bug in the pipe_write() code when multiple writers
are present that could cause garbage to be injected into the pipe due
to a resize possibly occuring while wpipe->pipe_buffer.cnt is non-zero.

12 years agoRecode the resource limit core (struct plimit) to fix a few races and
Matthew Dillon [Thu, 8 May 2008 01:26:01 +0000 (01:26 +0000)]
Recode the resource limit core (struct plimit) to fix a few races and
generally make it work better.  Also make changes with an eye towards
making it MPSAFE.

12 years agoClear the direction flag (CLD) on entry to the kernel, to support future
Matthew Dillon [Thu, 8 May 2008 01:21:06 +0000 (01:21 +0000)]
Clear the direction flag (CLD) on entry to the kernel, to support future
kernels compiled with GCC-4.3 or later.

12 years agoAdd some lines about lwkt_serialize_adaptive_enter().
Sascha Wildner [Wed, 7 May 2008 20:03:09 +0000 (20:03 +0000)]
Add some lines about lwkt_serialize_adaptive_enter().

Submitted-by: sephe
12 years agoBump base development version to 197700 so it is properly distinct from
Matthew Dillon [Wed, 7 May 2008 17:26:28 +0000 (17:26 +0000)]
Bump base development version to 197700 so it is properly distinct from
the 1.12 release version.

Reported-by: Hasso Tepper
12 years agoCorrect comments and minor variable naming and sysctl issues.
Matthew Dillon [Wed, 7 May 2008 17:19:47 +0000 (17:19 +0000)]
Correct comments and minor variable naming and sysctl issues.

Reported-by: Fabio Checconi <fabio@gandalf.sssup.it>
12 years agoFix a sizeof() the wrong variable name. The correct variable was the same
Matthew Dillon [Tue, 6 May 2008 21:40:40 +0000 (21:40 +0000)]
Fix a sizeof() the wrong variable name.  The correct variable was the same
size so no harm done, but get it right.

Submitted-by: Fabio Checconi <fabio@gandalf.sssup.it>
12 years agoThe vkernel's maximum number of CPUs is now 16.
Sascha Wildner [Tue, 6 May 2008 18:55:01 +0000 (18:55 +0000)]
The vkernel's maximum number of CPUs is now 16.

12 years agoEnable kern.trap_mpsafe and kern.syscall_mpsafe by default for vkernels.
Matthew Dillon [Tue, 6 May 2008 18:43:02 +0000 (18:43 +0000)]
Enable kern.trap_mpsafe and kern.syscall_mpsafe by default for vkernels.

12 years agoRemove the SMP_MAXCPU override for vkernels, causing the build to revert
Matthew Dillon [Tue, 6 May 2008 18:37:58 +0000 (18:37 +0000)]
Remove the SMP_MAXCPU override for vkernels, causing the build to revert
to the i386 limit of 16.  This is not because vkernels couldn't handle more
(up to 31), but because I want the installed world to be compatible between
vkernel and pc32.

This unbreaks programs like 'vmstat -m'.

12 years agoAdd strings for some AMD features
Sepherosa Ziehau [Tue, 6 May 2008 10:05:02 +0000 (10:05 +0000)]
Add strings for some AMD features

12 years agoHAMMER 41B/Many: Cleanup.
Matthew Dillon [Tue, 6 May 2008 00:21:08 +0000 (00:21 +0000)]
HAMMER 41B/Many: Cleanup.

* Disable (most) debugging kprintfs unless a hammer debug sysctl is set.

* Do not allow buffers to be synced on panic.

12 years agoHAMMER Utilities: Sync with recent changes.
Matthew Dillon [Tue, 6 May 2008 00:15:35 +0000 (00:15 +0000)]
HAMMER Utilities: Sync with recent changes.

* Add some missing crc initializations.

* Fix an assertion that was breaking newfs_hammer on large disks.

12 years agoKeep track of the number of buffers undgoing IO, and include that number
Matthew Dillon [Tue, 6 May 2008 00:14:12 +0000 (00:14 +0000)]
Keep track of the number of buffers undgoing IO, and include that number
in calculations involving numdirtybuffers.  This prevents the kernel from
believing that there are only a few dirty buffers when, in fact, all the
dirty buffers are running IOs.

12 years agoOnly call bwillwrite() for regular file write()s, instead of for all write()s.
Matthew Dillon [Mon, 5 May 2008 22:09:44 +0000 (22:09 +0000)]
Only call bwillwrite() for regular file write()s, instead of for all write()s.
This stops hicuping on things like pty's and sockets during heavy file

12 years agoHAMMER Utilities: Feature add
Matthew Dillon [Mon, 5 May 2008 20:34:52 +0000 (20:34 +0000)]
HAMMER Utilities: Feature add

* Check record crc and signature in extra-verbose mode

* Adjustments for structural changes

* Generate proper CRCs for structures laid down by newfs_hammer

12 years agoHAMMER 41/Many: Implement CRC checking (WARNING: On-media structures changed)
Matthew Dillon [Mon, 5 May 2008 20:34:48 +0000 (20:34 +0000)]
HAMMER 41/Many: Implement CRC checking (WARNING: On-media structures changed)

* Generate and check on-media CRC fields.

* Clean up the record modification API

* Add a header signature field for future critical recovery

* Rearrange CRC fields for on-media structures to make them easier to
  deal with.

* Adjust the ioctl API

* When trying to back-out of an operation that errored, free allocated
  data blocks.

12 years ago- Add lwkt_serialize_adaptive_enter(9), it is same as lwkt_serialize_enter(9)
Sepherosa Ziehau [Mon, 5 May 2008 12:35:03 +0000 (12:35 +0000)]
- Add lwkt_serialize_adaptive_enter(9), it is same as lwkt_serialize_enter(9)
  except that it spins a little bit before sleeping.
- Under debug sysctl tree, add sysctl nodes to tune various backoff related
  parameter for lwkt_serialize_adaptive_enter(9).
- Add ktr for serializer enter end, exit begin, spin backoff and spin backoff

Reviewed-by: corecode@
12 years agoUse mask instead of modulo, since bo->backoff is always power of 2
Sepherosa Ziehau [Mon, 5 May 2008 11:07:48 +0000 (11:07 +0000)]
Use mask instead of modulo, since bo->backoff is always power of 2

Suggested-by: dillon@
12 years agoHAMMER 40G/Many: UNDO cleanup & stabilization.
Matthew Dillon [Sun, 4 May 2008 19:57:42 +0000 (19:57 +0000)]
HAMMER 40G/Many: UNDO cleanup & stabilization.

* Fix a race in the undo record allocator that could result in a
  corrupted UNDO FIFO.

* Fix improperly placed calls to hammer_test_inode().

* Properly account for nlinks when a deleted ADD record is to be
  converted to a DEL record by the flush.  In this case the frontend's
  notion of nlinks accounts for the deletion but the backend must sync
  the record anyway, so the backend needs to bump the link count by one.

12 years agoHAMMER Utilities: enhanced show, timeout option
Matthew Dillon [Sun, 4 May 2008 19:18:17 +0000 (19:18 +0000)]
HAMMER Utilities: enhanced show, timeout option

* Enchange the show command when used with -vvv.  The command now reports
  directory entries and basic information about inodes.

* Add the [-t timeout] option.  The idea is to use this to limit the amount
  of time hammer spends reblocking or pruning a filesystem when running the
  command from a cron job.

* Adjust the format of the softlink option to be more consistent.

12 years agoAdjust to our current directory layout on pkgbox.
Sascha Wildner [Sun, 4 May 2008 17:07:49 +0000 (17:07 +0000)]
Adjust to our current directory layout on pkgbox.

Reported-by: aggelos and others
12 years agoHAMMER 40F/Many: UNDO cleanup & stabilization.
Matthew Dillon [Sun, 4 May 2008 09:06:45 +0000 (09:06 +0000)]
HAMMER 40F/Many: UNDO cleanup & stabilization.

* Properly classify UNDO zone buffers so they are flushed at the correct
  point in time.

* Minor rewrite of the code tracking the UNDO demark for the next flush.

* Introduce a considerably better backend flushing activation algorithm
  to avoid single-buffer flushes.

* Put a lock around the freemap allocator.

12 years agoThe direct-write pipe code has a bug in it somewhere when the system is
Matthew Dillon [Sun, 4 May 2008 08:42:03 +0000 (08:42 +0000)]
The direct-write pipe code has a bug in it somewhere when the system is
paging heavily.  Disable it for now.

12 years ago- Randomize spinlock exponential backoff value, which reduces the chance of
Sepherosa Ziehau [Sun, 4 May 2008 04:48:47 +0000 (04:48 +0000)]
- Randomize spinlock exponential backoff value, which reduces the chance of
  serious spinlock contention (probably) caused by same backoff steps
- Ktr spinlock backoff value and backoff failure
- Under debug sysctl tree, add sysctl node for spinlock backoff limit
- Break long lines

Reviewed-by: dillon@
12 years agoAdd more missing entries and fix more mistakes (some of which I introduced
Sascha Wildner [Sun, 4 May 2008 04:17:11 +0000 (04:17 +0000)]
Add more missing entries and fix more mistakes (some of which I introduced
in my last commit).

12 years agoFix some mistakes and add some missing entries.
Sascha Wildner [Sun, 4 May 2008 03:44:07 +0000 (03:44 +0000)]
Fix some mistakes and add some missing entries.

12 years agoPrint the 64 bit inode as a 64 bit quantity rather then a 32 bit quantity.
Matthew Dillon [Sun, 4 May 2008 02:28:28 +0000 (02:28 +0000)]
Print the 64 bit inode as a 64 bit quantity rather then a 32 bit quantity.

12 years agoAdd missing names and MLINKS.
Sascha Wildner [Sun, 4 May 2008 00:55:01 +0000 (00:55 +0000)]
Add missing names and MLINKS.

12 years agoCorrect a bug in seekdir/readdir which could cause the directory entry
Matthew Dillon [Sat, 3 May 2008 22:07:37 +0000 (22:07 +0000)]
Correct a bug in seekdir/readdir which could cause the directory entry
after a deleted entry to be skipped when seeking past the deleted entry.

NOTE: DragonFly has a specific issue even after this fix which currently
causes seekdirs to be unreliable if any files are deleted.  DragonFly
translates directory entries into a filesystem-independant form and if
the real filesystem collapses the entry, the offsets will not be maintained
in the machine-independant form.

Submitted-by: Marc Balmer <marc@msys.ch>
12 years agoHAMMER 40F/Many: Inode/link-count sequencer cleanup pass, UNDO cache.
Matthew Dillon [Sat, 3 May 2008 20:21:20 +0000 (20:21 +0000)]
HAMMER 40F/Many: Inode/link-count sequencer cleanup pass, UNDO cache.

* Implement an UNDO cache.  If we have already laid down an UNDO in the
  current flush cycle we do not have to lay down another one for the same
  address.  This greatly reduces the number of UNDOs we generate during
  a flush.

* Properly get the vnode in order to be able to issue vfsync()'s from the
  backend.  We may also have to acquire the vnode when doing an unload
  check for a file deletion.

* Properly generate UNDO records for the volume header.  During crash recovery
  we have to UNDO the volume header along with any partially written
  meta-data, because the volume header refers to the meta-data.

* Add another record type, GENERAL, representing inode or softlink records.

* Move the setting of HAMMER_INODE_WRITE_ALT to the backend, allowing
  the kernel to flush buffers up to the point where the backend syncs
  the inode.

12 years agoHAMMER 40E/Many: Inode/link-count sequencer cleanup pass.
Matthew Dillon [Sat, 3 May 2008 07:59:06 +0000 (07:59 +0000)]
HAMMER 40E/Many: Inode/link-count sequencer cleanup pass.

* An inode can go inactive before it is deleted, add an unload check
  in hammer_ip_del_directory to catch the nlinks == 0 case on an inactive
  inode.  Otherwise the inode would not be deleted on-media until umount.

* Add a missing resignaling case.

* Clean out a few more of the debug kprintf()'s

12 years agoHAMMER 40D/Many: Inode/link-count sequencer cleanup pass.
Matthew Dillon [Sat, 3 May 2008 05:28:55 +0000 (05:28 +0000)]
HAMMER 40D/Many: Inode/link-count sequencer cleanup pass.

* Move the vfsync from the frontend to the backend.  This allows the
  frontend to passively move inodes to the backend without having to
  actually start the flush, greatly improving performance.

* Use an inode lock to deal with directory entry syncing races between
  the frontend and the backend.  It isn't optimal but it's ok for now.

* Massively optimize the backend code by initializing a single cursor
  for an inode and passing the cursor to procedures, instead of having
  each procedure initialize its own cursor.

* Fix a sequencing issue with the backend.  While building the flush
  state for an inode another process could get in and initiate its own
  flush, screwing up the flush group and creating confusion.

* Don't lose track of HAMMER_FLUSH_SIGNAL flush requests.  If we get
  such a requet but have to flag a reflush, also flag that the reflush
  is to be signaled (done immediately when the current flush is done).

* Remove shared inode locks from hammer_vnops.c.  Their original purpose
  no longer exists.

* Simplify the arguments passed to numerous procedures (hammer_ip_first(),

12 years agoPrint the path even if we do not understand the filesystem type.
Matthew Dillon [Sat, 3 May 2008 04:13:12 +0000 (04:13 +0000)]
Print the path even if we do not understand the filesystem type.

Fix a switch/case compiler warning.

12 years agoElaborate a bit more on lexical conventions and ISA device configuration.
Sascha Wildner [Fri, 2 May 2008 22:10:58 +0000 (22:10 +0000)]
Elaborate a bit more on lexical conventions and ISA device configuration.

Taken-from: FreeBSD

12 years agoHAMMER 40C/Many: Inode/link-count sequencer cleanup pass.
Matthew Dillon [Fri, 2 May 2008 16:41:26 +0000 (16:41 +0000)]
HAMMER 40C/Many: Inode/link-count sequencer cleanup pass.

* Fix a forever-syncing inode issue by properly clearing the XDIRTY flag
  when the last record is removed from ip->rec_tree.

12 years ago- Put exit ktr in proper place
Sepherosa Ziehau [Fri, 2 May 2008 11:17:19 +0000 (11:17 +0000)]
- Put exit ktr in proper place
- Add sleep_{beg,end} and wakeup_{beg,end} ktr

12 years agoWhite space
Sepherosa Ziehau [Fri, 2 May 2008 10:57:33 +0000 (10:57 +0000)]
White space

12 years agoUse a list with tags.
Sascha Wildner [Fri, 2 May 2008 10:46:33 +0000 (10:46 +0000)]
Use a list with tags.

12 years agoIntroduce ETHER_INPUT_CHAIN option:
Sepherosa Ziehau [Fri, 2 May 2008 07:40:32 +0000 (07:40 +0000)]
Introduce ETHER_INPUT_CHAIN option:
1) During RXEOF, we aggregate packets, which have same target CPU, instead of
   calling lwkt_sendmsg() for each input packet.
2) At the end of RXEOF, low level ipiq sending is used to dispatch mbuf chain
   to the target CPU.
3) On the target CPU, the ipi function puts mbuf to their belonging msgport.
   Note, though lwkt_sendmsg() is used in ipi function, no further ipi activity
   will happen, since we are on target CPU.

em(4) is made to aware of this option.
This option is off by default and has no effect on vlan(4) operation.

12 years agoHAMMER 40B/Many: Inode/link-count sequencer cleanup pass.
Matthew Dillon [Fri, 2 May 2008 06:51:57 +0000 (06:51 +0000)]
HAMMER 40B/Many: Inode/link-count sequencer cleanup pass.

* Fix data record leakage w/ final inode disposition on disk.

* Fix numerous live locks with infinitely re-syncing inodes.

12 years agoSweep over our manual pages and remove .Pp before a .Bd or .Bl without
Sascha Wildner [Fri, 2 May 2008 02:05:08 +0000 (02:05 +0000)]
Sweep over our manual pages and remove .Pp before a .Bd or .Bl without
-compact because it has no effect.

12 years agoHAMMER 40A/Many: Inode/link-count sequencer.
Matthew Dillon [Fri, 2 May 2008 01:00:42 +0000 (01:00 +0000)]
HAMMER 40A/Many: Inode/link-count sequencer.

* Remove the hammer_depend structure and build the dependancies directly
  into the hammer_record structure.

* Attempt to implement layout rules to ensure connectivity is maintained.
  This means, for example, that before HAMMER can flush a newly created
  file it will make sure the file has namespace connectivity to the
  directory it was created it, recursively to the root.

NOTE: 40A destabilizes the filesystem a bit, it's going to take a few
passes to get everything working properly.  There are numerous issues
with this commit.

12 years agoProperly yield to userland processes.
Simon Schubert [Fri, 2 May 2008 00:19:52 +0000 (00:19 +0000)]
Properly yield to userland processes.

12 years agoMove text that doesn't belong to a list outside of it.
Sascha Wildner [Thu, 1 May 2008 23:36:43 +0000 (23:36 +0000)]
Move text that doesn't belong to a list outside of it.

12 years agoAdd .It
Sascha Wildner [Thu, 1 May 2008 23:29:10 +0000 (23:29 +0000)]
Add .It

12 years agoMove .Pp outside of .Bl
Sascha Wildner [Thu, 1 May 2008 22:06:06 +0000 (22:06 +0000)]
Move .Pp outside of .Bl

12 years agoReduce vertical space.
Sascha Wildner [Thu, 1 May 2008 21:51:43 +0000 (21:51 +0000)]
Reduce vertical space.

12 years agoRemove some obsolete lines.
Sascha Wildner [Thu, 1 May 2008 20:24:01 +0000 (20:24 +0000)]
Remove some obsolete lines.

12 years agoTurn off yy_flex_realloc() related warnings (such as the one issued when
Sascha Wildner [Thu, 1 May 2008 20:01:24 +0000 (20:01 +0000)]
Turn off yy_flex_realloc() related warnings (such as the one issued when
building usr.sbin/config) by marking the function unused.

There are a number of things which decide whether it's used or not, such
as using REJECT, %option yylineno, and some trailing context patterns.

See NetBSD's revisions 1.10 & 1.19.

Taken-from: NetBSD

12 years agoSet a sensible mode on /etc/upgrade/Makefile_upgrade.inc .
Thomas E. Spanjaard [Thu, 1 May 2008 19:44:37 +0000 (19:44 +0000)]
Set a sensible mode on /etc/upgrade/Makefile_upgrade.inc .

12 years agoRegenerate the pciconf(8) database from the following files:
Sascha Wildner [Thu, 1 May 2008 18:02:45 +0000 (18:02 +0000)]
Regenerate the pciconf(8) database from the following files:

Hart:    Jan 22, 2008 (version 671)
Boemler: May  1, 2008
Mares:   Mar  1, 2008

12 years agoAdd FreeBSD 7.1 (which is already referenced in cmx.4).
Sascha Wildner [Thu, 1 May 2008 13:04:51 +0000 (13:04 +0000)]
Add FreeBSD 7.1 (which is already referenced in cmx.4).

12 years agoMention that BCM430[69] chips do not work properly on channel 1/2/3
Sepherosa Ziehau [Thu, 1 May 2008 12:34:06 +0000 (12:34 +0000)]
Mention that BCM430[69] chips do not work properly on channel 1/2/3