DragonFly BSD

CVS log for src/sys/kern/lwkt_ipiq.c

[BACK] Up to [DragonFly] / src / sys / kern

Request diff between arbitrary revisions


Keyword substitution: kv
Default branch: MAIN


Revision 1.27: download - view: text, markup, annotated - select for diffs
Sun May 18 20:57:56 2008 UTC (6 years, 4 months ago) by nth
Branches: MAIN
CVS tags: HEAD, DragonFly_RELEASE_2_0_Slip, DragonFly_RELEASE_2_0, DragonFly_Preview
Diff to: previous 1.26: preferred, unified
Changes since revision 1.26: +0 -27 lines
Remove obsolete userland lwkt.

Approved-By: dillon@
(http://leaf.dragonflybsd.org/mailarchive/kernel/2008-05/msg00053.html)

Revision 1.26: download - view: text, markup, annotated - select for diffs
Thu May 1 09:37:48 2008 UTC (6 years, 5 months ago) by sephe
Branches: MAIN
Diff to: previous 1.25: preferred, unified
Changes since revision 1.25: +9 -0 lines
ktr the end of various ipiq sending operation.

Revision 1.25: download - view: text, markup, annotated - select for diffs
Thu May 1 02:11:39 2008 UTC (6 years, 5 months ago) by sephe
Branches: MAIN
Diff to: previous 1.24: preferred, unified
Changes since revision 1.24: +18 -6 lines
ktr cpu_send_ipiq

Revision 1.24: download - view: text, markup, annotated - select for diffs
Tue Apr 29 14:26:09 2008 UTC (6 years, 5 months ago) by sephe
Branches: MAIN
Diff to: previous 1.23: preferred, unified
Changes since revision 1.23: +1 -1 lines
Three int arguments are used in IPIQ_STRING

Revision 1.23: download - view: text, markup, annotated - select for diffs
Sun Nov 18 09:53:19 2007 UTC (6 years, 10 months ago) by sephe
Branches: MAIN
CVS tags: DragonFly_RELEASE_1_12_Slip, DragonFly_RELEASE_1_12
Diff to: previous 1.22: preferred, unified
Changes since revision 1.22: +27 -0 lines
Add a new light weight function to synchronize IPI queues on other CPUs by
broadcasting a NOP IPI to other CPUs; this is used be make sure that all
IPIs before the NOP one are processed.

Use this new function to fix a possible race between kfree() and
malloc_uninit():
kfree() may be in transitting state when malloc_uninit() is running.

Ideas-from: dillon@
Reviewed-by: dillon@

Revision 1.22: download - view: text, markup, annotated - select for diffs
Thu Jun 7 20:35:54 2007 UTC (7 years, 3 months ago) by dillon
Branches: MAIN
CVS tags: DragonFly_RELEASE_1_10_Slip, DragonFly_RELEASE_1_10
Diff to: previous 1.21: preferred, unified
Changes since revision 1.21: +6 -0 lines
Add KTR logging for SMP page table syncing ops.

Revision 1.21: download - view: text, markup, annotated - select for diffs
Mon Jan 22 19:37:04 2007 UTC (7 years, 8 months ago) by corecode
Branches: MAIN
CVS tags: DragonFly_RELEASE_1_8_Slip, DragonFly_RELEASE_1_8
Diff to: previous 1.20: preferred, unified
Changes since revision 1.20: +3 -3 lines
Pass structs by reference if you expect the callee to modify them.

This fixes kernel boot with gcc41.  The gpfault people were seeing comes from
vm86_bioscall() in init386().  The cause is that the assembler code passes the
struct vm86frame by value, i.e. simply creating it on the stack.  This worked
up to gcc34, but gcc41 now optimizes stores to unused memory locations away,
whis is allowed per the standards.  This led to an uninitialized stack frame
which in turn panicked the box.

Oooohh...-please-commit-by: dillon@

Revision 1.20: download - view: text, markup, annotated - select for diffs
Wed Dec 27 06:51:47 2006 UTC (7 years, 9 months ago) by dillon
Branches: MAIN
Diff to: previous 1.19: preferred, unified
Changes since revision 1.19: +0 -2 lines
Remove unused defines.

Revision 1.19: download - view: text, markup, annotated - select for diffs
Sat Dec 23 00:35:04 2006 UTC (7 years, 9 months ago) by swildner
Branches: MAIN
Diff to: previous 1.18: preferred, unified
Changes since revision 1.18: +1 -1 lines
Rename printf -> kprintf in sys/ and add some defines where necessary
(files which are used in userland, too).

Revision 1.18: download - view: text, markup, annotated - select for diffs
Tue Nov 7 18:50:06 2006 UTC (7 years, 10 months ago) by dillon
Branches: MAIN
Diff to: previous 1.17: preferred, unified
Changes since revision 1.17: +0 -1 lines
Remove system dependancies on <machine/ipl.h>.  Only architecture files
need it now.  SWI_* defines moved from the MD <machine/ipl.h> to the
MI <sys/interrupt.h> directory.

Revision 1.17: download - view: text, markup, annotated - select for diffs
Wed Oct 26 10:46:45 2005 UTC (8 years, 11 months ago) by sephe
Branches: MAIN
CVS tags: DragonFly_RELEASE_1_6_Slip, DragonFly_RELEASE_1_6, DragonFly_RELEASE_1_4_Slip, DragonFly_RELEASE_1_4
Diff to: previous 1.16: preferred, unified
Changes since revision 1.16: +8 -8 lines
unbreak kernel building by
- add arg2 to logipiq()
- adjust caller of logipiq()

Revision 1.16: download - view: text, markup, annotated - select for diffs
Tue Oct 25 17:26:54 2005 UTC (8 years, 11 months ago) by dillon
Branches: MAIN
Diff to: previous 1.15: preferred, unified
Changes since revision 1.15: +40 -49 lines
Remove the dummy IPI messaging routines for UP builds and properly
conditionalize the use of IPI messages in various core kernel modules.

Change the callback from func(arg, frameptr) to func(arg1, arg2, frameptr),
where the new argument (arg2) is an integer supplied by the originator.

Create wrappers for simpler versions of the callback: func(arg1), and
func(arg1, arg2) (for the moment we presume that GCC will generate code
for the full-sized three-argument callback which is compatible with one
and two-argument function pointers).

This extension to the IPI messaging code is needed to properly implement
MP-safe tsleep/wakeup code.  Although the extra argument is superfluous in
most cases, the overhead of doing an IPI is such that there should be no
noticeable impact on performance.

Revision 1.8.2.1: download - view: text, markup, annotated - select for diffs
Thu Jul 28 05:53:12 2005 UTC (9 years, 2 months ago) by dillon
Branches: DragonFly_RELEASE_1_2
CVS tags: DragonFly_RELEASE_1_2_Slip
Diff to: previous 1.8: preferred, unified; next MAIN 1.9: preferred, unified
Changes since revision 1.8: +5 -1 lines
MFC 1.15, fix a critical bug in the IPI messaging code, effecting SMP
systems.

Revision 1.15: download - view: text, markup, annotated - select for diffs
Sat Jul 23 07:17:42 2005 UTC (9 years, 2 months ago) by dillon
Branches: MAIN
Diff to: previous 1.14: preferred, unified
Changes since revision 1.14: +5 -1 lines
Fix a critical bug in the IPI messaging code, effecting SMP systems.  In
order to avoid placing a load fence in the FIFO scanning loop the FIFO write
index is cached and the code then loops the read index until it reaches
the cached write index, rather then real-time write index.  However, if a
FIFO full condition occurs during the callback AND additional IPIQ messages
are queued to the current cpu by a remote cpu at the same time, a recursive
call to lwkt_process_ipiq*() can occur and advance the read index past
the cached write index of the parent processing loop.

An exact comparison against the cached write index was being used which
resulted in the parent processing loop blowing past the actual write
index and re-executing stale IPI messages.  Fix the comparison.

The nature of this bug combined with other bugs in the token code and the
sockbuf code (which were causing crashes far more often) made this a
particularly nasty problem to find, with it taking upwards of a week to
generate a crash and the crash occuring at the worst place imagineable
(a hard IPI interrupt) and doing terrible things (re-executing a stale
IPI message).  It took KTR logging on both the sending and receiving
side of the IPI code to nail the problem.

Very special thanks to Peter Avalos and David rhodus for their debugging
help.  And, most especially, David Rhodus for helping track this down over
the last *THREE* months.

Reported-by: David Rhodus, Peter Avalos, YONETANI Tomokazu, Tomaz Borstnar
Special-thanks-to: David Rhodus and Peter Avalos

Revision 1.14: download - view: text, markup, annotated - select for diffs
Wed Jul 20 20:14:33 2005 UTC (9 years, 2 months ago) by dillon
Branches: MAIN
Diff to: previous 1.13: preferred, unified
Changes since revision 1.13: +25 -0 lines
Add some conditionalized debugging 'PANIC_DEBUG', to allow us to panic a
busy system from the worst place imagineable, the IPIQ loop.

Revision 1.13: download - view: text, markup, annotated - select for diffs
Tue Jun 21 05:25:17 2005 UTC (9 years, 3 months ago) by dillon
Branches: MAIN
Diff to: previous 1.12: preferred, unified
Changes since revision 1.12: +38 -10 lines
Add KTR support to the IPIQ code.

Revision 1.12: download - view: text, markup, annotated - select for diffs
Tue Jun 21 05:03:12 2005 UTC (9 years, 3 months ago) by dillon
Branches: MAIN
Diff to: previous 1.11: preferred, unified
Changes since revision 1.11: +11 -3 lines
Add additional sanity checks to IPIQ processing, do some cleanups,
and make minor changes to the function dispatch.

Revision 1.11: download - view: text, markup, annotated - select for diffs
Fri Jun 3 23:57:32 2005 UTC (9 years, 4 months ago) by dillon
Branches: MAIN
Diff to: previous 1.10: preferred, unified
Changes since revision 1.10: +21 -6 lines
Replace cpu_mb1() and cpu_mb2() with cpu_mfence(), cpu_lfence(), cpu_sfence(),
and cpu_ccfence().  These provide memory and compiler fences to guarentee
read/write ordering of memory and to prevent the compiler itself from
generating reordered code in particular cases.

Most of the cases where care must be taken are tail-chasing FIFO cases,
especially in the IPI messaging code.

At the moment on SMP boxes we use a locked bus cycle on (%%esp) for the
lfence and mfence.  The only other choices are to use a cpuid instruction
or one of the {S,L,M}FENCE instructions.  cpuid is usually horrible, and
the *FENCE instructions do not exist on older cpus.

Linux seems to use the locked bus cycle or *FENCE instruction method.

Although I have no conclusive evidence, a number of crash dumps provided
by David Rhodus has led me to believe that speculative reads by modern cpus,
in particular in HTT situations with Intel cpus, can survive many more
instructions then previously believed.  The only safe solution is to use
an instruction sequence which guarentees proper operation.

I would prefer to avoid the use of a locked bus cycle but at least in the
IPIQ case (the most common case we have to worry about), only one locked
bus cycle is required and then the entire IPIQ can be processed without
further locked cycles.

Revision 1.10: download - view: text, markup, annotated - select for diffs
Mon Apr 18 01:02:58 2005 UTC (9 years, 5 months ago) by dillon
Branches: MAIN
CVS tags: DragonFly_Stable
Diff to: previous 1.9: preferred, unified
Changes since revision 1.9: +2 -1 lines
Fix a bug in the last commit.  A FIFO calculation was incorrect (3/2 instead
of 2/3), resulting in overflows during heavy token handling activity.

Reported-by: =?ISO-8859-1?Q?Stefan_Kr=FCger?= <skrueger@meinberlikomm.de>

Revision 1.9: download - view: text, markup, annotated - select for diffs
Wed Apr 13 04:00:50 2005 UTC (9 years, 5 months ago) by dillon
Branches: MAIN
Diff to: previous 1.8: preferred, unified
Changes since revision 1.8: +126 -27 lines
Optimize lwkt_send_ipiq() - the IPI based inter-cpu messaging routine.

* Add a passive version which does not initiate any actual hardware IPI.
  The message will be handled the next time the target cpu polls the
  queue (on each tick typically).  Adjust the free() path to use this
  version when freeing memory owned by another cpu.

* Add an interlock to avoid reissuing and unnecessarily stalling on
  the hardware IPI if a prior hardware IPI to the target cpu has not
  yet completed processing.

  This feature theoretically means that two cpus can tightly couple a
  large number of pipelined messages with only a single actual IPI being
  sent.

* Reorganize the hystersis points in the IPIQ FIFOs.

* Change a token livelock warning into a panic if it occurs 10 times in
  a row.

* Add a call to lwkt_process_ipiq() just after the AP startup code enables
  a cpu, to process any messages that might have built up during startup.
  There shouldn't be any, but this may avoid surprises later.

Revision 1.8: download - view: text, markup, annotated - select for diffs
Fri Jul 16 05:51:10 2004 UTC (10 years, 2 months ago) by dillon
Branches: MAIN
CVS tags: DragonFly_Snap29Sep2004, DragonFly_Snap13Sep2004
Branch point for: DragonFly_RELEASE_1_2
Diff to: previous 1.7: preferred, unified
Changes since revision 1.7: +25 -17 lines
Update all my personal copyrights to the Dragonfly Standard Copyright.

Revision 1.7: download - view: text, markup, annotated - select for diffs
Sun Jul 4 22:44:27 2004 UTC (10 years, 2 months ago) by eirikn
Branches: MAIN
CVS tags: DragonFly_1_0_REL, DragonFly_1_0A_REL
Diff to: previous 1.6: preferred, unified
Changes since revision 1.6: +1 -1 lines
Rearrange the machine/cpufunc.h header and add it where needed to make libcaps
build again.

Revision 1.6: download - view: text, markup, annotated - select for diffs
Thu Apr 1 17:40:57 2004 UTC (10 years, 6 months ago) by dillon
Branches: MAIN
CVS tags: DragonFly_1_0_RC1
Diff to: previous 1.5: preferred, unified
Changes since revision 1.5: +3 -3 lines
Implement a convenient gd_cpumask so we don't have to do 1 << gd->gd_cpuid
all the time.

Revision 1.5: download - view: text, markup, annotated - select for diffs
Mon Mar 8 03:03:54 2004 UTC (10 years, 6 months ago) by dillon
Branches: MAIN
Diff to: previous 1.4: preferred, unified
Changes since revision 1.4: +2 -0 lines
libcaps now compiles ipiq and token in userland, make those files compile
properly in userland.

Revision 1.4: download - view: text, markup, annotated - select for diffs
Mon Mar 1 06:33:17 2004 UTC (10 years, 7 months ago) by dillon
Branches: MAIN
Diff to: previous 1.3: preferred, unified
Changes since revision 1.3: +56 -1 lines
Newtoken commit.  Change the token implementation as follows:  (1) Obtaining
a token no longer enters a critical section.  (2) tokens can be held through
schedular switches and blocking conditions and are effectively released and
reacquired on resume.  Thus tokens serialize access only while the thread
is actually running.  Serialization is not broken by preemptive interrupts.
That is, interrupt threads which preempt do no release the preempted thread's
tokens.  (3) Unlike spl's, tokens will interlock w/ interrupt threads on
the same or on a different cpu.

The vnode interlock code has been rewritten and the API has changed.  The
mountlist vnode scanning code has been consolidated and all known races have
been fixed.  The vnode interlock is now a pool token.

The code that frees unreferenced vnodes whos last VM page has been freed has
been moved out of the low level vm_page_free() code and moved to the
periodic filesystem sycer code in vfs_msycn().

The SMP startup code and the IPI code has been cleaned up considerably.
Certain early token interactions on AP cpus have been moved to the BSP.

The LWKT rwlock API has been cleaned up and turned on.

Major testing by: David Rhodus

Revision 1.3: download - view: text, markup, annotated - select for diffs
Tue Feb 17 19:38:49 2004 UTC (10 years, 7 months ago) by dillon
Branches: MAIN
Diff to: previous 1.2: preferred, unified
Changes since revision 1.2: +89 -46 lines
Introduce an MI cpu synchronization API, redo the SMP AP startup code,
and start cleaning up deprecated IPI and clock code.  Add a MMU/TLB page
table invalidation API (pmap_inval.c) which properly synchronizes page
table changes with other cpus in SMP environments.

    * removed (unused) gd_cpu_lockid
    * remove confusing invltlb() and friends, normalize use of cpu_invltlb()
      and smp_invltlb().
    * redo the SMP AP startup code to make the system work better in
      situations where all APs do not startup.
    * add memory barrier API, cpu_mb1() and cpu_mb2().
    * remove (obsolete, no longer used) old IPI hard and stat clock forwarding
      code.
    * add a cpu synchronization API which is capable of handling multiple
      simultanious requests without deadlocking or livelocking.
    * major changes to the PMAP code to use the new invalidation API.
    * remove (unused) all_procs_ipi() and self_ipi().
    * only use all_but_self_ipi() if it is known that all AP's started up,
      otherwise use a mask.
    * remove (obsolete, no longer usde) BETTER_CLOCK code
    * remove (obsolete, no longer used) Xcpucheckstate IPI code

Testing-by: David Rhodus and others

Revision 1.2: download - view: text, markup, annotated - select for diffs
Sun Feb 15 05:15:25 2004 UTC (10 years, 7 months ago) by dillon
Branches: MAIN
Diff to: previous 1.1: preferred, unified
Changes since revision 1.1: +88 -16 lines
Cleanup and augment the cpu synchronization API a bit.  Embed the maxcount
in the structure rather then returning it and requiring it to be passed
again, and document the procedures a bit more.

Revision 1.1: download - view: text, markup, annotated - select for diffs
Sun Feb 15 02:14:41 2004 UTC (10 years, 7 months ago) by dillon
Branches: MAIN
Split the IPIQ messaging out of lwkt_thread.c and move it to its own file,
lwkt_ipiq.c.

Add a MI synchronous cpu rendezvous API lwkt_cpusync_*().  This API allows the
kernel to synchronize an operation across any number of cpus.  Multiple cpus
can initiate synchronization operations simultaniously without creating a
deadlock.  The API utilizes the IPI messaging core and guarentees that
other synchronization and IPI messaging operations will continue to work
during any given synchronization op.  The API is a spin-blocking API, meaning
that it will not switch threads and can be used by mainline code, interrupts,
and other sensitive code.

This API is intended to replace smp_rendezvous(), Xcpustop, and other
hardwired IPI ops.  It will also be used to fix our TLB shootdown code.

As of this commit the API has not yet been connected to anything and has
been tested only a little.

Diff request

This form allows you to request diffs between any two revisions of a file. You may select a symbolic revision name using the selection box or you may type in a numeric name using the type-in text box.

Log view options