DragonFly commits List (threaded) for 2005-09
cvs commit: src/sys/sys journal.h src/sbin/jscan Makefile dump_debug.c dump_mirror.c dump_output.c dump_record.c jfile.c jscan.8 jscan.c jscan.h jstream.c subs.c
dillon 2005/09/05 23:42:44 PDT
DragonFly src repository
sbin/jscan Makefile dump_debug.c dump_mirror.c
jfile.c jscan.8 jscan.c jscan.h jstream.c
sbin/jscan dump_output.c dump_record.c
Rework and expand the algorithms in JSCAN, part 1/2. Implement a new
infrastructure in the userland jscan program to create a framework for
supporting the following:
* Pipe-throughs (without storing data in a buffering file).
* File-based buffering
* Streamed replication of journaled data.
* Sequenced output files limited to a specific size (to match against
* Transaction id tracking to support journal restarts and resynchronization.
This is needed to make off-site and networked backups reliable.
* Decoupled journal operations. e.g. (once finished) we will be able to run
a jscan in realtime to record journal data, and another one in batch to
take that data and use it to maintain a mirror or to send it to another
machine in a restartable and resynchronizable manner.
* Infrastructure to be able to scan a journal forwards or backwards by
an arbitrary number of raw records and maintain a mirror in both
the forwards (redo) and backwards (undo) direction based on that ability.
* Infrastructure that will aid in the preferred backup methodology of
maintaining a real time mirror and UNDO/REDO journal on a 100% full
backup partition such that one is able to free space on the partition
simply by removing the oldest journaling file.
* Infrastructure that will make it easier to solve the overlapping
meta-transaction problem illustrated below:
current synchronization point
[ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ][ ] <= Raw records
A <= meta transaction A
B B B B <= meta transaction B
C C C <= meta transaction C
D D <= meta transaction D
E E <= meta transaction E
As you can see, the point at which a journal is 'broken' could
be right smack in the middle of several meta transactions at
once, due to the fact that meta transactions are laid out as
their records can be generated and multiple processes could be
modifying the filesystem at the same time.
If we are attempting to mirror this data, only transactions A and E
will have been completed and issued to the mirror. If the journal
is broken at the sychnronization point and then restarted, in
order to pick up where we left off we have to scan existing
journal records all the way back to the first 'B' in order to
recover transactions B and C, then continue recording the journal
from the synchronization point until those transactions are complete
and can be issued to the mirror. The difficulty here is that we
have to carefully track exactly what needs to be synchronized.
In recovering transaction 'B' and 'C' we do not want to reissue
transaction E, for example, which had already been completed prior to
The problem exists in both the forwards and reverse direction, and
we need to be able to go in both directions to control the 'snapshot'
view the mirror is presenting to us.
NOTE: this is just part 1/2. This code and the new options are
NON WORKING at the moment.
Revision Changes Path
1.8 +2 -0 src/sys/sys/journal.h
1.3 +2 -1 src/sbin/jscan/Makefile
1.4 +5 -3 src/sbin/jscan/dump_debug.c
1.5 +2 -68 src/sbin/jscan/dump_mirror.c
1.6 +537 -75 src/sbin/jscan/jfile.c
1.6 +224 -11 src/sbin/jscan/jscan.8
1.5 +327 -48 src/sbin/jscan/jscan.c
1.6 +61 -28 src/sbin/jscan/jscan.h
1.5 +19 -174 src/sbin/jscan/jstream.c
1.7 +18 -0 src/sbin/jscan/subs.c