DragonFly BSD
DragonFly users List (threaded) for 2011-07
[Date Prev][Date Next]  [Thread Prev][Thread Next]  [Date Index][Thread Index]

Re: Real World DragonFlyBSD Hammer DeDup figures from HiFX - Reclaiming more than 1/4th ( 30% ) Disk Space from an Almost Full Drive

From: Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>
Date: Fri, 22 Jul 2011 09:55:37 -0700 (PDT)

:> >> take?
:> >>
:> >
:> > I ran them one by one. at my own pace but the biggest two
:> > simultaneously did not take more than 2 hrs.
:> > So I guess 2-3 hrs would be a nice approximation :-)
:> My experiences were different on a file system containing a lot of data  
:> (>2TB).
:> I didn't try dedup itself but a dedup-simulate already ran for more than  
:> two days (consuming a lot of memory in the process) before I finally  
:> cancelled it.
:	Most odd - I just tried a dedup-simulate on a 2TB filesystem with
:about 840GB used, it finished in about 30 seconds and reported a ratio of
:1.01 (dedup has been running automatically every night on this FS).
:Steve O'Hara-Smith                          |   Directable Mirror Arrays

    I think this could be a case of the more CRC collisions we have, the
    more I/O dedup (or dedup-simulate) needs to issue to determine if the
    collision is an actual dup or if it's just a CRC collision and the
    data is different.

    The memory use can be bounded with some additional work on the software,
    if someone wants to have a go at it.  Basically the way you limit memory
    use is by dynamically limiting the CRC range that you observe in a pass.
    As you reach a self-imposed memory limit you reduce the CRC range and
    throw away out-of-range records.  Once the pass is done you start a new
    pass with the remaining range.  Rinse, repeat until the whole thing is

    That would make it possible to run de-dup with bounded memory.  However,
    the extra I/O's required to verify duplicate data cannot be avoided.

					Matthew Dillon 

[Date Prev][Date Next]  [Thread Prev][Thread Next]  [Date Index][Thread Index]