DragonFly BSD
DragonFly users List (threaded) for 2011-04
[Date Prev][Date Next]  [Thread Prev][Thread Next]  [Date Index][Thread Index]

Re: Hammer deduplication needs for RAM size

From: Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>
Date: Fri, 22 Apr 2011 13:12:42 -0700 (PDT)

:Hi all,
:can someone compare/describe need of RAM size by deduplication in
:Hammer? There's something interesting about deduplication in ZFS

    The ram is basically needed to store matching CRCs.  The on-line dedup
    uses a limited fixed-sized hash table to remember CRCs, designed to
    match recently read data with future written data (e.g. 'cp').

    The off-line dedup (when you run 'hammer dedup ...' or
    'hammer dedup-simulate ...' will keep track of ALL data CRCs when
    it scans the filesystem B-Tree.  It will happily use lots of swap
    space if it comes down to it, which is probably a bug.  But that's
    how it works now.

    Actual file data is not persistently cached in memory.  It is read only
    when the dedup locates a potential match and sticks around in a limited
    cache before getting thrown away, and will be re-read as needed.

					Matthew Dillon 

[Date Prev][Date Next]  [Thread Prev][Thread Next]  [Date Index][Thread Index]