diff options
author | 2018-01-04 13:43:33 +0800 | |
---|---|---|
committer | 2018-01-05 11:59:39 +0800 | |
commit | 812ff6ca9fcd3e629e49d4328905f33eee8ca3f5 (patch) | |
tree | 04ece7b4da00d9d2f98093774594f4057ae561d4 /src/ceph/doc/cephfs/dirfrags.rst | |
parent | 15280273faafb77777eab341909a3f495cf248d9 (diff) |
initial code repo
This patch creates initial code repo.
For ceph, luminous stable release will be used for base code,
and next changes and optimization for ceph will be added to it.
For opensds, currently any changes can be upstreamed into original
opensds repo (https://github.com/opensds/opensds), and so stor4nfv
will directly clone opensds code to deploy stor4nfv environment.
And the scripts for deployment based on ceph and opensds will be
put into 'ci' directory.
Change-Id: I46a32218884c75dda2936337604ff03c554648e4
Signed-off-by: Qiaowei Ren <qiaowei.ren@intel.com>
Diffstat (limited to 'src/ceph/doc/cephfs/dirfrags.rst')
-rw-r--r-- | src/ceph/doc/cephfs/dirfrags.rst | 100 |
1 files changed, 100 insertions, 0 deletions
diff --git a/src/ceph/doc/cephfs/dirfrags.rst b/src/ceph/doc/cephfs/dirfrags.rst new file mode 100644 index 0000000..717553f --- /dev/null +++ b/src/ceph/doc/cephfs/dirfrags.rst @@ -0,0 +1,100 @@ + +=================================== +Configuring Directory fragmentation +=================================== + +In CephFS, directories are *fragmented* when they become very large +or very busy. This splits up the metadata so that it can be shared +between multiple MDS daemons, and between multiple objects in the +metadata pool. + +In normal operation, directory fragmentation is invisbible to +users and administrators, and all the configuration settings mentioned +here should be left at their default values. + +While directory fragmentation enables CephFS to handle very large +numbers of entries in a single directory, application programmers should +remain conservative about creating very large directories, as they still +have a resource cost in situations such as a CephFS client listing +the directory, where all the fragments must be loaded at once. + +All directories are initially created as a single fragment. This fragment +may be *split* to divide up the directory into more fragments, and these +fragments may be *merged* to reduce the number of fragments in the directory. + +Splitting and merging +===================== + +An MDS will only consider doing splits and merges if the ``mds_bal_frag`` +setting is true in the MDS's configuration file, and the allow_dirfrags +setting is true in the filesystem map (set on the mons). These settings +are both true by default since the *Luminous* (12.2.x) release of Ceph. + +When an MDS identifies a directory fragment to be split, it does not +do the split immediately. Because splitting interrupts metadata IO, +a short delay is used to allow short bursts of client IO to complete +before the split begins. This delay is configured with +``mds_bal_fragment_interval``, which defaults to 5 seconds. + +When the split is done, the directory fragment is broken up into +a power of two number of new fragments. The number of new +fragments is given by two to the power ``mds_bal_split_bits``, i.e. +if ``mds_bal_split_bits`` is 2, then four new fragments will be +created. The default setting is 3, i.e. splits create 8 new fragments. + +The criteria for initiating a split or a merge are described in the +following sections. + +Size thresholds +=============== + +A directory fragment is elegible for splitting when its size exceeds +``mds_bal_split_size`` (default 10000). Ordinarily this split is +delayed by ``mds_bal_fragment_interval``, but if the fragment size +exceeds a factor of ``mds_bal_fragment_fast_factor`` the split size, +the split will happen immediately (holding up any client metadata +IO on the directory). + +``mds_bal_fragment_size_max`` is the hard limit on the size of +directory fragments. If it is reached, clients will receive +ENOSPC errors if they try to create files in the fragment. On +a properly configured system, this limit should never be reached on +ordinary directories, as they will have split long before. By default, +this is set to 10 times the split size, giving a dirfrag size limit of +100000. Increasing this limit may lead to oversized directory fragment +objects in the metadata pool, which the OSDs may not be able to handle. + +A directory fragment is elegible for merging when its size is less +than ``mds_bal_merge_size``. There is no merge equivalent of the +"fast splitting" explained above: fast splitting exists to avoid +creating oversized directory fragments, there is no equivalent issue +to avoid when merging. The default merge size is 50. + +Activity thresholds +=================== + +In addition to splitting fragments based +on their size, the MDS may split directory fragments if their +activity exceeds a threshold. + +The MDS maintains separate time-decaying load counters for read and write +operations on directory fragments. The decaying load counters have an +exponential decay based on the ``mds_decay_halflife`` setting. + +On writes, the write counter is +incremented, and compared with ``mds_bal_split_wr``, triggering a +split if the threshold is exceeded. Write operations include metadata IO +such as renames, unlinks and creations. + +The ``mds_bal_split_rd`` threshold is applied based on the read operation +load counter, which tracks readdir operations. + +By the default, the read threshold is 25000 and the write threshold is +10000, i.e. 2.5x as many reads as writes would be required to trigger +a split. + +After fragments are split due to the activity thresholds, they are only +merged based on the size threshold (``mds_bal_merge_size``), so +a spike in activity may cause a directory to stay fragmented +forever unless some entries are unlinked. + |