diff options
| -rw-r--r-- | ceph.md | 217 |
1 files changed, 217 insertions, 0 deletions
@@ -0,0 +1,217 @@ +# Ceph Installation + +--- +## Intro +Ceph is used to build a storage system accross all machines + +## Architecture +We consider the following architecture + + TODO: add schema (4 machines: ceph-admin, 3 ceph-nodes: opensteak9{2,3,4}) + +Networks: +``` +192.168.0.0/24 is the cluster network (use for storage) +192.168.1.0/24 is the management network (use for admin task) +``` + + +## Ceph-admin machine preparation + +This is done on an Ubuntu 14.04 64b server + +### Install ceph-deploy +```bash +wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add - +echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list +sudo apt-get update && sudo apt-get install ceph-deploy +``` + +### Install ntp +```bash +sudo apt-get install ntp +sudo service ntp restart +``` + +### Create a ceph user on each node (ceph-admin included) +```bash +sudo useradd -d /home/ceph -m ceph +sudo passwd ceph +``` + +Add sudo rights: +```bash +echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph +sudo chmod 0440 /etc/sudoers.d/ceph +``` + +* *Note: if you think this can be a security treat, remove the ceph user from sudoers after installation is complete* + +* *Note 2: the ceph documentation ask for this user: http://ceph.com/docs/master/rados/deployment/preflight-checklist/?highlight=sudoers* + + +### Add each node in hosts file (ceph-admin included) +```bash +sudo bash -c ' cat << EOF >> /etc/hosts +192.168.1.200 ceph-admin +192.168.1.92 opensteak92 +192.168.1.93 opensteak93 +192.168.1.94 opensteak94 +EOF' +``` + +### Create and copy a passwordless ssh key to each node +```bash +ssh-keygen +ssh-copy-id ceph@ceph-admin +ssh-copy-id ceph@opensteak92 +ssh-copy-id ceph@opensteak93 +ssh-copy-id ceph@opensteak94 +``` + +### Create a .ssh/config file to connect automatically +```bash +cat << EOF >> .ssh/config +Host ceph-admin + Hostname ceph-admin + User ceph +Host opensteak92 + Hostname opensteak92 + User ceph +Host opensteak93 + Hostname opensteak93 + User ceph +Host opensteak94 + Hostname opensteak94 + User ceph +EOF +``` + +## Ceph storage cluster +All these commands must be run inside the ceph-admin machine as a regular user + +### Prepare folder +```bash +mkdir ceph-cluster +cd ceph-cluster/ +``` + +### Deploy initial monitor on first node +```bash +ceph-deploy new opensteak92 +``` + +### Configure ceph +We set default pool size to 2 and public/cluster networks: + +```bash +cat << EOF >> ceph.conf +osd pool default size = 2 +public network = 192.168.1.0/24 +cluster network = 192.168.0.0/24 +EOF +``` + +### Install ceph in all nodes +```bash +ceph-deploy --username ceph install ceph-admin opensteak92 opensteak93 opensteak94 +``` + +### Create initial monitor and gather the keys +```bash +ceph-deploy --username ceph mon create-initial +``` + +### Create and add OSD +We will use hard disk (/dev/sdb) for storage: http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-osd/ + +```bash +ceph-deploy --username ceph osd create opensteak93:sdb +ceph-deploy --username ceph osd create opensteak94:sdb +``` + +### Prepare all nodes to administer the cluster +Prepare all nodes with a ceph.conf and ceph.client.admin.keyring keyring so that it can administer the cluster: + +```bash +ceph-deploy admin ceph-admin opensteak92 opensteak93 opensteak94 +sudo chmod +r /etc/ceph/ceph.client.admin.keyring +``` + +### Add a metadata server in first node +```bash +ceph-deploy--username ceph mds create opensteak92 +``` + +## Extend +### Extend the OSD pool +We decided to extend OSD pool by adding the first node as well: + +```bash +ceph-deploy --username ceph osd create opensteak92:sdb +``` + +### Extend the monitors +In the same spirit, extend the monitor by adding the two last nodes and check the status +```bash +ceph-deploy --username ceph mon create opensteak93 opensteak94 +ceph quorum_status --format json-pretty +``` + +## Check status +```bash +ceph health +``` + +## Create a file system +Check osd pools: +```bash +ceph osd lspools +``` + +I you don't have data and metadata pools, create it: +```bash +ceph osd pool create cephfs_data 64 +ceph osd pool create cephfs_metadata 64 +``` + +Then enable filesystem on the cephfs_data pool: +```bash +ceph fs new cephfs cephfs_metadata cephfs_data +``` + +And check again: +```bash +ceph osd lspools +``` + +Should produce: +```bash +0 rbd,1 cephfs_data,2 cephfs_metadata, +``` + +You can check as well with: +```bash +$ ceph fs ls +name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] + +$ ceph mds stat +e5: 1/1/1 up {0=opensteak92=up:active} +``` + +## Mount file system +For each node you want to mount ceph in **/mnt/cephfs/**, run: +```bash +ssh opensteak9x "cat /etc/ceph/ceph.client.admin.keyring |grep key|awk '{print \$3}'|sudo tee /etc/ceph/ceph.client.admin.key" + +ssh opensteak9x "sudo mkdir /mnt/cephfs" + +ssh opensteak9x "echo '192.168.1.92:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/ceph.client.admin.key,noatime 0 2' | sudo tee --append /etc/fstab && sudo mount /mnt/cephfs" +``` + +This will add a line in fstab so the file system will automatically be mounted on boot. + +## TODO + +* create a python/bash script that will install & check that the cluster is well configured (do all of this automatically) +* create a conf file that will be used by the above script to describe the architecture? |