path: root/ceph.md

# Ceph Installation

---
## Intro
Ceph is used to build a storage system accross all machines

## Architecture
We consider the following architecture

    TODO: add schema (4 machines: ceph-admin, 3 ceph-nodes: opensteak9{2,3,4})

Networks:
```
192.168.0.0/24 is the cluster network (use for storage)
192.168.1.0/24 is the management network (use for admin task)
```


## Ceph-admin machine preparation

This is done on an Ubuntu 14.04 64b server

### Install ceph-deploy
```bash
wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
echo deb http://ceph.com/debian-firefly/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt-get update && sudo apt-get install ceph-deploy
```

### Install ntp
```bash
sudo apt-get install ntp
sudo service ntp restart
```

### Create a ceph user on each node (ceph-admin included)
```bash
sudo useradd -d /home/ceph -m ceph
sudo passwd ceph
```

Add sudo rights:
```bash
echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
sudo chmod 0440 /etc/sudoers.d/ceph
```

* *Note: if you think this can be a security treat, remove the ceph user from sudoers after installation is complete*

* *Note 2: the ceph documentation ask for this user: http://ceph.com/docs/master/rados/deployment/preflight-checklist/?highlight=sudoers*


### Add each node in hosts file (ceph-admin included)
```bash
sudo bash -c ' cat << EOF >> /etc/hosts
192.168.1.200 ceph-admin
192.168.1.92 opensteak92
192.168.1.93 opensteak93
192.168.1.94 opensteak94
EOF'
```

### Create and copy a passwordless ssh key to each node
```bash
ssh-keygen
ssh-copy-id ceph@ceph-admin
ssh-copy-id ceph@opensteak92
ssh-copy-id ceph@opensteak93
ssh-copy-id ceph@opensteak94
```

### Create a .ssh/config file to connect automatically
```bash
cat << EOF >> .ssh/config
Host ceph-admin
  Hostname ceph-admin
  User ceph
Host opensteak92
  Hostname opensteak92
  User ceph
Host opensteak93
  Hostname opensteak93
  User ceph
Host opensteak94
  Hostname opensteak94
  User ceph
EOF
```

## Ceph storage cluster
All these commands must be run inside the ceph-admin machine as a regular user

### Prepare folder
```bash
mkdir ceph-cluster
cd ceph-cluster/
```

### Deploy initial monitor on first node
```bash
ceph-deploy new opensteak92
```

### Configure ceph
We set default pool size to 2 and public/cluster networks:

```bash
cat << EOF >> ceph.conf
osd pool default size = 2
public network = 192.168.1.0/24
cluster network = 192.168.0.0/24
EOF
```

### Install ceph in all nodes
```bash
ceph-deploy --username ceph install ceph-admin opensteak92 opensteak93 opensteak94
```

### Create initial monitor and gather the keys
```bash
ceph-deploy --username ceph mon create-initial
```

### Create and add OSD
We will use hard disk (/dev/sdb) for storage: http://docs.ceph.com/docs/master/rados/deployment/ceph-deploy-osd/

```bash
ceph-deploy --username ceph osd create opensteak93:sdb
ceph-deploy --username ceph osd create opensteak94:sdb
```

### Prepare all nodes to administer the cluster
Prepare all nodes with a ceph.conf and ceph.client.admin.keyring keyring so that it can administer the cluster:

```bash
ceph-deploy admin ceph-admin opensteak92 opensteak93 opensteak94
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
```

### Add a metadata server in first node
```bash
ceph-deploy--username ceph mds create opensteak92
```

## Extend
### Extend the OSD pool
We decided to extend OSD pool by adding the first node as well:

```bash
ceph-deploy --username ceph osd create opensteak92:sdb
```

### Extend the monitors
In the same spirit, extend the monitor by adding the two last nodes and check the status
```bash
ceph-deploy --username ceph mon create opensteak93 opensteak94
ceph quorum_status --format json-pretty
```

## Check status
```bash
ceph health
```

## Create a file system
Check osd pools:
```bash
ceph osd lspools
```

I you don't have data and metadata pools, create it:
```bash
ceph osd pool create cephfs_data 64
ceph osd pool create cephfs_metadata 64
```

Then enable filesystem on the cephfs_data pool:
```bash
ceph fs new cephfs cephfs_metadata cephfs_data
```

And check again:
```bash
ceph osd lspools
```

Should produce:
```bash
0 rbd,1 cephfs_data,2 cephfs_metadata,
```

You can check as well with:
```bash
$ ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

$ ceph mds stat
e5: 1/1/1 up {0=opensteak92=up:active}
```

## Mount file system
For each node you want to mount ceph in **/mnt/cephfs/**, run:
```bash
ssh opensteak9x "cat /etc/ceph/ceph.client.admin.keyring |grep key|awk '{print \$3}'|sudo tee /etc/ceph/ceph.client.admin.key"

ssh opensteak9x "sudo mkdir /mnt/cephfs"

ssh opensteak9x "echo '192.168.1.92:6789:/ /mnt/cephfs ceph name=admin,secretfile=/etc/ceph/ceph.client.admin.key,noatime 0 2' | sudo tee --append /etc/fstab && sudo mount /mnt/cephfs"
```

This will add a line in fstab so the file system will automatically be mounted on boot.

## TODO

* create a python/bash script that will install & check that the cluster is well configured (do all of this automatically)
* create a conf file that will be used by the above script to describe the architecture?