diff options
10 files changed, 416 insertions, 1 deletions
diff --git a/docs/testing/user/userguide/opnfv_yardstick_tc057.rst b/docs/testing/user/userguide/opnfv_yardstick_tc057.rst new file mode 100644 index 000000000..2a4ce40c0 --- /dev/null +++ b/docs/testing/user/userguide/opnfv_yardstick_tc057.rst @@ -0,0 +1,165 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Yin Kanglin and others. +.. 14_ykl@tongji.edu.cn + +************************************* +Yardstick Test Case Description TC057 +************************************* + ++-----------------------------------------------------------------------------+ +|OpenStack Controller Cluster Management Service High Availability | ++==============+==============================================================+ +|test case id | | ++--------------+--------------------------------------------------------------+ +|test purpose | This test case will verify the quorum configuration of the | +| | cluster manager(pacemaker) on controller nodes. When a | +| | controller node , which holds all active application | +| | resources, failed to communicate with other cluster nodes | +| | (via corosync), the test case will check whether the standby | +| | application resources will take place of those active | +| | application resources which should be regarded to be down in | +| | the cluster manager. | ++--------------+--------------------------------------------------------------+ +|test method | This test case kills the processes of cluster messaging | +| | service(corosync) on a selected controller node(the node | +| | holds the active application resources), then checks whether | +| | active application resources are switched to other | +| | controller nodes and whether the Openstack commands are OK. | ++--------------+--------------------------------------------------------------+ +|attackers | In this test case, an attacker called "kill-process" is | +| | needed. This attacker includes three parameters: | +| | 1) fault_type: which is used for finding the attacker's | +| | scripts. It should be always set to "kill-process" in this | +| | test case. | +| | 2) process_name: which is the process name of the load | +| | balance service. If there are multiple processes use the | +| | same name on the host, all of them are killed by this | +| | attacker. | +| | 3) host: which is the name of a control node being attacked. | +| | | +| | In this case, this process name should set to "corosync" , | +| | for example | +| | -fault_type: "kill-process" | +| | -process_name: "corosync" | +| | -host: node1 | ++--------------+--------------------------------------------------------------+ +|monitors | In this test case, a kind of monitor is needed: | +| | 1. the "openstack-cmd" monitor constantly request a specific | +| | Openstack command, which needs two parameters: | +| | 1) monitor_type: which is used for finding the monitor class | +| | and related scripts. It should be always set to | +| | "openstack-cmd" for this monitor. | +| | 2) command_name: which is the command name used for request | +| | | +| | In this case, the command_name of monitor1 should be services| +| | that are managed by the cluster manager. (Since rabbitmq and | +| | haproxy are managed by pacemaker, most Openstack Services | +| | can be used to check high availability in this case) | +| | | +| | (e.g.) | +| | monitor1: | +| | -monitor_type: "openstack-cmd" | +| | -command_name: "nova image-list" | +| | monitor2: | +| | -monitor_type: "openstack-cmd" | +| | -command_name: "neutron router-list" | +| | monitor3: | +| | -monitor_type: "openstack-cmd" | +| | -command_name: "heat stack-list" | +| | monitor4: | +| | -monitor_type: "openstack-cmd" | +| | -command_name: "cinder list" | +| | | ++--------------+--------------------------------------------------------------+ +|checkers | In this test case, a checker is needed, the checker will | +| | the status of application resources in pacemaker and the | +| | checker have three parameters: | +| | 1) checker_type: which is used for finding the result | +| | checker class and related scripts. In this case the checker | +| | type will be "pacemaker-check-resource" | +| | 2) resource_name: the application resource name | +| | 3) resource_status: the expected status of the resource | +| | 4) expectedValue: the expected value for the output of the | +| | checker script, in the case the expected value will be the | +| | identifier in the cluster manager | +| | 3) condition: whether the expected value is in the output of | +| | checker script or is totally same with the output. | +| | (note: pcs is required to installed on controller node in | +| | order to run this checker) | +| | | +| | (e.g.) | +| | checker1: | +| | -checker_type: "pacemaker-check-resource" | +| | -resource_name: "p_rabbitmq-server" | +| | -resource_status: "Stopped" | +| | -expectedValue: "node-1" | +| | -condition: "in" | +| | checker2: | +| | -checker_type: "pacemaker-check-resource" | +| | -resource_name: "p_rabbitmq-server" | +| | -resource_status: "Master" | +| | -expectedValue: "node-2" | +| | -condition: "in" | ++--------------+--------------------------------------------------------------+ +|metrics | In this test case, there are two metrics: | +| | 1)service_outage_time: which indicates the maximum outage | +| | time (seconds) of the specified Openstack command request. | ++--------------+--------------------------------------------------------------+ +|test tool | None. Self-developed. | ++--------------+--------------------------------------------------------------+ +|references | ETSI NFV REL001 | ++--------------+--------------------------------------------------------------+ +|configuration | This test case needs two configuration files: | +| | 1) test case file: opnfv_yardstick_tc057.yaml | +| | -Attackers: see above "attackers" description | +| | -Monitors: see above "monitors" description | +| | -Checkers: see above "checkers" description | +| | -Steps: the test case execution step, see "test sequence" | +| | description below | +| | | +| | 2)POD file: pod.yaml | +| | The POD configuration should record on pod.yaml first. | +| | the "host" item in this test case will use the node name in | +| | the pod.yaml. | ++--------------+------+----------------------------------+--------------------+ +|test sequence | description and expected result | +| | | ++--------------+--------------------------------------------------------------+ +|step 1 | start monitors: | +| | each monitor will run with independently process | +| | | +| | Result: The monitor info will be collected. | +| | | ++--------------+--------------------------------------------------------------+ +|step 2 | do attacker: connect the host through SSH, and then execute | +| | the kill process script with param value specified by | +| | "process_name" | +| | | +| | Result: Process will be killed. | +| | | ++--------------+--------------------------------------------------------------+ +|step 3 | do checker: check whether the status of application | +| | resources on different nodes are updated | +| | | ++--------------+--------------------------------------------------------------+ +|step 4 | stop monitors after a period of time specified by | +| | "waiting_time" | +| | | +| | Result: The monitor info will be aggregated. | +| | | ++--------------+--------------------------------------------------------------+ +|step 5 | verify the SLA | +| | | +| | Result: The test case is passed or not. | +| | | ++--------------+------+----------------------------------+--------------------+ +|post-action | It is the action when the test cases exist. It will check the| +| | status of the cluster messaging process(corosync) on the | +| | host, and restart the process if it is not running for next | +| | test cases | ++--------------+------+----------------------------------+--------------------+ +|test verdict | Fails only if SLA is not passed, or if there is a test case | +| | execution problem. | ++--------------+--------------------------------------------------------------+ diff --git a/tests/opnfv/test_cases/opnfv_yardstick_tc057.yaml b/tests/opnfv/test_cases/opnfv_yardstick_tc057.yaml new file mode 100644 index 000000000..322e2bd76 --- /dev/null +++ b/tests/opnfv/test_cases/opnfv_yardstick_tc057.yaml @@ -0,0 +1,179 @@ +############################################################################## +# Copyright (c) 2017 14_ykl@tongji.edu.cn and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## +--- + +schema: "yardstick:task:0.1" +description: > + Test case for TC057 :OpenStack Controller Cluster Management Service High + Availability; + This test case is written by scenario-based HA testing framework. + +{% set file = file or '/etc/yardstick/pod.yaml' %} +{% set vip_mgmt = vip_mgmt or 'vip__management' %} +{% set vip_vrouter = vip_vrouter or 'vip__vrouter' %} +{% set attack_host = attack_host or 'node1' %} +{% set check_host = check_host or 'node2' %} +{% set monitor_time = monitor_time or 10 %} +{% set monitor_number = monitor_number or 3 %} + +scenarios: + - + type: "GeneralHA" + options: + attackers: + - + fault_type: "general-attacker" + host: {{attack_host}} + key: "kill-process" + attack_key: "kill-corosync" + action_parameter: + process_name: "corosync" + + monitors: + - + monitor_type: "openstack-cmd" + key: "check-nova-service" + command_name: "openstack image list" + monitor_time: {{monitor_time}} + monitor_number: {{monitor_number}} + sla: + max_outage_time: 5 + + - + monitor_type: "openstack-cmd" + key: "check-neutron-service" + command_name: "openstack network list" + monitor_time: {{monitor_time}} + monitor_number: {{monitor_number}} + sla: + max_outage_time: 5 + + - + monitor_type: "openstack-cmd" + key: "check-keystone-service" + command_name: "openstack user list" + monitor_time: {{monitor_time}} + monitor_number: {{monitor_number}} + sla: + max_outage_time: 5 + + - + monitor_type: "openstack-cmd" + key: "check-heat-service" + command_name: "openstack stack list" + monitor_time: {{monitor_time}} + monitor_number: {{monitor_number}} + sla: + max_outage_time: 5 + + operations: + - + operation_type: "general-operation" + key: "get-mgmt-vip-host" + operation_key: "get-vip-host" + host: {{check_host}} + action_parameter: + vip_name: {{vip_mgmt}} + return_parameter: + all: "$vip_mgmt_host" + + - + operation_type: "general-operation" + key: "get-router-vip-host" + operation_key: "get-vip-host" + host: {{check_host}} + action_parameter: + vip_name: {{vip_vrouter}} + return_parameter: + all: "$vip_router_host" + + resultCheckers: + - + checker_type: "general-result-checker" + key: "check-rabbitmq-master" + checker_key: "pacemaker-resource-checker" + host: {{check_host}} + parameter: + resource_name: "p_rabbitmq-server" + resource_host: "$vip_mgmt_host" + expectedValue: "Masters" + condition: "in" + + - + checker_type: "general-result-checker" + key: "check-conntrackd-master" + checker_key: "pacemaker-resource-checker" + host: {{check_host}} + parameter: + resource_name: "p_conntrackd" + resource_host: "$vip_router_host" + expectedValue: "Masters" + condition: "in" + + steps: + - + actionKey: "kill-process" + actionType: "attacker" + index: 1 + + - + actionKey: "check-nova-service" + actionType: "monitor" + index: 2 + + - + actionKey: "check-neutron-service" + actionType: "monitor" + index: 3 + + - + actionKey: "check-keystone-service" + actionType: "monitor" + index: 4 + + - + actionKey: "check-heat-service" + actionType: "monitor" + index: 5 + + - + actionKey: "get-mgmt-vip-host" + actionType: "operation" + index: 6 + + - + actionKey: "check-rabbitmq-master" + actionType: "resultchecker" + index: 7 + + - + actionKey: "get-router-vip-host" + actionType: "operation" + index: 8 + + - + actionKey: "check-conntrackd-master" + actionType: "resultchecker" + index: 9 + + + nodes: + {{attack_host}}: {{attack_host}}.LF + {{check_host}}: {{check_host}}.LF + runner: + type: Duration + duration: 1 + sla: + outage_time: 5 + action: monitor + +context: + type: Node + name: LF + file: {{file}} diff --git a/yardstick/benchmark/scenarios/availability/attacker_conf.yaml b/yardstick/benchmark/scenarios/availability/attacker_conf.yaml index aa144ab50..ee7ea7d83 100644 --- a/yardstick/benchmark/scenarios/availability/attacker_conf.yaml +++ b/yardstick/benchmark/scenarios/availability/attacker_conf.yaml @@ -40,3 +40,7 @@ stress-cpu: block-io: inject_script: ha_tools/disk/block_io.bash recovery_script: ha_tools/disk/recovery_disk_io.bash + +kill-corosync: + inject_script: ha_tools/fault_process_kill.bash + recovery_script: ha_tools/node/reboot_node.bash
\ No newline at end of file diff --git a/yardstick/benchmark/scenarios/availability/ha_tools/node/reboot_node.bash b/yardstick/benchmark/scenarios/availability/ha_tools/node/reboot_node.bash new file mode 100644 index 000000000..1ee8c9c2f --- /dev/null +++ b/yardstick/benchmark/scenarios/availability/ha_tools/node/reboot_node.bash @@ -0,0 +1,14 @@ +#!/bin/bash + +############################################################################## +# (c) OPNFV, Yin Kanglin and others. +# 14_ykl@tongji.edu.cn +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +# reboot node + +reboot
\ No newline at end of file diff --git a/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_resource_status.bash b/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_resource_status.bash new file mode 100644 index 000000000..68707cf4f --- /dev/null +++ b/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_resource_status.bash @@ -0,0 +1,14 @@ +#!/bin/bash + +############################################################################## +# (c) OPNFV, Yin Kanglin and others. +# 14_ykl@tongji.edu.cn +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +# get pacemaker resource status + +pcs resource show
\ No newline at end of file diff --git a/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_resource_status_host.bash b/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_resource_status_host.bash new file mode 100644 index 000000000..7a02ccf29 --- /dev/null +++ b/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_resource_status_host.bash @@ -0,0 +1,15 @@ +#!/bin/bash + +############################################################################## +# (c) OPNFV, Yin Kanglin and others. +# 14_ykl@tongji.edu.cn +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +# get pacemaker resource status of hosts +# parameter: $1 - resource name $2 status + +pcs resource show | grep $1 -A 3 | grep $2
\ No newline at end of file diff --git a/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_vip_host.bash b/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_vip_host.bash new file mode 100644 index 000000000..f4870fdae --- /dev/null +++ b/yardstick/benchmark/scenarios/availability/ha_tools/pacemaker/get_vip_host.bash @@ -0,0 +1,15 @@ +#!/bin/bash + +############################################################################## +# (c) OPNFV, Yin Kanglin and others. +# 14_ykl@tongji.edu.cn +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +# get vip host in pacemaker +# parameter: $1 - virtual ip name + +pcs resource show| grep -w $1 | awk '{print $4}'
\ No newline at end of file diff --git a/yardstick/benchmark/scenarios/availability/operation_conf.yaml b/yardstick/benchmark/scenarios/availability/operation_conf.yaml index 50664d087..dc5169196 100644 --- a/yardstick/benchmark/scenarios/availability/operation_conf.yaml +++ b/yardstick/benchmark/scenarios/availability/operation_conf.yaml @@ -30,3 +30,8 @@ nova-create-flavor: get-floatingip: action_script: ha_tools/nova/get_server_floatingip.bash rollback_script: ha_tools/nova/list_servers.bash + +get-vip-host: + action_script: ha_tools/pacemaker/get_vip_host.bash + rollback_script: ha_tools/pacemaker/get_resource_status.bash + diff --git a/yardstick/benchmark/scenarios/availability/result_checker_conf.yaml b/yardstick/benchmark/scenarios/availability/result_checker_conf.yaml index 0494a71a7..451cc0f11 100644 --- a/yardstick/benchmark/scenarios/availability/result_checker_conf.yaml +++ b/yardstick/benchmark/scenarios/availability/result_checker_conf.yaml @@ -18,4 +18,6 @@ service-checker: nova-instance-checker: verify_script: ha_tools/nova/show_instances.bash nova-flavor-checker: - verify_script: ha_tools/nova/show_flavors.bash
\ No newline at end of file + verify_script: ha_tools/nova/show_flavors.bash +pacemaker-resource-checker: + verify_script: ha_tools/pacemaker/get_resource_status_host.bash
\ No newline at end of file diff --git a/yardstick/benchmark/scenarios/availability/util.py b/yardstick/benchmark/scenarios/availability/util.py index 6fef622bd..d288fcbc1 100644 --- a/yardstick/benchmark/scenarios/availability/util.py +++ b/yardstick/benchmark/scenarios/availability/util.py @@ -51,6 +51,8 @@ def build_shell_command(param_config, remote=True, intermediate_variables=None): def read_stdout_item(stdout, key): + if key == "all": + return stdout for item in stdout.splitlines(): if key in item: attributes = item.split("|") |