diff options
| -rw-r--r-- | docs/testing/developer/devguide/index.rst | 1 | ||||
| -rw-r--r-- | docs/testing/developer/devguide/unit_coverage_test.rst | 190 | ||||
| -rw-r--r-- | requirements/verify.txt | 5 | ||||
| -rw-r--r-- | test/__init__.py | 0 | ||||
| -rw-r--r-- | testsuites/posca/testcase_cfg/posca_factor_multistack_storage_parallel.yaml | 32 | ||||
| -rw-r--r-- | testsuites/posca/testcase_script/posca_factor_multistack_storage_parallel.py | 164 | ||||
| -rw-r--r-- | utils/env_prepare/quota_prepare.py | 27 | ||||
| -rw-r--r-- | utils/env_prepare/stack_prepare.py | 14 | ||||
| -rw-r--r-- | utils/infra_setup/runner/docker_env.py | 2 | ||||
| -rwxr-xr-x | verify.sh | 40 |
10 files changed, 459 insertions, 16 deletions
diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index d3c871e4..37a4ea09 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -11,4 +11,5 @@ Bottlenecks Developer Guide ./quick_start.rst ./framework_guide.rst + ./unit_coverage_test.rst ./package_guide.rst diff --git a/docs/testing/developer/devguide/unit_coverage_test.rst b/docs/testing/developer/devguide/unit_coverage_test.rst new file mode 100644 index 00000000..87096fca --- /dev/null +++ b/docs/testing/developer/devguide/unit_coverage_test.rst @@ -0,0 +1,190 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Huawei Technologies Co.,Ltd and others. + +**************************************** +Bottlenecks - Unit & Coverage Test Guide +**************************************** + + +Introduction of the Rationale and Framework +=========================================== + +What are Unit & Coverage Tests +------------------------------ + +A unit test is an automated code-level test for a small and fairly isolated +part of functionality, mostly in terms of functions. +They should interact with external resources at their minimum, and includes +testing every corner cases and cases that do not work. + +Unit tests should always be pretty simple, by intent. There are +a couple of ways to integrate unit tests into your development style `[1]`_: + +* Test Driven Development, where unit tests are written prior to the functionality they're testing + +* During refactoring, where existing code -- sometimes code without any automated tests to start with -- is retrofitted with unit tests as part of the refactoring process + +* Bug fix testing, where bugs are first pinpointed by a targetted test and then fixed + +* Straight test enhanced development, where tests are written organically as the code evolves. + +Comprehensive and integrally designed unit tests serves valuably as +validator of your APIs, fuctionalities and the workflow that acctually +make them executable. It will make it possibe to deliver your codes +more quickly. + +In the meanwhile, Coverage Test is the tool for measuring code coverage of Python programs. Accompany with Unit Test, it monitors your program, noting which parts of the code have been executed, then analyzes the source to identify code that could have been executed but was not. + +Coverage measurement is typically used to gauge the effectiveness of tests. It can show which parts of your code are being exercised by tests, and which are not. + +Why We Use a Framework and Nose +------------------------------- + +People use unit test discovery and execution frameworks +so that they can forcus on add tests to existing code, +then the tests could be tirggerd, +resulting report could be obtained automatically. + +In addition to adding and running your tests, +frameworks can run tests selectively according to your requirements, add coverage and profiling information, generate comprehensive reports. + +There are many unit test frameworks in Python, and more arise every day. +It will take you some time to be falimiar with those +that are famous from among the ever-arising frameworks. +However, to us, it always matters more that you are actually +writing tests for your codes than how you write them. +Plus, nose is quite stable, it's been used by many projects and it could be adapted easily to mimic any other unit test discovery framework pretty easily. +So, why not? + +Principles of the Tests +----------------------- + +Before you actually implement test codes for your software, +please keep the following principles in mind `[2]`_ + +* A testing unit should focus on one tiny bit of functionality and prove it correct. + +* Each test unit must be fully independent. This is usually handled by setUp() and tearDown() methods. + +* Try hard to make tests that run fast. + +* Learn your tools and learn how to run a single test or a test case. Then, when developing a function inside a module, run this function’s tests frequently, ideally automatically when you save the code. + +* Always run the full test suite before a coding session, and run it again after. This will give you more confidence that you did not break anything in the rest of the code. + +* It is a good idea to implement a hook that runs all tests before pushing code to a shared repository. + +* If you are in the middle of a development session and have to interrupt your work, it is a good idea to write a broken unit test about what you want to develop next. When coming back to work, you will have a pointer to where you were and get back on track faster. + +* The first step when you are debugging your code is to write a new test pinpointing the bug, while it is not always possible to do. + +* Use long and descriptive names for testing functions. These function names are displayed when a test fails, and should be as descriptive as possible. + +* Welly designed tests could acts as an introduction to new developers (read tests or write tests first before going into functionality development) and demonstrations for maintainers. + + +Offline Test +============ + +There only are a few guidance for developing and testing your code on your +local server assuming that you already have python installed. +For more detailed introduction, +please refer to the wesites of nose and coverage `[3]`_ `[4]`_. + +Install Nose +------------ + +Install Nose using your OS's package manager. For example: + +.. code-block:: bash + + pip install nose + +As to creating tests and a quick start, please refer to `[5]`_ + +Run Tests +--------- + +Nose comes with a command line utility called 'nosetests'. +The simplest usage is to call nosetests from within your project directory +and pass a 'tests' directory as an argument. For example, + +.. code-block:: bash + + nosetests tests + +The outputs could be similar to the following summary: + +.. code-block:: bash + + % nosetests tests + .... + ---------------------------------------------------------------------- + Ran 4 tests in 0.003s OK + +Adding Code Coverage +-------------------- + +Coverage is the metric that could complete your unit tests by overseeing +your test codes themselves. +Nose support coverage test according the Coverage.py. + +.. code-block:: bash + + pip install coverage + +To generate a coverage report using the nosetests utility, +simply add the --with-coverage. By default, coverage generates data +for all modules found in the current directory. + +.. code-block:: bash + + nosetests --with-coverage + +% nosetests --with-coverage --cover-package a + +The --cover-package switch can be used multiple times to restrain the tests +only looking into the 3rd party package to avoid useless information. + +.. code-block:: bash + + nosetests --with-coverage --cover-package a --cover-package b + .... + Name Stmts Miss Cover Missing + ------------------------------------- + a 8 0 100% + ---------------------------------------------------------------------- + Ran 4 tests in 0.006sOK + + +OPNFV CI Verify Job +=================== + +Assuming that you have already got the main idea of unit testing +and start to programing you own tests under Bottlenecks repo. +The most important thing that should be clarified is that +unit tests under Bottlenecks should be either excutable offline and +by OPNFV CI pipeline. +When you submit patches to Bottlenecks repo, your patch should following certain ruls to enable the tests: + +* The Bottlenecks unit tests are triggered by OPNFV verify job of CI when you upload files to "test" directory. + +* You should add your --cover-package and test directory in ./verify.sh according to the above guides + +After meeting the two rules, your patch will automatically validated by +nose tests executed by OPNFV verify job. + + +Reference +========= + +_`[1]`: http://ivory.idyll.org/articles/nose-intro.html + +_`[2]`: https://github.com/kennethreitz/python-guide/blob/master/docs/writing/tests.rst + +_`[3]`: http://nose.readthedocs.io/en/latest/ + +_`[4]`: https://coverage.readthedocs.io/en/coverage-4.4.2 + +_`[5]`: http://blog.jameskyle.org/2010/10/nose-unit-testing-quick-start/ diff --git a/requirements/verify.txt b/requirements/verify.txt index e8d51f47..3c63c8ff 100644 --- a/requirements/verify.txt +++ b/requirements/verify.txt @@ -6,5 +6,8 @@ # which accompanies this distribution, and is available at # http://www.apache.org/licenses/LICENSE-2.0 ############################################################################## +# This file is for verify job excuted by OPNFV CI -flake8==3.0.4
\ No newline at end of file +flake8==3.0.4 +nose==1.3.7 +coverage==4.4.2 diff --git a/test/__init__.py b/test/__init__.py new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/test/__init__.py diff --git a/testsuites/posca/testcase_cfg/posca_factor_multistack_storage_parallel.yaml b/testsuites/posca/testcase_cfg/posca_factor_multistack_storage_parallel.yaml new file mode 100644 index 00000000..44147db8 --- /dev/null +++ b/testsuites/posca/testcase_cfg/posca_factor_multistack_storage_parallel.yaml @@ -0,0 +1,32 @@ +############################################################################## +# Copyright (c) 2017 HUAWEI TECHNOLOGIES CO.,LTD and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## + +load_manager: + scenarios: + tool: fio + # for this option we provide " write, read, rw, rr" + rw: "randrw" + bs: "4k" + size: "50g" + rwmixwrite: "50" + num_stack: 1, 3 + volume_num: "1" + num_jobs: "1" + direct: "1" + + runners: + stack_create: yardstick + flavor: + yardstick_test_dir: "samples" + yardstick_testcase: "storage_bottlenecks" + +contexts: + dashboard: "Bottlenecks-ELK" + yardstick: "Bottlenecks-Yardstick" + yardstick_envpre: True
\ No newline at end of file diff --git a/testsuites/posca/testcase_script/posca_factor_multistack_storage_parallel.py b/testsuites/posca/testcase_script/posca_factor_multistack_storage_parallel.py new file mode 100644 index 00000000..58f9dc8c --- /dev/null +++ b/testsuites/posca/testcase_script/posca_factor_multistack_storage_parallel.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python +############################################################################## +# Copyright (c) 2017 Huawei Technologies Co.,Ltd and others. +# +# All rights reserved. This program and the accompanying materials +# are made available under the terms of the Apache License, Version 2.0 +# which accompanies this distribution, and is available at +# http://www.apache.org/licenses/LICENSE-2.0 +############################################################################## +'''This file realize the function of run posca multistack storage stress test +This file contain several part: +First is create a script to realize several threading run''' + +import utils.logger as log +import uuid +import json +import os +import time +from utils.parser import Parser as conf_parser +import utils.env_prepare.quota_prepare as quota_prepare +import utils.env_prepare.stack_prepare as stack_prepare +import utils.infra_setup.runner.yardstick as yardstick_task + +import utils.infra_setup.runner.docker_env as docker_env + +# -------------------------------------------------- +# logging configuration +# -------------------------------------------------- +LOG = log.Logger(__name__).getLogger() + +test_dict = { + "action": "runTestCase", + "args": { + "opts": { + "task-args": {} + }, + "testcase": "multistack_storage_bottlenecks_parallel" + } +} +testfile = os.path.basename(__file__) +testcase, file_format = os.path.splitext(testfile) +cidr = "/home/opnfv/repos/yardstick/samples/storage_bottlenecks.yaml" +runner_DEBUG = True + + +def env_pre(test_config): + test_yardstick = False + if "yardstick" in test_config["contexts"].keys(): + test_yardstick = True + stack_prepare._prepare_env_daemon(test_yardstick) + quota_prepare.quota_env_prepare() + if(test_config["contexts"]['yardstick_envpre']): + cmd = ('yardstick env prepare') + LOG.info("yardstick environment prepare!") + yardstick_container = docker_env.yardstick_info['container'] + stdout = docker_env.docker_exec_cmd(yardstick_container, cmd) + LOG.debug(stdout) + + +def testcase_parser(out_file="yardstick.out", **parameter_info): + cmd = yardstick_task.yardstick_command_parser(debug=runner_DEBUG, + cidr=cidr, + outfile=out_file, + parameter=parameter_info) + return cmd + + +def do_test(test_config): + out_file = ("/tmp/yardstick_" + str(uuid.uuid4()) + ".out") + yardstick_container = docker_env.yardstick_info['container'] + cmd = testcase_parser(out_file=out_file, **test_config) + print(cmd) + stdout = docker_env.docker_exec_cmd(yardstick_container, cmd) + LOG.info(stdout) + loop_value = 0 + while loop_value < 60: + time.sleep(2) + loop_value = loop_value + 1 + with open(out_file) as f: + data = json.load(f) + if data["status"] == 1: + LOG.info("yardstick run success") + LOG.info("%s" % data["result"]["testcases"]) + break + elif data["status"] == 2: + LOG.error("yardstick error exit") + break + + save_data = config_to_result(test_config, data) + LOG.info(save_data) + return save_data + + +def config_to_result(test_config, test_result): + print(test_result) + out_data = test_result["result"]["testcases"] + test_data = out_data["storage_bottlenecks"]["tc_data"] + testdata = {} + testdata["read_iops"] = 0 + testdata["read_bw"] = 0 + testdata["read_lat"] = 0 + testdata["write_iops"] = 0 + testdata["write_bw"] = 0 + testdata["write_lat"] = 0 + print(testdata["read_iops"]) + for result in test_data: + testdata["read_iops"] += result["data"]["read_iops"] + testdata["read_bw"] += result["data"]["read_bw"] + if testdata["read_lat"] is 0: + testdata["read_lat"] = result["data"]["read_lat"] + else: + testdata["read_lat"] += result["data"]["read_lat"] + testdata["read_lat"] = testdata["read_lat"]/2 + testdata["write_iops"] += result["data"]["write_iops"] + testdata["write_bw"] += result["data"]["write_bw"] + if testdata["write_lat"] is 0: + testdata["write_lat"] = result["data"]["write_lat"] + else: + testdata["write_lat"] += result["data"]["write_lat"] + testdata["write_lat"] = testdata["write_lat"]/2 + return testdata + + +def run(test_config): + con_dic = test_config["load_manager"] + scenarios_conf = con_dic["scenarios"] + + if test_config["contexts"]["yardstick_ip"] is None: + con_dic["contexts"]["yardstick_ip"] =\ + conf_parser.ip_parser("yardstick_test_ip") + + env_pre(test_config) + LOG.info("yardstick environment prepare done!") + + test_num = conf_parser.str_to_list(scenarios_conf["num_stack"]) + rw = scenarios_conf["rw"] + bs = scenarios_conf["bs"] + size = scenarios_conf["size"] + rwmixwrite = scenarios_conf["rwmixwrite"] + numjobs = scenarios_conf["num_jobs"] + direct = scenarios_conf["direct"] + volume_num = scenarios_conf["volume_num"] + + result = [] + + for value in test_num: + case_config = {"stack_num": int(value), + "volume_num": volume_num, + "rw": rw, + "bs": bs, + "size": size, + "rwmixwrite": rwmixwrite, + "numjobs": numjobs, + "direct": direct} + data_reply = do_test(case_config) + result.append(data_reply) + + LOG.info("%s stack successful run" % (value)) + + conf_parser.result_to_file(data_reply, test_config["out_file"]) + + LOG.info('END POSCA stress multistack storage parallel testcase') + LOG.info("The result data is %s", result) + return result diff --git a/utils/env_prepare/quota_prepare.py b/utils/env_prepare/quota_prepare.py index 267e70ab..8dcdf3d6 100644 --- a/utils/env_prepare/quota_prepare.py +++ b/utils/env_prepare/quota_prepare.py @@ -42,14 +42,37 @@ nova_quota = {"ram": -1, "injected_file_path_bytes": -1} +def check_https_enabled(): + LOG.debug("Check if https is enabled in OpenStack") + os_auth_url = os.getenv('OS_AUTH_URL') + if os_auth_url.startswith('https'): + LOG.debug("https is enabled") + return True + LOG.debug("https is not enabled") + return False + + def quota_env_prepare(): + https_enabled = check_https_enabled() + insecure_option = '' + insecure = os.getenv('OS_INSECURE',) + if https_enabled: + LOG.info("https is enabled") + if insecure: + if insecure.lower() == "true": + insecure_option = ' --insecure ' + else: + LOG.warn("Env variable OS_INSECURE is {}: if https + no " + "credential used, it should be set as True." + .format(insecure)) + tenant_name = os.getenv("OS_TENANT_NAME") - cmd = ("openstack project list | grep " + + cmd = ("openstack {} project list | grep ".format(insecure_option) + tenant_name + " | awk '{print $2}'") result = commands.getstatusoutput(cmd) - if result[0] == 0: + if result[0] == 0 and 'exception' not in result[1]: LOG.info("Get %s project id is %s" % (tenant_name, result[1])) else: LOG.error("can't get openstack project id") diff --git a/utils/env_prepare/stack_prepare.py b/utils/env_prepare/stack_prepare.py index 25c2a29b..640afa13 100644 --- a/utils/env_prepare/stack_prepare.py +++ b/utils/env_prepare/stack_prepare.py @@ -37,6 +37,16 @@ def _prepare_env_daemon(test_yardstick): config.bottlenecks_config["yardstick_rc_dir"]) docker_env.docker_exec_cmd(yardstick_contain, cmd) + file_orig = ("/home/opnfv/repos/yardstick/etc" + "/yardstick/yardstick.conf.sample") + file_after = "/etc/yardstick/yardstick.conf" + cmd = "cp %s %s" % (file_orig, + file_after) + docker_env.docker_exec_cmd(yardstick_contain, + cmd) + cmd = "sed -i '12s/http/file/g' /etc/yardstick/yardstick.conf" + docker_env.docker_exec_cmd(yardstick_contain, + cmd) # update the external_network _source_file(rc_file) @@ -73,7 +83,9 @@ def _source_file(rc_file): p = subprocess.Popen(". %s; env" % rc_file, stdout=subprocess.PIPE, shell=True) output = p.communicate()[0] - env = dict((line.split('=', 1) for line in output.splitlines())) + output_lines = output.splitlines() + del output_lines[5] + env = dict((line.split('=', 1) for line in output_lines)) os.environ.update(env) return env diff --git a/utils/infra_setup/runner/docker_env.py b/utils/infra_setup/runner/docker_env.py index 438d3d19..54478ffc 100644 --- a/utils/infra_setup/runner/docker_env.py +++ b/utils/infra_setup/runner/docker_env.py @@ -45,7 +45,7 @@ def env_yardstick(docker_name): volume = get_self_volume() yardstick_tag = os.getenv("Yardstick_TAG") if yardstick_tag is None: - yardstick_tag = "latest" + yardstick_tag = "danube.3.1" env_docker = client.containers.run(image="opnfv/yardstick:%s" % yardstick_tag, privileged=True, @@ -14,7 +14,9 @@ getopts ":f" FILE_OPTION run_flake8() { - echo "Running flake8 ... " + echo "=========================================" + echo "Running flake8 for python style check: " + echo "-----------------------------------------" logfile=flake8_verify.log if [ $FILE_OPTION == "f" ]; then flake8 --append-config=flake8_cfg testsuites/posca/ utils/ > $logfile @@ -29,20 +31,36 @@ run_flake8() { fi exit 1 else - echo "OK" + echo "The patch has passed python style check " + echo "===================END===================" fi } -run_tests() { - echo "Running unittest ... " - #will add future -} -run_coverage() { - echo "Running coverage ..." - #will add future +run_nosetests() { + echo "=========================================" + echo "Running unit and coverage test: " + echo "-----------------------------------------" + nosetests --with-coverage --cover-tests \ + --cover-min-percentage 100 \ + test/__init__.py \ + testsuites/posca/__init__.py testsuites/__init__.py \ + testsuites/posca/testcase_cfg/__init__.py \ + testsuites/posca/testcase_dashboard/__init__.py \ + testsuites/posca/testcase_script/__init__.py \ + utils/__init__.py \ + utils/dashboard/__init__.py \ + utils/env_prepare/__init__.py \ + utils/infra_setup/__init__.py \ + monitor/__init__.py \ + monitor/config/__init__.py + echo "===================END===================" + } + +for((i=1;i<=1;i++));do echo -e "\n";done run_flake8 -run_tests -run_coverage + +for((i=1;i<=1;i++));do echo -e "\n";done +run_nosetests |