diff options
24 files changed, 513 insertions, 53 deletions
diff --git a/docker/Dockerfile b/docker/Dockerfile index eb85ea49..c4b11213 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -8,8 +8,11 @@ RUN \ apt-get update \ && \ apt-get install -y \ + build-essential \ gcc \ git \ + libssl-dev \ + libffi-dev \ vim \ python-dev \ python-mock \ diff --git a/docs/testing/developer/testscope/index.rst b/docs/testing/developer/testscope/index.rst index ab1577da..09901333 100644 --- a/docs/testing/developer/testscope/index.rst +++ b/docs/testing/developer/testscope/index.rst @@ -400,6 +400,14 @@ Verify high availability of OpenStack controller services | opnfv.ha.tc010.disk_I/O_block | opnfv.ha.tc011.load_balance_service_down +---------------------------------------- +Test Area vPing - Basic VNF Connectivity +---------------------------------------- + +| opnfv.vping.userdata +| opnfv.vping.ssh + + Optional CVP Test Areas ======================== diff --git a/docs/testing/user/testspecification/vping/index.rst b/docs/testing/user/testspecification/vping/index.rst new file mode 100644 index 00000000..d7a207c0 --- /dev/null +++ b/docs/testing/user/testspecification/vping/index.rst @@ -0,0 +1,279 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) Ericsson AB + +======================== +Vping test specification +======================== + +.. toctree:: + :maxdepth: 2 + +Scope +===== + +The vping test area evaluates basic NFVi capabilities of the system under test. +These capabilities include creating a small number of virtual machines, +establishing basic L3 connectivity between them and verifying connectivity by +means of ICMP packets. + + +References +========== + +- Neutron Client + + - https://docs.openstack.org/developer/python-neutronclient/usage/library.html + +- Nova Client + + - https://docs.openstack.org/developer/python-novaclient/ref/v2/servers.html + +- SSHClient + + - http://docs.paramiko.org/en/2.2/ + +- SCPClient + + - https://pypi.python.org/pypi/scp + + +Definitions and abbreviations +============================= + +The following terms and abbreviations are used in conjunction with this test +area + +- ICMP - Internet Control Message Protocol +- L3 - Layer 3 +- NFVi - Network functions virtualization infrastructure +- SCP - Secure Copy +- SSH - Secure Shell +- VM - Virtual machine + + +System Under Test (SUT) +======================= + +The system under test is assumed to be the NFVi and VIM in operation on a +Pharos compliant infrastructure. + + +Test Area Structure +=================== + +The test area is structured in two separate tests which are executed +sequentially. The order of the tests is arbitrary as there are no dependencies +across the tests. + + +Test Descriptions +================= + +-------------------------------------------------------------------- +Test Case 1 - vPing using userdata provided by nova metadata service +-------------------------------------------------------------------- + +Short name +---------- + +opnfv.vping.userdata + + +Use case specification +---------------------- + +This test evaluates the use case where an NFVi tenant boots up two VMs and +requires L3 connectivity between those VMs. The target IP is passed to the VM +that will initiate pings by using a custom userdata script provided by nova metadata service. + + +Test preconditions +------------------ + +At least one compute node is available. No further pre-configuration needed. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. The target IP is passed to the VM sending pings by using a +custom userdata script by means of the config driver mechanism provided by +Nova metadata service. Whether or not a ping was successful is determined by +checking the console output of the source VMs. + + +Test execution +'''''''''''''' + +* Test action 1: + * Create a private tenant network by using neutron client + * Create one subnet and one router in the network by neutron client + * Add one interface between the subnet and router + * Add one gateway route to the router by neutron client + * Store the network id in the response +* **Test assertion 1:** The network id, subnet id and router id can be found in the response +* Test action 2: + * Create an security group by using neutron client + * Store the security group id parameter in the response +* **Test assertion 2:** The security group id can be found in the response +* Test action 3: boot VM1 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 3:** The VM1 object can be found in the response +* Test action 4: Generate ping script with the IP of VM1 to be passed as userdata provided by + the **nova metadata service**. +* Test action 5: Boot VM2 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2, userdata created + in test action 4 +* **Test assertion 4:** The VM2 object can be found in the response +* Test action 6: Inside VM2, the ping script is executed automatically when booted and it contains a + loop doing the ping until the return code is 0 or timeout reached. For each ping, when the return + code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed. + Monitoring the console-log of VM2 to see the response generated by the script. +* **Test assertion 5:** "vPing OK" is detected, when monitoring the console-log in VM2 +* Test action 7: delete VM1, VM2 +* **Test assertion 6:** VM1 and VM2 are not present in the VM list +* Test action 8: delete security group, gateway, interface, router, subnet and network +* **Test assertion 7:** The security group, gateway, interface, router, subnet and network are + no longer present in the lists after deleting + + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates basic NFVi capabilities of the system under test. +Specifically, the test verifies that: + +* Neutron client network, subnet, router, interface create commands return valid "id" parameters + which are shown in the create response message +* Neutron client interface add command to add between subnet and router returns success code +* Neutron client gateway add command to add to router returns success code +* Neutron client security group create command returns valid "id" parameter which is shown in + the response message +* Nova client VM create command returns valid VM attributes response message +* Nova metadata server can transfer userdata configuration at nova client VM booting time +* Ping command from one VM to the other in same private tenant network returns valid code +* All items created using neutron client or nova client create commands are able to be removed by + using the returned identifiers + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +None + + +---------------------------------------------- +Test Case 2 - vPing using SSH to a floating IP +---------------------------------------------- + +Short name +---------- + +opnfv.vping.ssh + + +Use case specification +---------------------- + +This test evaluates the use case where an NFVi tenant boots up two VMs and requires +L3 connectivity between those VMs. An SSH connection is establised from the host to +a floating IP associated with VM2 and ``ping`` is executed on VM2 with the IP of VM1 as target. + + +Test preconditions +------------------ + +At least one compute node is available. There should exist an OpenStack external network +and can assign floating IP. + + +Basic test flow execution description and pass/fail criteria +------------------------------------------------------------ + +Methodology for verifying connectivity +'''''''''''''''''''''''''''''''''''''' + +Connectivity between VMs is tested by sending ICMP ping packets between +selected VMs. To this end, the test establishes an SSH connection from the host +running the test suite to a floating IP associated with VM2 and executes ``ping`` +on VM2 with the IP of VM1 as target. + + +Test execution +'''''''''''''' + + +* Test action 1: + * Create a private tenant network by neutron client + * Create one subnet and one router are created in the network by using neutron client + * Create one interface between the subnet and router + * Add one gateway route to the router by neutron client + * Store the network id in the response +* **Test assertion 1:** The network id, subnet id and router id can be found in the response +* Test action 2: + * Create an security group by using neutron client + * Store the security group id parameter in the response +* **Test assertion 2:** The security group id can be found in the response +* Test action 3: Boot VM1 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 3:** The VM1 object can be found in the response +* Test action 4: Boot VM2 by using nova client with configured name, image, flavor, private tenant + network created in test action 1, security group created in test action 2 +* **Test assertion 4:** The VM2 object can be found in the response +* Test action 5: create one floating IP by using neutron client, storing the floating IP address + returned in the response +* **Test assertion 5:** Floating IP address can be found in the response +* Test action 6: Assign the floating IP address created in test action 5 to VM2 by using nova client +* **Test assertion 6:** The assigned floating IP can be found in the VM2 console log file +* Test action 7: Establish SSH connection between the test host and VM2 through the floating IP +* **Test assertion 7:** SSH connection between the test host and VM2 is established within + 300 seconds +* Test action 8: Copy the Ping script from the test host to VM2 by using SCPClient +* **Test assertion 8:** The Ping script can be found inside VM2 +* Test action 9: Inside VM2, to execute the Ping script to ping VM1, the Ping script contains a + loop doing the ping until the return code is 0 or timeout reached, for each ping, when the return + code is 0, "vPing OK" is printed in the VM2 console-log, otherwise, "vPing KO" is printed. + Monitoring the console-log of VM2 to see the response generated by the script. +* **Test assertion 9:** "vPing OK" is detected, when monitoring the console-log in VM2 +* Test action 10: delete VM1, VM2 +* **Test assertion 10:** VM1 and VM2 are not present in the VM list +* Test action 11: delete floating IP, security group, gateway, interface, router, subnet and network +* **Test assertion 11:** The security group, gateway, interface, router, subnet and network are + no longer present in the lists after deleting + +Pass / fail criteria +'''''''''''''''''''' + +This test evaluates basic NFVi capabilities of the system under test. +Specifically, the test verifies that: + +* Neutron client network, subnet, router, interface create commands return valid "id" parameters + which are shown in the create response message +* Neutron client interface add command to add between subnet and router return success code +* Neutron client gateway add command to add to router return success code +* Neutron client security group create command returns valid "id" parameter which is shown in the + response message +* Nova client VM create command returns valid VM attributes response message +* Neutron client floating IP create command return valid floating IP address +* Nova client add floating IP command returns valid response message +* SSH connection can be established using a floating IP +* Ping command from one VM to another in same private tenant network returns valid code +* All items created using neutron client or nova client create commands are able to be removed by + using the returned identifiers + +In order to pass this test, all test assertions listed in the test execution +above need to pass. + + +Post conditions +--------------- + +None diff --git a/docs/testing/user/userguide/testing_guide.rst b/docs/testing/user/userguide/testing_guide.rst index 4f08a064..2d45b189 100644 --- a/docs/testing/user/userguide/testing_guide.rst +++ b/docs/testing/user/userguide/testing_guide.rst @@ -261,8 +261,8 @@ If the Test Host is online, you can directly pull. .. code-block:: bash - $ sudo docker pull opnfv/dovetail:cvp.0.3.0 - cvp.0.3.0: Pulling from opnfv/dovetail + $ sudo docker pull opnfv/dovetail:cvp.0.4.0 + cvp.0.4.0: Pulling from opnfv/dovetail 30d541b48fc0: Pull complete 8ecd7f80d390: Pull complete 46ec9927bb81: Pull complete @@ -275,9 +275,9 @@ If the Test Host is online, you can directly pull. bf7c644692de: Pull complete cdc345e3f363: Pull complete Digest: sha256:d571b1073b2fdada79562e8cc67f63018e8d89268ff7faabee3380202c05edee - Status: Downloaded newer image for opnfv/dovetail:cvp.0.3.0 + Status: Downloaded newer image for opnfv/dovetail:cvp.0.4.0 -An example of the <tag> is *cvp.0.3.0*. +An example of the <tag> is *cvp.0.4.0*. If the Test Host is offline, you will need to first pull the Dovetail Docker image, and all the dependent images that Dovetail uses, to a host that is online. The reason that you need @@ -287,9 +287,10 @@ offline, then all these dependencies will also need to be manually copied. .. code-block:: bash - $ sudo docker pull opnfv/dovetail:cvp.0.3.0 + $ sudo docker pull opnfv/dovetail:cvp.0.4.0 $ sudo docker pull opnfv/functest:cvp.0.2.0 $ sudo docker pull opnfv/yardstick:danube.3.1 + $ sudo docker pull opnfv/bottlenecks:cvp.0.4.0 $ sudo wget -nc http://artifacts.opnfv.org/sdnvpn/ubuntu-16.04-server-cloudimg-amd64-disk1.img -P {ANY_DIR} Once all these images are pulled, save the images, copy to the Test Host, and then load @@ -299,7 +300,8 @@ At the online host, save images. .. code-block:: bash - $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.3.0 opnfv/functest:cvp.0.2.0 opnfv/yardstick:danube.3.1 + $ sudo docker save -o dovetail.tar opnfv/dovetail:cvp.0.4.0 opnfv/functest:cvp.0.2.0 \ + opnfv/yardstick:danube.3.1 opnfv/bottlenecks:cvp.0.4.0 Copy dovetail.tar file to the Test Host, and then load the images on the Test Host. @@ -316,8 +318,9 @@ Now check to see that the Dovetail image has been pulled or loaded properly. $ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE opnfv/functest cvp.0.2.0 9eaeaea5f203 8 days ago 1.53GB - opnfv/dovetail cvp.0.3.0 5d25b289451c 8 days ago 516MB + opnfv/dovetail cvp.0.4.0 5d25b289451c 8 days ago 516MB opnfv/yardstick danube.3.1 574596b6ea12 8 days ago 1.2GB + opnfv/bottlenecks cvp.0.4.0 00450688bcae 3 hours ago 622 MB Regardless of whether you pulled down the Dovetail image directly online, or loaded from a static image tar file, you are ready to run Dovetail. diff --git a/dovetail/compliance/debug.yml b/dovetail/compliance/debug.yml index 0ae734ea..9f6d2389 100644 --- a/dovetail/compliance/debug.yml +++ b/dovetail/compliance/debug.yml @@ -12,6 +12,5 @@ debug: # - dovetail.ipv6.tc009 # - dovetail.ipv6.tc018 # - dovetail.ipv6.tc019 - - dovetail.nfvi.tc001 - - dovetail.nfvi.tc002 - - dovetail.nfvi.tc101 + - dovetail.vping.tc001 + - dovetail.vping.tc002 diff --git a/dovetail/compliance/proposed_tests.yml b/dovetail/compliance/proposed_tests.yml index 9d63cb2a..b148df68 100644 --- a/dovetail/compliance/proposed_tests.yml +++ b/dovetail/compliance/proposed_tests.yml @@ -5,6 +5,9 @@ proposed_tests: # defcore,defcore.tc001 used for auto-method, defcore.tc002 used for manually method - dovetail.defcore.tc001 # - dovetail.defcore.tc002 + # vping + - dovetail.vping.tc001 + - dovetail.vping.tc002 # ipv6 - dovetail.ipv6.tc001 - dovetail.ipv6.tc002 @@ -45,3 +48,5 @@ proposed_tests: - dovetail.sdnvpn.tc002 - dovetail.sdnvpn.tc004 - dovetail.sdnvpn.tc008 + # resiliency + - dovetail.resiliency.tc001 diff --git a/dovetail/conf/bottlenecks_config.yml b/dovetail/conf/bottlenecks_config.yml new file mode 100644 index 00000000..43df8c55 --- /dev/null +++ b/dovetail/conf/bottlenecks_config.yml @@ -0,0 +1,20 @@ +--- +bottlenecks: + image_name: opnfv/bottlenecks + docker_tag: cvp.0.4.0 + opts: '-id --privileged=true' + config: + dir: '/home/opnfv/userconfig' + pre_condition: + - 'echo test for precondition in bottlenecks' + cmds: + - '/home/opnfv/bottlenecks/run_tests.sh -c {{validate_testcase}}' + post_condition: + - 'echo test for postcondition in bottlenecks' + result: + dir: '/tmp' + file_path: 'bottlenecks.log' + openrc: '/tmp/admin_rc.sh' + extra_container: + - 'Bottlenecks-Yardstick' + - 'Bottlenecks-ELK' diff --git a/dovetail/conf/cmd_config.yml b/dovetail/conf/cmd_config.yml index a5b262d1..d862a910 100644 --- a/dovetail/conf/cmd_config.yml +++ b/dovetail/conf/cmd_config.yml @@ -21,14 +21,21 @@ cli: - '-y' path: - 'yardstick/docker_tag' - help: 'Overwrite tag for yardstick docker container (e.g. stable or latest)' + help: 'Overwrite tag for yardstick docker container (e.g. danube.3.1)' func_tag: flags: - '--func_tag' - '-f' path: - 'functest/docker_tag' - help: 'Overwrite tag for functest docker container (e.g. stable or latest)' + help: 'Overwrite tag for functest docker container (e.g. cvp.0.2.0)' + bott_tag: + flags: + - '--bott_tag' + - '-b' + path: + - 'bottlenecks/docker_tag' + help: 'Overwrite tag for bottlenecks docker container (e.g. cvp.0.4.0)' control: testsuite: flags: diff --git a/dovetail/conf/dovetail_config.yml b/dovetail/conf/dovetail_config.yml index fecc0ae1..98f9e814 100644 --- a/dovetail/conf/dovetail_config.yml +++ b/dovetail/conf/dovetail_config.yml @@ -30,9 +30,9 @@ testarea_supported: - example - ha - ipv6 - - nfvi - sdnvpn - - vimops + - vping + - resiliency functest_testsuite: - refstack_defcore @@ -67,10 +67,12 @@ parameters: include_config: - functest_config.yml - yardstick_config.yml + - bottlenecks_config.yml test_project: - 'yardstick' - 'functest' + - 'bottlenecks' validate_input: valid_docker_tag: @@ -83,3 +85,4 @@ validate_input: - 'cvp.0.1.0' - 'cvp.0.2.0' - 'cvp.0.3.0' + - 'cvp.0.4.0' diff --git a/dovetail/container.py b/dovetail/container.py index 1a5867a7..ede50a80 100644 --- a/dovetail/container.py +++ b/dovetail/container.py @@ -18,7 +18,8 @@ from utils.dovetail_config import DovetailConfig as dt_cfg class Container(object): container_list = {} - has_pull_latest_image = {'yardstick': False, 'functest': False} + has_pull_latest_image = {'yardstick': False, 'functest': False, + 'bottlenecks': False} logger = None @@ -123,6 +124,19 @@ class Container(object): return "%s %s %s" % (envs, log_vol, key_vol) @classmethod + def set_bottlenecks_config(cls, testcase_name): + dovetail_config = dt_cfg.dovetail_config + yard_tag = dovetail_config['yardstick']['docker_tag'] + docker_vol = '-v /var/run/docker.sock:/var/run/docker.sock' + env = ('-e Yardstick_TAG={} -e OUTPUT_FILE={}.out' + .format(yard_tag, testcase_name)) + report = "" + if dovetail_config['report_dest'].startswith("http"): + report = ("-e BOTTLENECKS_DB_TARGET={}" + .format(dovetail_config['report_dest'])) + return "{} {} {}".format(docker_vol, env, report) + + @classmethod def create(cls, type, testcase_name): dovetail_config = dt_cfg.dovetail_config docker_image = cls.get_docker_image(type) @@ -144,6 +158,8 @@ class Container(object): config = cls.set_functest_config(testcase_name) if type.lower() == "yardstick": config = cls.set_yardstick_config() + if type.lower() == "bottlenecks": + config = cls.set_bottlenecks_config(testcase_name) if not config: return None @@ -284,11 +300,23 @@ class Container(object): return image_id @classmethod - def clean(cls, container_id): - cmd1 = 'sudo docker stop %s' % (container_id) - dt_utils.exec_cmd(cmd1, cls.logger) - cmd2 = 'sudo docker rm %s' % (container_id) - dt_utils.exec_cmd(cmd2, cls.logger) + def check_container_exist(cls, container_name): + cmd = ('sudo docker ps -aq -f name={}'.format(container_name)) + ret, msg = dt_utils.exec_cmd(cmd, cls.logger) + if ret == 0 and msg: + return True + return False + + @classmethod + def clean(cls, container_id, valid_type): + cmd = ('sudo docker rm -f {}'.format(container_id)) + dt_utils.exec_cmd(cmd, cls.logger) + if valid_type.lower() == 'bottlenecks': + containers = dt_cfg.dovetail_config[valid_type]['extra_container'] + for container in containers: + if cls.check_container_exist(container): + cmd = ('sudo docker rm -f {}'.format(container)) + dt_utils.exec_cmd(cmd, cls.logger) @classmethod def exec_cmd(cls, container_id, sub_cmd, exit_on_error=False): diff --git a/dovetail/report.py b/dovetail/report.py index 08780b88..4743c256 100644 --- a/dovetail/report.py +++ b/dovetail/report.py @@ -25,7 +25,7 @@ from testcase import Testcase class Report(object): - results = {'functest': {}, 'yardstick': {}, 'shell': {}} + results = {'functest': {}, 'yardstick': {}, 'bottlenecks': {}, 'shell': {}} logger = None @@ -339,6 +339,53 @@ class YardstickCrawler(object): return None +class BottlenecksCrawler(object): + + logger = None + + def __init__(self): + self.type = 'bottlenecks' + self.logger.debug('Create crawler: {}'.format(self.type)) + + @classmethod + def create_log(cls): + cls.logger = \ + dt_logger.Logger(__name__ + '.BottlenecksCrawler').getLogger() + + def crawl(self, testcase=None): + report_dest = dt_cfg.dovetail_config['report_dest'] + if report_dest.lower() == 'file': + return self.crawl_from_file(testcase) + + if report_dest.lower().startswith('http'): + return self.crawl_from_url(testcase) + + def crawl_from_file(self, testcase=None): + file_path = os.path.join(dt_cfg.dovetail_config['result_dir'], + testcase.name() + '.out') + if not os.path.exists(file_path): + self.logger.error('Result file not found: {}'.format(file_path)) + return None + criteria = 'FAIL' + with open(file_path, 'r') as f: + for jsonfile in f: + data = json.loads(jsonfile) + try: + if 'PASS' == data["data_body"]["result"]: + criteria = 'PASS' + else: + criteria = 'FAIL' + break + except KeyError as e: + self.logger.exception('Pass flag not found {}'.format(e)) + json_results = {'criteria': criteria} + self.logger.debug('Results: {}'.format(str(json_results))) + return json_results + + def crawl_from_url(self, testcase=None): + return None + + class ShellCrawler(object): def __init__(self): @@ -364,6 +411,7 @@ class CrawlerFactory(object): CRAWLER_MAP = {'functest': FunctestCrawler, 'yardstick': YardstickCrawler, + 'bottlenecks': BottlenecksCrawler, 'shell': ShellCrawler} @classmethod @@ -466,6 +514,24 @@ class YardstickChecker(object): return +class BottlenecksChecker(object): + + logger = None + + @classmethod + def create_log(cls): + cls.logger = \ + dt_logger.Logger(__name__ + '.BottlenecksChecker').getLogger() + + @staticmethod + def check(testcase, result): + if not result: + testcase.passed('FAIL') + else: + testcase.passed(result['criteria']) + return + + class ShellChecker(object): @staticmethod @@ -480,6 +546,7 @@ class CheckerFactory(object): CHECKER_MAP = {'functest': FunctestChecker, 'yardstick': YardstickChecker, + 'bottlenecks': BottlenecksChecker, 'shell': ShellChecker} @classmethod diff --git a/dovetail/run.py b/dovetail/run.py index f89671f0..7515023b 100755 --- a/dovetail/run.py +++ b/dovetail/run.py @@ -22,8 +22,8 @@ from container import Container from testcase import Testcase from testcase import Testsuite from report import Report -from report import FunctestCrawler, YardstickCrawler -from report import FunctestChecker, YardstickChecker +from report import FunctestCrawler, YardstickCrawler, BottlenecksCrawler +from report import FunctestChecker, YardstickChecker, BottlenecksChecker from utils.dovetail_config import DovetailConfig as dt_cfg from test_runner import DockerRunner, ShellRunner @@ -90,6 +90,8 @@ def check_tc_result(testcase, logger): result_file = os.path.join(result_dir, testcase.name() + '.out') elif validate_type.lower() == 'functest': result_file = os.path.join(result_dir, functest_result) + elif validate_type.lower() == 'bottlenecks': + result_file = os.path.join(result_dir, testcase.name() + '.out') else: logger.error("Don't support {} now.".format(validate_type)) return @@ -107,6 +109,7 @@ def validate_input(input_dict, check_dict, logger): # for 'func_tag' and 'yard_tag' options func_tag = input_dict['func_tag'] yard_tag = input_dict['yard_tag'] + bott_tag = input_dict['bott_tag'] valid_tag = check_dict['valid_docker_tag'] if func_tag is not None and func_tag not in valid_tag: logger.error("The input option 'func_tag' can't be {}, " @@ -116,6 +119,10 @@ def validate_input(input_dict, check_dict, logger): logger.error("The input option 'yard_tag' can't be {}, " "valid values are {}.".format(yard_tag, valid_tag)) raise SystemExit(1) + if bott_tag is not None and bott_tag not in valid_tag: + logger.error("The input option 'bott_tag' can't be {}, " + "valid values are {}.".format(bott_tag, valid_tag)) + raise SystemExit(1) # for 'report' option report = input_dict['report'] @@ -164,8 +171,10 @@ def create_logs(): Report.create_log() FunctestCrawler.create_log() YardstickCrawler.create_log() + BottlenecksCrawler.create_log() FunctestChecker.create_log() YardstickChecker.create_log() + BottlenecksChecker.create_log() Testcase.create_log() Testsuite.create_log() DockerRunner.create_log() @@ -246,6 +255,7 @@ def main(*args, **kwargs): if(kwargs['report'].endswith('/')): kwargs['report'] = kwargs['report'][0:kwargs['report'].rfind('/')] dt_cfg.dovetail_config['report_dest'] = kwargs['report'] + dt_cfg.update_cmds() if kwargs['offline']: dt_cfg.dovetail_config['offline'] = True diff --git a/dovetail/test_runner.py b/dovetail/test_runner.py index b3fd7a3f..603156fe 100644 --- a/dovetail/test_runner.py +++ b/dovetail/test_runner.py @@ -111,7 +111,7 @@ class DockerRunner(object): ret, msg = Container.exec_cmd(container_id, cmd) self.testcase.cleaned(True) - Container.clean(container_id) + Container.clean(container_id, self.type) class FunctestRunner(DockerRunner): @@ -128,6 +128,13 @@ class YardstickRunner(DockerRunner): super(YardstickRunner, self).__init__(testcase) +class BottlenecksRunner(DockerRunner): + + def __init__(self, testcase): + self.type = 'bottlenecks' + super(BottlenecksRunner, self).__init__(testcase) + + class ShellRunner(object): logger = None @@ -190,6 +197,7 @@ class TestRunnerFactory(object): TEST_RUNNER_MAP = { "functest": FunctestRunner, "yardstick": YardstickRunner, + "bottlenecks": BottlenecksRunner, "shell": ShellRunner, } diff --git a/dovetail/testcase.py b/dovetail/testcase.py index 7b012c88..bdfd3d35 100644 --- a/dovetail/testcase.py +++ b/dovetail/testcase.py @@ -254,13 +254,30 @@ class FunctestTestcase(Testcase): class YardstickTestcase(Testcase): - validate_testcae_list = {} + validate_testcase_list = {} def __init__(self, testcase_yaml): super(YardstickTestcase, self).__init__(testcase_yaml) self.type = 'yardstick' +class BottlenecksTestcase(Testcase): + + validate_testcase_list = {} + + def __init__(self, testcase_yaml): + super(BottlenecksTestcase, self).__init__(testcase_yaml) + self.type = 'bottlenecks' + self._update_cmds() + + def _update_cmds(self): + if dt_cfg.dovetail_config['report_dest'].startswith("http"): + try: + self.testcase['validate']['cmds'][0] += ' --report' + except KeyError: + return + + class ShellTestcase(Testcase): validate_testcase_list = {} @@ -274,6 +291,7 @@ class TestcaseFactory(object): TESTCASE_TYPE_MAP = { 'functest': FunctestTestcase, 'yardstick': YardstickTestcase, + 'bottlenecks': BottlenecksTestcase, 'shell': ShellTestcase, } diff --git a/dovetail/testcase/nfvi.tc101.yml b/dovetail/testcase/nfvi.tc101.yml deleted file mode 100644 index 2fa47dee..00000000 --- a/dovetail/testcase/nfvi.tc101.yml +++ /dev/null @@ -1,9 +0,0 @@ ---- -dovetail.nfvi.tc101: - name: dovetail.nfvi.tc101 - objective: measure number of cores and threads, available memory size and cache size - validate: - type: yardstick - testcase: opnfv_yardstick_tc001 - report: - sub_testcase_list: diff --git a/dovetail/testcase/nfvi.tc102.yml b/dovetail/testcase/nfvi.tc102.yml deleted file mode 100644 index 6da52243..00000000 --- a/dovetail/testcase/nfvi.tc102.yml +++ /dev/null @@ -1,9 +0,0 @@ ---- -dovetail.nfvi.tc102: - name: dovetail.nfvi.tc102 - objective: measure number of cores and threads, available memory size and cache size - validate: - type: yardstick - testcase: opnfv_yardstick_tc002 - report: - sub_testcase_list: diff --git a/dovetail/testcase/resiliency.tc001.yml b/dovetail/testcase/resiliency.tc001.yml new file mode 100644 index 00000000..86399849 --- /dev/null +++ b/dovetail/testcase/resiliency.tc001.yml @@ -0,0 +1,11 @@ +--- +dovetail.resiliency.tc001: + name: dovetail.resiliency.tc001 + objective: > # This test case verifies the ability of the SUT setting up VM pairs + # for different tenants and providing acceptable capacity after the amount of + # VM pairs reaches certain quantity. + validate: + type: bottlenecks + testcase: posca_factor_ping + report: + sub_testcase_list: diff --git a/dovetail/testcase/nfvi.tc002.yml b/dovetail/testcase/vping.tc001.yml index 28a7adfd..e3b501be 100644 --- a/dovetail/testcase/nfvi.tc002.yml +++ b/dovetail/testcase/vping.tc001.yml @@ -1,6 +1,6 @@ --- -dovetail.nfvi.tc002: - name: dovetail.nfvi.tc002 +dovetail.vping.tc001: + name: dovetail.vping.tc001 objective: testing for vping using userdata validate: type: functest diff --git a/dovetail/testcase/nfvi.tc001.yml b/dovetail/testcase/vping.tc002.yml index 157fc8da..a800a4db 100644 --- a/dovetail/testcase/nfvi.tc001.yml +++ b/dovetail/testcase/vping.tc002.yml @@ -1,6 +1,6 @@ --- -dovetail.nfvi.tc001: - name: dovetail.nfvi.tc001 +dovetail.vping.tc002: + name: dovetail.vping.tc002 objective: testing for vping using ssh validate: type: functest diff --git a/dovetail/utils/dovetail_config.py b/dovetail/utils/dovetail_config.py index f8193e5d..5e100d69 100644 --- a/dovetail/utils/dovetail_config.py +++ b/dovetail/utils/dovetail_config.py @@ -54,3 +54,8 @@ class DovetailConfig(object): def update_non_envs(cls, path, value): if value: cls.set_leaf_dict(cls.dovetail_config, path, value) + + @classmethod + def update_cmds(cls): + if cls.dovetail_config['report_dest'].startswith("http"): + cls.dovetail_config['bottlenecks']['cmds'][0] += ' --report' diff --git a/dovetail/utils/local_db/launch_db.sh b/dovetail/utils/local_db/launch_db.sh index 722dc227..38a97331 100755 --- a/dovetail/utils/local_db/launch_db.sh +++ b/dovetail/utils/local_db/launch_db.sh @@ -59,8 +59,8 @@ echo "Create the testapi service." echo "==========================" set +e -# pull image opnfv/testapi:cvp.0.3.0 -testapi_img="opnfv/testapi:cvp.0.3.0" +# pull image opnfv/testapi:cvp.0.4.0 +testapi_img="opnfv/testapi:cvp.0.4.0" echo "Step1: pull the image $testapi_img." sudo docker pull $testapi_img set -e diff --git a/dovetail/utils/local_db/restart_db.sh b/dovetail/utils/local_db/restart_db.sh index 35096b99..668b4b84 100755 --- a/dovetail/utils/local_db/restart_db.sh +++ b/dovetail/utils/local_db/restart_db.sh @@ -26,4 +26,4 @@ export db_host_ip=${db_host_ip:-"$1"} sudo docker rm -f testapi sudo docker run -itd -p $testapi_port:8000 --name testapi \ - -e mongodb_url=mongodb://$db_host_ip:$mongodb_port/ opnfv/testapi:cvp.0.3.0 + -e mongodb_url=mongodb://$db_host_ip:$mongodb_port/ opnfv/testapi:cvp.0.4.0 diff --git a/dovetail/utils/offline/config.yaml b/dovetail/utils/offline/config.yaml index 11134dd1..bebd6f4a 100644 --- a/dovetail/utils/offline/config.yaml +++ b/dovetail/utils/offline/config.yaml @@ -2,7 +2,7 @@ docker_images: dovetail: domain: opnfv - tag: cvp.0.3.0 + tag: cvp.0.4.0 store_name: image_dovetail.docker functest: domain: opnfv @@ -12,9 +12,13 @@ docker_images: domain: opnfv tag: danube.3.1 store_name: image_yardstick.docker + bottlenecks: + domain: opnfv + tag: cvp.0.4.0 + store_name: image_bottlenecks.docker testapi: domain: opnfv - tag: cvp.0.3.0 + tag: cvp.0.4.0 store_name: image_testapi.docker mongo: tag: 3.2.1 @@ -1,6 +1,6 @@ [metadata] name = dovetail -version = 0.3.0 +version = 0.4.0 home-page = https://wiki.opnfv.org/display/dovetail [entry_points] |