diff options
22 files changed, 112 insertions, 139 deletions
diff --git a/cvp/3rd_party/static/testapi-ui/components/home/home.html b/cvp/3rd_party/static/testapi-ui/components/home/home.html index 8b97bd64..521c972a 100644 --- a/cvp/3rd_party/static/testapi-ui/components/home/home.html +++ b/cvp/3rd_party/static/testapi-ui/components/home/home.html @@ -1,74 +1,6 @@ <div class="row"> <div class="col-lg-12" style="font-size:15px"> - <h1>CVP Workflow</h1> - <hr> - <h2>Step 1: Applying</h2> - <br> - <p>A tester should start the process by completing an <a ng-click="ctrl.gotoApplication()">application</a>.</p> - - <!-- <ul> - <li>Organization name</li> - <li>Organization website (if public)</li> - <li>Product name and identifier</li> - <li>Product public documentation</li> - <li>Product categories, choose one: (i) software and hardware (ii) software and third party hardware (please specify)</li> - <li>Primary contact name, business email, postal address and phone number Only the primary email address should be used for official communication with OPNFV CVP.</li> - <li>User ID for CVP web portal The CVP web portal supports (i) Linux Foundation user ID, and (ii) Openstack user ID, in the current release. Specify either one as user ID for the CVP web portal. If a new user ID is needed, visit https://identity.linuxfoundation.org, or https://www.openstack.org/join/register.</li> - <li>Location where the verification testing is to be conducted. Choose one: (internal vendor lab, third-party lab)</li> - <li>If the test is to be conducted by a third-party lab, please specify name and contact information of the third-party lab.</li> - </ul> - - <p>Please email the completed application using the primary contact email account in order to establish identity.</p> - --> - - <p>Once the application information is received and in order, an email response will be sent to the primary contact with confirmation and next step information.</p> - - <!-- - <p>[Editor'note: No fee has been established at this time for CVP applications. Recommend we skip fee for the initial release of CVP.]</p> - --> - - <hr> - <h2>Step 2: Testing</h2> - <p>The following documents guide testers to prepare the test environment and run tests:</p> - - <ul> - <li><a href="http://artifacts.opnfv.org/dovetail/docs/testing_user_systempreparation/index.html">System Requirements and Preparation Guide for OPNFV CVP </a></li> - <li><a href="http://artifacts.opnfv.org/dovetail/docs/testing_developer_testscope/index.html">OPNFV CVP Teslt Case Specification </a></li> + <li><a href="http://artifacts.opnfv.org/dovetail/docs/testing_user_certificationworkflow/index.html">OPNFV Compliance Verification Program certification workflow </a></li> <li><a href="http://artifacts.opnfv.org/dovetail/docs/testing_user_userguide/index.html">Dovetail Test Tool User Guide </a></li> </ul> - <!-- - <p>[Editor's note: The above three documents are still work in progress. Names and references will be corrected when they are merged.]</p> - - <p>A unique Test ID is generated by the Dovetail tool for each test run. Please take a note of this ID for future reference.</p> - --> - - <hr> - <h2>Step 3: Submitting Test Results</h2> - <p>Testers can login to the CVP web portal <a href="https://cvp.opnfv.org">https://cvp.opnfv.org</a> with the LF ID or Openstack ID and upload the test results. By default, the results are visible only to the user who uploaded the data.</p> - - <!-- - <p>[Editor's note: the web portable URL is a placeholder and will be replaced once the actual URL is implemented.]</p> - --> - - <p>Testers can self-review the test results through the portal until they are ready to ask for CVP review. They may also update with or add new test results as needed.</p> - - <p>Once the tester is satisfied with the test result, the tester grants access to the test result for CVP review via the portal. The test result is identified by the unique Test ID.</p> - - <p>Notify via the official CVP email address that a review request has been made and reference the Test ID. This will alert the C&C Committee to start the CVP review process.</p> - - <hr> - <h2>Step 4: CVP Review</h2> - <p>Upon receiving the email notification and the Test ID, the C&C Committee conducts a peer based review of the test result. Persons employed by the same organization that submitted the test results or by affiliated organizations should not be part of the reviewers.</p> - - <p>The primary contact may be asked via email for any missing information or clarification of the application. The reviewers will make a determination and recommend to the C&C Committee. Normally, the outcome of the review should be communicated to the tester within 10 business days after all required information is in order.</p> - - <p>If an application is denied, an appeal can be made to the C&C Committee or ultimately to the Board of Directors of OPNFV.</p> - - <hr> - <h2>Step 5: Grant of Use of Logo</h2> - <p>If an application is approved, further information will be communicated to the tester on the guidelines of using OPNFV logos for promotional purposes.</p> - <hr> - </div> -</div> - diff --git a/cvp/3rd_party/static/testapi-ui/favicon.ico b/cvp/3rd_party/static/testapi-ui/favicon.ico Binary files differindex 156019aa..bbe55ab4 100644 --- a/cvp/3rd_party/static/testapi-ui/favicon.ico +++ b/cvp/3rd_party/static/testapi-ui/favicon.ico diff --git a/cvp/3rd_party/static/testapi-ui/index.html b/cvp/3rd_party/static/testapi-ui/index.html index c49276ce..1773ff08 100644 --- a/cvp/3rd_party/static/testapi-ui/index.html +++ b/cvp/3rd_party/static/testapi-ui/index.html @@ -21,8 +21,8 @@ <meta name="viewport" content="width=device-width"> <title>CVP</title> - <link rel="icon" type="image/png" href="https://www.opnfv.org/favicon.ico" sizes="16x16" /> - <link rel="icon" type="image/png" href="https://www.opnfv.org/favicon.ico" sizes="32x32" /> + <link rel="icon" type="image/png" href="testapi-ui/favicon.ico" sizes="16x16" /> + <link rel="icon" type="image/png" href="testapi-ui/favicon.ico" sizes="32x32" /> <link rel="stylesheet" href="testapi-ui/assets/lib/bootstrap/dist/css/bootstrap.min.css" /> <link rel="stylesheet" href="testapi-ui/assets/lib/angular-busy/dist/angular-busy.min.css" /> diff --git a/dovetail/compliance/proposed_tests.yml b/dovetail/compliance/proposed_tests.yml index 83b97d53..15fa7ba9 100644 --- a/dovetail/compliance/proposed_tests.yml +++ b/dovetail/compliance/proposed_tests.yml @@ -2,11 +2,22 @@ proposed_tests: name: proposed_tests testcases_list: + # mandatory test cases # osinterop - dovetail.osinterop.tc001 # vping - dovetail.vping.tc001 - dovetail.vping.tc002 + # HA + - dovetail.ha.tc001 + - dovetail.ha.tc002 + - dovetail.ha.tc003 + - dovetail.ha.tc004 + - dovetail.ha.tc005 + - dovetail.ha.tc006 + - dovetail.ha.tc007 + - dovetail.ha.tc008 + # optional test cases # ipv6 - dovetail.ipv6.tc001 - dovetail.ipv6.tc002 @@ -33,15 +44,6 @@ proposed_tests: - dovetail.ipv6.tc023 - dovetail.ipv6.tc024 - dovetail.ipv6.tc025 - # HA - - dovetail.ha.tc001 - - dovetail.ha.tc002 - - dovetail.ha.tc003 - - dovetail.ha.tc004 - - dovetail.ha.tc005 - - dovetail.ha.tc006 - - dovetail.ha.tc007 - - dovetail.ha.tc008 # sdnvpn - dovetail.sdnvpn.tc001 - dovetail.sdnvpn.tc002 @@ -53,5 +55,6 @@ proposed_tests: - dovetail.tempest.tc003 - dovetail.tempest.tc004 - dovetail.tempest.tc005 + # other test cases # resiliency # - dovetail.resiliency.tc001 diff --git a/dovetail/conf/cmd_config.yml b/dovetail/conf/cmd_config.yml index 58409f2b..470c7108 100644 --- a/dovetail/conf/cmd_config.yml +++ b/dovetail/conf/cmd_config.yml @@ -45,7 +45,7 @@ cli: testarea: flags: - '--testarea' - default: 'full' + multiple: 'True' help: 'compliance testarea within testsuite' debug: flags: diff --git a/dovetail/conf/dovetail_config.yml b/dovetail/conf/dovetail_config.yml index e93bd64a..b95c6d37 100644 --- a/dovetail/conf/dovetail_config.yml +++ b/dovetail/conf/dovetail_config.yml @@ -37,6 +37,9 @@ testarea_supported: - vping - resiliency - tempest + - optional + - mandatory + - full functest_testsuite: - refstack_defcore @@ -93,3 +96,13 @@ validate_input: - 'cvp.0.4.0' - 'cvp.0.5.0' - 'cvp.0.6.0' + +mandatory: + - osinterop + - ha + - vping + +optional: + - ipv6 + - tempest + - sdnvpn diff --git a/dovetail/container.py b/dovetail/container.py index ff92fa4a..e8840de3 100644 --- a/dovetail/container.py +++ b/dovetail/container.py @@ -68,16 +68,20 @@ class Container(object): # These are all just used by Functest's function push_results_to_db. # And has nothing to do with DoveTail running test cases. - ins_type = os.getenv('INSTALLER_TYPE', "unknown") - scenario = os.getenv('DEPLOY_SCENARIO', "unknown") + ins_type = "unknown" + scenario = "unknown" ins_type = ''.join([" -e INSTALLER_TYPE=", ins_type]) scenario = ''.join([" -e DEPLOY_SCENARIO=", scenario]) + ins_ip = os.getenv('INSTALLER_IP', "192.168.0.0") + ins_ip = " -e INSTALLER_IP={}".format(ins_ip) # vpn testcase only runs when scenario name includes bgpvpn # functest requirements if 'sdnvpn' in testcase_name: - ins_type = "-e INSTALLER_TYPE=netvirt" - scenario = " -e DEPLOY_SCENARIO=bgpvpn" - envs = "%s %s" % (ins_type, scenario) + ins_type = os.getenv('INSTALLER_TYPE', "netvirt") + ins_type = " -e INSTALLER_TYPE={}".format(ins_type) + scenario = os.getenv('DEPLOY_SCENARIO', "bgpvpn") + scenario = " -e DEPLOY_SCENARIO={}".format(scenario) + envs = "%s %s %s" % (ins_type, scenario, ins_ip) dovetail_config = dt_cfg.dovetail_config if dovetail_config['report_dest'].startswith("http"): diff --git a/dovetail/report.py b/dovetail/report.py index 535e5a25..41f14b53 100644 --- a/dovetail/report.py +++ b/dovetail/report.py @@ -55,11 +55,8 @@ class Report(object): report_obj['duration'] = duration report_obj['testcases_list'] = [] - testarea_list = [] - for value in testsuite_yaml['testcases_list']: - if value is not None and (testarea == 'full' or testarea in value): - testarea_list.append(value) - for testcase_name in testarea_list: + testcase_list = Testcase.get_testcase_list(testsuite_yaml, testarea) + for testcase_name in testcase_list: testcase = Testcase.get(testcase_name) testcase_inreport = {} testcase_inreport['name'] = testcase_name @@ -266,7 +263,8 @@ class FunctestCrawler(object): for jsonfile in f: try: data = json.loads(jsonfile) - if testcase_name == data['case_name'] and \ + if (testcase_name == data['case_name'] or + data['project_name'] == "sdnvpn") and \ build_tag == data['build_tag']: criteria = data['criteria'] timestart = data['start_date'] diff --git a/dovetail/run.py b/dovetail/run.py index 7de4c814..0306efe4 100755 --- a/dovetail/run.py +++ b/dovetail/run.py @@ -38,14 +38,10 @@ def load_testcase(): def run_test(testsuite, testarea, logger): - testarea_list = [] - for value in testsuite['testcases_list']: - if value is not None and (testarea == 'full' or testarea in value): - testarea_list.append(value) - + testcase_list = Testcase.get_testcase_list(testsuite, testarea) duration = 0 start_time = time.time() - for testcase_name in testarea_list: + for testcase_name in testcase_list: logger.info('>>[testcase]: {}'.format(testcase_name)) testcase = Testcase.get(testcase_name) if testcase is None: @@ -284,14 +280,11 @@ def main(*args, **kwargs): else: dt_cfg.dovetail_config['offline'] = False - testarea = kwargs['testarea'] + origin_testarea = kwargs['testarea'] testsuite_validation = False - testarea_validation = False - if (testarea == 'full') or \ - (testarea in dt_cfg.dovetail_config['testarea_supported']): - testarea_validation = True if kwargs['testsuite'] in dt_cfg.dovetail_config['testsuite_supported']: testsuite_validation = True + testarea_validation, testarea = Testcase.check_testarea(origin_testarea) if testsuite_validation and testarea_validation: testsuite_yaml = load_testsuite(kwargs['testsuite']) load_testcase() @@ -302,7 +295,7 @@ def main(*args, **kwargs): Report.save_logs() else: logger.error('Invalid input commands, testsuite {} testarea {}' - .format(kwargs['testsuite'], testarea)) + .format(kwargs['testsuite'], origin_testarea)) dt_cfg.load_config_files() diff --git a/dovetail/testcase.py b/dovetail/testcase.py index b6819964..99845484 100644 --- a/dovetail/testcase.py +++ b/dovetail/testcase.py @@ -243,6 +243,44 @@ class Testcase(object): return cls.testcase_list[testcase_name] return None + @classmethod + def check_testarea(cls, testarea): + area_no_duplicate = [] + area_full = ['full'] + + # testarea is empty, run full testsuite + if not testarea: + return True, area_full + + mandatory_list = dt_cfg.dovetail_config['mandatory'] + optional_list = dt_cfg.dovetail_config['optional'] + for area in testarea: + if area not in dt_cfg.dovetail_config['testarea_supported']: + return False, None + if area == 'full': + return True, area_full + if area == 'mandatory': + for mandatory_area in mandatory_list: + area_no_duplicate.append(mandatory_area) + continue + if area == 'optional': + for optional_area in optional_list: + area_no_duplicate.append(optional_area) + continue + area_no_duplicate.append(area) + area_no_duplicate = list(set(area_no_duplicate)) + return True, area_no_duplicate + + @classmethod + def get_testcase_list(cls, testsuite, testarea): + testcase_list = [] + for value in testsuite['testcases_list']: + for area in testarea: + if value is not None and (area == 'full' or area in value): + testcase_list.append(value) + break + return testcase_list + class FunctestTestcase(Testcase): diff --git a/dovetail/testcase/sdnvpn.tc001.yml b/dovetail/testcase/sdnvpn.tc001.yml index 9ab3d445..786892be 100644 --- a/dovetail/testcase/sdnvpn.tc001.yml +++ b/dovetail/testcase/sdnvpn.tc001.yml @@ -5,6 +5,9 @@ dovetail.sdnvpn.tc001: validate: type: functest testcase: bgpvpn + pre_condition: + - "sed -i 's/testcase_1/sdnvpn.test.functest.testcase_1/' /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/testcase_1.py" + - "sed -i 's/run_sdnvpn_tests.py/run_sdnvpn_tests.py -r/' /usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml" pre_copy: exist_src_file: sdnvpn_config_testcase1.yaml dest_path: /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/config.yaml diff --git a/dovetail/testcase/sdnvpn.tc002.yml b/dovetail/testcase/sdnvpn.tc002.yml index a5c70ba9..900fb461 100644 --- a/dovetail/testcase/sdnvpn.tc002.yml +++ b/dovetail/testcase/sdnvpn.tc002.yml @@ -5,6 +5,9 @@ dovetail.sdnvpn.tc002: validate: type: functest testcase: bgpvpn + pre_condition: + - "sed -i 's/testcase_2/sdnvpn.test.functest.testcase_2/' /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/testcase_2.py" + - "sed -i 's/run_sdnvpn_tests.py/run_sdnvpn_tests.py -r/' /usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml" pre_copy: exist_src_file: sdnvpn_config_testcase2.yaml dest_path: /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/config.yaml diff --git a/dovetail/testcase/sdnvpn.tc003.yml b/dovetail/testcase/sdnvpn.tc003.yml index c8c8b2be..8d28b57c 100644 --- a/dovetail/testcase/sdnvpn.tc003.yml +++ b/dovetail/testcase/sdnvpn.tc003.yml @@ -5,6 +5,9 @@ dovetail.sdnvpn.tc003: validate: type: functest testcase: bgpvpn + pre_condition: + - "sed -i 's/testcase_3/sdnvpn.test.functest.testcase_3/' /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/testcase_3.py" + - "sed -i 's/run_sdnvpn_tests.py/run_sdnvpn_tests.py -r/' /usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml" pre_copy: exist_src_file: sdnvpn_config_testcase3.yaml dest_path: /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/config.yaml diff --git a/dovetail/testcase/sdnvpn.tc004.yml b/dovetail/testcase/sdnvpn.tc004.yml index f6a4a6ff..236c39bd 100644 --- a/dovetail/testcase/sdnvpn.tc004.yml +++ b/dovetail/testcase/sdnvpn.tc004.yml @@ -5,6 +5,9 @@ dovetail.sdnvpn.tc004: validate: type: functest testcase: bgpvpn + pre_condition: + - "sed -i 's/testcase_4/sdnvpn.test.functest.testcase_4/' /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/testcase_4.py" + - "sed -i 's/run_sdnvpn_tests.py/run_sdnvpn_tests.py -r/' /usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml" pre_copy: exist_src_file: sdnvpn_config_testcase4.yaml dest_path: /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/config.yaml diff --git a/dovetail/testcase/sdnvpn.tc008.yml b/dovetail/testcase/sdnvpn.tc008.yml index 5713e102..a4d82316 100644 --- a/dovetail/testcase/sdnvpn.tc008.yml +++ b/dovetail/testcase/sdnvpn.tc008.yml @@ -5,6 +5,9 @@ dovetail.sdnvpn.tc008: validate: type: functest testcase: bgpvpn + pre_condition: + - "sed -i 's/testcase_8/sdnvpn.test.functest.testcase_8/' /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/testcase_8.py" + - "sed -i 's/run_sdnvpn_tests.py/run_sdnvpn_tests.py -r/' /usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml" pre_copy: exist_src_file: sdnvpn_config_testcase8.yaml dest_path: /usr/local/lib/python2.7/dist-packages/sdnvpn/test/functest/config.yaml diff --git a/dovetail/userconfig/sdnvpn_config_testcase1.yaml b/dovetail/userconfig/sdnvpn_config_testcase1.yaml index d9e4a1b4..5736dcff 100644 --- a/dovetail/userconfig/sdnvpn_config_testcase1.yaml +++ b/dovetail/userconfig/sdnvpn_config_testcase1.yaml @@ -2,13 +2,7 @@ defaults: flavor: m1.tiny # adapt to your environment testcases: - tempest: - enabled: true - description: Neutron BGPVPN tests in tempest - testname_db: functest_tempest # declared name in the test api - # http://testresults.opnfv.org/test/api/v1/projects/sdnvpn/cases - - testcase_1: + sdnvpn.test.functest.testcase_1: enabled: true description: VPN provides connectivity between subnets testname_db: functest_testcase_1 diff --git a/dovetail/userconfig/sdnvpn_config_testcase2.yaml b/dovetail/userconfig/sdnvpn_config_testcase2.yaml index 07b0adfa..2be3bdb5 100644 --- a/dovetail/userconfig/sdnvpn_config_testcase2.yaml +++ b/dovetail/userconfig/sdnvpn_config_testcase2.yaml @@ -2,13 +2,7 @@ defaults: flavor: m1.tiny # adapt to your environment testcases: - tempest: - enabled: true - description: Neutron BGPVPN tests in tempest - testname_db: functest_tempest # declared name in the test api - # http://testresults.opnfv.org/test/api/v1/projects/sdnvpn/cases - - testcase_2: + sdnvpn.test.functest.testcase_2: enabled: true description: Tenant separation testname_db: functest_testcase_2 diff --git a/dovetail/userconfig/sdnvpn_config_testcase3.yaml b/dovetail/userconfig/sdnvpn_config_testcase3.yaml index 60592cbb..c1ffc13b 100644 --- a/dovetail/userconfig/sdnvpn_config_testcase3.yaml +++ b/dovetail/userconfig/sdnvpn_config_testcase3.yaml @@ -2,13 +2,7 @@ defaults: flavor: m1.tiny # adapt to your environment testcases: - tempest: - enabled: true - description: Neutron BGPVPN tests in tempest - testname_db: functest_tempest # declared name in the test api - # http://testresults.opnfv.org/test/api/v1/projects/sdnvpn/cases - - testcase_3: + sdnvpn.test.functest.testcase_3: enabled: true description: Data center gateway integration testname_db: functest_testcase_3 diff --git a/dovetail/userconfig/sdnvpn_config_testcase4.yaml b/dovetail/userconfig/sdnvpn_config_testcase4.yaml index 1e221354..f3f5a56b 100644 --- a/dovetail/userconfig/sdnvpn_config_testcase4.yaml +++ b/dovetail/userconfig/sdnvpn_config_testcase4.yaml @@ -2,13 +2,7 @@ defaults: flavor: m1.tiny # adapt to your environment testcases: - tempest: - enabled: true - description: Neutron BGPVPN tests in tempest - testname_db: functest_tempest # declared name in the test api - # http://testresults.opnfv.org/test/api/v1/projects/sdnvpn/cases - - testcase_4: + sdnvpn.test.functest.testcase_4: enabled: true description: VPN provides connectivity between subnets using router association testname_db: functest_testcase_4 diff --git a/dovetail/userconfig/sdnvpn_config_testcase8.yaml b/dovetail/userconfig/sdnvpn_config_testcase8.yaml index c825997b..8712122b 100644 --- a/dovetail/userconfig/sdnvpn_config_testcase8.yaml +++ b/dovetail/userconfig/sdnvpn_config_testcase8.yaml @@ -2,13 +2,7 @@ defaults: flavor: m1.tiny # adapt to your environment testcases: - tempest: - enabled: true - description: Neutron BGPVPN tests in tempest - testname_db: functest_tempest # declared name in the test api - # http://testresults.opnfv.org/test/api/v1/projects/sdnvpn/cases - - testcase_8: + sdnvpn.test.functest.testcase_8: enabled: true description: Test floating IP and router assoc coexistence testname_db: functest_testcase_8 diff --git a/dovetail/utils/local_db/cases.json b/dovetail/utils/local_db/cases.json index f619c567..1253067c 100644 --- a/dovetail/utils/local_db/cases.json +++ b/dovetail/utils/local_db/cases.json @@ -1,3 +1,4 @@ {"testcases": [{"project_name": "functest", "run": {"class": "VPingSSH", "module": "functest.opnfv_tests.openstack.vping.vping_ssh"}, "description": "NFV \u201cHello World\u201d using SSH connection and floatting IP", "ci_loop": "daily", "tags": "ping", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "55e05dba514bc52914149f2e", "catalog_description": "vPing using SSH", "creation_date": "2015-08-28 13:10:18.478986", "domains": null, "dependencies": {"installer": "", "scenario": "^((?!bgpvpn|odl_l3).)*$"}, "version": ">arno.1.0", "criteria": null, "tier": "healthcheck", "trust": "silver", "blocking": true, "name": "vping_ssh"}, {"project_name": "functest", "run": {"class": "", "module": ""}, "description": "This test consists in running OpenStack Tempest fonctional suite.Initially based on the smoke suite, functest aims to define its own subset of tests", "ci_loop": "daily", "tags": "tempest,api,interfaces,openstack,vim", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "55effe6f514bc52b1791f966", "catalog_description": "OpenStack Tempest (smoke)", "creation_date": "2015-09-09 09:39:59.661046", "domains": "vim", "dependencies": {"installer": "", "scenario": ""}, "version": ">arno.1.0", "criteria": null, "tier": "smoke", "trust": "silver", "blocking": true, "name": "tempest_smoke_serial"}, {"project_name": "functest", "run": {"class": "ODLTests", "module": "functest.opnfv_tests.sdn.odl.odl"}, "description": "This test consists in creating, updating and deleting, networks, ports, subnets, routers in ODL and Neutron", "ci_loop": "daily", "tags": "odl,sdn", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "56169c4e514bc52b17f4adaa", "catalog_description": "OpenDaylight", "creation_date": "2015-10-08 16:39:42.727060", "domains": "networking", "dependencies": {"installer": "", "scenario": "odl"}, "version": ">arno.1.0", "criteria": null, "tier": "smoke", "trust": "silver", "blocking": true, "name": "odl"}, {"project_name": "functest", "run": {"class": "", "module": ""}, "description": "Tests of OpenStack performed with Rally framework. These tests aim to qualify OpenStack modules and their associated APIs", "ci_loop": "daily", "tags": "rally,api,interfaces,openstack,vim", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "561e504c514bc5355ba8b3a8", "catalog_description": "OpenStack Rally (sanity)", "creation_date": "2015-10-14 12:53:32.126750", "domains": "vim", "dependencies": {"installer": "", "scenario": "^((?!bgpvpn).)*$"}, "version": ">arno.1.0", "criteria": null, "tier": "smoke", "trust": "silver", "blocking": true, "name": "rally_sanity"}, {"project_name": "functest", "run": {"class": "", "module": ""}, "description": "Tests of the ONOS Virtual Network Northbound API", "ci_loop": "daily", "tags": "onos,sdn", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "565d6579514bc5087f2ddd0b", "catalog_description": "ONOS", "creation_date": "2015-12-01 09:16:41.343670", "domains": "networking", "dependencies": {"installer": "", "scenario": "onos"}, "version": ">brahmaputra.1.0", "criteria": null, "tier": "smoke", "trust": "silver", "blocking": true, "name": "onos"}, {"project_name": "functest", "run": {"class": "VPingUserdata", "module": "functest.opnfv_tests.openstack.vping.vping_userdata"}, "description": "vPing using userdata and cloudinit mechanisms", "ci_loop": "daily", "tags": "ping", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "569f60cf514bc541f885b206", "catalog_description": "vPing using userdata", "creation_date": "2016-01-20 10:26:23.160020", "domains": "", "dependencies": {"installer": "", "scenario": "^((?!lxd).)*$"}, "version": ">brahmaputra.1.0", "criteria": null, "tier": "smoke", "trust": "silver", "blocking": true, "name": "vping_userdata"}, {"project_name": "functest", "run": {"class": "", "module": ""}, "description": "Rally full extended suite", "ci_loop": "weekly", "tags": "openstack,vim", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "573c83ef9377c5287772dc81", "catalog_description": "OpenStack Rally (full)", "creation_date": "2016-05-18 15:02:07.153735", "domains": "vim", "dependencies": {"installer": "", "scenario": ""}, "version": ">colorado.1.0", "criteria": {"step1": "success_rate >=90"}, "tier": "components", "trust": "silver", "blocking": false, "name": "rally_full"}, {"project_name": "functest", "run": {"class": "", "module": ""}, "description": "Tempest full using parallel option", "ci_loop": "weekly", "tags": "openstack,vim,tempest", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "573c840d9377c5287772dc82", "catalog_description": "OpenStack Tempest (full)", "creation_date": "2016-05-18 15:02:37.517083", "domains": "vim", "dependencies": {"installer": "", "scenario": ""}, "version": ">colorado.1.0", "criteria": {"step1": "success_rate >=80"}, "tier": "components", "trust": "silver", "blocking": false, "name": "tempest_full_parallel"}, {"project_name": "functest", "run": {"class": "", "module": ""}, "description": "SFC tests for ONOS based scenarios", "ci_loop": "daily", "tags": "sfc,service", "url": "http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html", "_id": "578502b89377c54b278bbfe3", "catalog_description": "ONOS Service Function Chaining", "creation_date": "2016-07-12 14:46:16.149806", "domains": "networking", "dependencies": {"installer": "", "scenario": "onos-sfc"}, "version": ">colorado.1.0", "criteria": null, "tier": "feature", "trust": "silver", "blocking": true, "name": "onos_sfc"}, {"project_name": "functest", "run": {"class": "ConnectionCheck", "module": "functest.opnfv_tests.openstack.snaps.connection_check"}, "description": "This test case verifies the retrieval of OpenStack clients: Keystone, Glance, Neutron and Nova and may perform some simple queries. When the config value of snaps. use_keystone is True, functest must have access to the cloud's private network.", "ci_loop": "daily", "tags": "snaps,openstack,vim", "url": "http://artifacts.opnfv.org/functest/danube/docs/userguide/index.html", "_id": "584a5d4c1d2c6e000ab2d7e4", "catalog_description": "Healthcheck OpenStack connectivity", "creation_date": "2016-12-09 07:29:16.270583", "domains": "vim", "dependencies": {"installer": "", "scenario": ""}, "version": ">danube.1.0", "criteria": null, "tier": "healthcheck", "trust": "silver", "blocking": true, "name": "connection_check"}, {"project_name": "functest", "run": {"class": "ApiCheck", "module": "functest.opnfv_tests.openstack.snaps.api_check"}, "description": "This test case verifies the retrieval of OpenStack clients:Keystone, Glance, Neutron and Nova and may perform some simple queries. When the config value of snaps. use_keystone is True, functest must have access to the cloud's private network.", "ci_loop": "daily", "tags": "snaps,api,interfaces,openstack,vim", "url": "http://artifacts.opnfv.org/functest/danube/docs/userguide/index.html", "_id": "584a5e1d1d2c6e000ab2d7e5", "catalog_description": "Healthcheck OpenStack API", "creation_date": "2016-12-09 07:32:45.435360", "domains": "vim", "dependencies": {"installer": "", "scenario": ""}, "version": ">danube.1.0", "criteria": null, "tier": "healthcheck", "trust": "silver", "blocking": true, "name": "api_check"}, {"project_name": "functest", "run": {"class": "SnapsSmoke", "module": "functest.opnfv_tests.openstack.snaps.smoke"}, "description": "This test case contains tests that setup and destroy environments with VMs with and without Floating IPs with a newly created user and project. Set the config value snaps.use_floating_ips (True|False) to toggle this functionality. When the config value of snaps. use_keystone is True, functest must have access to the cloud's private network.", "ci_loop": "daily", "tags": "snaps,openstack,vim", "url": "http://artifacts.opnfv.org/functest/danube/docs/userguide/index.html", "_id": "584a5e731d2c6e000ab2d7e6", "catalog_description": "SNAPS (smoke)", "creation_date": "2016-12-09 07:34:11.982435", "domains": "vim", "dependencies": {"installer": "", "scenario": ""}, "version": ">danube.1.0", "criteria": null, "tier": "smoke", "trust": "silver", "blocking": true, "name": "snaps_smoke"}, {"project_name": "functest", "run": "daily", "description": "This test case creates executes the SimpleHealthCheck Python test class which creates an, image, flavor, network, and Cirros VM instance and observes the console output to validate the single port obtains the correct IP address.", "ci_loop": "daily", "tags": "dhcp", "url": "http://artifacts.opnfv.org/functest/danube/docs/userguide/index.html", "_id": "589c8f248cf551000c78211e", "catalog_description": "Healthcheck OpenStack (network)", "creation_date": "2017-02-09 15:47:48.739222", "domains": "vim", "dependencies": "", "version": ">danube", "criteria": "", "tier": "healthcheck", "trust": null, "blocking": true, "name": "snaps_health_check"}, {"project_name": "functest", "run": {"class": "AaaVnf", "module": "functest.opnfv_tests.vnf.aaa.aaa"}, "description": "AAA VNF sample", "ci_loop": "weekly", "tags": "aaa, freeradius, vnf", "url": "http://artifacts.opnfv.org/functest/danube/docs/userguide/index.html", "_id": "58a324e6584d06000b0157f2", "catalog_description": "VNF AAA", "creation_date": "2017-02-14 15:40:22.150387", "domains": "vnf", "dependencies": "", "version": ">Danube", "criteria": "", "tier": "vnf", "trust": null, "blocking": false, "name": "aaa"}, {"project_name": "functest", "run": {"class": "ImsVnf", "module": "functest.opnfv_tests.vnf.ims.cloudify_ims"}, "description": "Clearwater vIMS deployed with cloudify orchestrator", "ci_loop": "weekly", "tags": "vnf, ims", "url": "http://artifacts.opnfv.org/functest/danube/docs/userguide/index.html", "_id": "58a325e8584d06000b0157f5", "catalog_description": "VNF clearwater IMS with Cloudify orchestrator", "creation_date": "2017-02-14 15:44:40.235672", "domains": "vnf", "dependencies": "", "version": ">Danube", "criteria": "", "tier": "vnf", "trust": null, "blocking": false, "name": "cloudify_ims"}, {"project_name": "functest", "run": {"class": "ImsVnf", "module": "functest.opnfv_tests.vnf.ims.orchestra_ims"}, "description": "openIMS deployed with openBaton orchestrator", "ci_loop": "weekly", "tags": "vnf, ims", "url": "http://artifacts.opnfv.org/functest/danube/docs/userguide/index.html", "_id": "58a3263b584d06000b0157f6", "catalog_description": "VNF openIMS with Openbaton orchestrator", "creation_date": "2017-02-14 15:46:03.398028", "domains": "vnf", "dependencies": "", "version": ">Danube", "criteria": "", "tier": "vnf", "trust": null, "blocking": false, "name": "orchestra_openims"}, {"project_name": "functest", "run": "", "description": "Defcore List of tempest testcases", "ci_loop": "daily", "tags": "", "url": "", "_id": "58b438daf5a101000ac478ea", "catalog_description": "OpenStack Tempest (defcore)", "creation_date": "2017-02-27 14:34:02.219544", "domains": "", "dependencies": "", "version": "", "criteria": "", "tier": "components", "trust": null, "blocking": "false", "name": "tempest_defcore"}, {"project_name": "functest", "run": null, "description": "A sub group of tests of the OpenStack Defcore testcases by using refstack client", "ci_loop": "daily", "tags": null, "url": null, "_id": "58c91c64b91f23000a62ba0a", "catalog_description": "OpenStack RefStack", "creation_date": "2017-03-15 10:50:12.764115", "domains": null, "dependencies": null, "version": null, "criteria": null, "tier": "smoke", "trust": null, "blocking": "false", "name": "refstack_defcore"}, {"project_name": "functest", "run": {"class": "ImsVnf", "module": "functest.opnfv_tests.vnf.ims.opera_ims"}, "description": "Exemple of use of Open-O to deploy a VNF", "ci_loop": "weekly", "tags": "ims, open-o, onap, orchestration", "url": "", "_id": "58ca69abb91f23000a62baae", "catalog_description": "VNF clearwater IMS with Open-O orchestrator", "creation_date": "2017-03-16 10:32:11.203581", "domains": "orchestration, vnf", "dependencies": {"installer": "compass", "scenario": "os-nosdn-openo"}, "version": ">danube", "criteria": "", "tier": "vnf", "trust": null, "blocking": false, "name": "opera_ims"}, {"project_name": "functest", "run": {"class": "VrouterVnf", "module": "functest.opnfv_tests.vnf.router.vyos_vrouter"}, "description": "This test case is vRouter testing", "ci_loop": "", "tags": "vrouter, vnf", "url": "", "_id": "58ca6adeb91f23000a62baaf", "catalog_description": "VNF Vyatta vRouter with Cloudify orchestrator", "creation_date": "2017-03-16 10:37:18.293020", "domains": "network, vnf", "dependencies": {"installer": "fuel", "scenario": "nosdn-nofeature"}, "version": "", "criteria": "", "tier": "vnf", "trust": null, "blocking": "false", "name": "vyos_vrouter"}, {"project_name": "functest", "run": {"class": "ImsVnf", "module": "functest.opnfv_tests.vnf.ims.orchestra_ims"}, "description": "Clearwater vIMS deployed with Openbaton Mano orchestrator.", "ci_loop": "weekly", "tags": "vnf,ims", "url": "http://artifacts.opnfv.org/functest/euphrates/docs/userguide/index.html", "_id": "5922859a356833000a195187", "catalog_description": "VNF Clearwater IMS with OpenBaton orchestrator", "creation_date": "2017-05-22 06:30:50.512246", "domains": "mano,vnf", "dependencies": "", "version": ">euphrates", "criteria": 100, "tier": "vnf", "trust": null, "blocking": "false", "name": "orchestra_ims"}]} {"testcases": [{"project_name": "bottlenecks", "run": null, "description": "Stress tess on dataplane traffic using netperf.", "ci_loop": null, "tags": null, "url": "https://jira.opnfv.org/browse/BOTTLENECK-94", "_id": "58cf436a32c829000a11507e", "catalog_description": "Bandwidth Baseline", "creation_date": "2017-03-20 02:50:18.066854", "domains": null, "dependencies": null, "version": ">=danube.1.0", "criteria": null, "tier": null, "trust": null, "blocking": null, "name": "posca_factor_system_bandwidth"}, {"project_name": "bottlenecks", "run": null, "description": "Stress tess on performance life-cycle using ping.", "ci_loop": null, "tags": null, "url": "https://jira.opnfv.org/browse/BOTTLENECK-136", "_id": "58cf486332c829000a11508c", "catalog_description": "Ping Life-cycle Event", "creation_date": "2017-03-20 03:11:31.347177", "domains": null, "dependencies": null, "version": ">=danube.1.0", "criteria": null, "tier": null, "trust": null, "blocking": null, "name": "posca_factor_ping"}]} {"testcases": [{"project_name": "yardstick", "run": null, "description": "This is for ci to store test suites status", "ci_loop": "daily", "tags": null, "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/scenario_status.html", "_id": "592292b7356833000a195188", "catalog_description": "Not applicable (used for reporting)", "creation_date": "2017-05-22 07:26:47.523411", "domains": null, "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "scenario_status"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC001 config file; Measure network throughput using pktgen; Different amounts of flows are tested with, from 2 up to 1001000; All tests are run twice. First twice with the least amount of ports and further on.\n", "ci_loop": "daily", "tags": "pktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc001.html", "_id": "592292b8356833000a195189", "catalog_description": "network: throughput (pktgen)", "creation_date": "2017-05-22 07:26:48.714560", "domains": null, "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc001"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC002 config file; measure network latency using ping;\n", "ci_loop": "daily", "tags": "ping", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc002.html", "_id": "592292b9356833000a19518a", "catalog_description": "network: latency (ping)", "creation_date": "2017-05-22 07:26:49.820133", "domains": "network", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc002"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC005 config file; Measure Storage IOPS, throughput and latency using fio.\n", "ci_loop": "daily", "tags": "fio", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc005.html", "_id": "592292ba356833000a19518b", "catalog_description": "storage: disk IOPS, latency & throughput (fio)", "creation_date": "2017-05-22 07:26:50.927211", "domains": "storage", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc005"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC006 config file.\n", "ci_loop": "daily", "tags": "vtc_throughput", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc006.html", "_id": "592292bc356833000a19518c", "catalog_description": "vtc: data plane throughput", "creation_date": "2017-05-22 07:26:52.058010", "domains": "vtc", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc006"}, {"project_name": "yardstick", "run": null, "description": "Sample benchmark task config file; vTC.\n", "ci_loop": "daily", "tags": "vtc_throughput_noisy", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc007.html", "_id": "592292bd356833000a19518d", "catalog_description": "vtc: data plane throughput(Noisy neighbours)", "creation_date": "2017-05-22 07:26:53.143162", "domains": "vtc", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc007"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC008 config file; Measure network throughput and packet loss using Pktgen; Different amount of flows, from 2 up to 1001000, in combination with different packet sizes are run in each test. Each combination of packet size and flow amount is run 10 times. First 10 times with the smallest packet size, starting with the least amount of ports/flows, then next amount of ports with same packet size, and so on. The test sequence continues with the next packet size, with same ports/flows sequence as before.\n", "ci_loop": "daily", "tags": "pktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc008.html", "_id": "592292be356833000a19518e", "catalog_description": "network: throughput and packet loss extended(pktgen)", "creation_date": "2017-05-22 07:26:54.255469", "domains": null, "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc008"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC009 config file; Measure network throughput and packet loss using pktgen; Different amounts of flows are tested with, from 2 up to 1001000; All tests are run 10 times each. First 10 times with the least amount of ports, then 10 times with the next amount of ports, and so on until all packet sizes have been run with;\n", "ci_loop": "daily", "tags": "pktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc009.html", "_id": "592292bf356833000a19518f", "catalog_description": "network: throughput and packet loss (pktgen)", "creation_date": "2017-05-22 07:26:55.406119", "domains": null, "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc009"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC010 config file; measure memory read latency using lmbench.\n", "ci_loop": "daily", "tags": "lmbench", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc010.html", "_id": "592292c0356833000a195190", "catalog_description": "compute: memory read latency (lmbench)", "creation_date": "2017-05-22 07:26:56.562861", "domains": "compute", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc010"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC011 config file; Measure packet delay variation (jitter) using iperf3.\n", "ci_loop": "daily", "tags": "iperf3", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc011.html", "_id": "592292c1356833000a195191", "catalog_description": "network: performance latency (iperf)", "creation_date": "2017-05-22 07:26:57.673084", "domains": "network", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc011"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC012 config file; Measure memory read and write bandwidth using lmbench.\n", "ci_loop": "daily", "tags": "lmbench", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc012.html", "_id": "592292c2356833000a195192", "catalog_description": "compute: memory bandwith (lmbench)", "creation_date": "2017-05-22 07:26:58.871685", "domains": "compute", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc012"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC014 config file; Measure Processing speed using unixbench.\n", "ci_loop": "daily", "tags": "unixbench", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc014.html", "_id": "592292c4356833000a195193", "catalog_description": "compute: processing speed (unixbench)", "creation_date": "2017-05-22 07:27:00.007018", "domains": "compute", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc014"}, {"project_name": "yardstick", "run": null, "description": "Sample test case for the HA of controller node Openstack service.\n", "ci_loop": "daily", "tags": "serviceHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc019.html", "_id": "592292c5356833000a195194", "catalog_description": "HA: nova-api service down", "creation_date": "2017-05-22 07:27:01.147332", "domains": null, "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc019"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC020 config file.\n", "ci_loop": "daily", "tags": "vtc_instantiation_validation", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc020.html", "_id": "592292c6356833000a195195", "catalog_description": "vtc instantiation", "creation_date": "2017-05-22 07:27:02.271192", "domains": "vtc", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc020"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC021 config file.\n", "ci_loop": "daily", "tags": "vtc_instantiation_validation_noisy", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc021.html", "_id": "592292c7356833000a195196", "catalog_description": "vtc instantiation (Noisy neighbours)", "creation_date": "2017-05-22 07:27:03.420740", "domains": "vtc", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc021"}, {"project_name": "yardstick", "run": null, "description": "Sample test case for the HA of OpenStack Controll Node abnormally shutdown.\n", "ci_loop": "daily", "tags": "serviceHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc025.html", "_id": "592292c8356833000a195197", "catalog_description": "HA: controller node shutdown", "creation_date": "2017-05-22 07:27:04.577080", "domains": null, "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc025"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC027 config file; Measure IPV6 network latency using ping6.\n", "ci_loop": "daily", "tags": "ping6", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc027.html", "_id": "592292c9356833000a195198", "catalog_description": "network: IPv6 latency", "creation_date": "2017-05-22 07:27:05.736814", "domains": "ipv6", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc027"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC037 config file; Measure network throughput and packet loss using pktgen; Different amounts of flows are tested with, from 2 up to 1001000; All tests are run 2 times each. First 2 times with the least amount of ports, then 2 times with the next amount of ports, and so on until all packet sizes have been run with; During the measurements system load and network latency are recorded/measured using ping and mpstat, respectively;\n", "ci_loop": "daily", "tags": "cpuload,ping,oktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc037.html", "_id": "592292ca356833000a195199", "catalog_description": "network: throughput and packet loss extended (cpu&ping load)", "creation_date": "2017-05-22 07:27:06.875236", "domains": "network,compute", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc037"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC038 config file; Measure network throughput and packet loss using pktgen; Different amounts of flows are tested with, from 2 up to 1001000; All tests are run 10 times each. First 10 times with the least amount of ports, then 10 times with the next amount of ports, and so on until all packet sizes have been run with; During the measurements system load and network latency are recorded/measured using ping and mpstat, respectively;\n", "ci_loop": "daily", "tags": "cpuload,ping,pktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc038.html", "_id": "592292cb356833000a19519a", "catalog_description": "network: throughput packet loss (cpu&ping load)", "creation_date": "2017-05-22 07:27:07.988262", "domains": "network,compute", "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc038"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC040 config file; Running Parser Yang-to-Tosca module as a tool, validating output against expected outcome.\n", "ci_loop": "daily", "tags": "parser", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc040.html", "_id": "592292cd356833000a19519b", "catalog_description": "Parser Yang Tosca", "creation_date": "2017-05-22 07:27:09.104621", "domains": null, "dependencies": null, "version": ">brahmaputra.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc040"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC042 config file; Measure network latency using testpmd and pktgen-dpdk.\n", "ci_loop": "daily", "tags": "pktgenDPDKLatency", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc042.html", "_id": "592292ce356833000a19519c", "catalog_description": "network: throughput latency (dpdk-pktgen)", "creation_date": "2017-05-22 07:27:10.207188", "domains": null, "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc042"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC043 config file; Measure latency between NFVI nodes using ping.\n", "ci_loop": "daily", "tags": "ping", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc043.html", "_id": "592292cf356833000a19519d", "catalog_description": "network: node throughput latency(ping)", "creation_date": "2017-05-22 07:27:11.286317", "domains": "network", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc043"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC045 :Control node Openstack service down - neutron server.\n", "ci_loop": "daily", "tags": "serviceHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc045.html", "_id": "592292d0356833000a19519e", "catalog_description": "HA: neutron-server service down", "creation_date": "2017-05-22 07:27:12.399327", "domains": "ha", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc045"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC046 :Control node Openstack service down - keystone.\n", "ci_loop": "daily", "tags": "serviceHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc046.html", "_id": "592292d1356833000a19519f", "catalog_description": "HA: keystone service down", "creation_date": "2017-05-22 07:27:13.502066", "domains": "ha", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc046"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC047 :Control node Openstack service down - glance api.\n", "ci_loop": "daily", "tags": "serviceHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc047.html", "_id": "592292d2356833000a1951a0", "catalog_description": "HA: glance-api service down", "creation_date": "2017-05-22 07:27:14.610834", "domains": "ha", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc047"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC048 :Control node Openstack service down - cinder api.\n", "ci_loop": "daily", "tags": "serviceHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc048.html", "_id": "592292d3356833000a1951a1", "catalog_description": "HA: cinder-api service down", "creation_date": "2017-05-22 07:27:15.751416", "domains": "ha", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc048"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC049 :Control node Openstack service down - swift proxy.\n", "ci_loop": "daily", "tags": "serviceHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc049.html", "_id": "592292d4356833000a1951a2", "catalog_description": "HA: swift proxy service down", "creation_date": "2017-05-22 07:27:16.859383", "domains": "ha", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc049"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC050 :OpenStack Controller Node Network High Availability; This test case is written by new scenario-based HA testing framework.\n", "ci_loop": "daily", "tags": "GeneralHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc050.html", "_id": "592292d6356833000a1951a3", "catalog_description": "HA: network interface down", "creation_date": "2017-05-22 07:27:18.014215", "domains": null, "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc050"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC051 :OpenStack Controller Node CPU Overload High Availability; This test case is written by new scenario-based HA testing framework.\n", "ci_loop": "daily", "tags": "GeneralHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc051.html", "_id": "592292d7356833000a1951a4", "catalog_description": "HA: CPU overload", "creation_date": "2017-05-22 07:27:19.124525", "domains": null, "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc051"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC052 :OpenStack Controller Node Disk I/O Block High Availability; This test case is written by new scenario-based HA testing framework.\n", "ci_loop": "daily", "tags": "GeneralHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc052.html", "_id": "592292d8356833000a1951a5", "catalog_description": "HA: disk I/O block", "creation_date": "2017-05-22 07:27:20.288709", "domains": null, "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc052"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC053 :Openstack Controller Load Balance Service High Availability; This test case is written by new scenario-based HA testing framework.\n", "ci_loop": "daily", "tags": "GeneralHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc053.html", "_id": "592292de356833000a1951a6", "catalog_description": "HA: load balancer service down", "creation_date": "2017-05-22 07:27:26.504630", "domains": null, "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc053"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC054 :OpenStack VIP Master Node abnormally shutdown High Availability; This test case is written by new scenario-based HA testing framework.\n", "ci_loop": "daily", "tags": "GeneralHA", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc054.html", "_id": "592292df356833000a1951a7", "catalog_description": "HA: virtual IP node shutdown", "creation_date": "2017-05-22 07:27:27.763354", "domains": null, "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc054"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC055 config file; Collect hardware specification from /proc/cpuinfo /proc/meminfo; compute capacity and scale. the results have, number of CPUs, number of physical cores in a single CPU; number of logical cores, total memory size; cache size per CPU, total cache size; HT (Hyper-Thread) support status, 1 for open, 0 for close.\n", "ci_loop": "daily", "tags": "computeCapacity", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc055.html", "_id": "592292e1356833000a1951a8", "catalog_description": "compute: cpu & memory_specification", "creation_date": "2017-05-22 07:27:29.032130", "domains": "compute", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc055"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC063 config file; Measure disk size, block size and disk utilization using fdisk and iostat.\n", "ci_loop": "daily", "tags": "storageCapacity", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc063.html", "_id": "592292e2356833000a1951a9", "catalog_description": "storage: disk size&utilization", "creation_date": "2017-05-22 07:27:30.302236", "domains": "storage", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc063"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC069 config file; Measure memory read and write bandwidth using ramspeed.\n", "ci_loop": "daily", "tags": "ramspeed", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc069.html", "_id": "592292e3356833000a1951aa", "catalog_description": "compute: memory bandwith(ramspeed)", "creation_date": "2017-05-22 07:27:31.542189", "domains": "compute", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc069"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC070 config file; Measure network throughput and packet loss using pktgen; Different amounts of flows are tested with, from 2 up to 1001000; All tests are run 2 times each. First 2 times with the least amount of ports, then 2 times with the next amount of ports, and so on until all packet sizes have been run with; During the measurements memory usage statistics and network latency are recorded/measured using free and ping, respectively;\n", "ci_loop": "daily", "tags": "memoryload,ping,pktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc070.html", "_id": "592292e4356833000a1951ab", "catalog_description": "network: throughput packet loss (memory&ping load)", "creation_date": "2017-05-22 07:27:32.747354", "domains": "network,storage", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc070"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC071 config file; Measure cache hit/miss ratio and usage, network throughput and latency; Different amounts of flows are tested with, from 2 up to 1001000; All tests are run 2 times each. First 2 times with the least amount of ports, then 2 times with the next amount of ports, and so on until all packet sizes have been run with; During the measurements cache hit/miss ration, cache usage statistics and network latency are recorded/measured using cachestat and ping, respectively;\n", "ci_loop": "daily", "tags": "cachestat,ping,pktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc071.html", "_id": "592292e5356833000a1951ac", "catalog_description": "compute: cache hit&network throughput latency", "creation_date": "2017-05-22 07:27:33.957092", "domains": "network,storage", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc071"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC072 config file; Measure network throughput and packet loss using pktgen; Different amounts of flows are tested with, from 2 up to 1001000; All tests are run 2 times each. First 2 times with the least amount of ports, then 2 times with the next amount of ports, and so on until all packet sizes have been run with; During the measurements network usage statistics and network latency are recorded/measured using sar and ping, respectively;\n", "ci_loop": "daily", "tags": "netutilization,ping,pktgen", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc072.html", "_id": "592292e7356833000a1951ad", "catalog_description": "network: throughput(netutilization&pktgen load)", "creation_date": "2017-05-22 07:27:35.175762", "domains": "network,storage", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc072"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC073 config file; measure network latency and throughput using netperf; There are two sample scenarios: bulk test and request/response test; In bulk test, UDP_STREAM and TCP_STREAM can be used; send_msg_size and recv_msg_size are options of bulk test; In req/rsp test, TCP_RR TCP_CRR UDP_RR can be used; req_rsp_size is option of req/rsp test;\n", "ci_loop": "daily", "tags": "netperfnode", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc073.html", "_id": "592292e8356833000a1951ae", "catalog_description": "network: node throughput(netperf)", "creation_date": "2017-05-22 07:27:36.466462", "domains": "network", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc073"}, {"project_name": "yardstick", "run": null, "description": "Test case for TC074 StorPerf benchmark task config file; StorPerf is a tool to measure block and object storage performance in an NFVI.\n", "ci_loop": "daily", "tags": "storperf", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc074.html", "_id": "592292e9356833000a1951af", "catalog_description": "storage: storperf integration", "creation_date": "2017-05-22 07:27:37.665283", "domains": "storage", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc074"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC075 config file; Measure network capacity and scale. Measure number of connections, number of frames received;\n", "ci_loop": "daily", "tags": "networkCapacity", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc075.html", "_id": "592292ea356833000a1951b0", "catalog_description": "network: capacity connections number", "creation_date": "2017-05-22 07:27:38.857919", "domains": "network", "dependencies": null, "version": ">colorado.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc075"}, {"project_name": "yardstick", "run": null, "description": "Yardstick TC076 config file; Monitor network metrics provided by the kernel in a host and calculate IP datagram error rate, ICMP message error rate, TCP segment error rate and UDP datagram error rate.", "ci_loop": "daily", "tags": "ping,nstat", "url": "http://artifacts.opnfv.org/yardstick/docs/userguide/opnfv_yardstick_tc076.html", "_id": "592292ec356833000a1951b1", "catalog_description": "network: metrics (ping nstat)", "creation_date": "2017-05-22 07:27:40.163423", "domains": "network", "dependencies": null, "version": ">danube.1.0", "criteria": null, "tier": null, "trust": "silver", "blocking": null, "name": "opnfv_yardstick_tc076"}]} +{"testcases": [{"project_name": "sdnvpn", "run": null, "description": "RF tests for ODL VPN service using ODL REST API.", "ci_loop": null, "tags": null, "url": "https://wiki.opnfv.org/sdnvpn/main#functest", "_id": "565c7341514bc5087f2ddcfc", "catalog_description": null, "creation_date": "2015-11-30 16:03:13.149596", "domains": null, "dependencies": null, "version": null, "criteria": null, "tier": null, "trust": null, "blocking": null, "name": "odl_vpn"}, {"project_name": "sdnvpn", "run": null, "description": "RF tests for BGPVPN Neutron API extensions using Tempest framework.", "ci_loop": null, "tags": null, "url": "https://wiki.opnfv.org/display/sdnvpn/main#Main-OpenStackNeutronBGPVPNAPItests", "_id": "565c7348514bc5087f2ddcfd", "catalog_description": null, "creation_date": "2015-11-30 16:03:20.997647", "domains": null, "dependencies": null, "version": null, "criteria": null, "tier": null, "trust": null, "blocking": null, "name": "functest_tempest"}, {"project_name": "sdnvpn", "run": null, "description": "VPNs provide connectivity across Neutron networks and subnets if configured accordingly.", "ci_loop": null, "tags": null, "url": "https://wiki.opnfv.org/display/sdnvpn/main#Main-TestCase1-VPNprovidesconnectivitybetweensubnets", "_id": "565d5e45514bc5087f2ddd03", "catalog_description": null, "creation_date": "2015-12-01 08:45:57.900254", "domains": null, "dependencies": null, "version": null, "criteria": null, "tier": null, "trust": null, "blocking": null, "name": "functest_testcase_1"}, {"project_name": "sdnvpn", "run": null, "description": "Using VPNs to isolate tenants so that overlapping IP address ranges can be used.", "ci_loop": null, "tags": null, "url": "https://wiki.opnfv.org/display/sdnvpn/main#Main-TestCase2-tenantseparation", "_id": "565d5e60514bc5087f2ddd04", "catalog_description": null, "creation_date": "2015-12-01 08:46:24.194775", "domains": null, "dependencies": null, "version": null, "criteria": null, "tier": null, "trust": null, "blocking": null, "name": "functest_testcase_2"}, {"project_name": "sdnvpn", "run": {"class": "SdnVpnTests", "module": "functest.opnfv_tests.features.sdnvpn"}, "description": "This test case is to report the overall result of all the test cases executed in a specific CI run", "ci_loop": "daily", "tags": "bgpvpn", "url": "https://wiki.opnfv.org/display/sdnvpn/main", "_id": "578f743d9377c54b278bc466", "catalog_description": null, "creation_date": "2016-07-20 12:53:17.290365", "domains": "networking", "dependencies": {"installer": "(fuel)|(apex)", "scenario": "bgpvpn"}, "version": ">colorado.1.0", "criteria": null, "tier": "feature", "trust": "silver", "blocking": false, "name": "bgpvpn"}]} diff --git a/dovetail/utils/local_db/projects.json b/dovetail/utils/local_db/projects.json index 579ee4c9..25fb439d 100644 --- a/dovetail/utils/local_db/projects.json +++ b/dovetail/utils/local_db/projects.json @@ -16,5 +16,11 @@ "description": "The goal of yardstick is to verify the infrastructure compliance when running VNF applications. NFV Use Cases described in ETSI GS NFV 001 show a large variety of applications, each defining specific requirements and complex configuration on the underlying infrastructure and test tools.The Yardstick concept decomposes typical VNF work-load performance metrics into a number of characteristics/performance vectors, which each of them can be represented by distinct test-cases.", "name": "yardstick", "creation_date": "2015-10-14 06:56:09" + }, + { + "_id": "565c5939514bc5087f2ddcfa", + "description": "The project aims to integrate the OpenStack Neutron BGPVPN project and its supported backends into the OPNFV reference platform.", + "name": "sdnvpn", + "creation_date": "2015-11-30 14:12:09.101020" } ] |