summaryrefslogtreecommitdiffstats
path: root/dovetail/run.py
AgeCommit message (Collapse)AuthorFilesLines
2019-04-03Push CI results to OPNFV test DBxudan1-2/+13
In order to use OPNFV test results page to do the ayalysis of all OVP test cases, it needs to push all results generated by Dovetail CI jobs to test DB and then using http://testresults.opnfv.org/test/#/results to check all results. The following data must contain in the POST body: 1. project_name 2. case_name 3. details 4. installer 5. scenario 6. pod_name 7. build_tag 8. criteria 9. start_date 10. stop_date 11. version JIRA: DOVETAIL-767 Change-Id: I925ae249e24efd7bfb1c68a69150e9c22f0cdf36 Signed-off-by: xudan <xudan16@huawei.com>
2019-03-20Improve DockerfileStamatis Katsaounis1-3/+3
This patch improves the Dockerfile which produces the Dovetail image. Change-Id: I65e65f155afe237b11c668e6486e48b55cc7e96c Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-20Remove deprecated ONAP lifecycle test caseStamatis Katsaounis1-2/+0
This patch removes the ONAP lifecycle deprecated test case. Furthermore, it clears helper functions as well as unwanted configuration. Change-Id: Ib958457cf120ffb641a72be028d3a031741c44c6 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-14Dynamic results version and vnf type recognitionStamatis Katsaounis1-0/+4
This patch makes it able to fetch the version from the testsuite. Furthermore, in case of ONAP test cases it adds the vnf_type to the produced results.json file in order to be parsed from the results portal. Finally, it creates a testuite for ONAP TOSCA related tests. Change-Id: I613358c445cfe62730b98fee5e8c6fba7d322e90 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-01Add ONAP VVP Heat Template validation testsStamatis Katsaounis1-0/+2
Change-Id: I9a3d56932ce41191901381831013768d1c57a749 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-01-03Add ONAP VNF SDK verification testsxudan1-0/+2
Please refer to env_config.sh.onap.sample when testing it. The guide of preparing local env for onap package validation can be found here https://gerrit.onap.org/r/#/c/75119 Change-Id: I39d6b11544847756126623a7eb3b953dc800c470 Signed-off-by: xudan <xudan16@huawei.com>
2018-12-20Simplify project conf filesxudan1-4/+6
1. use copy commands instead of pre_copy 2. remove redundant items in project conf files 3. make the volume mapping clearer Change-Id: I6c6aa58fac65d7e40105e0a54f6544ee5c47db31 Signed-off-by: xudan <xudan16@huawei.com>
2018-11-23Integrate k8s test cassesxudan1-22/+2
In order to integrate k8s test cases, Dovetail framework should do some refactor and make it more general for k8s test cases as well as ONAP ones. Integrate 2 k8s test cases. JIRA: DOVETAIL-748 Change-Id: Ibd87754ffb5fb29f6b4ce79232af860c2ed2da9c Signed-off-by: xudan <xudan16@huawei.com>
2018-11-20Add missing unit tests for run fileStamatis Katsaounis1-51/+47
JIRA: DOVETAIL-724 This patch adds unit tests for run file methods of Dovetail which were missing. Change-Id: I1700c8c97430899abdc2b752a3dcbd4d09a334ac Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-06Update classes in report.pyxudan1-15/+18
Change-Id: Ia17bf7ef3e304eef7b3f0b2f7aa04e3be75bc8c3 Signed-off-by: xudan <xudan16@huawei.com>
2018-11-01Update classes in testcase.pyxudan1-6/+0
1. Remove the functions which are not called anywhere 2. Change some class methods to be static methods and instance methods Change-Id: I2e754cdff61aa34ffe899e352653ff767d4ac69d Signed-off-by: xudan <xudan16@huawei.com>
2018-07-24Making the results file a proper json fileGeorg Kunz1-3/+8
The results.json file is currently not a properly formated json file, but contains one json structure per line/testcase. Whit this change, all test results are being collected at runtime and then written to a properly formated json file at the end of a test run. Moreover, this patch introduces 'mandatory' and 'optional' sections to the test suites definition files in order to clearly indicate if a test is part of the corresponding test scope. The web portal needs to be adapted accordingly to be able to read the new results file. JIRA: DOVETAIL-672 Change-Id: Id0706c7e82622fc9c2c2fece26b2d6c07c1580be Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-07-17Update test suite ovp.nextxudan1-3/+6
Update test suite ovp.next according to the latest OVP scope. Modify Dovetail daily jobs to run all the scope. There should be 2 jobs. One for all mandatory test cases and the other for all optional ones. That's mainly because of that the total executed time will be too large (more than 300 minutes). Then the job will always failed because of time out. It's hard to enlarge the time because it's already larger than common 3 hours. Split it into 2 jobs can avoid the time out issue and make the results clearer for reviewing. JIRA: DOVETAIL-694 Change-Id: Ie0ea6221868941781af1477f7c7719f7cb4351a4 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-13Remove the yardstick_tag and functest_tagxudan1-23/+0
These 2 commands can't work now. Also there is no need to support them, so remove them. JIRA: DOVETAIL-695 Change-Id: I6707c7fc1d19fe06690cd14ef39f9d421d00a2bc Signed-off-by: xudan <xudan16@huawei.com>
2018-07-12Enable OVP tests to run on a DPDK-enabled systemPanagiotis Karalis1-1/+7
DPDK-enabled NFVIs require support for hugepages which in turn is configured by means of extra_specs in flavors. Currently, Functest enables hugepage support if the name of the scenario contains the substring "ovs". Dovetail allows end users to run all tests on DPDK-enabled systems and the documentation has been updates accordingly. JIRA: DOVETAIL-665 Change-Id: If4898da168f8b4c69a045ef3b00b8a8862f29b0c Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2018-07-02Fix syntax error on clean_results_dir methodPanagiotis Karalis1-1/+1
Change-Id: I9f33bb320452efab79be5a22e625efac820183eb Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2018-06-06Make exit status a constant in run.pyMiikka Koistinen1-4/+4
JIRA: DOVETAIL-625 Change-Id: Ib96e6e39bad708f495af97ed43884909e394da10 Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>
2018-06-06Fix error status when executing CLI run commandMiikka Koistinen1-0/+6
This commit fixes CLI run command to exit with non-zero exit status when bad arguments are given for '--testsuite' or '--testarea'. JIRA: DOVETAIL-660 Change-Id: I4539b19978d9e93d8a90be9000f36d0aa1895f71 Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>
2018-05-27Archive all test case result filesxudan1-23/+1
1. All detailed tempest test cases results are recorded with file tempest-report.html rather than tempest.log now. 2. Archive all results files include functest.log, yardstick.log ... for each test case. Otherwise, they will all be in one log file. 3. Support to set the source files and dest files need to be archived with the test case config files rather than hardcode with the source code. 4. Give the file which used to be parse the results (PASS/FAIL) with the test case config files rather than hardcode with the source code. Change-Id: I30f435d91ab90cf20def47007f177fe98187484d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-08Improve the CLI of Dovetail toolxudan1-44/+64
The CLI of dovetail tool needs some improvement. 1. Add an option '--testcase' to support specify test case. 2. Add an option '--no-clean' to keep functest/yardstick/bottlenecks containers for debugging after running the test cases. 3. Improve the local report section. a. Remove some info that is not necessary b. Make the info more clearly JIRA: DOVETAIL-588 Change-Id: Ic769a6e2788b407c3df08ccd593a0ecdaa35e5ea Signed-off-by: xudan <xudan16@huawei.com>
2018-05-04Remove local DBxudan1-53/+16
Remove local DB and can just report without launch local DB. Need to do a little modification with the web portal code. JIRA: DOVETAIL-593 Change-Id: I595db39cce4156596ee0522d4b7e642a23d78ab9 Signed-off-by: xudan <xudan16@huawei.com>
2018-04-17Implementation of API validaton exemption in dovetailGeorg Kunz1-2/+8
This patch adds the ability to Dovetail to disable strict API response validation in Tempest-based test cases run by Functest. Corresponding updates of the web portal and backporting of this patch to OVP 1.0.0 (Danube) will follow. JIRA: DOVETAIL-633 Change-Id: Iace99ea1b6224ea907a2c3af8676e9285e6ad3ee Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-03-29Integrate dovetail with ONAPMoshe1-0/+7
Change-Id: I6a1fa5f27df2f5127eb00156b3135dc79caf5e77 Issue-ID: DOVETAIL-629 Signed-off-by: Moshe <moshehoa@amdocs.com>
2018-03-15substitute image copy with docker volumeSerenaFeng1-0/+2
Change-Id: Ie470d03bd168c909fd02c316523049c67a6bb095 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2018-03-06Separate valid docker tags for functest and yardstickGeorg Kunz1-10/+12
Specifying separate valid docker tags for functest and yardstick for input validation. DOVETAIL-592 Change-Id: I3365cdbd9ab521bf1368750c3f34b6863b3f7ec8 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-03-06Bugfix: it crashes when check_tc_result returns Nonexudan1-2/+6
The function 'check_tc_result' may return 'None'. In this case it will throw an exception because of https://github.com/opnfv/dovetail/blob/master/dovetail/run.py#L65 The exception is like Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dovetail/run.py", line 339, in <module> main() File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 716, in _call_ return self.main(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 696, in main rv = self.invoke(ctx) File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 889, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 534, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/dovetail/run.py", line 304, in main duration = run_test(testsuite_yaml, testarea, logger, kwargs) File "/usr/local/lib/python2.7/dist-packages/dovetail/run.py", line 65, in run_test if stop_on_fail['criteria'] == "FAIL" and kwargs['stop']: TypeError: 'NoneType' object has no attribute '_getitem_' JIRA: DOVETAIL-622 Change-Id: I9e612b380c3dfb148692074a3bf8eb31875213ac Signed-off-by: xudan <xudan16@huawei.com>
2018-03-06leverage data_files to eliminate so many hardcoded abs_path_seekingsSerenaFeng1-16/+13
1. mv conf/ userconfig/ compliance/ patch to etc/ 2. after installation etc/ will be put under /etc/dovetail 3. leveraging /etc/dovetail in coding Change-Id: Icf9cb00ff7fc7cf35755f60d3e258b23e2c70839 Signed-off-by: SerenaFeng <feng.xiaowei@zte.com.cn>
2018-02-14Add stop on failure logic and CLI optionEddie Arrage1-7/+12
- Improve test execution control by adding stop on failure flag - Modify test check function to return value regardless of whether report or local file option is employed - Modify run test loop to check for failure after every test case if stop flag set Change-Id: I27a4f3aaaa8b8cb8c52eb3f092f52048980abb44 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-02-05Add default to --report CLI optionEddie Arrage1-6/+13
- Try new license header to deal with anteater failure - Ability to specify 'default' as argument to --report, which fetches host IP automatically and forms URL with 8000 as testapi port - This should not break the previous behavior of report option, if default option not specified Change-Id: I1e05bb52cf4fc211c59796d4e763e2e04177b406 Signed-off-by: Eddie Arrage <eddie.arrage@huawei.com>
2018-01-26Align container tags and test suite configuration yaml for initial releasexudan1-5/+5
Change default container tag to be: 1. Functest ovp.1.0.0 2. testapi ovp.1.0.0 Add test suite ovp.1.0.0 and set it to be the default test suite. Comment out bottlenecks CLI for initial release. JIRA: DOVETAIL-580 Change-Id: I956e49e43fc7ba0176c14079b7228089db5dc108 Signed-off-by: xudan <xudan16@huawei.com>
2017-11-02Get SUT's endpoint infoxudan1-0/+1
Get the admin endpoints of all service. JIRA: DOVETAIL-546 Change-Id: Ie0ac3bc5b4eb0885046e207b73217994a4f0e358 Signed-off-by: xudan <xudan16@huawei.com>
2017-11-02Get hardware info of SUTxudan1-0/+2
Hardware info: Use ansible to get all hardware info from all nodes. JIRA: DOVETAIL-540 Change-Id: I6a2dde1b2ebae2af8f702008c23110ebe63cc06f Signed-off-by: xudan <xudan16@huawei.com>
2017-09-29Merge "Support to run mandatory or optional test cases separately"Leo wang1-12/+5
2017-09-28Merge "Don't generate tar file when run without pushing results to DB"Leo wang1-1/+2
2017-09-28Don't generate tar file when run without pushing results to DBxudan1-1/+2
1. Dovetail generate a tar file for uploading results to CVP web portal. 2. If users don't run with option --report, it shouldn't generate the tar file. Because users can not upload results to CVP if they don't run with --report. 3. If it generates the tar file when user run dovetail without --report, the tar file is invalid because it lacks of results.json. JIRA: DOVETAIL-511 Change-Id: I4d21f21f0b278962564d810369df03100b38f003 Signed-off-by: xudan <xudan16@huawei.com>
2017-09-26Make Yardstick log level can be set by CLIxudan1-0/+9
1. Currently, the log level of yardstick is DEBUG and cannot be modified. 2. Make it can be changed from CLI. 3. When use dovetail run --debug, can see all Yardstick logs (including DEBUG) in dovetail.log. 4. When run without debug option, cannot see Yardstick debug logs in dovetail.logs. 5. yardstick.log always doesn't include debug logs. JIRA: DOVETAIL-506 Change-Id: Iaafb61232a22e16fff2fa4605bf7e262b022085e Signed-off-by: xudan <xudan16@huawei.com>
2017-09-25Support to run mandatory or optional test cases separatelyxudan1-12/+5
1. Currently, dovetail can run a test suite or a test area in that test suite. 2. The test areas in one test suite are not divided into optional and mandatory. 3. Split them and support to run just mandatory or optional. 4. Support to run multiple test areas. JIRA: DOVETAIL-505 Change-Id: I42cd7b4e11c3e3674c806e9bc999b782bf5c85c6 Signed-off-by: xudan <xudan16@huawei.com>
2017-09-06Add a patch to set ha SLA to be 30 secondsxudan1-0/+13
1. According to ha test specification doc, the SLA of process recover time is 30 seconds. 2. However, the SLA in Yardstick test cases are all 20 seconds. 3. Add a patch to change the SLA in yardstick test cases. JIRA: DOVETAIL-496 Change-Id: Ib8f229fdfbb3289bf70d6ee900132b3db3a21b17 Signed-off-by: xudan <xudan16@huawei.com>
2017-08-10create a tar file for logsgrakiss1-0/+1
JIRA: DOVETAIL-479 User may want to upload all logs along with the results, so others may check the running logs to make sure what' s going on during the test. Change-Id: Ice906264bbdf091d3353d6d2e146653aeb057f5f Signed-off-by: grakiss <grakiss.wanglei@huawei.com>
2017-08-03Add resiliency test caes into proposed_testsxudan1-2/+12
JIRA: DOVETAIL-474 Add Bottlenecks resiliency test case "posca_factor_ping" into proposed_tests. Change-Id: I76aea651014fb4722ddae07559f417d3f840eee2 Signed-off-by: xudan <xudan16@huawei.com>
2017-07-21Use uuid1 to replace uuid4xudan1-1/+1
JIRA: DOVETAIL-469 uuid1 is more unique than uuid4 Change-Id: Ic29ec7586ba518e67cb1750df1001cd390fcf313 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-28Modify Dovetail logsxudan1-18/+22
JIRA: DOVETAIL-450 Change-Id: I795679c9559d1bead3aecefff28d21b6567aac2f Signed-off-by: xudan <xudan16@huawei.com>
2017-06-22Bugfix: Move userconfig directory into dovetail/dovetailxudan1-1/+1
JITA: DOVETAIL-449 1. Dovetail cvp.0.1.0 link dovetail/dovetail to /usr/local/lib/python2.7/dist-packages/dovetail/ 2. The directory dovetail/userconfig is lost in this image 3. Move dovetail/userconfig into dovetail/dovetail/userconfig Change-Id: I9da8389e02d92f35879ff5c6a1db1b7d984dd862 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-21Merge "Docker version misalignment and move some irrelevant messages"Leo wang1-1/+1
2017-06-13Merge "Bugfix: KeyError when Functest fails to store results into file"Leo wang1-2/+2
2017-06-09Merge "https+credential support"hongbo tian1-0/+11
2017-06-08https+credential supportMatthewLi1-0/+11
JIRA: DOVETAIL-440 Change-Id: I69fb96673755545e8e0d78d940870195dbf20a0b Signed-off-by: MatthewLi <matthew.lijun@huawei.com>
2017-06-07Docker version misalignment and move some irrelevant messagesxudan1-1/+1
JIRA: DOVETAIL-437 JIRA: DOVETAIL-444 1. Docker client version and server version should be 1.12.3 or later. 2. If any of them is older than 1.12.3, there will be an ERROR message. 3. Delete the “rm -rf results” message. Change-Id: I0c10cf256ef139cac4d43f36108bf5f3f9da8398 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-06Bugfix: KeyError when Functest fails to store results into filexudan1-2/+2
JIRA: DOVETAIL-445 https://build.opnfv.org/ci/view/dovetail/job/dovetail-fuel-baremetal-proposed_tests-danube/19/console Change-Id: If933156bb3931e7bfc8ec31eb03ca5a3cb0bf8b0 Signed-off-by: xudan <xudan16@huawei.com>
2017-06-06Merge "Support Yardstick to push results to database"Leo wang1-12/+7