summaryrefslogtreecommitdiffstats
path: root/dovetail
AgeCommit message (Collapse)AuthorFilesLines
2019-04-03Push CI results to OPNFV test DBxudan5-12/+124
In order to use OPNFV test results page to do the ayalysis of all OVP test cases, it needs to push all results generated by Dovetail CI jobs to test DB and then using http://testresults.opnfv.org/test/#/results to check all results. The following data must contain in the POST body: 1. project_name 2. case_name 3. details 4. installer 5. scenario 6. pod_name 7. build_tag 8. criteria 9. start_date 10. stop_date 11. version JIRA: DOVETAIL-767 Change-Id: I925ae249e24efd7bfb1c68a69150e9c22f0cdf36 Signed-off-by: xudan <xudan16@huawei.com>
2019-03-28Fix VVP test caseStamatis Katsaounis3-8/+10
This patch adds check for no environment variables provided to container configuration. Furthermore, it changes the check results file to a list of check result files. Change-Id: I2112ee1fd4b3156608d8a27dd3b9c90dbe35a24c Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com> Co-Authored-By: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2019-03-20Improve DockerfileStamatis Katsaounis2-6/+6
This patch improves the Dockerfile which produces the Dovetail image. Change-Id: I65e65f155afe237b11c668e6486e48b55cc7e96c Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-03-18Replace subprocess commands with Docker SDKStamatis Katsaounis4-216/+202
JIRA: DOVETAIL-752 This patch replaces all python subprocess commands with the Docker SDK relevant commands. Change-Id: Iac7caffd80a863a8a022247d735b2a7f2792e49d Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-03-13Calculate checksum for input VNFStamatis Katsaounis4-5/+71
This patch adds checksum information inside ONAP related test case run results. The checksum is produced by the VNF input which can either be a CSAR file or an archive of Heat templates. Change-Id: I0ed58bdc9cc4031da08fd2ac220ef294520ef447 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-03-08Add Functest Rally test casesStamatis Katsaounis2-12/+81
This patch adds all Rally testcases of Functest to Dovetail. Change-Id: I8b20bce9966e924fd5aac41bde5a30442394db05 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-03-07Fix the verdict of TC in sub-testcase list resultPanagiotis Karalis2-0/+13
When the sub-test case list is defined, in the dovetail results, each test case is appeared with the its verdict next to its name. Once a sub-testcase is passed, the verdict PASS is appeared next to the name of sub testcase. Once a sub-testcase is skipped or failed, the verdict FAIL is appeared next to the name of sub-testcase. This fix updates the code so that the verdict of skipped and failed sub test cases to be appeared respectively next to its sub-tsetcase name. JIRA: DOVETAIL-763 Change-Id: Ib05b3793a57acbaa67b89321a840ae11251c2417 Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2019-03-07Provide a list of check results files instead of oneStamatis Katsaounis2-72/+80
This patchs gives the ability to provide more than one check results file. Furthermore, it does not alter the behavior of the existing tests but prepares dovetail to be able to fetch information from many sources. Change-Id: I23cd23d2908ad6bdddd94b977813d0e81dc0c05a Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-20Remove deprecated ONAP lifecycle test caseStamatis Katsaounis11-365/+9
This patch removes the ONAP lifecycle deprecated test case. Furthermore, it clears helper functions as well as unwanted configuration. Change-Id: Ib958457cf120ffb641a72be028d3a031741c44c6 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-14Dynamic results version and vnf type recognitionStamatis Katsaounis3-3/+14
This patch makes it able to fetch the version from the testsuite. Furthermore, in case of ONAP test cases it adds the vnf_type to the produced results.json file in order to be parsed from the results portal. Finally, it creates a testuite for ONAP TOSCA related tests. Change-Id: I613358c445cfe62730b98fee5e8c6fba7d322e90 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-02-01Add ONAP VVP Heat Template validation testsStamatis Katsaounis9-34/+348
Change-Id: I9a3d56932ce41191901381831013768d1c57a749 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2019-01-03Add ONAP VNF SDK verification testsxudan8-10/+352
Please refer to env_config.sh.onap.sample when testing it. The guide of preparing local env for onap package validation can be found here https://gerrit.onap.org/r/#/c/75119 Change-Id: I39d6b11544847756126623a7eb3b953dc800c470 Signed-off-by: xudan <xudan16@huawei.com>
2018-12-20Simplify project conf filesxudan6-180/+35
1. use copy commands instead of pre_copy 2. remove redundant items in project conf files 3. make the volume mapping clearer Change-Id: I6c6aa58fac65d7e40105e0a54f6544ee5c47db31 Signed-off-by: xudan <xudan16@huawei.com>
2018-12-17make item 'extra_container' available for every projectxudan2-4/+11
The item 'extra_container' used to be available for Bottlenecks only. Make it open for Dovetail framework. Change-Id: If4f2c032924447b97e401d497aca0309c0f5d847 Signed-off-by: xudan <xudan16@huawei.com>
2018-12-11Enable ShellRunnerxudan5-104/+49
1. fix the bug 'ShellRunner' object has no attribute 'archive_logs' 2. remove the testarea_supported because it's not very necessary for the test cases themselves and can make it a little more simple to add a test case with new test area JIRA: DOVETAIL-750 Change-Id: I71ee74615200376adca2a0db040753e5fce329bc Signed-off-by: xudan <xudan16@huawei.com>
2018-11-23Integrate k8s test cassesxudan12-485/+419
In order to integrate k8s test cases, Dovetail framework should do some refactor and make it more general for k8s test cases as well as ONAP ones. Integrate 2 k8s test cases. JIRA: DOVETAIL-748 Change-Id: Ibd87754ffb5fb29f6b4ce79232af860c2ed2da9c Signed-off-by: xudan <xudan16@huawei.com>
2018-11-20Add missing unit tests for report fileStamatis Katsaounis2-19/+1176
JIRA: DOVETAIL-724 This patch adds unit tests for report file methods of Dovetail which were missing. Change-Id: I2ba7c8f9fcfb962988b2904b01538e9f89a0d849 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-20Add missing unit tests for run fileStamatis Katsaounis3-51/+738
JIRA: DOVETAIL-724 This patch adds unit tests for run file methods of Dovetail which were missing. Change-Id: I1700c8c97430899abdc2b752a3dcbd4d09a334ac Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-15Add missing unit tests for testcase fileStamatis Katsaounis2-2/+661
JIRA: DOVETAIL-724 This patch adds unit tests for Tescases classes methods of Dovetail which were missing. Change-Id: Ie1b665249ce8afaa76b59b0dbe7f132e404a5569 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-14Add missing unit tests for test_runner fileStamatis Katsaounis2-8/+660
JIRA: DOVETAIL-724 This patch adds unit tests for Runners classes methods of Dovetail which were missing. Change-Id: Ib985a44db86be02a7b9cefb8deca187bb50e78cc Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-14Add missing unit tests for container fileStamatis Katsaounis2-3/+768
JIRA: DOVETAIL-724 This patch adds unit tests for container class methods of Dovetail which were missing. Change-Id: Iaba3bbfbb8838278e20ac9f63ef6e01f9ada1cbc Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-14Add missing unit tests for cli filesStamatis Katsaounis5-1/+284
JIRA: DOVETAIL-724 This patch adds unit tests for cli methods of Dovetail which were missing. Change-Id: I99c584007659401e298e58aebb1764df1c543894 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-14Add missing unit tests for utils filesStamatis Katsaounis7-29/+1615
JIRA: DOVETAIL-724 This patch adds unit tests which were missing from dovetail code base. Furthermore it updates the test_parser existing unit test. Finally, it fixes some minor issues in dovetail_utils itself. Change-Id: I8fd7cd4f6b1ede11664914746d2279f062511639 Signed-off-by: Stamatis Katsaounis <mokats@intracom-telecom.com>
2018-11-08Remove the offline scriptsxudan3-156/+0
As we discussed in Dovetail weekly meeting, there is no need to maintain all the offline scripts. Change-Id: I360d1a211856626af91310f142c1777a77398a12 Signed-off-by: xudan <xudan16@huawei.com>
2018-11-06Update classes in report.pyxudan2-48/+46
Change-Id: Ia17bf7ef3e304eef7b3f0b2f7aa04e3be75bc8c3 Signed-off-by: xudan <xudan16@huawei.com>
2018-11-01Update classes in testcase.pyxudan3-63/+12
1. Remove the functions which are not called anywhere 2. Change some class methods to be static methods and instance methods Change-Id: I2e754cdff61aa34ffe899e352653ff767d4ac69d Signed-off-by: xudan <xudan16@huawei.com>
2018-10-31Update class in container.pyxudan2-110/+98
Change-Id: I4fb24f86c6d15aea5a62baaca20531b2891437ff Signed-off-by: xudan <xudan16@huawei.com>
2018-10-31CI verify jobs failed due to the update of toxxudan3-10/+8
The LF pod used to run the CI verify jobs have updated the version of tox. So the CI verify jobs failed. See https://build.opnfv.org/ci/view/dovetail/job/dovetail-merge-master/476/console Fix the python format errors Ignore error 'W503 line break before binary operator'. Change-Id: Ic2df070447bbea39c877ce7a394d733a95064bb1 Signed-off-by: xudan <xudan16@huawei.com>
2018-10-01Bugfix: There is an exception when crawling Yardstick resultsxudan1-1/+2
If there is no pod.yaml, the results data looks like, {"status": 2, "result": "[Errno 2] No such file or directory: u'/home/opnfv/userconfig/pre_config/pod.yaml'"} It will get an exception when reading value from file, File "/usr/local/lib/python2.7/dist-packages/dovetail/report.py", line 327, in crawl_from_file criteria = data['result']['criteria'] TypeError: string indices must be integers https://build.opnfv.org/ci/view/dovetail/job/dovetail-apex-baremetal-default-mandatory-master/52/console Change-Id: I73ca3e5ae73433a2dc139789df4d67e8c583be08 Signed-off-by: xudan <xudan16@huawei.com>
2018-09-13Update code base and docs to 2018.09xudan1-1/+1
The official name of the next release is going to be 2018.09. We need to rename all occurrences of 2018.08 in the code base, the documentation, and the web portal. JIRA: DOVETAIL-740 Change-Id: Ie0e532d6671b7590454e9afd960a2934f313a022 Signed-off-by: xudan <xudan16@huawei.com>
2018-09-05Enable the web portal to show all endpointsxudan2-11/+29
The endpoints info for the 2018.08 has been changed. The web portal needs some adaptions for the new data format. It keeps the same as 2018.01 and doesn't need to change dovetail-webportal. JIRA: DOVETAIL-725 Change-Id: I74cde3aa6032c7afac4b6ce1d2146e09a0f99fe5 Signed-off-by: xudan <xudan16@huawei.com>
2018-08-28Change Bottlenecks volume mappingxudan1-0/+1
If mapping to host's /tmp, it won't clean all .log and .out files before run a new test. It causes the results file including all previous results. See https://build.opnfv.org/ci/view/dovetail/job/dovetail-apex-baremetal-default-mandatory-master/26/artifact/results/stress_logs/bottlenecks.stress.ping.out/*view*/ It will affect the results check. Mapping to DOVETAIL_HOME/results/bottlenecks, which will always be cleaned before running another test. Change-Id: I22bfc871d6f041ae38308bf8e8f685993a376973 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-31Renaming test cases to include test framework namesGeorg Kunz1-3/+1
Replacing the generic "dovetail" prefix of the testcase definitions with the name of the OPNFV test framework which hosts the test. JIRA: DOVETAIL-719 Signed-off-by: Georg Kunz <georg.kunz@ericsson.com> Change-Id: I8f90423b8908197bdc7adddf82c61ff394893f52
2018-07-24Making the results file a proper json fileGeorg Kunz3-35/+49
The results.json file is currently not a properly formated json file, but contains one json structure per line/testcase. Whit this change, all test results are being collected at runtime and then written to a properly formated json file at the end of a test run. Moreover, this patch introduces 'mandatory' and 'optional' sections to the test suites definition files in order to clearly indicate if a test is part of the corresponding test scope. The web portal needs to be adapted accordingly to be able to read the new results file. JIRA: DOVETAIL-672 Change-Id: Id0706c7e82622fc9c2c2fece26b2d6c07c1580be Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-07-22Remove functest_testsuite and functest_testcase in dovetail_config.ymlxudan3-21/+7
The 2 sections 'functest_testsuite' and 'functest_testcase' are used by all Functest test cases. They aim to parse the results of Tempest sub test cases. Remove them, and check the sub test case list, 1. if it's empty, that means no need to check sub test cases 2. if it is not empty, check the sub test cases one by one. Additional, remove some useless keys in dovetail_config.yml JIRA: DOVETAIL-658 Change-Id: Ic829264b0c8eaedd8e2b3fa55b38a34644482c20 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-17Update test suite ovp.nextxudan4-43/+68
Update test suite ovp.next according to the latest OVP scope. Modify Dovetail daily jobs to run all the scope. There should be 2 jobs. One for all mandatory test cases and the other for all optional ones. That's mainly because of that the total executed time will be too large (more than 300 minutes). Then the job will always failed because of time out. It's hard to enlarge the time because it's already larger than common 3 hours. Split it into 2 jobs can avoid the time out issue and make the results clearer for reviewing. JIRA: DOVETAIL-694 Change-Id: Ie0ea6221868941781af1477f7c7719f7cb4351a4 Signed-off-by: xudan <xudan16@huawei.com>
2018-07-13Remove the yardstick_tag and functest_tagxudan1-23/+0
These 2 commands can't work now. Also there is no need to support them, so remove them. JIRA: DOVETAIL-695 Change-Id: I6707c7fc1d19fe06690cd14ef39f9d421d00a2bc Signed-off-by: xudan <xudan16@huawei.com>
2018-07-12Enable OVP tests to run on a DPDK-enabled systemPanagiotis Karalis2-1/+9
DPDK-enabled NFVIs require support for hugepages which in turn is configured by means of extra_specs in flavors. Currently, Functest enables hugepage support if the name of the scenario contains the substring "ovs". Dovetail allows end users to run all tests on DPDK-enabled systems and the documentation has been updates accordingly. JIRA: DOVETAIL-665 Change-Id: If4898da168f8b4c69a045ef3b00b8a8862f29b0c Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2018-07-10CI ERROR - File contains violation on test_runnerPanagiotis Karalis1-1/+1
Proposed solution by CI server: "Avoid dangerous file parsing & serialization libs, use yaml.safe_load" Change-Id: Iff9764601bc135cec271e16444d0bbbb4da8d459 Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2018-07-02Fix syntax error on clean_results_dir methodPanagiotis Karalis1-1/+1
Change-Id: I9f33bb320452efab79be5a22e625efac820183eb Signed-off-by: Panagiotis Karalis <pkaralis@intracom-telecom.com>
2018-06-29Update licence headers to comply with guidelinesGeorg Kunz12-11/+42
Updating the licence headers to comply with the community guidelines [1] and anteater checks. [1] https://wiki.opnfv.org/display/DEV/Contribution+Guidelines#ContributionGuidelines-License Change-Id: Idec787e32daf76c0b1bd6108977b78828125da5e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-06-14Replace openstack commands with shadexudan4-54/+37
Use openstack shade to get endpoint infomation rather than executing openstack commands directly. The next step can replace openstack commands by shade for some test cases that need to create openstack resources (images, flavors) if we agree to use shade. JIRA: DOVETAIL-659 Change-Id: I72b302fb2d4a61e9249a7e53f81ea9464944dcd3 Signed-off-by: xudan <xudan16@huawei.com>
2018-06-06Make exit status a constant in run.pyMiikka Koistinen1-4/+4
JIRA: DOVETAIL-625 Change-Id: Ib96e6e39bad708f495af97ed43884909e394da10 Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>
2018-06-06Fix error status when executing CLI run commandMiikka Koistinen2-1/+7
This commit fixes CLI run command to exit with non-zero exit status when bad arguments are given for '--testsuite' or '--testarea'. JIRA: DOVETAIL-660 Change-Id: I4539b19978d9e93d8a90be9000f36d0aa1895f71 Signed-off-by: Miikka Koistinen <miikka.koistinen@nokia.com>
2018-05-27Archive all test case result filesxudan4-83/+66
1. All detailed tempest test cases results are recorded with file tempest-report.html rather than tempest.log now. 2. Archive all results files include functest.log, yardstick.log ... for each test case. Otherwise, they will all be in one log file. 3. Support to set the source files and dest files need to be archived with the test case config files rather than hardcode with the source code. 4. Give the file which used to be parse the results (PASS/FAIL) with the test case config files rather than hardcode with the source code. Change-Id: I30f435d91ab90cf20def47007f177fe98187484d Signed-off-by: xudan <xudan16@huawei.com>
2018-05-18Got an exception when test suite is emptyxudan1-1/+5
1. Return empty test case list when test suite is empty. 2. Don't quite understand why comment out all proposed_tests. 3. Move stress ping test case at the end of the list, because it may broken the message queue. Change-Id: Ib717fc91e3763be03f243f82e53679462ecdae04 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-11Adding Neutron trunk port tests to proposed_tests suiteGeorg Kunz1-0/+4
Adding the Neutron trunk port tempest tests to Dovetail and the proposed_tests test suite. JIRA: DOVETAIL-648 Change-Id: Ia77980a245fd1bca6e5f0e567467432e12bf974e Signed-off-by: Georg Kunz <georg.kunz@ericsson.com>
2018-05-08Improve the CLI of Dovetail toolxudan3-72/+74
The CLI of dovetail tool needs some improvement. 1. Add an option '--testcase' to support specify test case. 2. Add an option '--no-clean' to keep functest/yardstick/bottlenecks containers for debugging after running the test cases. 3. Improve the local report section. a. Remove some info that is not necessary b. Make the info more clearly JIRA: DOVETAIL-588 Change-Id: Ic769a6e2788b407c3df08ccd593a0ecdaa35e5ea Signed-off-by: xudan <xudan16@huawei.com>
2018-05-07Decoupling Bottlenecks Docker config items from source codexudan4-106/+76
There are some special Docker setting items when creating Bottlenecks Docker Cntainers. Move the special settings to config files. Disable the function to push results to DB. JIRA: DOVETAIL-645 Change-Id: Icc033222edf74c0fe4853dede2794f4226fd4252 Signed-off-by: xudan <xudan16@huawei.com>
2018-05-04Remove local DBxudan12-517/+47
Remove local DB and can just report without launch local DB. Need to do a little modification with the web portal code. JIRA: DOVETAIL-593 Change-Id: I595db39cce4156596ee0522d4b7e642a23d78ab9 Signed-off-by: xudan <xudan16@huawei.com>