diff options
Diffstat (limited to 'docs')
67 files changed, 3555 insertions, 2624 deletions
diff --git a/docs/com/css/theme/OPNFV-Berlin.css b/docs/com/css/theme/OPNFV-Berlin.css index 9f957d6f1..34f73b4bd 100644 --- a/docs/com/css/theme/OPNFV-Berlin.css +++ b/docs/com/css/theme/OPNFV-Berlin.css @@ -31,13 +31,13 @@ body { bottom: 1em; left: 1em; font-size: 0.5em; - + } .reveal .slides > section, .reveal .slides > section > section { line-height: 1.3; font-weight: inherit; } - + .reveal .opnfv-title { background-image: url("../../img/title-bg-berlin.png" no-repeat); } @@ -87,7 +87,7 @@ body { .reveal h1 { text-shadow: none; } - + /********************************************* * OTHER diff --git a/docs/com/css/theme/OPNFV.css b/docs/com/css/theme/OPNFV.css index 745a2a8de..c59c59f8d 100644 --- a/docs/com/css/theme/OPNFV.css +++ b/docs/com/css/theme/OPNFV.css @@ -30,13 +30,13 @@ body { bottom: 1em; left: 1em; font-size: 0.5em; - + } .reveal .slides > section, .reveal .slides > section > section { line-height: 1.3; font-weight: inherit; } - + .reveal .opnfv-title { background-image: url("../../img/title-bg.png" no-repeat); } @@ -83,7 +83,7 @@ body { .reveal h1 { text-shadow: none; } - + /********************************************* * OTHER diff --git a/docs/com/pres/Summit/Berlin-2016/conversation.html b/docs/com/pres/Summit/Berlin-2016/conversation.html index 356c2ad1e..5c1e18450 100644 --- a/docs/com/pres/Summit/Berlin-2016/conversation.html +++ b/docs/com/pres/Summit/Berlin-2016/conversation.html @@ -215,7 +215,7 @@ </div> <div class='footer'> - <img src="../../../img/logo-OPNFV-Berlin.png" alt="OPNFV logo"> + <img src="../../../img/logo-OPNFV-Berlin.png" alt="OPNFV logo"> </div> </div> diff --git a/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html b/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html index 97fa66c18..2b3b3b4ef 100644 --- a/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html +++ b/docs/com/pres/Summit/Berlin-2016/summit-Berlin.html @@ -81,7 +81,7 @@ </section> <section data-markdown> ## so concretely... - * Validate scenarios for the release + * Validate scenarios for the release * Give confidence on OPNFV fresh releases * Ensure consistancy towards installers * Ensure End to End interoperability @@ -135,7 +135,7 @@ * 2 installers: Foreman/Fuel * Lots of manual operations (reporting, documentation) </section> - </section> + </section> <section> <section data-markdown> # Brahmaputra @@ -146,8 +146,8 @@ * 3 feature projects: doctor, promise, sdnvpn * 13 Scenarios * 4 Installers (Apex, Compass, Fuel, Joid) - * Test result collection - * Automatic dashboard + * Test result collection + * Automatic dashboard </section> <section> <h3>Jiras</h3> @@ -196,7 +196,7 @@ </tr> </tbody> </table> - * difference of duration due to POD and test suites + * difference of duration due to POD and test suites </section> <section data-markdown> @@ -204,7 +204,7 @@ * Tempest/Rally * harmonizing installer related OpenStack configuration differences * creating customized test lists - * main challenge was resolving SDN controller interworking problems + * main challenge was resolving SDN controller interworking problems * vIMS * complete but complex test case * very interesting to automate (pre MANO, most of Telco needs met there) @@ -212,7 +212,7 @@ </section> - <section> + <section> <section data-markdown> # Colorado </section> @@ -221,7 +221,7 @@ ## What's new? * New internal test cases: healthcheck, security * New controler: OpenContrail? - * New feature projects (domino, multisites, movie, parser, moon, copper, models, onos-sfc...) + * New feature projects (domino, multisites, movie, parser, moon, copper, models, onos-sfc...) * ARM Support (congratulations to ENEA team!) * Simplified feature project integration journey </section> @@ -255,19 +255,19 @@ * New VNFs (vEPC, vCDN, vWhatever...) * Multi sites (e.g bgpvpn with different back ends) * Better test coverage - </section> + </section> <section data-markdown> ## A Functional testing As a Service framework * Scenario owner can select their relevant test cases * Test duration estimation * Agile dashboarding * Analytics - </section> + </section> <section data-markdown> ## Upstream * Rally: time to give back more - * Functest description for ETSI? - </section> + * Functest description for ETSI? + </section> </section> <section> diff --git a/docs/com/pres/Summit/Berlin-2016/testapi.html b/docs/com/pres/Summit/Berlin-2016/testapi.html index c40637cbf..5e56d5052 100644 --- a/docs/com/pres/Summit/Berlin-2016/testapi.html +++ b/docs/com/pres/Summit/Berlin-2016/testapi.html @@ -69,7 +69,7 @@ * Test cases * Results </section> - + <section data-markdown> ## Status Currently * From many projects @@ -87,7 +87,7 @@ </aside> </script> </section> - + <section data-markdown> ## Help building * Dashboards @@ -135,8 +135,8 @@ <li>Scripts to support databse backup/restore/update</li> <li>Support installation</li> <aside class='notes'> - So you can discover and understand the capabilities of the service without - access to source code, documentation, or through network traffic inspection, + So you can discover and understand the capabilities of the service without + access to source code, documentation, or through network traffic inspection, and also you can interact with the TestAPI directly through swagger website. </aside> </ul> @@ -148,12 +148,12 @@ <section data-markdown> # API Evoluation </section> - + <section> <h2> URI changes...</h2> - <div style="text-align:left""> + <div style="text-align:left""> <p> testresults.opnfv.org/<span style="color:lightblue">TestAPI</span> => <br>testresults.opnfv.org/<span style="color:yellow">test/api/v1</span> </p> - + <p> /test/api/v1/<b>pods</b></p> <p> /test/api/v1/<b>projects</b></p> <p> /test/api/v1/projects/qtip/<b>cases</b></p> @@ -200,7 +200,7 @@ <section> <h2>unit tests</h2> <pre><code class="hljs" data-trim contenteditable> -umry8364@umry8364-Latitude-E6400:~/Dev/OPNFV/releng/utils/test/result_collection_api$ ./run_test.sh +umry8364@umry8364-Latitude-E6400:~/Dev/OPNFV/releng/utils/test/result_collection_api$ ./run_test.sh Tests running... WARNING:tornado.general:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1): Project name missing WARNING:tornado.access:404 GET /dashboard/v1/results?case=vPing&pod=zte-pod1&version=C&installer=fuel&period=5 (127.0.0.1) 2.30ms diff --git a/docs/com/pres/compliance/compliance.md b/docs/com/pres/compliance/compliance.md new file mode 100644 index 000000000..cbe0dbe17 --- /dev/null +++ b/docs/com/pres/compliance/compliance.md @@ -0,0 +1,97 @@ +# From Verification to CNTT Compliance + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2019/12/04 + + + +## Infrastructure Verification + + +### Functest in a nutshell + +- verify any kind of OpenStack and Kubernetes deployments (OPNFV model) + including production environments +- conform with upstream rules (OpenStack gate jobs and Kubernetes conformance + tests) +- ensure that the platforms meet Network Functions Virtualization requirements + + +### Functest suites + +- all functional tests (3000+) as defined by the upstream communities + (e.g. Tempest, neutron-tempest-api, Barbican, Patrole...) +- 3 hours upstream API and dataplane benchmarking tests (Rally, VMTP and + Shaker) +- Virtual Network Function deployments and testing (vIMS, vRouter and vEPC) + + + +## CNTT Compliance + + +### Verification vs Compliance + +- verification allows skipping test if optional services (Gnocchi, Barbican, + etc.) or capabilities (remote console access, Neutron BGPVPN or SFC, etc.) + are missing +- compliance forces here the full API descriptions as currently proposed by + CNTT (please see + [Interfaces and APIs](https://github.com/cntt-n/CNTT/blob/master/doc/ref_arch/openstack/chapters/chapter05.md)) +- then the testcase descriptions should forbid skipping any test and cover only + the mandatory services (and their mandatory capabilities) + + +### Changelog + +- all the logics were already in Functest and the underlying frameworks (Rally, + Tempest, etc.) +- 2 new CNTT-related containers including the new testcase descriptions were + published to easily verify the compliance +- all 3 Functest SUTs (Rocky, Stein/Train and Master) are now compliant with + CNTT API to ensure the continuous integration +- the benchmarking testcases doesn't validate any KPI as nothing is written in + CNTT documentation + + +### RI verification and Compliance + +- CNTT Reference Implementation 1 is already continuously verified + ([continuous integration model](https://build.opnfv.org/ci/view/cntt/job/cntt-latest-daily/)) +- the conformance is currently failing due to a few bugs in deployments and + missing features + +**Be free to [deploy your own CNTT Compliance CI/CD toolchain](https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004) + in a few commands** + + + +## Conclusion + + +### Next steps + +- fix reference implementation 1 deployments and then achieve the compliance +- port existing OPNFV testcases to Xtesting and then add them in the continuous + integration loop +- update the testcase descritions according to the CNTT progress (KPI, API + changes) + + +### Takeaways + +- Functest allows verifying any production Infrastructure and now checking the + CNTT API Compliance +- all containers can be already consumed +- any third-party certification should reuse the Functest CNTT-related + containers as they are +- [CNTT RI continuous integration](https://build.opnfv.org/ci/view/cntt/job/cntt-latest-daily/) + is in place and any testcase can be smoothly added if they leverage on + Xtesting + +**Try it, and you will love it!** + + + +## Thank you diff --git a/docs/com/pres/compliance/index.html b/docs/com/pres/compliance/index.html new file mode 100644 index 000000000..94fc121d6 --- /dev/null +++ b/docs/com/pres/compliance/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>From Verification to Compliance</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="compliance.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/dockerslicing/dockerslicing.md b/docs/com/pres/dockerslicing/dockerslicing.md index a66453248..0d1ce5e21 100644 --- a/docs/com/pres/dockerslicing/dockerslicing.md +++ b/docs/com/pres/dockerslicing/dockerslicing.md @@ -63,7 +63,7 @@ ### 8 Functest containers -```bash +```shell $ sudo docker search opnfv |grep functest- opnfv/functest-core OPNFV Functest core image opnfv/functest-restapi OPNFV Functest restapi image diff --git a/docs/com/pres/euphrates_functest_evolution/euphrates.md b/docs/com/pres/euphrates_functest_evolution/euphrates.md index 49aab16cd..2c3105539 100644 --- a/docs/com/pres/euphrates_functest_evolution/euphrates.md +++ b/docs/com/pres/euphrates_functest_evolution/euphrates.md @@ -120,7 +120,7 @@ export OS_IDENTITY_API_VERSION=3 #### Retrieve all the images ``` -mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | bash -s -- images && ls -1 images/* +mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | sh -s -- images && ls -1 images/* images/CentOS-7-aarch64-GenericCloud.qcow2 images/CentOS-7-aarch64-GenericCloud.qcow2.xz images/CentOS-7-x86_64-GenericCloud.qcow2 diff --git a/docs/com/pres/functest2019/functest2019.md b/docs/com/pres/functest2019/functest2019.md new file mode 100644 index 000000000..48f01cb0c --- /dev/null +++ b/docs/com/pres/functest2019/functest2019.md @@ -0,0 +1,105 @@ +# Functest 2019 + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2019/01/08 + + + +## Gates + + +### Today's Verify -1 + +- one error is detected by yamllint, pep8, pylint, ansible-lint, bashate or + doc8 +- one unit test fails (py27 and py35) +- specific modules are not rated 10/10 by pylint +- our Framework is not fully covered +- unix permissions are wrong +- one doc cannot be generated without warning + +Please see [tox.ini](https://git.opnfv.org/functest/tree/tox.ini) for details + + +### Great! But + +- the current gate checks all coding rules, our Framework and all interfaces +to third-parties but **not our testcases** +- it doesn't break **the circular dependencies** between Installers and +Functest (Installers ask for a trustable healthcheck but we need results of +"Installer runs" as prerequisites) + +Our testcases have still **to be verified by hand** what differs from +OpenStack Workflow. + + +### Functional gating + +- we need to run all Functest patches before merge against reference +plateforms (OpenStack and Kubernetes) +- a voting job must forbid the merge if one test fails (as the workflow +process implemented by OpenStack) + + +### Improve the current daily model + +- all Functest jobs are linked to the OPNFV installers and can hardly be + reused by endusers +- all test suites are run sequentially +- rally_full is excluded due to its duration (~3 hours) +- all possible remaining resources are cleant when installing the scenarios + +**It's fine for gating installers but not for verifying that Functest supports parallel and live testing** + + +### New Xtesting Ansible role + +- deploy anywhere the full OPNFV CI/CD toolchain in few commands +- easily add external bots voting in reviews +- produce all Xtesting-based fonctional jobs (Xtesting, Functest and Functest + Kubernetes) in Releng +- could be instanciated for testing services out of the infrastructure domain + +**It's already in a [good shape](https://lists.opnfv.org/g/opnfv-tech-discuss/message/22552). [Try it!](https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004)** + + + +## Quality Assurance + +**Functest is matching the OpenStack Quality Golden rules** + + +### Pylint and coverage (started from E release) +- only need to refactor several vnf testcases and to remove duplicated code +- document well and cover Functest utils: they are reused by +OPNFV third-parties (SFC, SDNVPN) + +** Functest will be rated 10/10 very soon** + + +### releasing + +- Functest could be also released as a classical Python packages like Xtesting( +[PyPI](https://pypi.python.org/pypi)) +- Xtesting and Functest could be released as GNU/Linux distribution packages + + + +## other challenges + + +### proposals + +- integrate Rally and Tempest as core modules in Functest (we do merge tempest.py and + conf_utils.py) +- deploy cloud-native VNF (Clearwater IMS?) +- why not leveraging on OpenStack middlewares such as + [oslo.config](https://docs.openstack.org/oslo.config/latest/)? +- promote our VNF testcases in the upstream communities + +**Any new testcase is more than welcome!** + + + +## Thank you diff --git a/docs/com/pres/functest2019/index.html b/docs/com/pres/functest2019/index.html new file mode 100644 index 000000000..33dc9a9f2 --- /dev/null +++ b/docs/com/pres/functest2019/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>Functest 2019</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="functest2019.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/functest2020/functest2020.md b/docs/com/pres/functest2020/functest2020.md new file mode 100644 index 000000000..a81667939 --- /dev/null +++ b/docs/com/pres/functest2020/functest2020.md @@ -0,0 +1,61 @@ +# Functest 2020 + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2020/01/16 + + + +## What's new in Jerma? + + +### Better test case verification + +- switch to Ceph in our Functest SUTs +- add Neutron features in our SUTs to improve the case verification +- test all capabilities possible (e.g. vnc_console) +- verify automatically the CNTT-related containers into additional to the + classical ones +- harden xrally_kubernetes + +**Functest SUTs are compliant to CNTT RC** + + +### New test cases + +- tempest_horizon +- tempest_keystone +- tempest_cinder +- refstack_platform +- refstack_object +- octavia +- xrally_kubernetes + + +### New usage + +- **support CNTT RC (API testing, API and dataplane benchmarking, VNF onboarding and testing)** +- verify ONAP WindRiver OpenLab via Functest CI in a VM ("Inception model") +- allow minimal l2-only testing via Rally + +**still pushing the limit!** + + + +## Kali (K-release) + + +### Ideas: + +- finish updating to Alpine 3.11 and Python 3.8 +- finish KloudBuster integration (it has to be updated to Python3 first) +- add tempest-stress +- update and possibly add heat-tempest-plugin +- add cyborg-tempest-plugin? ironic-tempest-plugin ? xxx-tempest-plugin? +- add CNF into Kubernetes testing + +**continuously hardening the gates and improving code quality** + + + +## Thank you diff --git a/docs/com/pres/functest2020/index.html b/docs/com/pres/functest2020/index.html new file mode 100644 index 000000000..88195becb --- /dev/null +++ b/docs/com/pres/functest2020/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>Functest 2020</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="functest2020.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/gambia/gambia.md b/docs/com/pres/gambia/gambia.md new file mode 100644 index 000000000..4a65e9905 --- /dev/null +++ b/docs/com/pres/gambia/gambia.md @@ -0,0 +1,178 @@ +# Functest on steroids + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2019/01/08 + + + +## OPNFV + + +<!-- .element: style="border: 0; width: 90%" --> + + + +## Verify OpenStack and Kubernetes + + +### Functest in a nutshell + +- verify any kind of OpenStack and Kubernetes deployments +- conform with upstream rules (OpenStack gate jobs and Kubernetes conformance + tests) +- ensure that the platforms meet Network Functions Virtualization requirements + + +### Functest suites + +- as many upstream functional tests as possible (e.g. Tempest, + neutron-tempest-api, Barbican, Patrole...) +- upstream API and dataplane benchmarking tools (Rally, Vmtp and Shaker) +- additional VNF deployments and testing (vIMS, vRouter and vEPC) + +**Dovetail only runs few Functest functional tests and then we have to verify +any OPNFV-certified scenarios via Functest anyway** + + +### What's new in Functest? + +- new testcases were quickly integrated in Functest: Patrole, Barbican, Shaker, + ... +- all testcases can run in parallel to decrease the overall duration +- the resources cleaning has been improved +- **our testcases may be run vs VIM in production** +- **it includes most of the OpenStack gate jobs** + + +### Support of OS and K8s master + +| Functest | OpenStack | Kubernetes | +| :------: | :---------: | :--------: | +| master | master | master | +| hunter | rocky | v1.11.3 | +| gambia | queens | v1.11.3 | + + +### < 50 Euros + + +<!-- .element: style="border: 0; width: 70%" --> + + + +## Reuse of OPNFV + + +### Xtesting in a nutshell + +- allow the developer to work only on the test suites without diving into CI/CD + integration +- simplify test integration in a complete LFN-based CI/CD toolchain (e.g. + Jenkins, Testing Containers, Test API and dashboard) +- allow a proper design and verify multiple components in the same CI/CD + toolchain (OpenStack, Kubernetes, ONAP, etc.) + +**Easy to use and very useful for any CI/CD toochain (unlinked to Infrastrure)** + + +### A user story ONAP + +- all tests are run by a specialized Docker container(**<100 MB**) instead of +the classical ONAP testing virtual machine (**> 1GB**). +- the container mainly inherits from opnfv/xtesting and is completed by: + - Python dependencies + - all ONAP Robot Framework files retrieved from the original repositories + - testcases.yaml describing the testcases + +[Orange-OpenSource/xtesting-onap-robot](https://github.com/Orange-OpenSource/xtesting-onap-robot/) + + +### What's new in Xtesting? + +- new ansible roles and playbooks have been developed to allow **deploying your + full CI/CD toolchains in few minutes** (Jenkins, Minio, TestAPI, MongoDB and + Docker registry) + +```shell +virtualenv xtesting +. xtesting/bin/activate +pip install ansible docker +ansible-galaxy install collivier.xtesting +git clone https://gerrit.opnfv.org/gerrit/functest-xtesting functest-xtesting-src +ansible-playbook functest-xtesting-src/ansible/site.yml +deactivate +``` + +**They are already reused in Functest and by Orange out of the +Infrastrure domain** + + + +## Collect results + + +### OPNFV Test Database in a nutshell + +- it's a fair comparison of Neutron implementations (Agents vs SDN + controller) +- it stores all verification results and all performance data from different + hardware over the world which could be easily postprocessed +- it could be very useful to select the adequate opensource solutions + regarding metrics and capabilities + + +### which Neutron backend? + +- most Neutron standalone and OVN scenarios pass Functest decently +- no ODL scenarios pass the advanced testcases (benchmarking tools and + VNFs). It's still unclear if it's due to the Installers, ODL or POD + misconfigurations. +- no Tungsten Fabric is released in Gambia + +**We expected that ODL results would have improved before the first Gambia + corrective** + + +### Contrail testing (out of OPNFV) + +- **3.X 4.X**: mostly verified except some functional tests about + visibility which fail due to the falsy admin role (they can be easily + blacklisted) +- **5.X**: a limited set of bugs in Contrail mostly forbid running few + functional tests and benchmarking tools: + - wrong external network listing + - Contrail doesn't allow booting a VM without network (and elects the wrong + network) + + +### And performance? + +- OVS DPDK is not fully integrated by an OPNFV installer (Fuel is in a good + shape to support it) +- the only scenario including VPP is not part of Gambia + +**From the time being, we can't evaluate the benefits of OVS DPDK or VPP thanks + to OPNFV** + + + +## Conclusion + + +### Gambia + +- Functest and Xtesting are powerful and easy to (re)use (containers, jenkins + jobs, ansible playbooks, Raspberry PI, etc.). +- the number of installers and scenarios decreased in Gambia (it's still + unclear regarding the overall quality). **What about OPNFV Test Database if + it decreases again in 2019?** + + +### And beyond + +- test frameworks are now considered as crucial for OPNFV (see [Last OPNFV Marketing update](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Marketing%20Update%20091818.pptx?version=1&modificationDate=1537228648000&api=v2)) and Functest and Xtesting could be +already widely reused out of OPNFV +- the new test-driven approach as proposed by the [OPNFV Strategic Plan](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Strategy%20and%20Plan%20v0.5.pptx?version=1&modificationDate=1540961098000&api=v2) could increase **the quality of all +scenarios**. But we are also suggesting to let the installers decide their +test cases. diff --git a/docs/com/pres/gambia/index.html b/docs/com/pres/gambia/index.html new file mode 100644 index 000000000..db3a6aaa2 --- /dev/null +++ b/docs/com/pres/gambia/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>Functest on steroids</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="gambia.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/gambia/raspberrypi.jpg b/docs/com/pres/gambia/raspberrypi.jpg Binary files differnew file mode 100644 index 000000000..60a2638f3 --- /dev/null +++ b/docs/com/pres/gambia/raspberrypi.jpg diff --git a/docs/com/pres/gambiachallenges/gambia.md b/docs/com/pres/gambiachallenges/gambia.md new file mode 100644 index 000000000..078ec61ce --- /dev/null +++ b/docs/com/pres/gambiachallenges/gambia.md @@ -0,0 +1,79 @@ +# Functest Gambia, new challenges + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2018/03/22 + + + +## Gating + + +### Our gating is very simple + +- today all test suites are run sequentially +- all resources are not necessarily cleant until the next OpenStack deployment + +**It's fine for gating installers but not for verifying a production +environment (e.g. multi users running the same testcase)** + + +### Parallel and Live testing + +- all resources should be allocated in their own OpenStack projects (already + mostly the case) +- no testcase should create singletons on resource attributes (name, ip, etc.) +- all resources have to be cleant at the end + +**According to [Orange testing results](https://wiki.opnfv.org/pages/viewpage.action?pageId=13211751), +Functest is already in a good shape to meet this goal** + + + +## Quality Assurance + +**Gambia release will match the OpenStack Quality Golden rules** + + +### Pylint and coverage (started from E release) +- need to refactor several vnf testcases and to remove duplicated code +- document well and cover Functest utils: they are reused by +OPNFV third-parties (SFC, SDNVPN) + +** Functest will be rated 10/10** + + +### releasing + +- Functest could be also released as a classical Python package ( +[PyPI](https://pypi.python.org/pypi)) +- its API docs could be published in [Read the Docs](https://readthedocs.org/) + + +### Current technical debt +- former python modules (e.g. cli) or containers (e.g. restapi) have to be +removed +- functest shouldn't contain any OPNFV Installer logic. **All Jenkins Jobs +must set the right env vars as all endusers** +- vnf descriptors should be hosted by Functest instead of using external github +repositories + + + +## other challenges + + +### proposals + +- ease integrating tempest plugins with a minimum effort (new tempest driver in +Functest Framework and new opnfv/functest-tempest container) +- add opnfv/functest-components in verify jobs +- why not leveraging on OpenStack middlewares such as + [oslo.config](https://docs.openstack.org/oslo.config/latest/) or + [stevedore](https://docs.openstack.org/stevedore/latest/)? + +**Any new testcase is more than welcome!** + + + +## Thank you diff --git a/docs/com/pres/gambiachallenges/index.html b/docs/com/pres/gambiachallenges/index.html new file mode 100644 index 000000000..592ccf2f5 --- /dev/null +++ b/docs/com/pres/gambiachallenges/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>Gambia challenges</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="gambia.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/gates/gates.md b/docs/com/pres/gates/gates.md new file mode 100644 index 000000000..aaf087743 --- /dev/null +++ b/docs/com/pres/gates/gates.md @@ -0,0 +1,82 @@ +# Functest Gates + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2018/04/19 + + + +### Why gating? + +- maintain an overall high quality code whatever the skills involved in our +open community +- detect the errors as soon as possible (before merge operations) +- verify automatically most of Functest milestones + + +### Today's Verify -1 + +- one error is detected by yamllint, pylint or doc8 +- one unit test fails (py27 and py35) +- our Framework is not fully covered +- specific modules are not rated 10/10 by pylint +- unix permissions are wrong +- one doc cannot be generated without warning + +Please see [tox.ini](https://git.opnfv.org/functest/tree/tox.ini) for details + + +### Gambia's Verify -1 + +- enforce an overall 10/10 by pylint +- check the full coverage of all third-party interfaces +- integrate other linters like ShellScript (if they meet tox basics) + +Please see +[Gambia Challenges](http://testresults.opnfv.org/functest/gambiachallenges/) +for details + + +### Great! But + +- the current gate checks all coding rules, our Framework and all interfaces +to third-parties but **not our testcases** +- it doesn't break **the circular dependencies** between Installers and +Functest (Installers ask for a trustable healthcheck but we need results of +"Installer runs" as prerequisites) + +Our testcases have still **to be verified by hand** what differs from +OpenStack Workflow + + + +### functional gating + +- we need to run all Functest patches before merge against reference +plateforms (OpenStack and Kubernetes) +- a voting job must forbid the merge if one test fails (as the workflow +process implemented by OpenStack) + + +### reference platform + +- it could be baremetal or virtual if all testcases (including all VNFs) can +be tested successfully +- any compliant Installer could be selected as long as the appropriate target +VIM version is proposed at the beginning of the release: + - devstack (OpenStack gates) + - kolla-ansible (Orange ONAP OpenLab) + - XCI + - APEX + + +### Verify +2 + +Only the functional check will take hours once the patchset is accepted +by core reviewers (each patchset will be verified in ~10 minutes). + +**Always stable!** + + + +## Thank you! diff --git a/docs/com/pres/gates/index.html b/docs/com/pres/gates/index.html new file mode 100644 index 000000000..b64e371f7 --- /dev/null +++ b/docs/com/pres/gates/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>Functest Gates</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="gates.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/oran/ftth.png b/docs/com/pres/oran/ftth.png Binary files differnew file mode 100644 index 000000000..5b441d1de --- /dev/null +++ b/docs/com/pres/oran/ftth.png diff --git a/docs/com/pres/oran/oran.md b/docs/com/pres/oran/oran.md new file mode 100644 index 000000000..87fba1306 --- /dev/null +++ b/docs/com/pres/oran/oran.md @@ -0,0 +1,90 @@ +--- +author: Cédric Ollivier +title: OPNFV/CNTT CI assets +date: 2020/12/01 +--- +## Continuous integration + + +### A few CNTT reqs + +- OPNFV has built a complete CI/CD toolchain for continuously deploying and + testing cloud infrastructure +- Reference conformance only requires for the local deployment of the + same components +- all test cases must be delivered as **Docker containers** including the + common test case execution and the unified way to manage all the interactions + +**Integrate smoothly and deploy fastly** + + +### How OPNFV/CNTT helps? + +- **Functest** offers a collection of state-of-the-art virtual infrastructure + test suites +- **Xtesting** helps assembling sparse test cases and accelerating the adoption + of CI/CD best practices +- **XtestingCI** eases deploying anywhere plug-and-play CI/CD toolchains in a + few commands +- **CNTT** defines conformance suites and its playbooks leveraging this 3 + testing frameworks + +**Leverage best opensource tools and practices** + + +### Testing [FTTH](https://en.wikipedia.org/wiki/Fiber_to_the_x) equipment + + + + +### Deploy your CI toolchain + +``` +virtualenv xtesting +. xtesting/bin/activate +pip install ansible +ansible-galaxy install collivier.xtesting +git clone https://gerrit.opnfv.org/gerrit/functest-xtesting functest-xtesting-src +ansible-playbook functest-xtesting-src/ansible/site.yml +deactivate +rm -rf functest-xtesting-src xtesting +``` + +**More scenarios in [[1]](https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004)** + + + +## Testing + + +### Verif', Compliance and Interop' + +- verification allows skipping test if optional services or capabilities are + missing +- compliance forces the full API descriptions as currently proposed by + CNTT and then skips all optional capability testing +- the opensource certifications are driven by simple interoperability testing + (RefStack and OVP are about less than 10% of Functest. **trustability?**) + +**Fine tune the same test frameworks** + + +### [CNTT RC1](https://build.opnfv.org/ci/view/functest/job/functest-leguer-daily/12/) + + + + + +## Conclusion + + +### Key takeaways + +- **test integration requirements** are crucial to smoothly assemble all + test cases +- leverage **existing** OPNFV testing knowledge (projects) and experience + (history) by utilising the OPNFV toolchain design already in-place +- **3000+** functional tests, **3 hours** upstream API and dataplane benchmarks + and VNFs automatically onboarded and tested **can be reused asis** + +**Keep it simple and do not reivent the wheel** diff --git a/docs/com/pres/oran/rc1.png b/docs/com/pres/oran/rc1.png Binary files differnew file mode 100644 index 000000000..9179f3bb6 --- /dev/null +++ b/docs/com/pres/oran/rc1.png diff --git a/docs/com/pres/reveal.js b/docs/com/pres/reveal.js -Subproject a349ff43c58c23f9c837b8ea9b5fc7d4761b8de +Subproject 2c5396b7d347f8ee1344016f15b93d4f7840156 diff --git a/docs/com/pres/rfp/index.html b/docs/com/pres/rfp/index.html new file mode 100644 index 000000000..538c0c8c8 --- /dev/null +++ b/docs/com/pres/rfp/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>OPNFV and CNTT in Orange RFP</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="rfp.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/rfp/rfp.md b/docs/com/pres/rfp/rfp.md new file mode 100644 index 000000000..66f7ba9e7 --- /dev/null +++ b/docs/com/pres/rfp/rfp.md @@ -0,0 +1,97 @@ +# OPNFV and CNTT in Orange RFP + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2020/09/19 + + + +## Vision and contributions + + +### Our guidelines + +- **automate** to bring determinism and to meet the new + software release rate +- test all software layers **independently** (OpenStack, Kubernetes and VNFs) +- run all deployment and verification jobs in **our** continous integration + chains +- leverage best **opensource** tools and practices + +**Integrate smoothly and deploy everywhere fastly** + + +### How OpenSource helps? + +- **Functest** offers a collection of state-of-the-art virtual infrastructure + test suites +- **Xtesting** helps assembling sparse test cases and accelerating the adoption + of CI/CD best practices +- **XtestingCI** eases deploying anywhere plug-and-play CI/CD toolchains in a + few commands +- **CNTT** defines conformance suites and its playbooks leveraging this 3 + softwares + +**Any contribution is more than welcome! +[[1]](https://www.linkedin.com/pulse/call-functest-cntt-rc1-contributions-c%25C3%25A9dric-ollivier/) +[[2]](https://www.linkedin.com/pulse/call-functest-cntt-rc2-contributions-c%25C3%25A9dric-ollivier/)** + + + +## CNTT/OPNFV in Orange + + +### A couple of RFP requirements + +- the **full** CNTT reference conformance for OpenStack results and outputs + (Orange CNTT Field Trial is in a very good shape + [[1]](http://testresults.opnfv.org/functest/field_trial/) + [[2]](https://www.linkedin.com/pulse/cntt-field-trials-c%C3%A9dric-ollivier/)) +- the **success** of the Functest Kubernetes test suites (now released as + part of CNTT RC2 Baraque) +- **first** VNF test cases running in **our** continuous integration chain + thanks to Xtesting and XtestingCI + +**It's implementing Orange and CNTT targets** + + +### Orange CNTT RC1 Field Trial + +- helped detecting a couple of issues in CNTT RC1 +- integrated cinder backup and nova instance_password in Orange IaaS +- to fix 10 remaining single test failures (out 2000+ functional tests, 3 hours + benchmarking and 3 VNFs automatically onboarded and tested) +- to enhance Functest juju_epc to pass proxies + +**99,999%** + + +### Wish list + +- to integrate **more benchmarks** in CNTT conformance (e.g. disk benchmarking) +- to switch from the current Kubernetes interoperability testing to a **true** + CNTT conformance suite +- to build the first **VNF and CNF** conformance suites (**high priority**) + +**We need your contribution helps! +[[1]](https://www.linkedin.com/pulse/call-functest-cntt-rc1-contributions-c%25C3%25A9dric-ollivier/) +[[2]](https://www.linkedin.com/pulse/call-functest-cntt-rc2-contributions-c%25C3%25A9dric-ollivier/)** + + + +## Conclusion + + +### Take aways + +- Orange leverages OPNFV and CNTT in RFP +- we keep contributing in both specification and implementation streams for + the success of Network Function Virtualization +- we expect more OPNFV and CNTT contributions especially for VNF and CNF + conformance suites, our initial CNTT target + +**Try CNTT reference suites, you will love them!** + + + +## Thank you! diff --git a/docs/com/pres/stockholm/chainedci.png b/docs/com/pres/stockholm/chainedci.png Binary files differnew file mode 100644 index 000000000..df594cd3a --- /dev/null +++ b/docs/com/pres/stockholm/chainedci.png diff --git a/docs/com/pres/stockholm/ftth.png b/docs/com/pres/stockholm/ftth.png Binary files differnew file mode 100644 index 000000000..5b441d1de --- /dev/null +++ b/docs/com/pres/stockholm/ftth.png diff --git a/docs/com/pres/stockholm/index.html b/docs/com/pres/stockholm/index.html new file mode 100644 index 000000000..65fa37a3a --- /dev/null +++ b/docs/com/pres/stockholm/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>From Paris to Stockholm</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="stockholm.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/stockholm/stockholm.md b/docs/com/pres/stockholm/stockholm.md new file mode 100644 index 000000000..2fe99cd6e --- /dev/null +++ b/docs/com/pres/stockholm/stockholm.md @@ -0,0 +1,164 @@ +# From Paris to Stockholm + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2019/06/09 + + + +## Agenda + + +### Agenda + +- Functest and Xtesting in a nutshell +- what's new from Paris (Iruya and J-release)? + - code improvement + - functional gates + - switch to python3 + - Raspberry PI +- what's next? GSMA, CNF, ETSI... + + + +## Infrastructure verification + + +### Functest in a nutshell + +- verify any kind of OpenStack and Kubernetes deployments (OPNFV model) + including production environments +- conform with upstream rules (OpenStack gate jobs and Kubernetes conformance + tests) +- ensure that the platforms meet Network Functions Virtualization requirements + + +### Functest suites + +- all functional tests (2000+) as defined by the upstream communities + (e.g. Tempest, neutron-tempest-api, Barbican, Patrole...) +- upstream API and dataplane benchmarking tools (Rally, VMTP and Shaker) +- Virtual Network Function deployments and testing (vIMS, vRouter and vEPC) + + + +### Network automation journey + + +### Xtesting in a nutshell + +- allow the developer to work only on the test suites without diving into + CI/CD integration +- simplify test integration in a complete LFN-based CI/CD toolchain (e.g. + Jenkins, Testing Containers, Test API and dashboard) +- allow a proper design and verify multiple components in the same CI/CD + toolchain (OpenStack, Kubernetes, ONAP, etc.) + +**Easy to use and very useful for any CI/CD toolchain** + + +### Chained CI + + +<!-- .element: style="border: 0" --> + + +### Testing [FTTH](https://en.wikipedia.org/wiki/Fiber_to_the_x) equipment + + +<!-- .element: style="border: 0; width: 90%" --> + + +### Demo + +- describe the testcase execution +- build and publish your container +- list your test in the toolchain description +- deploy your own toolchain in a few commands + +https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004 + + + +## What's new from Paris (Iruya and J-release) + + +### Code improvement + +- all patches are now verified by [bandit](https://github.com/PyCQA/bandit) (in + addition to yamllint, pep8, pylint, ansible-lint and bashate. Please see + [tox.ini](https://git.opnfv.org/functest/tree/tox.ini) for details). +- 10/10 by pylint is now enforced for all + Functest packages + +**we still need python3.6 in build servers to run our unit tests vs python3** + + +### Functional Gates + +- live-migration, VMTP and Shaker are now verified in all gates (pre and post + merge) +- Functest is continuously executed up to 4 tests in parallel verifying our + reference OpenStack/Kubernetes +- we continuously check the remaining resources after multiple runs (they only + remain a few region and volume leaks which are being fixed) + + +### Switch to python3 + +- all latest Functest containers switched to python3 as default (Functest has + supported both python2 and python3 via six for a while) +- all missing decode/encode calls are backported to Hunter and Iruya +- VMTP has been updated to keep the same testcase list +- our patch about Cloudify REST client will be published upstream if the + project is still active + + +### Raspberry PI + +- travis-ci.org allows us to continuously cross-compile containers (Hunter, + Iruya and latest) for arm (Raspberry PI) and arm64 +- no additional configuration is required (Docker manifest allows an unify way + to run the containers whatever the architectures) +- only testcases leveraging on go have not yet been ported (juju_epc and + Kubernetes conformance tests) + +**Try it, and you will love it!** + + +### New testcases + +- refstack_platform and refstack_object (already part of tempest_full) are now + dedicated testcases in Functest +- tempest_slow is on our agenda to keep conforming with OpenStack gates + + +### New opportunities + +- a few indentified opportunities about +[CNF](https://www.cncf.io/announcement/2019/02/25/cncf-launches-cloud-native-network-functions-cnf-testbed/): +[Clearwater Ims via Kubernetes](https://github.com/Metaswitch/clearwater-docker/) +and [xrally-kubernetes](https://github.com/xrally/xrally-kubernetes) +- GSMA should come with new testcases needs (Functest can already be executed + vs the current profiles) +- ETSI could leverage on Xtesting to package its testcases + + + +## Conclusion + + +### Takeaways + +- Xtesting is a key helper from first tests to full end2end service testing +- Functest allows verifying production Infrastructure and is breaking the + classical preproduction/production model +- And we are still pushing the limits: + - python packages + - GNU/Linux packages + +**Upstream first! Try them, and you will love them! ** + + + +## Thank you diff --git a/docs/com/pres/vevent202004/index.html b/docs/com/pres/vevent202004/index.html new file mode 100644 index 000000000..7c1118242 --- /dev/null +++ b/docs/com/pres/vevent202004/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>2020 April Virtual Technical Event</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="vevent202004.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/vevent202004/vevent202004.md b/docs/com/pres/vevent202004/vevent202004.md new file mode 100644 index 000000000..a7f8b5dc4 --- /dev/null +++ b/docs/com/pres/vevent202004/vevent202004.md @@ -0,0 +1,99 @@ +# CNTT RC April 2020 and beyond + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2020/04/21 + + + +## CNTT RC status + + +### Functest SUTs + +- ease verifying automatically the CNTT-related containers (5 SUTs in total) +- mostly conformed with OpenStack reference implementations except: + - all use Ceph and Ceph RadosGW + - one SUT leverages on OVN to check RA1 Chapter5 vs SDN Controllers +- OpenStack and Kubernetes are not redeployed between 2 runs as opposed to + [CNTT RI](https://build.opnfv.org/ci/view/cntt/job/cntt-latest-daily/102/) + +**Functest SUTs are compliant to CNTT RC** + + +### RC verification vs Neutron + +- all branches are successful vs Neutron reference implementation: + - [Functest Hunter (Rocky)](https://build.opnfv.org/ci/view/functest/job/functest-hunter-daily/485/) + - [Functest Iruya (Stein)](https://build.opnfv.org/ci/view/functest/job/functest-iruya-daily/452/) + - [Functest Jerma (Train)](https://build.opnfv.org/ci/view/functest/job/functest-jerma-daily/249/) + - [Functest Master (next Ussuri)](https://build.opnfv.org/ci/view/functest/job/functest-latest-daily/597/) + + +### RC verification vs OVN + +- the overall results are in a good shape as Functest master includes all latest software: + - [CNTT Conformance passed sucessfully](https://build.opnfv.org/ci/view/functest/job/functest-ovn-latest-daily/17/) + - [IaaS verification](https://build.opnfv.org/ci/view/functest/job/functest-ovn-latest-daily/17/) + failed due to one Rally task hanging iteration (under analysis) + +**RA1 Chapter 5 is compatible with external SDN controllers** + + + +## Field trial results + + +### Orange IaaS + +- Orange IaaS verification has leveraged on the Functest containers before CNTT +- Functest containers are also executed to verify the deployment before any VNF + onboarding +- a few mandatory features are missing to fully pass CNTT Compliance + ([cinder backup](https://cntt-n.github.io/CNTT/doc/ref_arch/openstack/chapters/chapter05.html) + and [keystone security_compliance](https://cntt-n.github.io/CNTT/doc/ref_arch/openstack/chapters/chapter05.html)) +- a few extra RadosGW configurations are needed + +**the CNTT Conformance is in a good shape** + + + +## What's next? + + +### last minute's changes: + +- tempest_heat was recently added in Functest IaaS verification and will be + tuned for CNTT compliance +- tempest_horizon should be added into CNTT RC document (mandatory features) + +**they would be added into CNTT RC very soon** + + +### and beyond + +- NFVBench and VSperf are being xtesting-ready and should be added in RC +- a few CNTT issues are opened for developping Functest and then RC: + - [Run fio or dd in Functest](https://github.com/cntt-n/CNTT/issues/1477) + - [Add ceph bench and rbd bench into Functest](https://github.com/cntt-n/CNTT/issues/1476) + - [Add swift-bench in Functest](https://github.com/cntt-n/CNTT/issues/1474) + +**[help is needed!](https://github.com/cntt-n/CNTT/issues?q=is%3Aopen+is%3Aissue+project%3Acntt-n%2FCNTT%2F4)** + + + +## Conclusion + + +### Take aways + +- CNTT RC1 1.0 seems good and stable +- [RA1 Chapter 5]((https://cntt-n.github.io/CNTT/doc/ref_arch/openstack/chapters/chapter05.html) + is compatible with external SDN controllers +- Orange IaaS is closed to be compliant + +**any contribution and RC results are more than welcome!** + + + +## Thank you diff --git a/docs/com/pres/vevent202010/index.html b/docs/com/pres/vevent202010/index.html new file mode 100644 index 000000000..a24c721d2 --- /dev/null +++ b/docs/com/pres/vevent202010/index.html @@ -0,0 +1,52 @@ +<html> +<head> +<title>Orange CNTT RC1 Field Trial feedbacks</title> +<meta name="author" content="Cédric Ollivier"> +<meta name="viewport" + content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"> +<link rel="stylesheet" href="../reveal.js/css/reveal.css"> +<link rel="stylesheet" href="../reveal.js/css/theme/white.css"> +<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css"> +<script> +var link = document.createElement( 'link' ); +link.rel = 'stylesheet'; +link.type = 'text/css'; +link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css'; +document.getElementsByTagName( 'head' )[0].appendChild( link ); +</script> +</head> +<body> + <div class="reveal"> + <div class="slides"> + <section data-markdown="vevent202010.md" data-separator="^\n\n\n" + data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section> + </div> + </div> + <script src="../reveal.js/lib/js/head.min.js"></script> + <script src="../reveal.js/js/reveal.js"></script> + <script> + Reveal.initialize({ + dependencies : [ { + src : '../reveal.js/plugin/markdown/marked.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src : '../reveal.js/plugin/markdown/markdown.js', + condition : function() { + return !!document.querySelector('[data-markdown]'); + } + }, { + src: '../reveal.js/plugin/highlight/highlight.js', + async: true, + callback: function() { + hljs.initHighlightingOnLoad(); + } + }, { + src: '../reveal.js/plugin/notes/notes.js', + async: true + } ] + }); + </script> +</body> +</html> diff --git a/docs/com/pres/vevent202010/vevent202010.md b/docs/com/pres/vevent202010/vevent202010.md new file mode 100644 index 000000000..b128210ac --- /dev/null +++ b/docs/com/pres/vevent202010/vevent202010.md @@ -0,0 +1,62 @@ +# Orange CNTT RC1 Field Trial + +[Cédric Ollivier](mailto:cedric.ollivier@orange.com) + +2020/10/14 + + + +## CNTT RC1 Field Trial + + +### Orange CNTT RC1 Field Trial + +- helped detecting a couple of issues in CNTT RC1 +- integrated cinder backup and nova instance_password in Orange IaaS +- to fix 10 remaining single test failures (out 2000+ functional tests, 3 hours + benchmarking and 3 VNFs automatically onboarded and tested) +- to enhance Functest juju_epc to pass proxies + +**99,999%** + + +### Orange CNTT RC1 feedbacks + +- we easily executed the CNTT RC1 playbook asis and no clear bug was found in + CNTT RC1 Baldy +- CNTT RC1 is now used in our Orange IaaS verification in addition to the + classical Functest containers +- CNTT RC1 is also executed to verify all deployments before onboarding any VNF +- CNTT RC1 Baldy and CNTT RC2 Baraque are listed in **Orange RFP requirements** + (see last ONES + [[1]](http://testresults.opnfv.org/functest/ONES2020NA_OPNFV-CNTT_RFP.1080p.mp4)) + + + +## What else? + + +### Wish list + +- to integrate **more benchmarks** in CNTT conformance (e.g. disk benchmarking) +- to switch from the current Kubernetes interoperability testing to a **true** + CNTT conformance suite +- to build the first **VNF and CNF** conformance suites (**high priority**) + +**We need your contribution helps! +[[2]](https://www.linkedin.com/pulse/call-functest-cntt-rc1-contributions-c%25C3%25A9dric-ollivier/) +[[3]](https://www.linkedin.com/pulse/call-functest-cntt-rc2-contributions-c%25C3%25A9dric-ollivier/)** + + +### New Functest opportunities + +- to implement live monitoring of your OpenStack and Kubernetes deployments via + a subset of Functest (healthcheck? new Rally tasks?) +- to implement new functest testcases to validate and verify OpenStack upgrades + (new Rally tasks)? + +**Try Functest Leguer, you will love it [[4]](https://www.linkedin.com/pulse/opnfv-functest-leguer-out-c%25C3%25A9dric-ollivier/)** + + + +## Thank you! diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 000000000..7a8d1858d --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,179 @@ +# -*- coding: utf-8 -*- +# +# Functest documentation build configuration file, created +# by sphinx-quickstart on Tue Apr 3 03:51:57 2018. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +# needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Functest' +copyright = u'2018, Functest <opnfv-tech-discuss@lists.opnfv.org>' +author = u'Functest <opnfv-tech-discuss@lists.opnfv.org>' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = u'master' +# The full version, including alpha/beta/rc tags. +release = u'master' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = [] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'sphinx_rtd_theme' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +# html_theme_path = [] + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = [] + +# Custom sidebar templates, must be a dictionary that maps document names +# to template names. +# +# This is required for the alabaster theme +# refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars +html_sidebars = { + '**': [ + 'relations.html', # needs 'show_related': True theme option to display + 'searchbox.html', + ] +} + + +# -- Options for HTMLHelp output ------------------------------------------ + +# Output file base name for HTML help builder. +htmlhelp_basename = 'Functestdoc' + + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { + # The paper size ('letterpaper' or 'a4paper'). + # + # 'papersize': 'letterpaper', + + # The font size ('10pt', '11pt' or '12pt'). + # + # 'pointsize': '10pt', + + # Additional stuff for the LaTeX preamble. + # + # 'preamble': '', + + # Latex figure (float) alignment + # + # 'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, + 'Functest.tex', + u'Functest Documentation', + u'Functest \\textless{}opnfv-tech-discuss@lists.opnfv.org\\textgreater{}', + 'manual'), +] + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, + 'functest', + u'Functest Documentation', + [author], + 1) +] + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, + 'Functest', + u'Functest Documentation', + author, + 'Functest', + 'One line description of project.', + 'Miscellaneous'), +] + +spelling_word_list_filename = 'spelling_wordlist.txt' diff --git a/docs/images/clearwater-architecture-v2.png b/docs/images/clearwater-architecture-v2.png Binary files differnew file mode 100644 index 000000000..e4d209a13 --- /dev/null +++ b/docs/images/clearwater-architecture-v2.png diff --git a/docs/images/concepts_mapping_fraser.png b/docs/images/concepts_mapping_fraser.png Binary files differnew file mode 100644 index 000000000..a566e39db --- /dev/null +++ b/docs/images/concepts_mapping_fraser.png diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 000000000..4d0fcd8ba --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,15 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +.. _functest: + +************** +OPNFV Functest +************** + +.. toctree:: + :maxdepth: 1 + + testing/user/configguide/index + testing/user/userguide/index + testing/developer/devguide/index + release/release-notes/index diff --git a/docs/release/release-notes/functest-release.rst b/docs/release/release-notes/functest-release.rst index 78ec8f227..b5e228caa 100644 --- a/docs/release/release-notes/functest-release.rst +++ b/docs/release/release-notes/functest-release.rst @@ -1,83 +1,92 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 -======= -License -======= - -OPNFV Euphrates release note for Functest Docs -are licensed under a Creative Commons Attribution 4.0 International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -============================================= -OPNFV Euphrates 5.1 release note for Functest -============================================= +====================================== +OPNFV master release note for Functest +====================================== Abstract ======== This document contains the release notes of the Functest project. - -OPNFV Euphrates Release -====================== +OPNFV master Release +==================== Functest deals with functional testing of the OPNFV solution. It includes test cases developed within the project, test cases developed in -other OPNFV projects and it also intgrates test cases from other upstream +other OPNFV projects and it also integrates test cases from other upstream communities. +OpenStack +--------- + The internal test cases are: * connection_check - * api_check - * snaps_health_check + * tenantnetwork1 + * tenantnetwork2 + * vmready1 + * vmready2 + * singlevm1 + * singlevm2 * vping_ssh * vping_userdata - * tempest_smoke_serial - * refstack_defcore - * snaps_smoke - * rally_sanity + * cinder_test + * tempest_smoke + * tempest_horizon * odl - * tempest_full_parallel + * tempest_neutron + * tempest_cinder + * tempest_keystone + * tempest_heat + * tempest_telemetry + * rally_sanity + * refstack_compute + * refstack_object + * refstack_platform + * tempest_full + * tempest_scenario + * tempest_slow + * patrole_admin + * patrole_user + * patrole_reader + * tempest_barbican + * tempest_octavia + * tempest_cyborg * rally_full + * rally_jobs + * vmtp + * shaker + * cloudify * cloudify_ims + * heat_ims * vyos_vrouter + * juju_epc -The OPNFV projects integrated into Functest framework for automation are: +Kubernetes +---------- - * barometer - * bgpvpn - * doctor - * domino - * fds - * odl-sfc - * odl-netvirt - * parser - * promise - * orchestra_openims - * orchestra_clearwaterims +The internal test cases are: + * k8s_quick + * k8s_smoke + * k8s_conformance + * xrally_kubernetes + * kube_hunter + * kube_bench_master + * kube_bench_node + * xrally_kubernetes_full + * k8s_vims + * helm_vims + * cnf_conformance Release Data ============ +--------------------------------------+--------------------------------------+ | **Project** | functest | -| | | -+--------------------------------------+--------------------------------------+ -| **Repo/tag** | opnfv-5.1.0 | -| | | +--------------------------------------+--------------------------------------+ -| **Release designation** | Euphrates 5.1 release | -| | | -+--------------------------------------+--------------------------------------+ -| **Release date** | December 15th 2017 | -| | | -+--------------------------------------+--------------------------------------+ -| **Purpose of the delivery** | Euphrates second release | -| | | +| **Repository branch** | master | +--------------------------------------+--------------------------------------+ Deliverables @@ -86,164 +95,66 @@ Deliverables Software -------- - Functest Docker images: + Functest Docker images (OpenStack): * https://hub.docker.com/r/opnfv/functest-healthcheck * https://hub.docker.com/r/opnfv/functest-smoke - * https://hub.docker.com/r/opnfv/functest-features - * https://hub.docker.com/r/opnfv/functest-components + * https://hub.docker.com/r/opnfv/functest-benchmarking * https://hub.docker.com/r/opnfv/functest-vnf - * https://hub.docker.com/r/opnfv/functest-parser - * https://hub.docker.com/r/opnfv/functest-restapi + * https://hub.docker.com/r/opnfv/functest-smoke-cntt + * https://hub.docker.com/r/opnfv/functest-benchmarking-cntt - TestAPI Docker image: + Functest Docker images (Kubernetes): - * https://hub.docker.com/r/opnfv/testapi + * https://hub.docker.com/r/opnfv/functest-kubernetes-healthcheck + * https://hub.docker.com/r/opnfv/functest-kubernetes-smoke + * https://hub.docker.com/r/opnfv/functest-kubernetes-security + * https://hub.docker.com/r/opnfv/functest-kubernetes-benchmarking + * https://hub.docker.com/r/opnfv/functest-kubernetes-cnf -Docker tag for Euphrates 5.1 release: opnfv-5.1.0 -Docker tag for Euphrates with latest bugfixes: euphrates +Docker tag for master: latest Documents --------- - - Installation/configuration guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/configguide/index.html - - - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/userguide/index.html - - - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/developer/devguide/index.html - - - API Docs: http://artifacts.opnfv.org/functest/docs/index.html - - - Functest Framework presentation: http://testresults.opnfv.org/functest/framework/index.html - + * Functest Guides: https://functest.readthedocs.io/en/latest/ + * API Docs: https://functest-api.readthedocs.io/en/latest/ Version change ============== -Functest now delivers light-weigth Docker images based on Alpine 3.7. The test cases are grouped into several categories -or tiers and must be run from the corresponding container. For example, to run the test case healthcheck, the image -opnfv/functest-healthcheck shall be used. The tiers and the tests within them are explained in detail in the User Guide. - -The former Ubuntu image is not longer maintained. - -The Parser test case has its own dedicated Docker image since it requires libraries released for OpenStack Pike and -Euphrates is based on Ocata. - -The Docker images do not contain OS images (Cirros, Ubuntu, Centos, ..) anymore. A script has been created under the -ci directory (download_images.sh) which contains all the needed images for all the tests. This file can be modified by -the user since not all the images might be used. It must be executed before starting Functest and attach the needed -images as a Docker volume. See Configuration Guide for more information. - -The requirements have been split into 3 files: - * requirements.txt : lists all abstract dependencies of the OPNFV packages - * test-requirements.txt : lists all abstract dependencies required by Functest Unit Tests - * upper-constraints.txt : lists all concrete upstream dependencies required by Functest Docker container - -OPNFV (test-)requirements.txt have been updated according to stable/ocata global-requirements.txt. -Functest uses (and completes) stable/ocata upper-constraints.txt in Dockerfiles and tox configuration. -The project relies on pbr, which injects requirements into the install_requires, tests_require and/or dependency_links -arguments to setup. It also supports conditional dependencies which can be added to the requirements (e.g. dnspython>=1.14.0;python_version=='2.7') - -The way to manage logging has been centralized to a configuration file (logging.ini) which might be modified by the user. -By default, the output of executing the test cases is redirected to log files and is not displayed on the console, only result -messages and summary tables are displayed. - -The framework has been refactored and all the test cases inherit from a core class TestCase. For Feature projects who develop -test cases, 2 sub-classes have been created: - - Feature: it implements all the needed functions and the developer must only overwrite the method "execute" (e.g. Barometer) - - BashFeature: it is used if the third party test case is a shell script. This way, the execution command must be specified in - testcases.yaml as the argument (e.g. Domino, Doctor) +Key changes +----------- -An internal REST API has been introduced in Euphrates. The goal is to trigger Functest operations through an API in addition of the CLI. -This could be considered as a first step towards a pseudo micro services approach where the different test projects could expose and -consume APIs to the other test projects. + * update testcases and containers to `OpenStack master`_ and to + `Kubernetes master`_ -Euphrates 5.1 improvements -========================== +.. _`OpenStack master`: https://github.com/openstack/requirements/blob/master/upper-constraints.txt +.. _`Kubernetes master`: https://github.com/kubernetes/kubernetes -* Alpine images are now supported for ARM (arm64). -* Added Vyos_router test case. -* Updated of Rally 0.9.1 and fixed some bugs in cinder scenarios. -* Patch to allow building containers from a gerrit change. -* Selection of a subset of SNAPS test cases. -* Reorder VNF test cases and adjust timeouts in VNFs. +Key benefits +------------ + * the enduser can easily deploy its own `Functest toolchains`_ in few commands + * everyone can pick stable Functest rolling releases (latest included) + * Functest can verify VIM in production even on `Raspberry PI`_ + * all testcases can run in parallel (tested with 4 executors in our gates) + * no remaining resources detected in our gates after multiple runs +.. _`Functest toolchains`: https://github.com/collivier/ansible-role-xtesting +.. _`Raspberry PI`: https://github.com/opnfv/functest/blob/master/PI.md -Euphrates 5.1 known restrictions/issues -======================================= -+--------------+-----------+----------------------------------------------+ -| Installer | Scenario | Issue | -+==============+===========+==============================================+ -| fuel@aarch64 | any | VNF tier not supported yet. | -+--------------+-----------+----------------------------------------------+ -| | | The test cases belonging to the VNF tier | -| any | any | have been only tested on os-nosdn-nofeature | -| | | scenarios and baremetal deployments. | -+--------------+-----------+----------------------------------------------+ -| Joid | k8 | Functest does not offer test suites for | -| Compass | | Kubernetes scenarios yet. | -+--------------+-----------+----------------------------------------------+ - - -Test and installer/scenario dependencies -======================================== - -It is not always possible to run all the test cases on all the scenarios. -The scenario dependencies (installer or scenario) are detailed -in the different testcases.yaml for each tier: - - * https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/smoke/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/features/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/components/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable/euphrates - * https://git.opnfv.org/functest/tree/docker/parser/testcases.yaml?h=stable/euphrates - - -Test results -============ - -The Functest scenario status on December 15, 2017 can be seen on -http://testresults.opnfv.org/functest/euphrates/ - -Test logs are available in: - - - test results logs from CI: http://artifacts.opnfv.org (within different directories 'logs_functest_X') - - - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ - - - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/ - - - -Open JIRA tickets -================= - -+------------------+-----------------------------------------------+ -| JIRA | Description | -+==================+===============================================+ -| | | -| | | -+------------------+-----------------------------------------------+ - -All the tickets that are not blocking have been fixed or postponed -the next release. +Code quality +------------ + * pylint: 10.00/10 + * code coverage: 70% Useful links ============ - - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing - - - wiki Functest Euphrates page: https://wiki.opnfv.org/display/functest/5.+Euphrates - - - Functest repo: https://git.opnfv.org/cgit/functest - - - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ - - - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 - - - Functest IRC chan: #opnfv-functest - - - Reporting page: http://testresults.opnfv.org/reporting/euphrates.html + * Functest git repository: https://github.com/opnfv/functest + * Functest CI dashboard: http://104.154.71.112:8080/view/functest/ + * JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 + * Functest IRC channel: #opnfv-functest + * Reporting page: http://testresults.opnfv.org/reporting/master/functest/functest.html diff --git a/docs/release/release-notes/index.rst b/docs/release/release-notes/index.rst index 411e09bdf..25c2cbdfb 100644 --- a/docs/release/release-notes/index.rst +++ b/docs/release/release-notes/index.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 .. _functest-releasenotes: @@ -8,13 +7,6 @@ Functest Release Notes ********************** .. toctree:: - :numbered: :maxdepth: 4 functest-release.rst - -Revision: _sha1_ - -:Author: Jose Lausuch (jalausuch@suse.com) - -Build date: |today| diff --git a/docs/results/euphrates/5.0/apex.html b/docs/results/euphrates/5.0/apex.html index 2a75054c3..34f0c1503 100644 --- a/docs/results/euphrates/5.0/apex.html +++ b/docs/results/euphrates/5.0/apex.html @@ -307,13 +307,13 @@ $(document).ready(function (){ <th width="10%">Iteration</th> </tr> <tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=http://testresultS.opnfv.org/reporting>os-odl-fdio-ha</a></td> <td><div id="gaugeScenario5"></div></td> @@ -327,7 +327,7 @@ $(document).ready(function (){ <td>4/18</td> <td>3</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-euphrates/108/console>os-nosdn-ovs_dpdk-ha</a></td> <td><div id="gaugeScenario8"></div></td> @@ -341,9 +341,9 @@ $(document).ready(function (){ <td>16/18</td> <td>2</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-euphrates/139/console>os-nosdn-fdio-ha</a></td> <td><div id="gaugeScenario12"></div></td> @@ -357,11 +357,11 @@ $(document).ready(function (){ <td>18/20</td> <td>3</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-euphrates/121/console>os-nosdn-bar-ha</a></td> <td><div id="gaugeScenario17"></div></td> @@ -369,7 +369,7 @@ $(document).ready(function (){ <td>26/27</td> <td>4</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-euphrates/134/console>os-nosdn-nofeature-ha</a></td> <td><div id="gaugeScenario19"></div></td> @@ -413,9 +413,9 @@ $(document).ready(function (){ <td>8/30</td> <td>4</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-euphrates/114/console>os-ovn-nofeature-noha</a></td> <td><div id="gaugeScenario7"></div></td> @@ -423,9 +423,9 @@ $(document).ready(function (){ <td>6/18</td> <td>2</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-euphrates/136/console>os-odl-nofeature-noha</a></td> <td><div id="gaugeScenario10"></div></td> @@ -439,9 +439,9 @@ $(document).ready(function (){ <td>4/18</td> <td>3</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=http://testresultS.opnfv.org/reporting>os-odl-fdio_dvr-noha</a></td> <td><div id="gaugeScenario14"></div></td> @@ -461,7 +461,7 @@ $(document).ready(function (){ <td>18/18</td> <td>3</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-apex-baremetal-daily-euphrates/128/console>os-nosdn-calipso-noha</a></td> <td><div id="gaugeScenario18"></div></td> @@ -469,7 +469,7 @@ $(document).ready(function (){ <td>18/18</td> <td>3</td> </tr><tr class="tr-ok"> - + </tr> </table> @@ -489,31 +489,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -534,31 +534,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Domino * @@ -582,37 +582,37 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> FDS - + </th><th> SNAPS - + </th><th> Domino * @@ -636,34 +636,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Doctor * @@ -693,37 +693,37 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> FDS - + </th><th> SNAPS - + </th><th> Domino * @@ -747,31 +747,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -792,31 +792,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Doctor * @@ -843,31 +843,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -888,31 +888,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Doctor * @@ -942,34 +942,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Doctor * @@ -996,31 +996,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -1041,31 +1041,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Domino * @@ -1089,34 +1089,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Doctor * @@ -1143,37 +1143,37 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> FDS - + </th><th> SNAPS - + </th><th> Domino * @@ -1197,31 +1197,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Doctor * @@ -1251,31 +1251,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Doctor * @@ -1302,31 +1302,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Doctor * @@ -1356,31 +1356,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Doctor * @@ -1407,31 +1407,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Doctor * diff --git a/docs/results/euphrates/5.0/compass.html b/docs/results/euphrates/5.0/compass.html index d62ba4d2e..b7f7a20ea 100644 --- a/docs/results/euphrates/5.0/compass.html +++ b/docs/results/euphrates/5.0/compass.html @@ -223,11 +223,11 @@ $(document).ready(function (){ <th width="10%">Iteration</th> </tr> <tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-daily-euphrates/81/console>os-odl_l3-nofeature-ha</a></td> <td><div id="gaugeScenario4"></div></td> @@ -247,7 +247,7 @@ $(document).ready(function (){ <td>26/30</td> <td>9</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-daily-euphrates/72/console>os-nosdn-ovs_dpdk-ha</a></td> <td><div id="gaugeScenario8"></div></td> @@ -255,9 +255,9 @@ $(document).ready(function (){ <td>2/18</td> <td>3</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-euphrates/24/console>os-odl_l2-moon-ha</a></td> <td><div id="gaugeScenario11"></div></td> @@ -301,11 +301,11 @@ $(document).ready(function (){ <td>24/30</td> <td>4</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-daily-euphrates/86/console>os-nosdn-nofeature-noha</a></td> <td><div id="gaugeScenario7"></div></td> @@ -313,7 +313,7 @@ $(document).ready(function (){ <td>26/27</td> <td>6</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-compass-virtual-daily-euphrates/82/console>os-nosdn-ovs_dpdk-noha</a></td> <td><div id="gaugeScenario9"></div></td> @@ -327,9 +327,9 @@ $(document).ready(function (){ <td>14/18</td> <td>3</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr> </table> @@ -349,31 +349,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Domino * @@ -397,34 +397,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> SFC * @@ -451,34 +451,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Domino * @@ -502,31 +502,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Domino * @@ -550,31 +550,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Domino * @@ -598,34 +598,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> SFC * @@ -652,31 +652,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Domino * @@ -700,31 +700,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -745,31 +745,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -790,31 +790,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Domino * @@ -838,34 +838,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Domino * @@ -889,31 +889,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Domino * diff --git a/docs/results/euphrates/5.0/daisy.html b/docs/results/euphrates/5.0/daisy.html index 6de8d0895..09580fc19 100644 --- a/docs/results/euphrates/5.0/daisy.html +++ b/docs/results/euphrates/5.0/daisy.html @@ -127,9 +127,9 @@ $(document).ready(function (){ <th width="10%">Iteration</th> </tr> <tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr> </table> @@ -149,31 +149,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Domino * @@ -206,34 +206,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Domino * diff --git a/docs/results/euphrates/5.0/fuel@aarch64.html b/docs/results/euphrates/5.0/fuel@aarch64.html index c8a2f2db0..aa9f1eb66 100644 --- a/docs/results/euphrates/5.0/fuel@aarch64.html +++ b/docs/results/euphrates/5.0/fuel@aarch64.html @@ -115,7 +115,7 @@ $(document).ready(function (){ <th width="10%">Iteration</th> </tr> <tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-fuel-armband-baremetal-arm-daily-euphrates/31/console>os-nosdn-nofeature-ha</a></td> <td><div id="gaugeScenario2"></div></td> @@ -147,9 +147,9 @@ $(document).ready(function (){ <td>1/9</td> <td>1</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr> </table> @@ -169,31 +169,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * @@ -220,31 +220,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * @@ -283,34 +283,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Promise * diff --git a/docs/results/euphrates/5.0/fuel@x86.html b/docs/results/euphrates/5.0/fuel@x86.html index 3b918dd74..52b67b37f 100644 --- a/docs/results/euphrates/5.0/fuel@x86.html +++ b/docs/results/euphrates/5.0/fuel@x86.html @@ -151,7 +151,7 @@ $(document).ready(function (){ <th width="10%">Iteration</th> </tr> <tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-euphrates/40/console>os-odl-nofeature-ha</a></td> <td><div id="gaugeScenario2"></div></td> @@ -159,9 +159,9 @@ $(document).ready(function (){ <td>29/30</td> <td>6</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-fuel-baremetal-daily-euphrates/38/console>os-nosdn-ovs-ha</a></td> <td><div id="gaugeScenario5"></div></td> @@ -193,7 +193,7 @@ $(document).ready(function (){ <td>25/27</td> <td>10</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-daily-euphrates/91/console>os-odl-nofeature-noha</a></td> <td><div id="gaugeScenario3"></div></td> @@ -207,9 +207,9 @@ $(document).ready(function (){ <td>27/27</td> <td>9</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr> </table> @@ -229,31 +229,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * @@ -280,34 +280,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Promise * @@ -334,34 +334,34 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> ODL - + </th><th> SNAPS - + </th><th> Promise * @@ -388,31 +388,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * @@ -439,31 +439,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * @@ -490,31 +490,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * diff --git a/docs/results/euphrates/5.0/joid.html b/docs/results/euphrates/5.0/joid.html index d58efcca1..9b6d0f4d0 100644 --- a/docs/results/euphrates/5.0/joid.html +++ b/docs/results/euphrates/5.0/joid.html @@ -163,7 +163,7 @@ $(document).ready(function (){ <th width="10%">Iteration</th> </tr> <tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-euphrates/84/console>os-ocl-nofeature-ha</a></td> <td><div id="gaugeScenario2"></div></td> @@ -171,7 +171,7 @@ $(document).ready(function (){ <td>3/27</td> <td>10</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-euphrates/67/console>os-nosdn-openbaton-ha</a></td> <td><div id="gaugeScenario4"></div></td> @@ -185,7 +185,7 @@ $(document).ready(function (){ <td>2/15</td> <td>4</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-euphrates/81/console>os-nosdn-nofeature-ha</a></td> <td><div id="gaugeScenario7"></div></td> @@ -211,7 +211,7 @@ $(document).ready(function (){ <td>3/27</td> <td>9</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-euphrates/90/console>os-nosdn-nofeature-noha</a></td> <td><div id="gaugeScenario3"></div></td> @@ -219,9 +219,9 @@ $(document).ready(function (){ <td>15/27</td> <td>9</td> </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> - + </tr><tr class="tr-ok"> <td><a href=https://build.opnfv.org/ci/view/functest/job/functest-joid-baremetal-daily-euphrates/88/console>os-nosdn-lxd-noha</a></td> <td><div id="gaugeScenario6"></div></td> @@ -229,7 +229,7 @@ $(document).ready(function (){ <td>3/15</td> <td>9</td> </tr><tr class="tr-ok"> - + </tr> </table> @@ -249,31 +249,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -294,31 +294,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th> </tr> <tr class="tr-weather-weather"> @@ -339,31 +339,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * @@ -390,31 +390,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * @@ -441,19 +441,19 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> vPing (ssh) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> Promise * @@ -480,19 +480,19 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> vPing (ssh) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> Promise * @@ -519,31 +519,31 @@ $(document).ready(function (){ <tr> <th> Health (connection) - + </th><th> Health (api) - + </th><th> Health (dhcp) - + </th><th> vPing (ssh) - + </th><th> vPing (userdata) - + </th><th> Tempest (smoke) - + </th><th> Rally (smoke) - + </th><th> Refstack - + </th><th> SNAPS - + </th><th> Promise * diff --git a/docs/results/js/default.css b/docs/results/js/default.css index e32fa5fba..cbb564326 100644 --- a/docs/results/js/default.css +++ b/docs/results/js/default.css @@ -55,7 +55,7 @@ td{ background-color: #0095a2; } -h1 { +h1 { display: block; font-size: 2em; margin-top: 0.67em; diff --git a/docs/results/js/trend.js b/docs/results/js/trend.js index f24213382..47ee4d906 100644 --- a/docs/results/js/trend.js +++ b/docs/results/js/trend.js @@ -63,13 +63,13 @@ var trend = function(container, trend_data) { .attr("d", valueline(trend_data)) .attr("stroke", "steelblue") .attr("fill", "none"); - + trend_svg.selectAll(".dot") .data(trend_data) .enter().append("circle") .attr("r", 2.5) .attr("cx", function(d) { return trend_x(d.date); }) - .attr("cy", function(d) { return trend_y(d.score); }); + .attr("cy", function(d) { return trend_y(d.score); }); return trend; } diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt new file mode 100644 index 000000000..f4dcea507 --- /dev/null +++ b/docs/spelling_wordlist.txt @@ -0,0 +1,151 @@ +ABot +admin +adminURL +api +auth +backend +barbican +benchmarking +Benchmarking +bgpvpn +cacerts +chan +ci +Clearwater +cloudify +Cloudify +cnf +cntt +conf +config +dashboarding +defcore +Defcore +Deliverables +dhcp +dockerhub +enduser +env +EnvironmentVariable +epc +fds +frontend +functest +Functest +functionalities +guestbook +Guestbook +healthcheck +Healthcheck +http +https +ims +ini +init +interop +Interop +interoperability +intra +iptables +jenkins +Joid +juju +Juju +jumphost +Jumphost +kube +kubernetes +Kubernetes +kvm +linux +login +metadata +Metadata +middleware +nfv +nofeature +noha +nosdn +octavia +odl +onboarding +ons +Opendaylight +openstack +Openstack +opnfv +orchestrator +Orchestrator +organised +os +ovs +patrole +Patrole +plugin +pre +proxified +Reachability +reconnection +redis +refactored +refactoring +refstack +Refstack +repo +resolv +restapi +Restconf +runnable +screenshot +sdn +sfc +signalling +singlevm +src +su +subfolder +subnet +Subnet +subnets +sudo +systemctl +Telco +tenantnetwork +testcase +testcases +TLS +txt +un +url +userdata +userid +username +usr +util +utils +UUID +vdb +vEPC +verifier +versioned +vIMS +vims +Virtualised +Virtualized +vm +VM +vmready +vmtp +vnf +Vnf +vping +vPing +vrouter +vRouter +vyos +wconsole +wiki +Workflow +xrally +xtesting +yaml diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 19a1bdcb8..da5485ef0 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 ************************ @@ -6,7 +5,6 @@ Functest Developer Guide ************************ .. toctree:: - :numbered: :maxdepth: 2 ============ @@ -54,10 +52,7 @@ to the different tiers: * functest-core: https://hub.docker.com/r/opnfv/functest-core/ * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/ * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/ - * functest-features: https://hub.docker.com/r/opnfv/functest-features/ - * functest-components: https://hub.docker.com/r/opnfv/functest-components/ * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/ - * functest-parser: https://hub.docker.com/r/opnfv/functest-parser/ * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/ Standalone functest dockers are maintained for Euphrates but Alpine containers @@ -84,17 +79,14 @@ Functest internal test cases The internal test cases in Euphrates are: - * api_check * connection_check - * snaps_health_check * vping_ssh * vping_userdata * odl - * odl_netvirt * rally_full * rally_sanity - * tempest_smoke_serial - * tempest_full_parallel + * tempest_smoke + * tempest_full * cloudify_ims By internal, we mean that this particular test cases have been developed and/or @@ -108,59 +100,22 @@ The structure of this repository is detailed in `[1]`_. The main internal test cases are in the opnfv_tests subfolder of the repository, the internal test cases can be grouped by domain: - * sdn: odl, odl_netvirt, odl_fds - * openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_* + * sdn: odl, odl_fds + * openstack: connection_check, vping_ssh, vping_userdata, tempest_*, rally_* * vnf: cloudify_ims -If you want to create a new test case you will have to create a new folder under -the testcases directory (See next section for details). - -Functest external test cases -============================ -The external test cases are inherited from other OPNFV projects, especially the -feature projects. - -The external test cases are: - - * barometer - * bgpvpn - * doctor - * domino - * fds - * parser - * promise - * refstack_defcore - * snaps_smoke - * functest-odl-sfc - * orchestra_clearwaterims - * orchestra_openims - * vyos_vrouter - * juju_vepc - -External test cases integrated in previous versions but not released in -Euphrates: - - * copper - * moon - * netready - * security_scan - - -The code to run these test cases is hosted in the repository of the project. -Please note that orchestra test cases are hosted in Functest repository and not -in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in -functest as there are no dedicated projects. - +If you want to create a new test case you will have to create a new folder +under the testcases directory (See next section for details). Functest framework ================== Functest is a framework. -Historically Functest is released as a docker file, including tools, scripts and -a CLI to prepare the environment and run tests. -It simplifies the integration of external test suites in CI pipeline and provide -commodity tools to collect and display results. +Historically Functest is released as a docker file, including tools, scripts +and a CLI to prepare the environment and run tests. +It simplifies the integration of external test suites in CI pipeline and +provide commodity tools to collect and display results. Since Colorado, test categories also known as **tiers** have been created to group similar tests, provide consistent sub-lists and at the end optimize @@ -171,8 +126,8 @@ The definition of the tiers has been agreed by the testing working group. The tiers are: * healthcheck * smoke + * benchmarking * features - * components * vnf Functest abstraction classes @@ -214,35 +169,6 @@ Feature, unit and vnf_base inherit from testcase:: +--------------------+ +---------+ +------------------------+ +-----------------+ -Testcase --------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html - -Feature -------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html - -Unit ----- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html - -VNF ---- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html - -Robotframework --------------- -.. raw:: html - :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html - - -see `[5]`_ to get code samples - - Functest util classes ===================== @@ -250,8 +176,8 @@ In order to simplify the creation of test cases, Functest develops also some functions that are used by internal test cases. Several features are supported such as logger, configuration management and Openstack capabilities (tacker,..). -These functions can be found under <repo>/functest/utils and can be described as -follows:: +These functions can be found under <repo>/functest/utils and can be described +as follows:: functest/utils/ |-- config.py @@ -262,14 +188,11 @@ follows:: |-- openstack_tacker.py `-- openstack_utils.py -It is recommended to use the SNAPS-OO library for deploying OpenStack instances. -SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. - TestAPI ======= Functest is using the Test collection framework and the TestAPI developed by -the OPNFV community. See `[6]`_ for details. +the OPNFV community. See `[4]`_ for details. Reporting @@ -281,17 +204,10 @@ jinja2 templates `[3]`_. Dashboard ========= -Additional dashboarding is managed at the testing group level, see `[7]`_ for +Additional dashboarding is managed at the testing group level, see `[5]`_ for details. -======= -How TOs -======= - -See How to section on Functest wiki `[8]`_ - - ========== References ========== @@ -300,16 +216,10 @@ _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide -_`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting - -_`[4]`: https://git.opnfv.org/snaps/ - -_`[5]` : http://testresults.opnfv.org/functest/framework/index.html - -_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html +_`[3]`: https://github.com/opnfv/releng-testresults/tree/master/reporting -_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba +_`[4]`: https://wiki.opnfv.org/display/functest/2017+Beijing?preview=%2F11699623%2F11700523%2FTestAPI+-+test+results+collection+service.pptx -_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 +_`[5]`: https://lfanalytics.io/projects/lfn%2Fopnfv/dashboard IRC support chan: #opnfv-functest diff --git a/docs/testing/developer/internship/security_group/index.rst b/docs/testing/developer/internship/security_group/index.rst deleted file mode 100644 index d1cdbdd8f..000000000 --- a/docs/testing/developer/internship/security_group/index.rst +++ /dev/null @@ -1,70 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -================================== -Functest Security group test cases -================================== - -Author: Girish Sukhatankar -mentors: D.Blaisonneau, J.Lausuch, M.Richomme - -Abstract -======== - - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-??-?? | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= - - - - -Problem Statement: ------------------- - - - -Curation Phase --------------- - - - - - -Schedule: -========= - - - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| December - January | ........ | -+--------------------------+------------------------------------------+ -| January - february | ........ | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Security+groups+test+case+in+Functest - diff --git a/docs/testing/developer/internship/testapi_evolution/index.rst b/docs/testing/developer/internship/testapi_evolution/index.rst deleted file mode 100644 index 3038d0ac6..000000000 --- a/docs/testing/developer/internship/testapi_evolution/index.rst +++ /dev/null @@ -1,237 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -================= -TestAPI evolution -================= - -Author: Sakala Venkata Krishna Rohit -Mentors: S. Feng, J.Lausuch, M.Richomme - -Abstract -======== - -The TestAPI is used by all the test opnfv projects to report results. -It is also used to declare projects, test cases and labs. A major refactoring -has been done in Colorado with the introduction of swagger. The TestAPI is defined in Functest -developer guide. The purpose of this project is to add more features to the TestAPI that automate -the tasks that are done manually now, though there are tasks other than automation. - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ -| 2017-02-17 | 0.0.2 | S.V.K Rohit | End of the Internship | -| | | (IIIT Hyderabad) | | -+------------+----------+------------------+------------------------+ - -Overview: -========= - -The internhip time period was from Nov 14th to Feb 17th. The project prosposal page is here `[1]`_. -The intern project was assigned to Svk Rohit and was mentored by S. Feng, J.Lausuch, M.Richomme. -The link to the patches submitted is `[2]`_. The internship was successfully completed and the -documentation is as follows. - -Problem Statement: ------------------- - -The problem statement could be divided into pending features that needed to be added into TestAPI -repo. The following were to be accomplished within the internship time frame. - -* **Add verification jenkins job for the TestAPI code** - The purpose of this job is to verify whehter the unit tests are successful or not with the - inclusion of the patchset submitted. - -* **Automatic update of opnfv/testapi docker image** - The docker image of TestAPI is hosted in the opnfv docker hub. To ensure that the TestAPI image - is always updated with the repository, automatic updation of the image is necessary and a job - is triggered whenever a new patch gets merged. - -* **Automation deployment of testresults.opnfv.org/test/ website** - In the same manner as the docker image of TestAPI is updated, the TestAPI website needs to be - in sync with the repository code. So, a job has been added to the opnfv jenkins ci for the - updation of the testresults website. - -* **Generate static documentation of TestAPI calls** - The purpose of this is to give an static/offline view of TestAPI. If someone wants to have a - look at the Restful APIs of TestAPI, he/she does't need to go to the website, he can download - a html page and view it anytime. - -* **Backup MongoDB of TestAPI** - The mongoDB needs to be backed up every week. Till now it was done manually, but due to this - internship, it is now automated using a jenkins job. - -* **Add token based authorization to the TestAPI calls** - The token based authorization was implemented to ensure that only ci_pods could access the - database. Authentication has been added to only delete/put/post requests. - -Curation Phase: ---------------- - -The curation phase was the first 3 to 4 weeks of the internship. This phase was to get familiar -with the TestAPI code and functionality and propose the solutions/tools for the tasks mentioned -above. Swagger codegen was choosen out of the four tools proposed `[3]`_ for generating static -documentaion. - -Also, specific amount of time was spent on the script flow of the jenkins jobs. The automatic -deployment task involves accessing a remote server from inside the jenkins build. The deployment -had to be done only after the docker image update is done. For these constraints to satisfy, a -multijob jenkins job was choosen instead of a freestyle job. - -Important Links: ----------------- - -* MongoDB Backup Link - `[4]`_ -* Static Documentation - `[5]`_ -* TestAPI Token addition to ci_pods - `[6]`_ - -Schedule: -========= - -The progress and completion of the tasks is described in the below table. - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| Nov 14th - Dec 31st | Understand TestAPI code and the | -| | requirements. | -+--------------------------+------------------------------------------+ -| Jan 1st - Jan 7th | Add jenkins job to create static | -| | documentation and write build scripts. | -+--------------------------+------------------------------------------+ -| Jan 8th - Jan 21st | Add verification jenkins job for unit | -| | tests. | -+--------------------------+------------------------------------------+ -| Jan 22nd - Jan 28th | Add jenkins job for mongodb backup | -| | | -+--------------------------+------------------------------------------+ -| Jan 29th - Feb 11th | Enable automatic deployment of | -| | testresults.opnfv.org/test/ | -+--------------------------+------------------------------------------+ -| Feb 12th - Feb 17th | Add token based authentication | -| | | -+--------------------------+------------------------------------------+ - -FAQ's -===== - -This section lists the problems that I have faced and the understanding that I have acquired during -the internship. This section may help other developers in solving any errors casused because of the -code written as a part of this internship. - - -TestAPI -------- - -What is the difference between defining data_file as "/etc/.." and "etc/.." in setup.cfg ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If in the setup.cfg, it is defined as - -[files] -data_files = -etc/a.conf = etc/a.conf.sample - -then it ends up installed in the /usr/etc/. With this configuration, it would be installed -correctly within a venv. but when it is defined as - -[files] -data_files = -/etc/a.conf = etc/a.conf.sample - -then it ends up installed on the root of the filesystem instead of properly be installed within the -venv. - -Which attribute does swagger-codegen uses as the title in the generation of document generation ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It uses the nickname of the api call in swagger as the title in the generation of the document -generation. - -Does swagger-codegen take more than one yaml file as input ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -No, swagger-codegen only takes one yaml file as input to its jar file. If there more than one yaml -file, one needs to merge them and give it as an input keeping mind the swagger specs. - - -Jenkins & JJB -------------- - -Which scm macro is used for verification jenkins jobs ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are two macros for scm one is git-scm and other git-scm-gerrit. git-scm-gerrit is used for -verification jenkins job. - -Does the virtualenv created in one build script exists in other build scripts too ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -No, the virtualenv created in one build script only exists in that build script/shell. - -What parameters are needed for the scm macros ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Project and Branch are the two parameters needed for scm macros. - -What is the directory inside the jenkins build ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The directory of the jenkins build is the directory of the repo. `ls $WORKSPACE` command will give -you all the contents of the directory. - -How to include a bash script in jenkins job yaml file ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An example might be apt here as an answer. - -builders: - - shell: - !include-raw: include-raw001-hello-world.sh - - -How do you make a build server run on a specific machine ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It can be done by defining a label parameter 'SLAVE_LABEL' or in OPNFV , there are macros for each -server, one can use those parameter macros. -Ex: opnfv-build-defaults. Note, if we use macro, then no need to define GIT_BASE, but if one uses -SLAVE_LABEL, one needs to define a parameter GIT_BASE. This is because macro already has GIT_BASE -defined. - -What job style should be used when there is a situation like one build should trigger other builds -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -or when different build scripts need to be run on different machines ? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -MultiJob style should be used as it has phases where each phase can be taken as a build scipt and -can have its own parameters by which one can define the SLAVE_LABEL parameter. - -References: -=========== - -_`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+testapi+evolution - -_`[2]` : https://gerrit.opnfv.org/gerrit/#/q/status:merged+owner:%22Rohit+Sakala+%253Crohitsakala%2540gmail.com%253E%22 - -_`[3]` : https://docs.google.com/document/d/1jWwVZ1ZpKgKcOS_zSz2KzX1nwg4BXxzBxcwkesl7krw/edit?usp=sharing - -_`[4]` : http://artifacts.opnfv.org/testapibackup.html - -_`[5]` : http://artifacts.opnfv.org/releng/docs/testapi.html - -_`[6]` : http://artifacts.opnfv.org/functest/docs/devguide/index.html#test-api-authorization diff --git a/docs/testing/developer/internship/unit_tests/index.rst b/docs/testing/developer/internship/unit_tests/index.rst deleted file mode 100644 index a117c8609..000000000 --- a/docs/testing/developer/internship/unit_tests/index.rst +++ /dev/null @@ -1,126 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -=================== -Functest Unit tests -=================== - -Author: Ashish Kumar -Mentors: H.Yao, J.Lausuch, M.Richomme - -Abstract -======== - - -Version history -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-11-14 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ -| 2017-03-31 | 0.0.2 | Ashish Kumar | During the | -| | | (IIIT Hyderabad) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= -Functest project is developing and integrating functional test cases for OPNFV -and it is part of OPNFV since the beginning. Functest develops its own testcases -and framework. This framework includes several utility libraries. The Project is -growing rapidly with more features, tests added as per requirement. It becomes -the responsibility of every developer to maintain the integrity of code i.e. new -patch should not break the previous functionality of the project. To automate this -process of software development, we should write unit tests and add them to CI so -that when a new patch is ready to merge, we shouldn't allow those which are breaking -previous unit tests or decreasing the coverage. - - - -Problem Statement: ------------------- -The goal of the intership consists in creating unit test suites on Functest code -with good code coverage (>80%) and integrate it in continuous integration in order -to consolidate existing code. - - -Curation Phase --------------- -The curation phase was the first 3 to 4 weeks of the internship. This phase was to get -familiar with the functest code and functionality and explore the solutions for unit -testing in other projects and come up with the strategy for writing unit tests in functest. - -In this phase we decided, -- Coverage should be 80%. There are some functions like __init__, getter, setter and other - private methods for which writing unit test is a tedious job, so we are leaving these methods - for now. -- Do method wise testing for every module. -- Use mock for external or third party services, system calls and other external library calls - which could impact the behaviour of system during the run of unit test. -- Add it in jenkins as passing criteria for patches. -- Write tests in modular way so that it can help to serve as a form of documentation. - - - -Schedule: -========= -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| Nov 14th - Nov 28th | 1. Learn Functest Project Business | -| | 2. Set up the development environment | -| | 3. Run Functest code | -+--------------------------+------------------------------------------+ -| Nov 28th - Dec.9th | 1. Explore Unit Testing Strategy, | -| | 2. Learn about Mock in python | -+--------------------------+------------------------------------------+ -| Dec 12th - Dec 23rd | Implement Unit Tests for CLI | -| | | -+--------------------------+------------------------------------------+ -| Dec 26th - Jan 6th | Implement Unit Tests for Utils | -| | | -+--------------------------+------------------------------------------+ -| Jan 9th - Jan 20th | Implement Unit Tests for CI | -| | | -+--------------------------+------------------------------------------+ -| Jan 23rd - Feb 3rd | Implement Unit Tests for Core | -| | | -+--------------------------+------------------------------------------+ -| Feb 6th - Feb 17th | Implement Unit Tests for | -| | opnfv_tests/openstack/tempest | -+--------------------------+------------------------------------------+ -| Feb 20th - Mar 3rd | Implement Unit Tests for | -| | opnfv_tests/openstack/rally | -+--------------------------+------------------------------------------+ -| Mar 6th - Mar 17th | Implement Unit Tests for | -| | opnfv_tests/vnf/ims | -+--------------------------+------------------------------------------+ -| Mar 20th - Mar 31st | Recheck and Increase Coverage for all | -| | modules > 80% | -+--------------------------+------------------------------------------+ -| Apr 3rd - Apr 14th | Add CI Gating for unit tests | -| | | -+--------------------------+------------------------------------------+ -| Apr 17th - Apr 28th | Use Tox Utility, Documentation | -| | | -+--------------------------+------------------------------------------+ -| Apr 28th - End | Bug Fixing | -| | | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Functest+unit+tests diff --git a/docs/testing/developer/internship/vnf_catalog/index.rst b/docs/testing/developer/internship/vnf_catalog/index.rst deleted file mode 100644 index df7633391..000000000 --- a/docs/testing/developer/internship/vnf_catalog/index.rst +++ /dev/null @@ -1,170 +0,0 @@ -======= -License -======= - -Functest Docs are licensed under a Creative Commons Attribution 4.0 -International License. -You should have received a copy of the license along with this. -If not, see <http://creativecommons.org/licenses/by/4.0/>. - -======================= -Open Source VNF Catalog -======================= - -Author: Kumar Rishabh -Mentors: B.Souville, M.Richomme, J.Lausuch - -Abstract -======== - - - -Version hissory -=============== - -+------------+----------+------------------+------------------------+ -| **Date** | **Ver.** | **Author** | **Comment** | -| | | | | -+------------+----------+------------------+------------------------+ -| 2016-12-12 | 0.0.1 | Morgan Richomme | Beginning of the | -| | | (Orange) | Internship | -+------------+----------+------------------+------------------------+ - - -Overview: -========= - - -This project aims to create an Open Source catalog for reference and -classification of Virtual Network Functions (VNFs)s available on -Internet. The classification method proposed will be in sync with the -requirements of Telcos active in NFV landscape. The project aims to have -running web platform similar to [1] by the mid of internship (2nd week -of March). By the penultimate month of internship I aim to have fully -functional implementation of an Open Source VNF in functest. - - -Problem Statement: ------------------- - -OPNFV aims to be the reference platform for development, -standardization and integration of Open Source NFV components across -various Open Source Platforms. It mainly deals with the infrastructure -through the Network Function Virtualization Infrastructure (NFVI) and -Virtual Infrastructure manager (VIM). The MANO (Management and -orchestration) stacks have been introduced recently. VNFs are not -directly in OPNFV scope, however VNFs are needed to test and qualify the -infrastructure. In this regard having a common curated Open Source -Reference VNF catalog would be of immense importance to community. - -Since major focus of OPNFV is Telcos, a curated platform targeted from -industry point of view would be very useful. We plan to divide the -entire project into three major phases(with some iterative improvements -and overlaps) - - -Curation Phase --------------- -This phase pertains to studying various Open Source VNFs available and -classification of them based on certain parameters. The parameters that -I currently have in mind are: - * Developer Metrics: These pertain to repo characteristics of VNF under - study - * Usage Statistics - Activity, Number of Commits, stars - * Maturity Statistics - For instance if an NFV community decides code - coverage is important for them, it shows the NFV community is serious - about taking the project forward - * Technical Tagging: These are the tags that pertain to technical - characteristics of a VNF - * Broad Use Cases - Whether the VNF fits strictly in IaaS, PaaS or - SaaS layer or is an hybrid of two/all. - * Generic Use Cases - This in my opinion is the broadest - classification category. For instance a VNF could be built with a - broad idea of powering IOT devices at home or from usage perspective - of Telco Operators (vFW, vEPC, vIMS, vCDN, vAAA, vCPE,...).`[2]`_ - * Fields of Application - * Library Status - Whether APIs are standardized, support RESTful - services. - * Dependency Forwarding Graph - This is pretty complex tagging - mechanism. It essentially tries to establish a graph relationship - between the VNFs (elementary VNFs are used in Service Function - Chaining chains such as Firewall, DPI, content enrichment,..). In my - opinion this is useful immensely. This will allow users to go to - platform and ask a question like - “I have this X tech stack to - support, Y and Z are my use cases, which NFVs should I use to support - this. - * Visitor Score - Based on `[1]`_ I plan to evolve a visitor score for - the platform. This will allow users to score an NFV on certain - parameters, may be post comments. - -**I plan to use the above three scores and evolve cumulative score which -will be displayed next to each of the NFV on the platform.** - - * Platform building phase - This will involve erecting a Web Platform - which will be similar to this `[1]`_. I am decently familiar with - Django and hence I will write the platform in Django. There are two - action plans that I have in mind right now. Either I can start writing - the platform simultaneously which will help keep track of my progress - or I can write the platform after 1.5 - 2 months into the internship. - Either way I aim to have the Web Platform ready by March 12. - - * Functest VNF implementation phase - This is the last phase that will - involve writing a fully functional implementation of an Open Source VNF - into Functest. I will undertake this after I am 3 months into the - internship. I have a decent familiarity with python and hence I think - it shouldn’t be too difficult. I need to decide how complex the VNFI - should undertake this exercise for (e.g. AAA such as free radius sounds - relatively easy, vCDN is much more challenging). - This will be decided in consent with my mentors. - - - - -Schedule: -========= -I plan to take this project in 6 months time frame as I want to use it -as a chance to read more about NFVs in particular and SDN in general - - -+--------------------------+------------------------------------------+ -| **Date** | **Comment** | -| | | -+--------------------------+------------------------------------------+ -| December 12 - January 12 | Study the above mentioned metrics | -| | Decide which of them are important for | -| | community (and which are not). | -+--------------------------+------------------------------------------+ -| January 12 - January 27 | Make a database for the above studied | -| | metrics and evolve it further based on | -| | Mentors’ input. + associated API | -+--------------------------+------------------------------------------+ -| January 27 - February 5 | Compile the data collected above and make| -| | it public. Although I can keep everything| -| | public from the beginning too. My | -| | rationale of not making the entire data | -| | public in initial stage as the errors | -| | caused by me could be misleading for | -| | developers. | -+--------------------------+------------------------------------------+ -| February 5 - March 5 | Erect the Web Platform and release it | -| | for restricted group for alpha testing. | -+--------------------------+------------------------------------------+ -| March 5 - March 12 | Make it public. Release it to public for | -| | beta testing. Fix Bugs. | -+--------------------------+------------------------------------------+ -| March 12 - April 12 | Start working on implementation of an | -| | Open Source VNF in Functest. | -+--------------------------+------------------------------------------+ -| April 12 - May 12 | I will decided what to do here based on | -| | discussion with mentors. | -+--------------------------+------------------------------------------+ - - -References: -=========== - -.. _`[1]` : Openhub: https://www.openhub.net/explore/projects - -.. _`[2]` : ETSI NFV White Paper: https://portal.etsi.org/Portals/0/TBpages/NFV/Docs/NFV_White_Paper3.pdf - -.. _`[3]` : https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Open+Source+VNF+catalog diff --git a/docs/testing/user/configguide/ci.rst b/docs/testing/user/configguide/ci.rst deleted file mode 100644 index f3901d867..000000000 --- a/docs/testing/user/configguide/ci.rst +++ /dev/null @@ -1,13 +0,0 @@ -Integration in CI -================= -In CI we use the Docker images and execute the appropriate commands within the -container from Jenkins. - -4 steps have been defined:: - * functest-cleanup: clean existing functest dockers on the jumphost - * functest-daily: run dockers opnfv/functest-* (healthcheck, smoke, features, vnf) - * functest-store-results: push logs to artifacts - -See `[3]`_ for details. - -.. _`[3]`: https://git.opnfv.org/releng/tree/jjb/functest/functest-daily-jobs.yml diff --git a/docs/testing/user/configguide/configguide.rst b/docs/testing/user/configguide/configguide.rst index d63286475..47c25adb1 100644 --- a/docs/testing/user/configguide/configguide.rst +++ b/docs/testing/user/configguide/configguide.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 Installation and configuration @@ -10,201 +9,348 @@ the refactoring on dependency management should allow the creation of light and fully customized docker images. -Functest Dockers ----------------- +Functest Dockers for OpenStack deployment +----------------------------------------- Docker images are available on the dockerhub: - * opnfv/functest-core * opnfv/functest-healthcheck * opnfv/functest-smoke - * opnfv/functest-features - * opnfv/functest-components + * opnfv/functest-smoke-cntt + * opnfv/functest-benchmarking + * opnfv/functest-benchmarking-cntt * opnfv/functest-vnf - * opnfv/functest-parser - * opnfv/functest-restapi - -The tag "opnfv-5.0.0" is the official release image in Euphrates, but you can also pull "euphrates" -tag as it is being maintained by Functest team and might include bugfixes. - -The Functest docker container environment can -in principle- be also -used with non-OPNFV official installers (e.g. 'devstack'), with the -**disclaimer** that support for such environments is outside of the -scope and responsibility of the OPNFV project. Preparing your environment --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^ cat env:: - INSTALLER_TYPE=XXX - INSTALLER_IP=XXX - EXTERNAL_NETWORK=XXX - DEPLOY_SCENARIO=XXX + DEPLOY_SCENARIO=XXX # if not os-nosdn-nofeature-noha scenario + NAMESERVER=XXX # if not 8.8.8.8 + EXTERNAL_NETWORK=XXX # if not first network with router:external=True + DASHBOARD_URL=XXX # else tempest_horizon will be skipped + NEW_USER_ROLE=XXX # if not member + SDN_CONTROLLER_IP=XXX # if odl scenario + VOLUME_DEVICE_NAME=XXX # if not vdb + FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios See section on environment variables for details. -cat openstack.creds:: +cat env_file:: export OS_AUTH_URL=XXX export OS_USER_DOMAIN_NAME=XXX export OS_PROJECT_DOMAIN_NAME=XXX export OS_USERNAME=XXX - export OS_TENANT_NAME=XXX export OS_PROJECT_NAME=XXX export OS_PASSWORD=XXX - export OS_VOLUME_API_VERSION=XXX - export OS_IDENTITY_API_VERSION=XXX - export OS_IMAGE_API_VERSION=XXX + export OS_IDENTITY_API_VERSION=3 + export OS_REGION_NAME=XXX See section on OpenStack credentials for details. Create a directory for the different images (attached as a Docker volume):: - mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh?h=stable/euphrates | bash -s -- images && ls -1 images/* - - images/CentOS-7-aarch64-GenericCloud.qcow2 - images/CentOS-7-aarch64-GenericCloud.qcow2.xz - images/CentOS-7-x86_64-GenericCloud.qcow2 - images/cirros-0.4.0-x86_64-disk.img - images/cirros-0.4.0-x86_64-lxc.tar.gz - images/cirros-d161201-aarch64-disk.img - images/cirros-d161201-aarch64-initramfs - images/cirros-d161201-aarch64-kernel - images/cloudify-manager-premium-4.0.1.qcow2 - images/img - images/trusty-server-cloudimg-amd64-disk1.img + mkdir -p images && wget -q -O- https://git.opnfv.org/functest/plain/functest/ci/download_images.sh | sh -s -- images && ls -1 images/* + + images/cirros-0.5.1-aarch64-disk.img + images/cirros-0.5.1-x86_64-disk.img + images/cloudify-docker-manager-community-19.01.24.tar + images/Fedora-Cloud-Base-30-1.2.x86_64.qcow2 + images/shaker-image-1.3.0+stretch.qcow2 images/ubuntu-14.04-server-cloudimg-amd64-disk1.img images/ubuntu-14.04-server-cloudimg-arm64-uefi1.img images/ubuntu-16.04-server-cloudimg-amd64-disk1.img - images/vyos-1.1.7.img - + images/vyos-1.1.8-amd64.qcow2 Testing healthcheck suite --------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^ Run healthcheck suite:: sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ opnfv/functest-healthcheck Results shall be displayed as follows:: - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:02 | PASS | - | api_check | functest | healthcheck | 04:57 | PASS | - | snaps_health_check | functest | healthcheck | 00:51 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ - NOTE: the duration is a reference and it might vary depending on your SUT. + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:03 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:05 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | + | vmready1 | functest | healthcheck | 00:06 | PASS | + | vmready2 | functest | healthcheck | 00:08 | PASS | + | singlevm1 | functest | healthcheck | 00:32 | PASS | + | singlevm2 | functest | healthcheck | 00:37 | PASS | + | vping_ssh | functest | healthcheck | 00:46 | PASS | + | vping_userdata | functest | healthcheck | 00:39 | PASS | + | cinder_test | functest | healthcheck | 01:05 | PASS | + | tempest_smoke | functest | healthcheck | 05:39 | PASS | + | tempest_horizon | functest | healthcheck | 01:05 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ + +NOTE: the duration is a reference and it might vary depending on your SUT. Testing smoke suite -------------------- +^^^^^^^^^^^^^^^^^^^ Run smoke suite:: sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ opnfv/functest-smoke Results shall be displayed as follows:: - +------------------------------+------------------+---------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------+------------------+----------------+ - | vping_ssh | functest | smoke | 01:19 | PASS | - | vping_userdata | functest | smoke | 01:56 | PASS | - | tempest_smoke_serial | functest | smoke | 26:30 | PASS | - | rally_sanity | functest | smoke | 19:42 | PASS | - | refstack_defcore | functest | smoke | 22:00 | PASS | - | snaps_smoke | functest | smoke | 41:14 | PASS | - | odl | functest | smoke | 00:16 | PASS | - | odl_netvirt | functest | smoke | 00:00 | SKIP | - | fds | functest | smoke | 00:00 | SKIP | - +------------------------------+------------------+---------------+------------------+----------------+ - Note: if the scenario does not support some tests, they are indicated as SKIP. - See User guide for details. - -Testing features suite ----------------------- - -Run features suite:: + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron | functest | smoke | 15:30 | PASS | + | tempest_cinder | functest | smoke | 02:01 | PASS | + | tempest_keystone | functest | smoke | 01:17 | PASS | + | tempest_heat | functest | smoke | 22:14 | PASS | + | tempest_telemetry | functest | smoke | 00:00 | SKIP | + | rally_sanity | functest | smoke | 17:24 | PASS | + | refstack_compute | functest | smoke | 07:03 | PASS | + | refstack_object | functest | smoke | 02:09 | PASS | + | refstack_platform | functest | smoke | 07:31 | PASS | + | tempest_full | functest | smoke | 41:52 | PASS | + | tempest_scenario | functest | smoke | 08:42 | PASS | + | tempest_slow | functest | smoke | 43:42 | PASS | + | patrole_admin | functest | smoke | 21:06 | PASS | + | patrole_member | functest | smoke | 21:23 | PASS | + | patrole_reader | functest | smoke | 21:56 | PASS | + | tempest_barbican | functest | smoke | 02:30 | PASS | + | tempest_octavia | functest | smoke | 00:00 | SKIP | + | tempest_cyborg | functest | smoke | 00:00 | SKIP | + +---------------------------+------------------+---------------+------------------+----------------+ + +Note: if the scenario does not support some tests, they are indicated as SKIP. +See User guide for details. + +Testing smoke CNTT suite +^^^^^^^^^^^^^^^^^^^^^^^^ + +Run smoke-cntt suite:: sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-features + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-smoke-cntt Results shall be displayed as follows:: - +---------------------------+--------------------------+------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +---------------------------+--------------------------+------------------+------------------+----------------+ - | promise | promise | features | 00:00 | SKIP | - | bgpvpn | sdnvpn | features | 00:00 | SKIP | - | security_scan | securityscanning | features | 00:00 | SKIP | - | functest-odl-sfc | sfc | features | 00:00 | SKIP | - | domino-multinode | domino | features | 00:00 | SKIP | - | barometercollectd | barometer | features | 00:00 | SKIP | - +---------------------------+--------------------------+------------------+------------------+----------------+ - Note: if the scenario does not support some tests, they are indicated as SKIP. - See User guide for details. + +-------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron_cntt | functest | smoke | 11:35 | PASS | + | tempest_cinder_cntt | functest | smoke | 01:58 | PASS | + | tempest_keystone_cntt | functest | smoke | 01:13 | PASS | + | tempest_heat_cntt | functest | smoke | 22:32 | PASS | + | rally_sanity_cntt | functest | smoke | 17:16 | PASS | + | tempest_full_cntt | functest | smoke | 41:13 | PASS | + | tempest_scenario_cntt | functest | smoke | 08:57 | PASS | + | tempest_slow_cntt | functest | smoke | 35:58 | PASS | + +-------------------------------+------------------+---------------+------------------+----------------+ -Testing components suite ------------------------- +Testing benchmarking suite +^^^^^^^^^^^^^^^^^^^^^^^^^^ -Run components suite:: +Run benchmarking suite:: sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-components + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-benchmarking Results shall be displayed as follows:: - +-------------------------------+------------------+--------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +-------------------------------+------------------+--------------------+------------------+----------------+ - | tempest_full_parallel | functest | components | 102:48 | PASS | - | rally_full | functest | components | 160:58 | PASS | - +-------------------------------+------------------+--------------------+------------------+----------------+ + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 93:03 | PASS | + | rally_jobs | functest | benchmarking | 27:05 | PASS | + | vmtp | functest | benchmarking | 17:56 | PASS | + | shaker | functest | benchmarking | 24:02 | PASS | + +--------------------+------------------+----------------------+------------------+----------------+ + +Note: if the scenario does not support some tests, they are indicated as SKIP. +See User guide for details. + +Testing benchmarking CNTT suite +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Run benchmarking-cntt suite:: + + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-benchmarking-cntt + +Results shall be displayed as follows:: + + +-------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+----------------------+------------------+----------------+ + | rally_full_cntt | functest | benchmarking | 89:52 | PASS | + | rally_jobs_cntt | functest | benchmarking | 19:39 | PASS | + | vmtp | functest | benchmarking | 16:59 | PASS | + | shaker | functest | benchmarking | 23:43 | PASS | + +-------------------------+------------------+----------------------+------------------+----------------+ Testing vnf suite ------------------ +^^^^^^^^^^^^^^^^^ Run vnf suite:: -sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-vnf + sudo docker run --env-file env \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ + -v $(pwd)/images:/home/opnfv/functest/images \ + opnfv/functest-vnf + +Results shall be displayed as follows:: + + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify | functest | vnf | 05:08 | PASS | + | cloudify_ims | functest | vnf | 24:46 | PASS | + | heat_ims | functest | vnf | 33:12 | PASS | + | vyos_vrouter | functest | vnf | 15:53 | PASS | + | juju_epc | functest | vnf | 27:52 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ + +Functest Dockers for Kubernetes deployment +------------------------------------------ +Docker images are available on the dockerhub: + + * opnfv/functest-kubernetes-healthcheck + * opnfv/functest-kubernetes-smoke + * opnfv/functest-kubernetes-security + * opnfv/functest-kubernetes-benchmarking + * opnfv/functest-kubernetes-cnf + +Preparing your environment +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +cat env:: + + DEPLOY_SCENARIO=k8s-XXX + +Testing healthcheck suite +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Run healthcheck suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-healthcheck + +A config file in the current dir 'config' is also required, which should be +volume mapped to ~/.kube/config inside kubernetes container. + +Results shall be displayed as follows:: + + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_quick | functest | healthcheck | 00:18 | PASS | + | k8s_smoke | functest | healthcheck | 01:14 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ + +Testing smoke suite +^^^^^^^^^^^^^^^^^^^ + +Run smoke suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-smoke + +Results shall be displayed as follows:: + + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 94:26 | PASS | + | xrally_kubernetes | functest | smoke | 13:05 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ + +Testing security suite +^^^^^^^^^^^^^^^^^^^^^^ + +Run smoke suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-security + +Results shall be displayed as follows:: + + +---------------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+------------------+------------------+----------------+ + | kube_hunter | functest | security | 00:19 | PASS | + | kube_bench_master | functest | security | 00:02 | PASS | + | kube_bench_node | functest | security | 00:01 | PASS | + +---------------------------+------------------+------------------+------------------+----------------+ + +Testing benchmarking suite +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Run benchmarking suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-benchmarking Results shall be displayed as follows:: - +---------------------------------+------------------+--------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +---------------------------------+------------------+--------------+------------------+----------------+ - | cloudify_ims | functest | vnf | 21:25 | PASS | - | orchestra_openims | functest | vnf | 11:02 | FAIL | - | orchestra_clearwaterims | functest | vnf | 09:13 | FAIL | - +---------------------------------+------------------+--------------+------------------+----------------+ + +--------------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------------+------------------+----------------------+------------------+----------------+ + | xrally_kubernetes_full | functest | benchmarking | 34:16 | PASS | + +--------------------------------+------------------+----------------------+------------------+----------------+ + +Testing cnf suite +^^^^^^^^^^^^^^^^^ +Run cnf suite:: + + sudo docker run -it --env-file env \ + -v $(pwd)/config:/root/.kube/config \ + opnfv/functest-kubernetes-cnf + +Results shall be displayed as follows:: + + +-------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+--------------+------------------+----------------+ + | k8s_vims | functest | cnf | 09:06 | PASS | + | helm_vims | functest | cnf | 08:54 | PASS | + | cnf_conformance | functest | cnf | 02:00 | PASS | + +-------------------------+------------------+--------------+------------------+----------------+ Environment variables ===================== -Several environement variables may be specified: - * INSTALLER_TYPE=(apex|compass|daisy|fuel|joid) +Several environment variables may be specified: + * INSTALLER_IP=<Specific IP Address> * DEPLOY_SCENARIO=<vim>-<controller>-<nfv_feature>-<ha_mode> + * NAMESERVER=XXX # if not 8.8.8.8 + * VOLUME_DEVICE_NAME=XXX # if not vdb + * EXTERNAL_NETWORK=XXX # if not first network with router:external=True + * NEW_USER_ROLE=XXX # if not member - -INSTALLER IP may be required by some test cases like SFC or Barometer in order -to access the installer node and the deployment. +INSTALLER_IP is required by Barometer in order to access the installer node and +the deployment. The format for the DEPLOY_SCENARIO env variable can be described as follows: * vim: (os|k8s) = OpenStack or Kubernetes @@ -221,8 +367,8 @@ release note. **NOTE:** The scenario name is mainly used to automatically detect if a test suite is runnable or not (e.g. it will prevent ODL test suite to be -run on 'nosdn' scenarios). If not set, Functest will try to run the default test -cases that might not include SDN controller or a specific feature +run on 'nosdn' scenarios). If not set, Functest will try to run the default +test cases that might not include SDN controller or a specific feature. **NOTE:** An HA scenario means that 3 OpenStack controller nodes are deployed. It does not necessarily mean that the whole system is HA. See @@ -234,9 +380,9 @@ to the Functest Docker Container, using the -e only relevant to Jenkins CI invoked testing and **should not be used** when performing manual test scenarios: - * NODE_NAME = <Test POD Name> - * BUILD_TAG = <Jenkins Build Tag> - * CI_DEBUG = <DebugTraceValue> + * INSTALLER_TYPE=(apex|compass|daisy|fuel) + * NODE_NAME=<Test POD Name> + * BUILD_TAG=<Jenkins Build Tag> where: @@ -256,11 +402,6 @@ where: which are independently pushed to the results database from different Jenkins jobs. DO NOT USE THIS OPTION IN MANUAL TEST SCENARIOS. - * <DebugTraceValue> = "true" or "false" - Default = "false", if not specified - If "true" is specified, then additional debug trace - text can be sent to the test results file / log files - and also to the standard console output. Openstack credentials @@ -268,7 +409,7 @@ Openstack credentials OpenStack credentials are mandatory and must be provided to Functest. When running the command "functest env prepare", the framework will automatically look for the Openstack credentials file -"/home/opnfv/functest/conf/openstack.creds" and will exit with +"/home/opnfv/functest/conf/env_file" and will exit with error if it is not present or is empty. There are 2 ways to provide that file: @@ -276,7 +417,7 @@ There are 2 ways to provide that file: * by using a Docker volume with -v option when creating the Docker container. This is referred to in docker documentation as "Bind Mounting". See the usage of this parameter in the following chapter. - * or creating manually the file '/home/opnfv/functest/conf/openstack.creds' + * or creating manually the file '/home/opnfv/functest/conf/env_file' inside the running container and pasting the credentials in it. Consult your installer guide for further details. This is however not instructed in this document. @@ -317,194 +458,6 @@ be:: export OS_INSECURE=true -Functest docker container directory structure -============================================= -Inside the Functest docker container, the following directory structure -should now be in place:: - - `-- - |- home - | |-- opnfv - | | `- functest - | | |-- conf - | | `-- results - | `-- repos - | `-- vnfs - |- src - | |-- tempest - | |-- vims-test - | |-- odl_test - | `-- fds - `- usr - `- lib - `- python2.7 - `- site-packages - `- functest - |-- ... - -Underneath the '/home/opnfv/functest' directory, the Functest docker container -includes two main directories: - - * The **conf** directory stores configuration files (e.g. the - OpenStack creds are stored in path '/home/opnfv/functest/conf/openstack.creds'), - * the **results** directory stores some temporary result log files - -src and repos directories are used to host third party code used for the tests. - -The structure of functest repo can be described as follows:: - - |-- INFO - |-- LICENSE - |-- api - | `-- apidoc - |-- build.sh - |-- commons - |-- docker - | |-- components - | |-- core - | |-- features - | |-- healthcheck - | |-- smoke - | |-- vnf - | |-- parser - | |-- restapi - | |-- thirdparty-requirements.txt - |-- docs - | |-- com - | |-- images - | |-- release - | | `-- release-notes - | |-- results - | | testing - | | |-- developer - | | `-- user - | | |-- configguide - | | `-- userguide - `-- functest - |-- api - | |-- base.py - | |-- server.py - | |-- urls.py - | |-- common - | | |-- api_utils.py - | | |-- thread.py - | `-- resources - | `-- v1 - | |-- creds.py - | |-- envs.py - | |-- testcases.py - | |-- tiers.py - | |-- tasks.py - | `-- database - | |-- db.py - | `-- v1 - | |-- handlers.py - | |-- models.py - | `-- swagger - |-- ci - │ |-- check_deployment.py - │ |-- config_aarch64_patch.yaml - │ |-- config_functest.yaml - │ |-- config_patch.yaml - │ |-- download_images.sh - │ |-- logging.ini - │ |-- rally_aarch64_patch.conf - │ |-- run_tests.py - │ |-- testcases.yaml - │ |-- tier_builder.py - │ |-- tier_handler.py - |-- cli - │ |-- cli_base.py - │ |-- commands - │ │ |-- cli_env.py - │ │ |-- cli_os.py - │ │ |-- cli_testcase.py - │ │ |-- cli_tier.py - │ |-- functest-complete.sh - |-- core - │ |-- feature.py - │ |-- robotframework.py - │ |-- testcase.py - │ |-- unit.py - │ |-- vnf.py - |-- energy - │ |-- energy.py - |-- opnfv_tests - │ `-- openstack - │ |-- rally - │ |-- refstack_client - │ |-- snaps - │ |-- tempest - │ |-- vping - │ `-- sdn - │ │ `-- odl - │ `-- vnf - │ |-- ims - │ `-- router - |-- tests - │ `-- unit - │ |-- ci - │ |-- cli - │ |-- core - │ |-- energy - │ |-- features - │ |-- odl - │ |-- openstack - │ |-- opnfv_tests - │ |-- test_utils.py - │ |-- utils - │ `-- vnf - |-- utils - | |-- config.py - | |-- constants.py - | |-- decorators.py - | |-- env.py - | |-- functest_utils.py - | |-- functest_vacation.py - | |-- openstack_clean.py - | |-- openstack_tacker.py - | `-- openstack_utils.py - |-- requirements.txt - |-- setup.cfg - |-- setup.py - |-- test-requirements.txt - |-- tox.ini - |-- upper-constraints.txt - - (Note: All *.pyc files removed from above list for brevity...) - -We may distinguish several directories, the first level has 5 directories: - -* **api**: This directory is dedicated to the API (framework) documentations. -* **commons**: This directory is dedicated for storage of traffic profile or - any other test inputs that could be reused by any test project. -* **docker**: This directory includes the needed files and tools to - build the Functest Docker images. -* **docs**: This directory includes documentation: Release Notes, - User Guide, Configuration Guide and Developer Guide. -* **functest**: This directory contains all the code needed to run - functest internal cases and OPNFV onboarded feature or VNF test cases. - -Functest directory has 7 sub-directories, which is located under -/usr/lib/python2.7/site-packages/functest: - * **api**: This directory is dedicated for the internal Functest API. - * **ci**: This directory contains test structure definition files - (e.g <filename>.yaml) and bash shell/python scripts used to - configure and execute Functional tests. The test execution script - can be executed under the control of Jenkins CI jobs. - * **cli**: This directory holds the python based Functest CLI utility - source code, which is based on the Python 'click' framework. - * **core**: This directory holds the python based Functest core - source code. Three abstraction classes have been created to ease - the integration of internal, feature or vnf cases. - * **opnfv_tests**: This directory includes the scripts required by - Functest internal test cases and other feature projects test cases. - * **tests**: This directory includes the functest unit tests. - * **utils**: this directory holds Python source code for some general - purpose helper utilities, which testers can also re-use in their - own test code. See for an example the Openstack helper utility: - 'openstack_utils.py'. - Logs ==== @@ -512,7 +465,7 @@ By default all the logs are put un /home/opnfv/functest/results/functest.log. If you want to have more logs in console, you may edit the logging.ini file manually. Connect on the docker then edit the file located in -/usr/lib/python2.7/site-packages/functest/ci/logging.ini +/usr/lib/python3.8/site-packages/xtesting/ci/logging.ini Change wconsole to console in the desired module to get more traces. @@ -522,7 +475,8 @@ Configuration You may also directly modify the python code or the configuration file (e.g. testcases.yaml used to declare test constraints) under -/usr/lib/python2.7/site-packages/functest +/usr/lib/python3.8/site-packages/xtesting and +/usr/lib/python3.8/site-packages/functest Tips @@ -580,7 +534,7 @@ and credentials are working as expected. Once the credentials are there inside the container, they should be sourced before running any Openstack commands:: - source /home/opnfv/functest/conf/openstack.creds + source /home/opnfv/functest/conf/env_file After this, try to run any OpenStack command to see if you get any output, for instance:: @@ -614,6 +568,8 @@ If the OpenStack command still does not show anything or complains about connectivity issues, it could be due to an incorrect url given to the OS_AUTH_URL environment variable. Check the deployment settings. +.. _`Proxy support`: + Proxy support ------------- If your Jumphost node is operating behind a http proxy, then there are @@ -623,7 +579,7 @@ succeed: #. Initial installation of docker engine First, try following the official Docker documentation for Proxy settings. Some issues were experienced on CentOS 7 based Jumphost. Some tips are documented - in section: `Docker Installation on CentOS behind http proxy`_ + in section: :ref:`Docker Installation on CentOS behind http proxy` below. If that is the case, make sure the resolv.conf and the needed @@ -631,14 +587,14 @@ http_proxy and https_proxy environment variables, as well as the 'no_proxy' environment variable are set correctly:: # Make double sure that the 'no_proxy=...' line in the - # 'openstack.creds' file is commented out first. Otherwise, the + # 'env_file' file is commented out first. Otherwise, the # values set into the 'no_proxy' environment variable below will # be ovewrwritten, each time the command - # 'source ~/functest/conf/openstack.creds' is issued. + # 'source ~/functest/conf/env_file' is issued. cd ~/functest/conf/ - sed -i 's/export no_proxy/#export no_proxy/' openstack.creds - source ./openstack.creds + sed -i 's/export no_proxy/#export no_proxy/' env_file + source ./env_file # Next calculate some IP addresses for which http_proxy # usage should be excluded: @@ -669,7 +625,7 @@ docker container:: Note: In a Jumphost node based on the CentOS family OS, the **nc** commands might not work. You can use the **curl** command instead. - curl http://www.opnfv.org:80 + curl https://www.opnfv.org/ <HTML><HEAD><meta http-equiv="content-type" . @@ -686,6 +642,8 @@ commands might not work. You can use the **curl** command instead. (Ignore the content. If command returns a valid HTML page, it proves the connection.) +.. _`Docker Installation on CentOS behind http proxy`: + Docker Installation on CentOS behind http proxy ----------------------------------------------- This section is applicable for CentOS family OS on Jumphost which @@ -725,11 +683,10 @@ and install the **docker-engine**. The instructions conclude with a work with the above pre-requisite actions. -.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html .. _`dockerdocs`: https://docs.docker.com/ .. _`Proxy`: https://docs.docker.com/engine/admin/systemd/#http-proxy .. _`Install Docker on CentOS`: https://docs.docker.com/engine/installation/linux/centos/ .. _`Functest User Guide`: http://docs.opnfv.org/en/stable-danube/submodules/functest/docs/testing/user/userguide/index.html -.. _`images/CentOS-7-x86_64-GenericCloud.qcow2` http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img -.. _`images/cirros-0.4.0-x86_64-disk.img` https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img -.. _`images/ubuntu-14.04-server-cloudimg-amd64-disk1.img` https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 +.. _`images/CentOS-7-x86_64-GenericCloud.qcow2`: https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 +.. _`images/cirros-0.5.1-x86_64-disk.img`: http://download.cirros-cloud.net/0.5.1/cirros-0.5.1-x86_64-disk.img +.. _`images/ubuntu-14.04-server-cloudimg-amd64-disk1.img`: https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img diff --git a/docs/testing/user/configguide/index.rst b/docs/testing/user/configguide/index.rst index fd997344b..e5963435e 100644 --- a/docs/testing/user/configguide/index.rst +++ b/docs/testing/user/configguide/index.rst @@ -1,130 +1,22 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 +.. _functest-install-guide: *************************** Functest Installation Guide *************************** .. toctree:: - :numbered: :maxdepth: 2 - -Introduction -============ -This document describes how to install and configure Functest in OPNFV. - -High level architecture ------------------------ - -The high level architecture of Functest within OPNFV can be described as -follows:: - - CIMC/Lights+out management Admin Mgmt/API Public Storage Private - PXE - + + + + + + - | | | | | | - | +----------------------------+ | | | | | - | | | | | | | | - +-----+ Jumphost | | | | | | - | | | | | | | | - | | +--------------------+ | | | | | | - | | | | | | | | | | - | | | Tools | +----------------+ | | | - | | | - Rally | | | | | | | - | | | - Robot | | | | | | | - | | | - RefStack | | | | | | | - | | | | |-------------------------+ | | - | | | Testcases | | | | | | | - | | | - VIM | | | | | | | - | | | | | | | | | | - | | | - SDN Controller | | | | | | | - | | | | | | | | | | - | | | - Features | | | | | | | - | | | | | | | | | | - | | | - VNF | | | | | | | - | | | | | | | | | | - | | +--------------------+ | | | | | | - | | Functest Docker + | | | | | - | | | | | | | | - | | | | | | | | - | | | | | | | | - | +----------------------------+ | | | | | - | | | | | | - | +----------------+ | | | | | - | | 1 | | | | | | - +----+ +--------------+-+ | | | | | - | | | 2 | | | | | | - | | | +--------------+-+ | | | | | - | | | | 3 | | | | | | - | | | | +--------------+-+ | | | | | - | | | | | 4 | | | | | | - | +-+ | | +--------------+-+ | | | | | - | | | | | 5 +-------------+ | | | | - | +-+ | | nodes for | | | | | | - | | | | deploying +---------------------+ | | | - | +-+ | OPNFV | | | | | | - | | | +------------------------------+ | | - | +-+ SUT | | | | | | - | | +--------------------------------------+ | - | | | | | | | | - | | +----------------------------------------------+ - | +----------------+ | | | | | - | | | | | | - + + + + + + - SUT = System Under Test - -Note connectivity to management network is not needed for most of the testcases. -But it may be needed for some specific snaps tests. - -All the libraries and dependencies needed by all of the Functest tools are -pre-installed into the Docker images. This allows running Functest on any -platform. - -The automated mechanisms inside the Functest Docker containers will: - - * Prepare the environment according to the System Under Test (SUT) - * Perform the appropriate functional tests - * Push the test results into the OPNFV test result database (optional) - -The OpenStack credentials file must be provided to the container. - -These Docker images can be integrated into CI or deployed independently. - -Please note that the Functest Docker images have been designed for OPNFV, -however, it would be possible to adapt them to any OpenStack based VIM + -controller environment, since most of the test cases are integrated from -upstream communities. - -The functional test cases are described in the Functest User Guide `[2]`_ - -.. include:: ./prerequisites.rst - -.. include:: ./configguide.rst - -.. include:: ./ci.rst - + intro.rst + prerequisites.rst + configguide.rst References ========== -`[1]`_ : Keystone and public end point constraint - -`[2]`_ : Functest User guide - -`[3]`_ : Functest Jenkins jobs - -`[4]`_ : Functest Configuration guide - -`[5]`_ : OPNFV main site - -`[6]`_ : Functest wiki page +`[1]`_ : Functest Jenkins jobs IRC support channel: #opnfv-functest -.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ -.. _`[2]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[3]`: https://git.opnfv.org/releng/tree/jjb/functest/functest-daily-jobs.yml -.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[5]`: http://www.opnfv.org -.. _`[6]`: https://wiki.opnfv.org/functest +.. _`[1]`: https://github.com/opnfv/releng/blob/master/jjb/functest/functest.yaml diff --git a/docs/testing/user/configguide/intro.rst b/docs/testing/user/configguide/intro.rst new file mode 100644 index 000000000..02329995c --- /dev/null +++ b/docs/testing/user/configguide/intro.rst @@ -0,0 +1,86 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Introduction +============ +This document describes how to install and configure Functest in OPNFV. + +High level architecture +----------------------- + +The high level architecture of Functest within OPNFV can be described as +follows:: + + CIMC/Lights+out management Admin Mgmt/API Public Storage Private + PXE + + + + + + + + | | | | | | + | +----------------------------+ | | | | | + | | | | | | | | + +-----+ Jumphost | | | | | | + | | | | | | | | + | | +--------------------+ | | | | | | + | | | | | | | | | | + | | | Tools | +----------------+ | | | + | | | - Rally | | | | | | | + | | | - Robot | | | | | | | + | | | - RefStack | | | | | | | + | | | | |-------------------------+ | | + | | | Testcases | | | | | | | + | | | - VIM | | | | | | | + | | | | | | | | | | + | | | - SDN Controller | | | | | | | + | | | | | | | | | | + | | | - Features | | | | | | | + | | | | | | | | | | + | | | - VNF | | | | | | | + | | | | | | | | | | + | | +--------------------+ | | | | | | + | | Functest Docker + | | | | | + | | | | | | | | + | | | | | | | | + | | | | | | | | + | +----------------------------+ | | | | | + | | | | | | + | +----------------+ | | | | | + | | 1 | | | | | | + +----+ +--------------+-+ | | | | | + | | | 2 | | | | | | + | | | +--------------+-+ | | | | | + | | | | 3 | | | | | | + | | | | +--------------+-+ | | | | | + | | | | | 4 | | | | | | + | +-+ | | +--------------+-+ | | | | | + | | | | | 5 +-------------+ | | | | + | +-+ | | nodes for | | | | | | + | | | | deploying +---------------------+ | | | + | +-+ | OPNFV | | | | | | + | | | +------------------------------+ | | + | +-+ SUT | | | | | | + | | +--------------------------------------+ | + | | | | | | | | + | | +----------------------------------------------+ + | +----------------+ | | | | | + | | | | | | + + + + + + + + SUT = System Under Test + +All the libraries and dependencies needed by all of the Functest tools are +pre-installed into the Docker images. This allows running Functest on any +platform. + +The automated mechanisms inside the Functest Docker containers will: + + * Prepare the environment according to the System Under Test (SUT) + * Perform the appropriate functional tests + * Push the test results into the OPNFV test result database (optional) + +The OpenStack credentials file must be provided to the container. + +These Docker images can be integrated into CI or deployed independently. + +Please note that the Functest Docker images have been designed for OPNFV, +however, it would be possible to adapt them to any OpenStack based VIM + +controller environment, since most of the test cases are integrated from +upstream communities. + +The functional test cases are described in the :ref:`functest-userguide` diff --git a/docs/testing/user/configguide/prerequisites.rst b/docs/testing/user/configguide/prerequisites.rst index 8289803bb..dab4a9e6b 100644 --- a/docs/testing/user/configguide/prerequisites.rst +++ b/docs/testing/user/configguide/prerequisites.rst @@ -1,7 +1,9 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + Prerequisites ============= The OPNFV deployment is out of the scope of this document but it can be -found in http://docs.opnfv.org. +found in https://docs.opnfv.org/en/stable-hunter/. The OPNFV platform is considered as the SUT in this document. Several prerequisites are needed for Functest: @@ -12,12 +14,6 @@ Several prerequisites are needed for Functest: #. An admin/management network created on the SUT #. Connectivity from the Jumphost to the SUT public/external network -Some specific SNAPS tests may require a connectivity from the Jumphost to the -SUT admin/management network but most of the test cases do not. This requirement -can be changed by overriding the 'interface' attribute (OS_INTERFACE) value -to 'public' in the credentials file. Another means to circumvent this issue -would be to change the 'snaps.use_keystone' value from True to False. - WARNING: Connectivity from Jumphost is essential and it is of paramount importance to make sure it is working before even considering to install and run Functest. Make also sure you understand how your networking is @@ -29,7 +25,7 @@ deployment has been triggered previously, but it could be any server with proper connectivity to the SUT. NOTE: If your Jumphost is operating behind a company http proxy and/or -firewall, please consult first the section `Proxy Support`_, towards +firewall, please consult first the section :ref:`Proxy support`, towards the end of this document. The section details some tips/tricks which *may* be of help in a proxified environment. @@ -44,8 +40,8 @@ commands below are offered as a short reference. *Tip:* For running docker containers behind the proxy, you need first some extra configuration which is described in section -`Docker Installation on CentOS behind http proxy`_. You should follow -that section before installing the docker engine. +:ref:`Docker Installation on CentOS behind http proxy`. You should follow that +section before installing the docker engine. Docker installation needs to be done as root user. You may use other userid's to create and run the actual containers later if so desired. @@ -96,7 +92,4 @@ should thus be known. Ensure you can reach each node in the SUT, from the Jumphost using the 'ping' command using the respective IP address on the public/external network for each node in the SUT. The details of how to determine the needed IP addresses for each node in the SUT may vary according -to the used installer and are therefore ommitted here. - -.. _`[1]`: https://ask.openstack.org/en/question/68144/keystone-unable-to-use-the-public-endpoint/ -.. _`[4]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html +to the used installer and are therefore omitted here. diff --git a/docs/testing/user/userguide/index.rst b/docs/testing/user/userguide/index.rst index 66dfd3e7b..1e73cd622 100644 --- a/docs/testing/user/userguide/index.rst +++ b/docs/testing/user/userguide/index.rst @@ -1,8 +1,7 @@ -.. _functest-userguide: - -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. SPDX-License-Identifier: CC-BY-4.0 +.. _functest-userguide: + ******************* Functest User Guide ******************* @@ -10,39 +9,17 @@ Functest User Guide .. toctree:: :maxdepth: 2 - - -Introduction -============ - -The goal of this document is to describe the OPNFV Functest test cases and to -provide a procedure to execute them. - -**IMPORTANT**: It is assumed here that Functest has been properly deployed -following the installation guide procedure `[1]`_. - -.. include:: ./test_overview.rst - -.. include:: ./test_details.rst - -.. include:: ./runfunctest.rst - -.. include:: ./test_results.rst - -.. include:: ./reporting.rst - -.. figure:: ../../../images/functest-reporting-status.png - :align: center - :alt: Functest reporting portal Fuel status page - -.. include:: ./troubleshooting.rst + intro.rst + test_overview.rst + test_details.rst + test_results.rst + reporting.rst + troubleshooting.rst References ========== -`[1]`_: Functest configuration guide - `[2]`_: OpenStack Tempest documentation `[3]`_: Rally documentation @@ -51,7 +28,7 @@ References `[5]`_: Clearwater vIMS blueprint -`[6]`_: NIST web site +`[6]`_: Security Content Automation Protocol `[7]`_: OpenSCAP web site @@ -63,17 +40,13 @@ References `[11]`_: Robot Framework web site -`[12]`_: Functest User guide - -`[13]`_: SNAPS wiki +`[13]`_: SNAPS `[14]`_: vRouter `[15]`_: Testing OpenStack Tempest part 1 -`[16]`_: Running Functest through internal REST API - -`[17]`_: OPNFV Test API +`[16]`_: OPNFV Test API `OPNFV main site`_: OPNFV official web site @@ -81,26 +54,23 @@ References IRC support chan: #opnfv-functest -.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf +.. _`[2]`: https://docs.openstack.org/tempest/latest/ +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html +.. _`[4]`: https://events.static.linuxfound.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml -.. _`[6]`: https://scap.nist.gov/ +.. _`[6]`: https://en.wikipedia.org/wiki/Security_Content_Automation_Protocol .. _`[7]`: https://github.com/OpenSCAP/openscap .. _`[8]`: https://github.com/openstack/refstack-client -.. _`[9]`: https://github.com/openstack/defcore +.. _`[9]`: https://github.com/openstack/interop .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO +.. _`[11]`: https://robotframework.org/ +.. _`[13]`: https://git.opnfv.org/snaps/ .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ -.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API -.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html -.. _`OPNFV main site`: http://www.opnfv.org -.. _`Functest page`: https://wiki.opnfv.org/functest +.. _`[16]`: http://testresults.opnfv.org/test/ +.. _`OPNFV main site`: https://www.opnfv.org/ +.. _`Functest page`: https://github.com/opnfv/functest/ .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html -.. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml +.. _`config_functest.yaml` : https://github.com/opnfv/functest/blob/master/functest/ci/config_functest.yaml .. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html diff --git a/docs/testing/user/userguide/intro.rst b/docs/testing/user/userguide/intro.rst new file mode 100644 index 000000000..c001afb2f --- /dev/null +++ b/docs/testing/user/userguide/intro.rst @@ -0,0 +1,10 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Introduction +============ + +The goal of this document is to describe the OPNFV Functest test cases and to +provide a procedure to execute them. + +**IMPORTANT**: It is assumed here that Functest has been properly deployed +following the installation guide procedure :ref:`functest-install-guide`. diff --git a/docs/testing/user/userguide/reporting.rst b/docs/testing/user/userguide/reporting.rst index 88f5e865a..8fad55d33 100644 --- a/docs/testing/user/userguide/reporting.rst +++ b/docs/testing/user/userguide/reporting.rst @@ -1,4 +1,3 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. .. http://creativecommons.org/licenses/by/4.0 Test reporting @@ -26,7 +25,7 @@ and features) corresponding to this scenario. +---------------------+---------+---------+---------+---------+ | vPing_userdata | X | X | X | X | +---------------------+---------+---------+---------+---------+ - | tempest_smoke_serial| X | X | X | X | + | tempest_smoke | X | X | X | X | +---------------------+---------+---------+---------+---------+ | rally_sanity | X | X | X | X | +---------------------+---------+---------+---------+---------+ @@ -42,12 +41,14 @@ and features) corresponding to this scenario. +---------------------+---------+---------+---------+---------+ | copper | X | | | X | +---------------------+---------+---------+---------+---------+ - src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix version) + + src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix + version) All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature scenarios. Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration -addons and, as a consequence, specific scenario. +add-ons and, as a consequence, specific scenario. There are not considered as runnable on the generic odl_l2 scenario. @@ -69,16 +70,20 @@ scoring. In fact the success criteria are not always easy to define and may require specific hardware configuration. Please also note that all the test cases have the same "weight" for the score -calculation whatever the complexity of the test case. Concretely a vping has the -same weith than the 200 tempst tests. +calculation whatever the complexity of the test case. Concretely a vping has +the same weight than the 200 tempest tests. Moreover some installers support more features than others. The more cases your scenario is dealing with, the most difficult to rich a good scoring. Therefore the scoring provides 3 types of indicators: - * the richness of the scenario: if the target scoring is high, it means that the scenario includes lots of features - * the maturity: if the percentage (scoring/target scoring * 100) is high, it means that all the tests are PASS - * the stability: as the number of iteration is included in the calculation, the pecentage can be high only if the scenario is run regularly (at least more than 4 iterations over the last 10 days in CI) + * the richness of the scenario: if the target scoring is high, it means that + the scenario includes lots of features + * the maturity: if the percentage (scoring/target scoring * 100) is high, it + means that all the tests are PASS + * the stability: as the number of iteration is included in the calculation, + the percentage can be high only if the scenario is run regularly (at least + more than 4 iterations over the last 10 days in CI) In any case, the scoring is used to give feedback to the other projects and does not represent an absolute value of the scenario. @@ -86,5 +91,8 @@ does not represent an absolute value of the scenario. See `reporting page`_ for details. For the status, click on the version, Functest then the Status menu. - .. _`reporting page`: http://testresults.opnfv.org/reporting/ + +.. figure:: ../../../images/functest-reporting-status.png + :align: center + :alt: Functest reporting portal Fuel status page diff --git a/docs/testing/user/userguide/runfunctest.rst b/docs/testing/user/userguide/runfunctest.rst deleted file mode 100644 index 89580e564..000000000 --- a/docs/testing/user/userguide/runfunctest.rst +++ /dev/null @@ -1,140 +0,0 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 - - -Executing Functest suites -========================= - -As mentioned in the configuration guide `[1]`_, Alpine docker containers have -been introduced in Euphrates. -Tier containers have been created. -Assuming that you pulled the container and your environement is ready, you can -simply run the tiers by typing (e.g. with functest-healthcheck):: - - sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck - -You should get:: - - +----------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +----------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:02 | PASS | - | api_check | functest | healthcheck | 03:19 | PASS | - | snaps_health_check | functest | healthcheck | 00:46 | PASS | - +----------------------------+------------------+---------------------+------------------+----------------+ - -You can run functest-healcheck, functest-smoke, functest-features, -functest-components and functest-vnf. - -The result tables show the results by test case, it can be:: - - * PASS - * FAIL - * SKIP: if the scenario/installer does not support the test case - - -Manual run -========== -If you want to run the test step by step, you may add docker option then run the -different commands within the docker. - -Considering the healthcheck example, running functest manaully means:: - - sudo docker run -ti --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ - -v $(pwd)/images:/home/opnfv/functest/images \ - opnfv/functest-healthcheck /bin/bash - -The docker prompt shall be returned. Then within the docker run the following -commands:: - - $ source /home/opnfv/functest/conf/openstack.creds - -Tier ----- -Each Alpine container provided on the docker hub matches with a tier. -The following commands are available:: - - # functest tier list - - 0. healthcheck: - ['connection_check', 'api_check', 'snaps_health_check'] - # functest tier show healthcheck - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | TIERS | ORDER | CI LOOP | DESCRIPTION | TESTCASES | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - | healthcheck | 0 | (daily)|(weekly) | First tier to be executed to verify the | connection_check api_check | - | | | | basic operations in the VIM. | snaps_health_check | - +---------------------+---------------+--------------------------+-------------------------------------------------+------------------------------------+ - -To run all the cases of the tier, type:: - - # functest tier run healthcheck - -Testcase --------- -Testcases can be listed, shown and run though the CLI:: - - # functest testcase list - connection_check - api_check - snaps_health_check - # functest testcase show api_check - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | TEST CASE | DESCRIPTION | CRITERIA | DEPENDENCY | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - | api_check | This test case verifies the retrieval of | 100 | ^((?!netvirt).)*$ | - | | OpenStack clients: Keystone, Glance, | | | - | | Neutron and Nova and may perform some | | | - | | simple queries. When the config value of | | | - | | snaps.use_keystone is True, functest | | | - | | must have access to the cloud's private | | | - | | network. | | | - +-------------------+--------------------------------------------------+------------------+---------------------------+ - # functest testcase run connection_check - ... - # functest run all - -You can also type run_tests -t all to run all the tests. - -Note the list of test cases depend on the installer and the scenario. - - -Reporting results to the test Database -====================================== -In OPNFV CI we collect all the results from CI. A test APi shall be available -as well as a test database `[17]`_. - -Functest internal API -===================== - -An internal API has been introduced in Euphrates. The goal is to trigger -Functest operations through an API in addition of the CLI. -This could be considered as a first step towards a pseudo micro services -approach where the different test projects could expose and consume APIs to the -other test projects. - -In Euphrates the main method of the APIs are: - - * Show credentials - * Update openrc file - * Show environment - * Update hosts info for domain name - * Prepare environment - * List all testcases - * Show a testcase - * Run a testcase - * List all tiers - * Show a tier - * List all testcases within given tier - * Get the result of the specified task - * Get the log of the specified task - -See `[16]`_ to get examples on how to use the API. - - -.. _`[1]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html -.. _`[16]`: https://wiki.opnfv.org/display/functest/Running+test+cases+via+new+Functest+REST+API -.. _`[17]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html diff --git a/docs/testing/user/userguide/test_details.rst b/docs/testing/user/userguide/test_details.rst index 97c4688cc..98247d488 100644 --- a/docs/testing/user/userguide/test_details.rst +++ b/docs/testing/user/userguide/test_details.rst @@ -1,8 +1,7 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 - -The different test cases are described in the remaining sections of this document. +The different test cases are described in the remaining sections of this +document. VIM (Virtualized Infrastructure Manager) ---------------------------------------- @@ -25,20 +24,10 @@ The tests are: * *connection_check* - * *api_check* - * *snaps_health_check* -Connection_check consists in 9 test cases (test duration < 5s) checking the +Connection_check consists in test cases (test duration < 5s) checking the connectivity with Glance, Keystone, Neutron, Nova and the external network. -Api_check verifies the retrieval of OpenStack clients: Keystone, Glance, -Neutron and Nova and may perform some simple queries. When the config value of -snaps.use_keystone is True, functest must have access to the cloud's private -network. This suite consists in 49 tests (test duration < 2 minutes). - -Snaps_health_check creates a VM with a single port with an IPv4 address that -is assigned by DHCP and then validates the expected IP with the actual. - Self-obviously, successful completion of the 'healthcheck' testcase is a necessary pre-requisite for the execution of all other test Tiers. @@ -59,8 +48,8 @@ Given the script **ping.sh**:: The goal of this test is to establish an SSH connection using a floating IP -on the Public/External network and verify that 2 instances can talk over a Private -Tenant network:: +on the Public/External network and verify that 2 instances can talk over a +Private Tenant network:: vPing_ssh test case +-------------+ +-------------+ @@ -105,7 +94,8 @@ vPing_userdata This test case is similar to vPing_ssh but without the use of Floating IPs and the Public/External network to transfer the ping script. -Instead, it uses Nova metadata service to pass it to the instance at booting time. +Instead, it uses Nova metadata service to pass it to the instance at booting +time. As vPing_ssh, it checks that 2 instances can talk to each other on a Private Tenant network:: @@ -158,17 +148,32 @@ updates the appropriate parameters into the configuration file. When the Tempest suite is executed, each test duration is measured and the full console output is stored to a *log* file for further analysis. -The Tempest testcases are distributed across two +The Tempest testcases are distributed across three Tiers: - * Smoke Tier - Test Case 'tempest_smoke_serial' - * Components Tier - Test case 'tempest_full_parallel' + * Smoke Tier - Test Case 'tempest_smoke' + * Components Tier - Test case 'tempest_full' + * Neutron Trunk Port - Test case 'neutron_trunk' + * OpenStack interop testcases - Test case 'refstack_defcore' + * Testing and verifying RBAC policy enforcement - Test case 'patrole' + +NOTE: Test case 'tempest_smoke' executes a defined set of tempest smoke +tests. Test case 'tempest_full' executes all defined Tempest tests. + +NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN +separated networks using a single NIC. The feature neutron trunk ports have +been supported by Apex, Fuel and Compass, so the tempest testcases have been +integrated normally. -NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke -tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel' -executes all defined Tempest tests using several concurrent threads -(i.e. parallel mode). The number of threads activated corresponds to the number -of available logical CPUs. +NOTE: Rally is also used to run Openstack Interop testcases `[9]`_, which focus +on testing interoperability between OpenStack clouds. + +NOTE: Patrole is a tempest plugin for testing and verifying RBAC policy +enforcement. It runs Tempest-based API tests using specified RBAC roles, thus +allowing deployments to verify that only intended roles have access to those +APIs. Patrole currently offers testing for the following OpenStack services: +Nova, Neutron, Glance, Cinder and Keystone. Currently in functest, only neutron +and glance are tested. The goal of the Tempest test suite is to check the basic functionalities of the different OpenStack components on an OPNFV fresh installation, using the @@ -182,10 +187,11 @@ Rally `[3]`_ is a benchmarking tool that answers the question: *How does OpenStack work at scale?* -The goal of this test suite is to benchmark all the different OpenStack modules and -get significant figures that could help to define Telco Cloud KPIs. +The goal of this test suite is to benchmark all the different OpenStack modules +and get significant figures that could help to define Telco Cloud KPIs. -The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios: +The OPNFV Rally scenarios are based on the collection of the actual Rally +scenarios: * authenticate * cinder @@ -195,7 +201,6 @@ The OPNFV Rally scenarios are based on the collection of the actual Rally scenar * neutron * nova * quotas - * ceilometer A basic SLA (stop test on errors) has been implemented. @@ -208,104 +213,6 @@ NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test cases. Test case 'rally_full' executes the full defined set of Rally tests. -Refstack-client to run OpenStack interop testcases --------------------------------------------------- - -Refstack-client `[8]`_ is a command line utility that allows you to -execute Tempest test runs based on configurations you specify. -It is the official tool to run Openstack Interop (previously known as Defcore) -testcases `[9]`_, which focus on testing interoperability between OpenStack -clouds. - -Refstack-client is integrated in Functest, consumed by Dovetail, which -intends to define and provide a set of OPNFV related validation criteria -that will provide input for the evaluation of the use of OPNFV trademarks. -This progress is under the guideline of Compliance Verification Program(CVP). - -Running methods -^^^^^^^^^^^^^^^ - -Two running methods are provided after refstack-client integrated into -Functest, Functest command line and manually, respectively. - -By default, for Defcore test cases run by Functest command line, -are run followed with automatically generated -configuration file, i.e., refstack_tempest.conf. In some circumstances, -the automatic configuration file may not quite satisfied with the SUT, -Functest also inherits the refstack-client command line and provides a way -for users to set its configuration file according to its own SUT manually. - -*command line* - -Inside the Functest container, first to prepare Functest environment: - -:: - - functest env prepare - -then to run default defcore testcases by using refstack-client: - -:: - - functest testcase run refstack_defcore - -In OPNFV Continuous Integration(CI) system, the command line method is used. - -*manually* - -Prepare the tempest configuration file and the testcases want to run with the SUT, -run the testcases with: - -:: - - ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list> - -using help for more information: - -:: - - ./refstack-client --help - ./refstack-client test --help - -Reference tempest configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -*command line method* - -When command line method is used, the default tempest configuration file -is generated by Rally. - -*manually* - -When running manually is used, recommended way to generate tempest configuration -file is: - -:: - - cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client - python tempest_conf.py - -a file called tempest.conf is stored in the current path by default, users can do -some adjustment according to the SUT: - -:: - - vim refstack_tempest.conf - -a reference article can be used `[15]`_. - - -snaps_smoke ------------- - -This test case contains tests that setup and destroy environments with VMs with -and without Floating IPs with a newly created user and project. Set the config -value snaps.use_floating_ips (True|False) to toggle this functionality. -Please note that When the configuration value of snaps.use_keystone is True, Functest must have access -the cloud's private network. -This suite consists in 38 tests (test duration < 10 minutes) - - SDN Controllers --------------- @@ -320,31 +227,40 @@ OpenDaylight and Neutron. The list of tests can be described as follows: * Basic Restconf test cases + * Connect to Restconf URL * Check the HTTP code status * Neutron Reachability test cases + * Get the complete list of neutron resources (networks, subnets, ports) * Neutron Network test cases + * Check OpenStack networks * Check OpenDaylight networks - * Create a new network via OpenStack and check the HTTP status code returned by Neutron + * Create a new network via OpenStack and check the HTTP status code returned + by Neutron * Check that the network has also been successfully created in OpenDaylight * Neutron Subnet test cases + * Check OpenStack subnets * Check OpenDaylight subnets - * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron + * Create a new subnet via OpenStack and check the HTTP status code returned + by Neutron * Check that the subnet has also been successfully created in OpenDaylight * Neutron Port test cases + * Check OpenStack Neutron for known ports * Check OpenDaylight ports - * Create a new port via OpenStack and check the HTTP status code returned by Neutron + * Create a new port via OpenStack and check the HTTP status code returned by + Neutron * Check that the new port has also been successfully created in OpenDaylight * Delete operations + * Delete the port previously created via OpenStack * Check that the port has been also successfully deleted in OpenDaylight * Delete previously subnet created via OpenStack @@ -356,49 +272,6 @@ Note: the checks in OpenDaylight are based on the returned HTTP status code returned by OpenDaylight. -Features --------- - -Functest has been supporting several feature projects since Brahpamutra: - - -+-----------------+---------+----------+--------+-----------+ -| Test | Brahma | Colorado | Danube | Euphrates | -+=================+=========+==========+========+===========+ -| barometer | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| bgpvpn | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| copper | | X | | | -+-----------------+---------+----------+--------+-----------+ -| doctor | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| domino | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| fds | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| moon | | X | | | -+-----------------+---------+----------+--------+-----------+ -| multisite | | X | X | | -+-----------------+---------+----------+--------+-----------+ -| netready | | | X | | -+-----------------+---------+----------+--------+-----------+ -| odl_sfc | | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| opera | | | X | | -+-----------------+---------+----------+--------+-----------+ -| orchestra | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| parser | | | X | X | -+-----------------+---------+----------+--------+-----------+ -| promise | X | X | X | X | -+-----------------+---------+----------+--------+-----------+ -| security_scan | | X | X | | -+-----------------+---------+----------+--------+-----------+ - -Please refer to the dedicated feature user guides for details. - - VNF --- @@ -409,9 +282,9 @@ The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an architectural framework for delivering IP multimedia services. vIMS has been integrated in Functest to demonstrate the capability to deploy a -relatively complex NFV scenario on the OPNFV platform. The deployment of a complete -functional VNF allows the test of most of the essential functions needed for a -NFV platform. +relatively complex NFV scenario on the OPNFV platform. The deployment of a +complete functional VNF allows the test of most of the essential functions +needed for a NFV platform. The goal of this test suite consists of: @@ -422,48 +295,31 @@ The goal of this test suite consists of: The Clearwater architecture is described as follows: -.. figure:: ../../../images/clearwater-architecture.png +.. figure:: ../../../images/clearwater-architecture-v2.png :align: center :alt: vIMS architecture +heat_ims +^^^^^^^^ +The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an +architectural framework for delivering IP multimedia services. -cloudify_ims_perf -^^^^^^^^^^^^^^^^^ -This testcase extends the cloudify_ims test case. -The first part is similar but the testing part is different. -The testing part consists in automating a realistic signaling load on the vIMS -using an Ixia loader (proprietary tools) - - You need to have access to an Ixia licence server defined in the configuration - file and have ixia image locally. - -This test case is available but not declared in testcases.yaml. The declaration -of the testcase is simple, connect to your functest-vnf docker, add the following -section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml:: - - - - case_name: cloudify_ims_perf - project_name: functest - criteria: 80 - blocking: false - description: >- - Stress tests based on Cloudify. Ixia loader images and access to Ixia - server license. - dependencies: - installer: '' - scenario: 'os-nosdn-nofeature-ha' - run: - module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf' - class: 'CloudifyImsPerf' - -orchestra_openims -^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of OpenIMS with OpenBaton -orchestrator. +vIMS has been integrated in Functest to demonstrate the capability to deploy a +relatively complex NFV scenario on the OPNFV platform. The deployment of a +complete functional VNF allows the test of most of the essential functions +needed for a NFV platform. -orchestra_clearwaterims -^^^^^^^^^^^^^^^^^^^^^^^ -Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton -orchestrator. +The goal of this test suite consists of: + +* deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF using + OpenStack Heat orchestrator based on a HOT template defined in `[17]`_ +* run suite of signaling tests on top of this VNF + +The Clearwater architecture is described as follows: + +.. figure:: ../../../images/clearwater-architecture-v2.png + :align: center + :alt: vIMS architecture vyos-vrouter ^^^^^^^^^^^^ @@ -484,14 +340,73 @@ The Workflow is as follows: The vyos-vrouter architecture is described in `[14]`_ +juju_epc +^^^^^^^^ +The Evolved Packet Core (EPC) is the main component of the System Architecture +Evolution (SAE) which forms the core of the 3GPP LTE specification. + +vEPC has been integrated in Functest to demonstrate the capability to deploy a +complex mobility-specific NFV scenario on the OPNFV platform. The OAI EPC +supports most of the essential functions defined by the 3GPP Technical Specs; +hence the successful execution of functional tests on the OAI EPC provides a +good endorsement of the underlying NFV platform. + +This integration also includes ABot, a Test Orchestration system that enables +test scenarios to be defined in high-level DSL. ABot is also deployed as a +VM on the OPNFV platform; and this provides an example of the automation +driver and the Test VNF being both deployed as separate VNFs on the underlying +OPNFV platform. + +The Workflow is as follows: + * Deploy Orchestrator + Deploy Juju controller using Bootstrap command. + * Deploy VNF + Deploy ABot orchestrator and OAI EPC as Juju charms. + Configuration of ABot and OAI EPC components is handled through + built-in Juju relations. + * Test VNF + Execution of ABot feature files triggered by Juju actions. + This executes a suite of LTE signalling tests on the OAI EPC. + * Reporting + ABot test results are parsed accordingly and pushed to Functest Db. + +Details of the ABot test orchestration tool may be found in `[15]`_ + +Kubernetes (K8s) +---------------- + +Kubernetes testing relies on sets of tests, which are part of the Kubernetes +source tree, such as the Kubernetes End-to-End (e2e) tests `[16]`_. + +The kubernetes testcases are distributed across various Tiers: + + * Healthcheck Tier + + * k8s_smoke Test Case: Creates a Guestbook application that contains redis + server, 2 instances of redis slave, frontend application, frontend service + and redis master service and redis slave service. Using frontend service, + the test will write an entry into the guestbook application which will + store the entry into the backend redis database. Application flow MUST + work as expected and the data written MUST be available to read. + + * Smoke Tier + + * k8s_conformance Test Case: Runs a series of k8s e2e tests expected to + pass on any Kubernetes cluster. It is a subset of tests necessary to + demonstrate conformance grows with each release. Conformance is thus + considered versioned, with backwards compatibility guarantees and are + designed to be run with no cloud provider configured. + -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html +.. _`[2]`: https://docs.openstack.org/tempest/latest/ +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml .. _`[8]`: https://github.com/openstack/refstack-client +.. _`[9]`: https://github.com/openstack/interop .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst -.. _`[11]`: http://robotframework.org/ -.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html -.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO +.. _`[11]`: https://robotframework.org/ +.. _`[13]`: https://git.opnfv.org/snaps/ .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter -.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/ +.. _`[15]`: https://github.com/RebacaInc/abot_charm +.. _`[16]`: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md +.. _`[17]`: https://github.com/Metaswitch/clearwater-heat/blob/release-129/clearwater.yaml diff --git a/docs/testing/user/userguide/test_overview.rst b/docs/testing/user/userguide/test_overview.rst index a22a5067f..bc3e79dcb 100644 --- a/docs/testing/user/userguide/test_overview.rst +++ b/docs/testing/user/userguide/test_overview.rst @@ -1,5 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 Overview of the Functest suites =============================== @@ -9,158 +8,114 @@ In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure. -The current list of test suites can be distributed over 4 main domains: +The current list of test suites can be distributed over 5 main domains: * VIM (Virtualised Infrastructure Manager) * Controllers (i.e. SDN Controllers) - * Features * VNF (Virtual Network Functions) + * Kubernetes Functest test suites are also distributed in the OPNFV testing categories: -healthcheck, smoke, features, components, performance, VNF, Stress tests. +healthcheck, smoke, benchmarking, VNF, Stress tests. -All the Healthcheck and smoke tests of a given scenario must be succesful to +All the Healthcheck and smoke tests of a given scenario must be successful to validate the scenario for the release. -+-------------+---------------+----------------+----------------------------------+ -| Domain | Tier | Test case | Comments | -+=============+===============+================+==================================+ -| VIM | healthcheck | connection | Check OpenStack connectivity | -| | | _check | through SNAPS framework | -| | +----------------+----------------------------------+ -| | | api_check | Check OpenStack API through | -| | | | SNAPS framework | -| | +----------------+----------------------------------+ -| | | snaps_health | basic instance creation, check | -| | | \_check | DHCP | -| +---------------+----------------+----------------------------------+ -| | smoke | vping_ssh | NFV "Hello World" using an SSH | -| | | | connection to a destination VM | -| | | | over a created floating IP | -| | | | address on the SUT Public / | -| | | | External network. Using the SSH | -| | | | connection a test script is then | -| | | | copied to the destination | -| | | | VM and then executed via SSH. | -| | | | The script will ping another | -| | | | VM on a specified IP address over| -| | | | the SUT Private Tenant network | -| | +----------------+----------------------------------+ -| | | vping_userdata | Uses Ping with given userdata | -| | | | to test intra-VM connectivity | -| | | | over the SUT Private Tenant | -| | | | network. The correct operation | -| | | | of the NOVA Metadata service is | -| | | | also verified in this test | -| | +----------------+----------------------------------+ -| | | tempest_smoke | Generate and run a relevant | -| | | \_serial | Tempest Test Suite in smoke mode.| -| | | | The generated test set is | -| | | | dependent on the OpenStack | -| | | | deployment environment | -| | +----------------+----------------------------------+ -| | | rally_sanity | Run a subset of the OpenStack | -| | | | Rally Test Suite in smoke mode | -| | +----------------+----------------------------------+ -| | | snaps_smoke | Run the SNAPS-OO integration | -| | | | tests | -| | +----------------+----------------------------------+ -| | | refstack | Reference RefStack suite | -| | | \_defcore | tempest selection for NFV | -| +---------------+----------------+----------------------------------+ -| | components | tempest_full | Generate and run a full set of | -| | | \_parallel | the OpenStack Tempest Test Suite.| -| | | | See the OpenStack reference test | -| | | | suite `[2]`_. The generated | -| | | | test set is dependent on the | -| | | | OpenStack deployment environment | -| | +----------------+----------------------------------+ -| | | rally_full | Run the OpenStack testing tool | -| | | | benchmarking OpenStack modules | -| | | | See the Rally documents `[3]`_ | -+-------------+---------------+----------------+----------------------------------+ -| Controllers | smoke | odl | Opendaylight Test suite | -| | | | Limited test suite to check the | -| | | | basic neutron (Layer 2) | -| | | | operations mainly based on | -| | | | upstream testcases. See below | -| | | | for details | -| | +----------------+----------------------------------+ -| | | odl_netvirt | Test Suite for the OpenDaylight | -| | | | SDN Controller when the NetVirt | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| Features | features | bgpvpn | Implementation of the OpenStack | -| | | | bgpvpn API from the SDNVPN | -| | | | feature project. It allows for | -| | | | the creation of BGP VPNs. | -| | | | See `SDNVPN User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | doctor | Doctor platform, as of Colorado | -| | | | release, provides the three | -| | | | features: | -| | | | * Immediate Notification | -| | | | * Consistent resource state | -| | | | awareness for compute host down | -| | | | * Valid compute host status | -| | | | given to VM owner | -| | | | See `Doctor User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | odl-sfc | SFC testing for odl scenarios | -| | | | See `SFC User Guide`_ for details| -| | +----------------+----------------------------------+ -| | | parser | Parser is an integration project | -| | | | which aims to provide | -| | | | placement/deployment templates | -| | | | translation for OPNFV platform, | -| | | | including TOSCA -> HOT, POLICY ->| -| | | | TOSCA and YANG -> TOSCA. it | -| | | | deals with a fake vRNC. | -| | | | See `Parser User Guide`_ for | -| | | | details | -| | +----------------+----------------------------------+ -| | | fds | Test Suite for the OpenDaylight | -| | | | SDN Controller when the GBP | -| | | | features are installed. It | -| | | | integrates some test suites from | -| | | | upstream using Robot as the test | -| | | | framework | -+-------------+---------------+----------------+----------------------------------+ -| VNF | vnf | cloudify_ims | Example of a real VNF deployment | -| | | | to show the NFV capabilities of | -| | | | the platform. The IP Multimedia | -| | | | Subsytem is a typical Telco test | -| | | | case, referenced by ETSI. | -| | | | It provides a fully functional | -| | | | VoIP System | -| | +----------------+----------------------------------+ -| | | orchestra | OpenIMS deployment using | -| | | \_openims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | orchestra | Cleawater IMS deployment using | -| | | \_cleawaterims | Openbaton orchestrator | -| | +----------------+----------------------------------+ -| | | vyos_vrouter | vRouter testing | -| | +----------------+----------------------------------+ -| | | cloudify_ims | Based on cloudify_ims test case | -| | | perf | cloudify_ims_perf substitutes | -| | | | the signaling test suite by an | -| | | | automatic deployment of an Ixia | -| | | | loader and generic SIP stress | -| | | | tests. | -| | | | This work has been initiated | -| | | | during the plugfest and allows | -| | | | realistic load tests on top of | -| | | | cloudify_ims. Please note that | -| | | | this test is available but not | -| | | | declared in testcases.yaml as it | -| | | | requires access to proprietary | -| | | | resources (Ixia loader) | -+-------------+---------------+----------------+----------------------------------+ ++-------------+---------------+------------+----------------------------------+ +| Domain | Tier | Test case | Comments | ++=============+===============+============+==================================+ +| VIM | healthcheck | connection | Check OpenStack connectivity | +| | | \_check | | +| +---------------+------------+----------------------------------+ +| | smoke | vping_ssh | NFV "Hello World" using an SSH | +| | | | connection to a destination VM | +| | | | over a created floating IP | +| | | | address on the SUT Public / | +| | | | External network. Using the SSH | +| | | | connection a test script is then | +| | | | copied to the destination | +| | | | VM and then executed via SSH. | +| | | | The script will ping another | +| | | | VM on a specified IP address over| +| | | | the SUT Private Tenant network | +| | +------------+----------------------------------+ +| | | vping | Uses Ping with given userdata | +| | | \_userdata | to test intra-VM connectivity | +| | | | over the SUT Private Tenant | +| | | | network. The correct operation | +| | | | of the NOVA Metadata service is | +| | | | also verified in this test | +| | +------------+----------------------------------+ +| | | tempest | Generate and run a relevant | +| | | \_smoke | Tempest Test Suite in smoke mode.| +| | | | The generated test set is | +| | | | dependent on the OpenStack | +| | | | deployment environment | +| | +------------+----------------------------------+ +| | | rally | Run a subset of the OpenStack | +| | | \_sanity | Rally Test Suite in smoke mode | +| | +------------+----------------------------------+ +| | | refstack | Reference RefStack suite | +| | | \_defcore | tempest selection for NFV | +| | +------------+----------------------------------+ +| | | patrole | Patrole is a tempest plugin for | +| | | | testing and verifying RBAC policy| +| | | | enforcement, which offers testing| +| | | | for the following OpenStack | +| | | | services: Nova, Neutron, Glance, | +| | | | Cinder and Keystone | +| +---------------+------------+----------------------------------+ +| | | neutron | The neutron trunk port testcases | +| | | \_trunk | have been introduced and they are| +| | | | supported by installers : | +| | | | Apex, Fuel and Compass. | +| +---------------+------------+----------------------------------+ +| | components | tempest | Generate and run a full set of | +| | | \_full | the OpenStack Tempest Test Suite.| +| | | \_parallel | See the OpenStack reference test | +| | | | suite `[2]`_. The generated | +| | | | test set is dependent on the | +| | | | OpenStack deployment environment | +| | +------------+----------------------------------+ +| | | rally_full | Run the OpenStack testing tool | +| | | | benchmarking OpenStack modules | +| | | | See the Rally documents `[3]`_ | ++-------------+---------------+------------+----------------------------------+ +| Controllers | smoke | odl | Opendaylight Test suite | +| | | | Limited test suite to check the | +| | | | basic neutron (Layer 2) | +| | | | operations mainly based on | +| | | | upstream testcases. See below | +| | | | for details | ++-------------+---------------+------------+----------------------------------+ +| VNF | vnf | cloudify | Example of a real VNF deployment | +| | | \_ims | to show the NFV capabilities of | +| | | | the platform. The IP Multimedia | +| | | | Subsystem is a typical Telco test| +| | | | case, referenced by ETSI. | +| | | | It provides a fully functional | +| | | | VoIP System | +| | +------------+----------------------------------+ +| | | vyos | vRouter testing | +| | | \_vrouter | | +| | +------------+----------------------------------+ +| | | juju_epc | Validates deployment of a complex| +| | | | mobility VNF on OPNFV Platform. | +| | | | Uses Juju for deploying the OAI | +| | | | EPC and ABot for defining test | +| | | | scenarios using high-level DSL. | +| | | | VNF tests reference 3GPP | +| | | | Technical Specs and are executed | +| | | | through protocol drivers provided| +| | | | by ABot. | ++-------------+---------------+------------+----------------------------------+ +| Kubernetes | healthcheck | k8s_smoke | Test a running Kubernetes | +| | | | cluster and ensure it satisfies | +| | | | minimal functional requirements | +| +---------------+------------+----------------------------------+ +| | smoke | k8s\_ | Run a subset of Kubernetes | +| | | conformance| End-to-End tests, expected to | +| | | | pass on any Kubernetes cluster | ++-------------+---------------+------------+----------------------------------+ As shown in the above table, Functest is structured into different 'domains', @@ -169,29 +124,30 @@ As shown in the above table, Functest is structured into different 'domains', Test cases also have an implicit execution order. For example, if the early 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke' -Tier testcases, there is little point to launch a full testcase execution round. +Tier testcases, there is little point to launch a full testcase execution +round. In Danube, we merged smoke and sdn controller tiers in smoke tier. An overview of the Functest Structural Concept is depicted graphically below: -.. figure:: ../../../images/concepts_mapping_final.png +.. figure:: ../../../images/concepts_mapping_fraser.png :align: center :alt: Functest Concepts Structure Some of the test cases are developed by Functest team members, whereas others are integrated from upstream communities or other OPNFV projects. For example, -`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the +`Tempest <https://docs.openstack.org/tempest/latest/>`_ is the OpenStack integration test suite and Functest is in charge of the selection, integration and automation of those tests that fit suitably to OPNFV. -The Tempest test suite is the default OpenStack smoke test suite but no new test -cases have been created in OPNFV Functest. +The Tempest test suite is the default OpenStack smoke test suite but no new +test cases have been created in OPNFV Functest. The results produced by the tests run from CI are pushed and collected into a -NoSQL database. The goal is to populate the database with results from different -sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot -of a live Functest Dashboard is shown below: +NoSQL database. The goal is to populate the database with results from +different sources and scenarios and to show them on a `Functest Dashboard`_. A +screenshot of a live Functest Dashboard is shown below: .. figure:: ../../../images/FunctestDashboardEuphrates.png :align: center @@ -217,12 +173,11 @@ combinations (which may change from one version to another): Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available -deployed features. The system uses the environment variables (INSTALLER_TYPE and -DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given -environment. +deployed services. The system uses the environment variables to automatically +determine the valid test cases, for each given environment. A convenience Functest CLI utility is also available to simplify setting up the -Functest evironment, management of the OpenStack environment (e.g. resource +Functest environment, management of the OpenStack environment (e.g. resource clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allows execution of a single specified @@ -230,10 +185,6 @@ testcase, all test cases in a specified Tier, or the special case of execution of **ALL** testcases. The Functest CLI is introduced in more details in next section. -.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html -.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html -.. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html -.. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html -.. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html -.. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/ -.. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html +.. _`[2]`: https://docs.openstack.org/tempest/latest/ +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html +.. _`Functest Dashboard`: http://testresults.opnfv.org/ diff --git a/docs/testing/user/userguide/test_results.rst b/docs/testing/user/userguide/test_results.rst index 3941ba0a1..10f87d8ec 100644 --- a/docs/testing/user/userguide/test_results.rst +++ b/docs/testing/user/userguide/test_results.rst @@ -1,3 +1,5 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + Test results ============ @@ -8,40 +10,154 @@ In manual mode test results are displayed in the console and result files are put in /home/opnfv/functest/results. If you want additional logs, you may configure the logging.ini under -/usr/lib/python2.7/site-packages/functest/ci. +/usr/lib/python3.8/site-packages/xtesting/ci. Automated testing --------------- +----------------- + +In automated mode, tests are run within split Alpine containers, and test +results are displayed in jenkins logs. The result summary is provided at the +end of each suite and can be described as follow. + +Healthcheck suite:: + + +--------------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------+------------------+---------------------+------------------+----------------+ + | connection_check | functest | healthcheck | 00:03 | PASS | + | tenantnetwork1 | functest | healthcheck | 00:05 | PASS | + | tenantnetwork2 | functest | healthcheck | 00:06 | PASS | + | vmready1 | functest | healthcheck | 00:06 | PASS | + | vmready2 | functest | healthcheck | 00:08 | PASS | + | singlevm1 | functest | healthcheck | 00:32 | PASS | + | singlevm2 | functest | healthcheck | 00:37 | PASS | + | vping_ssh | functest | healthcheck | 00:46 | PASS | + | vping_userdata | functest | healthcheck | 00:39 | PASS | + | cinder_test | functest | healthcheck | 01:05 | PASS | + | tempest_smoke | functest | healthcheck | 05:39 | PASS | + | tempest_horizon | functest | healthcheck | 01:05 | PASS | + | odl | functest | healthcheck | 00:00 | SKIP | + +--------------------------+------------------+---------------------+------------------+----------------+ + +Smoke suite:: + + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron | functest | smoke | 15:30 | PASS | + | tempest_cinder | functest | smoke | 02:01 | PASS | + | tempest_keystone | functest | smoke | 01:17 | PASS | + | tempest_heat | functest | smoke | 22:14 | PASS | + | tempest_telemetry | functest | smoke | 00:00 | SKIP | + | rally_sanity | functest | smoke | 17:24 | PASS | + | refstack_compute | functest | smoke | 07:03 | PASS | + | refstack_object | functest | smoke | 02:09 | PASS | + | refstack_platform | functest | smoke | 07:31 | PASS | + | tempest_full | functest | smoke | 41:52 | PASS | + | tempest_scenario | functest | smoke | 08:42 | PASS | + | tempest_slow | functest | smoke | 43:42 | PASS | + | patrole_admin | functest | smoke | 21:06 | PASS | + | patrole_member | functest | smoke | 21:23 | PASS | + | patrole_reader | functest | smoke | 21:56 | PASS | + | tempest_barbican | functest | smoke | 02:30 | PASS | + | tempest_octavia | functest | smoke | 00:00 | SKIP | + | tempest_cyborg | functest | smoke | 00:00 | SKIP | + +---------------------------+------------------+---------------+------------------+----------------+ + +Smoke CNTT suite:: + + +-------------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------------+------------------+---------------+------------------+----------------+ + | tempest_neutron_cntt | functest | smoke | 11:35 | PASS | + | tempest_cinder_cntt | functest | smoke | 01:58 | PASS | + | tempest_keystone_cntt | functest | smoke | 01:13 | PASS | + | tempest_heat_cntt | functest | smoke | 22:32 | PASS | + | rally_sanity_cntt | functest | smoke | 17:16 | PASS | + | tempest_full_cntt | functest | smoke | 41:13 | PASS | + | tempest_scenario_cntt | functest | smoke | 08:57 | PASS | + | tempest_slow_cntt | functest | smoke | 35:58 | PASS | + +-------------------------------+------------------+---------------+------------------+----------------+ + +Benchmarking suite:: + + +--------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------+------------------+----------------------+------------------+----------------+ + | rally_full | functest | benchmarking | 93:03 | PASS | + | rally_jobs | functest | benchmarking | 27:05 | PASS | + | vmtp | functest | benchmarking | 17:56 | PASS | + | shaker | functest | benchmarking | 24:02 | PASS | + +--------------------+------------------+----------------------+------------------+----------------+ + +Benchmarking CNTT suite:: + + +-------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+----------------------+------------------+----------------+ + | rally_full_cntt | functest | benchmarking | 89:52 | PASS | + | rally_jobs_cntt | functest | benchmarking | 19:39 | PASS | + | vmtp | functest | benchmarking | 16:59 | PASS | + | shaker | functest | benchmarking | 23:43 | PASS | + +-------------------------+------------------+----------------------+------------------+----------------+ + +Vnf suite:: + + +----------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +----------------------+------------------+--------------+------------------+----------------+ + | cloudify | functest | vnf | 05:08 | PASS | + | cloudify_ims | functest | vnf | 24:46 | PASS | + | heat_ims | functest | vnf | 33:12 | PASS | + | vyos_vrouter | functest | vnf | 15:53 | PASS | + | juju_epc | functest | vnf | 27:52 | PASS | + +----------------------+------------------+--------------+------------------+----------------+ + +Kubernetes healthcheck suite:: + + +-------------------+------------------+---------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------+------------------+---------------------+------------------+----------------+ + | k8s_quick | functest | healthcheck | 00:18 | PASS | + | k8s_smoke | functest | healthcheck | 01:14 | PASS | + +-------------------+------------------+---------------------+------------------+----------------+ + +Kubernetes smoke suite:: + + +---------------------------+------------------+---------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+---------------+------------------+----------------+ + | k8s_conformance | functest | smoke | 94:26 | PASS | + | xrally_kubernetes | functest | smoke | 13:05 | PASS | + +---------------------------+------------------+---------------+------------------+----------------+ + +Kubernetes security suite:: + + +---------------------------+------------------+------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +---------------------------+------------------+------------------+------------------+----------------+ + | kube_hunter | functest | security | 00:19 | PASS | + | kube_bench_master | functest | security | 00:02 | PASS | + | kube_bench_node | functest | security | 00:01 | PASS | + +---------------------------+------------------+------------------+------------------+----------------+ + +Kubernetes benchmarking suite:: + + +--------------------------------+------------------+----------------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +--------------------------------+------------------+----------------------+------------------+----------------+ + | xrally_kubernetes_full | functest | benchmarking | 34:16 | PASS | + +--------------------------------+------------------+----------------------+------------------+----------------+ + +Kubernetes cnf suite:: -In automated mode, test results are displayed in jenkins logs, a summary is provided -at the end of the job and can be described as follow:: - - +-------------------------+----------------------------------------------------------+ - | ENV VAR | VALUE | - +-------------------------+----------------------------------------------------------+ - | INSTALLER_TYPE | daisy | - | DEPLOY_SCENARIO | os-nosdn-nofeature-ha | - | BUILD_TAG | jenkins-functest-daisy-baremetal-daily-master-67 | - | CI_LOOP | daily | - +-------------------------+----------------------------------------------------------+ - - +------------------------------+------------------+---------------------+------------------+----------------+ - | TEST CASE | PROJECT | TIER | DURATION | RESULT | - +------------------------------+------------------+---------------------+------------------+----------------+ - | connection_check | functest | healthcheck | 00:08 | PASS | - | api_check | functest | healthcheck | 04:22 | PASS | - | snaps_health_check | functest | healthcheck | 00:35 | PASS | - | vping_ssh | functest | smoke | 00:54 | PASS | - | vping_userdata | functest | smoke | 00:27 | PASS | - | tempest_smoke_serial | functest | smoke | 19:39 | FAIL | - | rally_sanity | functest | smoke | 15:16 | PASS | - | refstack_defcore | functest | smoke | 15:55 | PASS | - | snaps_smoke | functest | smoke | 26:45 | FAIL | - | cloudify_ims | functest | vnf | 23:56 | PASS | - | orchestra_openims | orchestra | vnf | 15:07 | PASS | - | orchestra_clearwaterims | orchestra | vnf | 19:10 | PASS | - | vyos_vrouter | functest | vnf | 00:00 | SKIP | - +------------------------------+------------------+---------------------+------------------+----------------+ + +-------------------------+------------------+--------------+------------------+----------------+ + | TEST CASE | PROJECT | TIER | DURATION | RESULT | + +-------------------------+------------------+--------------+------------------+----------------+ + | k8s_vims | functest | cnf | 09:06 | PASS | + | helm_vims | functest | cnf | 08:54 | PASS | + | cnf_conformance | functest | cnf | 02:00 | PASS | + +-------------------------+------------------+--------------+------------------+----------------+ Results are automatically pushed to the test results database, some additional result files are pushed to OPNFV artifact web sites. diff --git a/docs/testing/user/userguide/troubleshooting.rst b/docs/testing/user/userguide/troubleshooting.rst index 1e342b653..d857ed4c4 100644 --- a/docs/testing/user/userguide/troubleshooting.rst +++ b/docs/testing/user/userguide/troubleshooting.rst @@ -1,5 +1,4 @@ -.. This work is licensed under a Creative Commons Attribution 4.0 International License. -.. http://creativecommons.org/licenses/by/4.0 +.. SPDX-License-Identifier: CC-BY-4.0 Troubleshooting =============== @@ -15,19 +14,18 @@ credentials:: or:: - source /home/opnfv/functest/conf/openstack.creds + source /home/opnfv/functest/conf/env_file VIM --- This section covers the test cases related to the VIM (healthcheck, vping_ssh, -vping_userdata, tempest_smoke_serial, tempest_full_parallel, rally_sanity, -rally_full). +vping_userdata, tempest_smoke, tempest_full, rally_sanity, rally_full). vPing common ^^^^^^^^^^^^ -For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are -similar: +For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first +steps are similar: * Create Glance image * Create Network @@ -37,15 +35,17 @@ similar: After these actions, the test cases differ and will be explained in their respective section. -These test cases can be run inside the container, using new Functest CLI as follows:: +These test cases can be run inside the container, using new Functest CLI as +follows:: - $ functest testcase run vping_ssh - $ functest testcase run vping_userdata + $ run_tests -t vping_ssh + $ run_tests -t vping_userdata The Functest CLI is designed to route a call to the corresponding internal -python scripts, located in paths -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py -and /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py +python scripts, located in paths:: + + /usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py + /usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py Notes: @@ -73,10 +73,11 @@ Some of the common errors that can appear in this test case are:: vPing_ssh- ERROR - There has been a problem when creating the neutron network.... -This means that there has been some problems with Neutron, even before creating the -instances. Try to create manually a Neutron network and a Subnet to see if that works. -The debug messages will also help to see when it failed (subnet and router creation). -Example of Neutron commands (using 10.6.0.0/24 range for example):: +This means that there has been some problems with Neutron, even before creating +the instances. Try to create manually a Neutron network and a Subnet to see if +that works. The debug messages will also help to see when it failed (subnet and +router creation). Example of Neutron commands (using 10.6.0.0/24 range for +example):: neutron net-create net-test neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \ @@ -85,7 +86,8 @@ Example of Neutron commands (using 10.6.0.0/24 range for example):: neutron router-interface-add <ROUTER_ID> test_subnet neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME> -Another related error can occur while creating the Security Groups for the instances:: +Another related error can occur while creating the Security Groups for the +instances:: vPing_ssh- ERROR - Failed to create the security group... @@ -100,9 +102,9 @@ In this case, proceed to create it manually. These are some hints:: --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0 The next step is to create the instances. The image used is located in -*/home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is created -with the name **functest-vping**. If booting the instances fails (i.e. the status -is not **ACTIVE**), you can check why it failed by doing:: +*/home/opnfv/functest/data/cirros-0.5.1-x86_64-disk.img* and a Glance image is +created with the name **functest-vping**. If booting the instances fails (i.e. +the status is not **ACTIVE**), you can check why it failed by doing:: nova list nova show <INSTANCE_ID> @@ -113,7 +115,8 @@ It might show some messages about the booting failure. To try that manually:: This will spawn a VM using the network created previously manually. In all the OPNFV tested scenarios from CI, it never has been a problem with the -previous actions. Further possible problems are explained in the following sections. +previous actions. Further possible problems are explained in the following +sections. vPing_SSH @@ -122,7 +125,7 @@ This test case creates a floating IP on the external network and assigns it to the second instance **opnfv-vping-2**. The purpose of this is to establish a SSH connection to that instance and SCP a script that will ping the first instance. This script is located in the repository under -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh +/usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/vping/ping.sh and takes an IP as a parameter. When the SCP is completed, the test will do a SSH call to that script inside the second instance. Some problems can happen here:: @@ -130,8 +133,8 @@ here:: vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting If this is displayed, stop the test or wait for it to finish, if you have used -the special method of test invocation with specific supression of OpenStack -resource clean-up, as explained earler. It means that the Container can not +the special method of test invocation with specific suppression of OpenStack +resource clean-up, as explained earlier. It means that the Container can not reach the Public/External IP assigned to the instance **opnfv-vping-2**. There are many possible reasons, and they really depend on the chosen scenario. For most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known @@ -143,19 +146,21 @@ from the DHCP agent. It can be checked by doing:: nova console-log opnfv-vping-2 If the message *Sending discover* and *No lease, failing* is shown, it probably -means that the Neutron dhcp-agent failed to assign an IP or even that it was not -responding. At this point it does not make sense to try to ping the floating IP. +means that the Neutron dhcp-agent failed to assign an IP or even that it was +not responding. At this point it does not make sense to try to ping the +floating IP. -If the instance got an IP properly, try to ping manually the VM from the container:: +If the instance got an IP properly, try to ping manually the VM from the +container:: nova list <grab the public IP> ping <public IP> -If the ping does not return anything, try to ping from the Host where the Docker -container is running. If that solves the problem, check the iptable rules because -there might be some rules rejecting ICMP or TCP traffic coming/going from/to the -container. +If the ping does not return anything, try to ping from the Host where the +Docker container is running. If that solves the problem, check the iptables +rules because there might be some rules rejecting ICMP or TCP traffic +coming/going from/to the container. At this point, if the ping does not work either, try to reproduce the test manually with the steps described above in the vPing common section with the @@ -176,7 +181,8 @@ vPing_userdata This test case does not create any floating IP neither establishes an SSH connection. Instead, it uses nova-metadata service when creating an instance to pass the same script as before (ping.sh) but as 1-line text. This script -will be executed automatically when the second instance **opnfv-vping-2** is booted. +will be executed automatically when the second instance **opnfv-vping-2** is +booted. The only known problem here for this test to fail is mainly the lack of support of cloud-init (nova-metadata service). Check the console of the instance:: @@ -219,79 +225,69 @@ In the upstream OpenStack CI all the Tempest test cases are supposed to pass. If some test cases fail in an OPNFV deployment, the reason is very probably one of the following -+-----------------------------+-----------------------------------------------------+ -| Error | Details | -+=============================+=====================================================+ -| Resources required for test | Such resources could be e.g. an external network | -| case execution are missing | and access to the management subnet (adminURL) from | -| | the Functest docker container. | -+-----------------------------+-----------------------------------------------------+ -| OpenStack components or | Check running services in the controller and compute| -| services are missing or not | nodes (e.g. with "systemctl" or "service" commands).| -| configured properly | Configuration parameters can be verified from the | -| | related .conf files located under '/etc/<component>'| -| | directories. | -+-----------------------------+-----------------------------------------------------+ -| Some resources required for | The tempest.conf file, automatically generated by | -| execution test cases are | Rally in Functest, does not contain all the needed | -| missing | parameters or some parameters are not set properly. | -| | The tempest.conf file is located in directory | -| | 'root/.rally/verification/verifier-<UUID> | -| | /for-deployment-<UUID>' | -| | in the Functest Docker container. Use the "rally | -| | deployment list" command in order to check the UUID | -| | the UUID of the current deployment. | -+-----------------------------+-----------------------------------------------------+ ++----------------------------+------------------------------------------------+ +| Error | Details | ++============================+================================================+ +| Resources required for | Such resources could be e.g. an external | +| testcase execution are | network and access to the management subnet | +| missing | (adminURL) from the Functest docker container. | ++----------------------------+------------------------------------------------+ +| OpenStack components or | Check running services in the controller and | +| services are missing or | compute nodes (e.g. with "systemctl" or | +| not configured properly | "service" commands). | +| | Configuration parameters can be verified from | +| | the related .conf files located under | +| | '/etc/<component>' directories. | ++----------------------------+------------------------------------------------+ +| Some resources required | The tempest.conf file, automatically generated | +| for execution test cases | by Rally in Functest, does not contain all the | +| are missing | needed parameters or some parameters are not | +| | set properly. | +| | The tempest.conf file is located in directory | +| | 'root/.rally/verification/verifier-<UUID> | +| | /for-deployment-<UUID>' | +| | in the Functest Docker container. Use the | +| | "rally deployment list" command in order to | +| | check the UUID of the current deployment. | ++----------------------------+------------------------------------------------+ When some Tempest test case fails, captured traceback and possibly also the -related REST API requests/responses are output to the console. More detailed debug -information can be found from tempest.log file stored into related Rally deployment -folder. +related REST API requests/responses are output to the console. More detailed +debug information can be found from tempest.log file stored into related Rally +deployment folder. Functest offers a possibility to test a customized list of Tempest test cases. -To enable that, add a new entry in docker/components/testcases.yaml on the "components" container -with the following content:: - - - - case_name: tempest_custom - project_name: functest - criteria: 100 - blocking: false - description: >- - The test case allows running a customized list of tempest - test cases - dependencies: - installer: '' - scenario: '' - run: - module: 'functest.opnfv_tests.openstack.tempest.tempest' - class: 'TempestCustom' - -Also, a list of the Tempest test cases must be provided to the container or modify -the existing one in -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt - -Example of custom list of tests 'my-custom-tempest-tests.txt':: - - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke] - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke] +To enable that, add a new entry in docker/smoke/testcases.yaml on the +"smoke" container with the following content:: + + - + case_name: tempest_custom + project_name: functest + criteria: 100 + blocking: false + description: >- + The test case allows running a customized list of tempest + test cases + run: + name: tempest_common + args: + mode: "tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops|\ + tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops" This is an example of running a customized list of Tempest tests in Functest:: sudo docker run --env-file env \ - -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \ + -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \ -v $(pwd)/images:/home/opnfv/functest/images \ - -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python2.7/site-packages/functest/ci/testcases.yaml \ - -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \ - opnfv/functest-components run_tests -t tempest_custom - + -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python3.8/site-packages/xtesting/ci/testcases.yaml \ + opnfv/functest-smoke run_tests -t tempest_custom Rally ^^^^^ -The same error causes which were mentioned above for Tempest test cases, may also -lead to errors in Rally as well. +The same error causes which were mentioned above for Tempest test cases, may +also lead to errors in Rally as well. Possible scenarios are: * authenticate @@ -301,18 +297,19 @@ Possible scenarios are: * keystone * neutron * nova - * ceilometer * quotas * vm -To know more about what those scenarios are doing, they are defined in directory: -/usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/rally/scenario -For more info about Rally scenario definition please refer to the Rally official -documentation. `[3]`_ +To know more about what those scenarios are doing, they are defined in +directory: +/usr/lib/python3.8/site-packages/functest/opnfv_tests/openstack/rally/scenario +For more info about Rally scenario definition please refer to the Rally +official documentation. `[3]`_ To check any possible problems with Rally, the logs are stored under */home/opnfv/functest/results/rally/* in the Functest Docker container. +.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html Controllers ----------- @@ -372,7 +369,7 @@ described in the following table: | the VM | the vIMS VNF installation fails | +-----------------------------------+------------------------------------+ -Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), it -is there fore not recommended to run it on a light configuration. +Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)), +it is there fore not recommended to run it on a light configuration. .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/testing_developer_devguide/index.html# |
