1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
# Functest on steroids
[Cédric Ollivier](mailto:cedric.ollivier@orange.com)
2019/01/08
## OPNFV
<!-- .element: style="border: 0; width: 90%" -->
## Verify OpenStack and Kubernetes
### Functest in a nutshell
- verify any kind of OpenStack and Kubernetes deployments
- conform with upstream rules (OpenStack gate jobs and Kubernetes conformance
tests)
- ensure that the platforms meet Network Functions Virtualization requirements
### Functest suites
- as many upstream functional tests as possible (e.g. Tempest,
neutron-tempest-api, Barbican, Patrole...)
- upstream API and dataplane benchmarking tools (Rally, Vmtp and Shaker)
- additional VNF deployments and testing (vIMS, vRouter and vEPC)
**Dovetail only runs few Functest functional tests and then we have to verify
any OPNFV-certified scenarios via Functest anyway**
### What's new in Functest?
- new testcases were quickly integrated in Functest: Patrole, Barbican, Shaker,
...
- all testcases can run in parallel to decrease the overall duration
- the resources cleaning has been improved
- **our testcases may be run vs VIM in production**
- **it includes most of the OpenStack gate jobs**
### Support of OS and K8s master
| Functest | OpenStack | Kubernetes |
| :------: | :---------: | :--------: |
| master | master | master |
| hunter | rocky | v1.11.3 |
| gambia | queens | v1.11.3 |
### < 50 Euros
<!-- .element: style="border: 0; width: 70%" -->
## Reuse of OPNFV
### Xtesting in a nutshell
- allow the developer to work only on the test suites without diving into CI/CD
integration
- simplify test integration in a complete LFN-based CI/CD toolchain (e.g.
Jenkins, Testing Containers, Test API and dashboard)
- allow a proper design and verify multiple components in the same CI/CD
toolchain (OpenStack, Kubernetes, ONAP, etc.)
**Easy to use and very useful for any CI/CD toochain (unlinked to Infrastrure)**
### A user story ONAP
- all tests are run by a specialized Docker container(**<100 MB**) instead of
the classical ONAP testing virtual machine (**> 1GB**).
- the container mainly inherits from opnfv/xtesting and is completed by:
- Python dependencies
- all ONAP Robot Framework files retrieved from the original repositories
- testcases.yaml describing the testcases
[Orange-OpenSource/xtesting-onap-robot](https://github.com/Orange-OpenSource/xtesting-onap-robot/)
### What's new in Xtesting?
- new ansible roles and playbooks have been developed to allow **deploying your
full CI/CD toolchains in few minutes** (Jenkins, Minio, TestAPI, MongoDB and
Docker registry)
```bash
virtualenv xtesting
. xtesting/bin/activate
pip install ansible docker
ansible-galaxy install collivier.xtesting
git clone https://gerrit.opnfv.org/gerrit/functest-xtesting functest-xtesting-src
ansible-playbook functest-xtesting-src/ansible/site.yml
deactivate
```
**They are already reused in Functest and by Orange out of the
Infrastrure domain**
## Collect results
### OPNFV Test Database in a nutshell
- it's a fair comparison of Neutron implementations (Agents vs SDN
controller)
- it stores all verification results and all performance data from different
hardware over the world which could be easily postprocessed
- it could be very useful to select the adequate opensource solutions
regarding metrics and capabilities
### which Neutron backend?
- most Neutron standalone and OVN scenarios pass Functest decently
- no ODL scenarios pass the advanced testcases (benchmarking tools and
VNFs). It's still unclear if it's due to the Installers, ODL or POD
misconfigurations.
- no Tungsten Fabric is released in Gambia
**We expected that ODL results would have improved before the first Gambia
corrective**
### Contrail testing (out of OPNFV)
- **3.X 4.X**: mostly verified except some functional tests about
visibility which fail due to the falsy admin role (they can be easily
blacklisted)
- **5.X**: a limited set of bugs in Contrail mostly forbid running few
functional tests and benchmarking tools:
- wrong external network listing
- Contrail doesn't allow booting a VM without network (and elects the wrong
network)
### And performance?
- OVS DPDK is not fully integrated by an OPNFV installer (Fuel is in a good
shape to support it)
- the only scenario including VPP is not part of Gambia
**From the time being, we can't evaluate the benefits of OVS DPDK or VPP thanks
to OPNFV**
## Conclusion
### Gambia
- Functest and Xtesting are powerful and easy to (re)use (containers, jenkins
jobs, ansible playbooks, Raspberry PI, etc.).
- the number of installers and scenarios decreased in Gambia (it's still
unclear regarding the overall quality). **What about OPNFV Test Database if
it decreases again in 2019?**
### And beyond
- test frameworks are now considered as crucial for OPNFV (see [Last OPNFV Marketing update](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Marketing%20Update%20091818.pptx?version=1&modificationDate=1537228648000&api=v2)) and Functest and Xtesting could be
already widely reused out of OPNFV
- the new test-driven approach as proposed by the [OPNFV Strategic Plan](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Strategy%20and%20Plan%20v0.5.pptx?version=1&modificationDate=1540961098000&api=v2) could increase **the quality of all
scenarios**. But we are also suggesting to let the installers decide their
test cases.
|
