blob: 48f01cb0cb6e090f2ad249a272ca73b4d32e718a (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
|
# Functest 2019
[Cédric Ollivier](mailto:cedric.ollivier@orange.com)
2019/01/08
## Gates
### Today's Verify -1
- one error is detected by yamllint, pep8, pylint, ansible-lint, bashate or
doc8
- one unit test fails (py27 and py35)
- specific modules are not rated 10/10 by pylint
- our Framework is not fully covered
- unix permissions are wrong
- one doc cannot be generated without warning
Please see [tox.ini](https://git.opnfv.org/functest/tree/tox.ini) for details
### Great! But
- the current gate checks all coding rules, our Framework and all interfaces
to third-parties but **not our testcases**
- it doesn't break **the circular dependencies** between Installers and
Functest (Installers ask for a trustable healthcheck but we need results of
"Installer runs" as prerequisites)
Our testcases have still **to be verified by hand** what differs from
OpenStack Workflow.
### Functional gating
- we need to run all Functest patches before merge against reference
plateforms (OpenStack and Kubernetes)
- a voting job must forbid the merge if one test fails (as the workflow
process implemented by OpenStack)
### Improve the current daily model
- all Functest jobs are linked to the OPNFV installers and can hardly be
reused by endusers
- all test suites are run sequentially
- rally_full is excluded due to its duration (~3 hours)
- all possible remaining resources are cleant when installing the scenarios
**It's fine for gating installers but not for verifying that Functest supports parallel and live testing**
### New Xtesting Ansible role
- deploy anywhere the full OPNFV CI/CD toolchain in few commands
- easily add external bots voting in reviews
- produce all Xtesting-based fonctional jobs (Xtesting, Functest and Functest
Kubernetes) in Releng
- could be instanciated for testing services out of the infrastructure domain
**It's already in a [good shape](https://lists.opnfv.org/g/opnfv-tech-discuss/message/22552). [Try it!](https://wiki.opnfv.org/pages/viewpage.action?pageId=32015004)**
## Quality Assurance
**Functest is matching the OpenStack Quality Golden rules**
### Pylint and coverage (started from E release)
- only need to refactor several vnf testcases and to remove duplicated code
- document well and cover Functest utils: they are reused by
OPNFV third-parties (SFC, SDNVPN)
** Functest will be rated 10/10 very soon**
### releasing
- Functest could be also released as a classical Python packages like Xtesting(
[PyPI](https://pypi.python.org/pypi))
- Xtesting and Functest could be released as GNU/Linux distribution packages
## other challenges
### proposals
- integrate Rally and Tempest as core modules in Functest (we do merge tempest.py and
conf_utils.py)
- deploy cloud-native VNF (Clearwater IMS?)
- why not leveraging on OpenStack middlewares such as
[oslo.config](https://docs.openstack.org/oslo.config/latest/)?
- promote our VNF testcases in the upstream communities
**Any new testcase is more than welcome!**
## Thank you
|
