From 8b1cb4abd504e6286a691575c31c6a69aa6f8e50 Mon Sep 17 00:00:00 2001 From: Leo Wang Date: Sun, 9 Oct 2016 21:53:00 -0400 Subject: Add documentation for dovetail tool JIRA:DOVETAIL-28 Change-Id: I2478655d951a65a11f39e9ceba10969e56274cc3 Signed-off-by: Leo Wang --- .../installationprocedure/feature.configuation.rst | 45 +++++++ docs/installationprocedure/index.rst | 14 ++ .../installation.instruction.rst | 142 +++++++++++++++++++++ docs/userguide/01-introduction.rst | 4 +- docs/userguide/04-dovetail_config_template.rst | 134 +++++++++++++++++++ docs/userguide/index.rst | 1 + 6 files changed, 338 insertions(+), 2 deletions(-) create mode 100644 docs/installationprocedure/feature.configuation.rst create mode 100644 docs/installationprocedure/index.rst create mode 100644 docs/installationprocedure/installation.instruction.rst create mode 100644 docs/userguide/04-dovetail_config_template.rst diff --git a/docs/installationprocedure/feature.configuation.rst b/docs/installationprocedure/feature.configuation.rst new file mode 100644 index 00000000..0bc4ae33 --- /dev/null +++ b/docs/installationprocedure/feature.configuation.rst @@ -0,0 +1,45 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +Testcase Template Syntax +========================= + +The testcases certification are defined in ``dovetail/dovetail/testcase``. +Take testcase ``ipv6.tc001.yml`` as an example. + +Basic template syntax +--------------------- + +:: + + dovetail.ipv6.tc001: + name: dovetail.ipv6.tc001 + objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation + scripts: + type: functest + testcase: tempest_smoke_serial + sub_testcase_list: + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet + +The testcase needs at least three keys named as 'name', 'objective' and 'scripts'. The whole +dovetail project can just recognize testcases by the 'name' section, which here is +'dovetail.ipv6.tc001'. The 'objective' describes what exactly this testcase does. In the +'scripts' part, there are some subsections such as 'type', 'testcase' and 'sub_testcase_list'. +Dovetail now supports two kinds of types, functest and yardstick. If you define the type as +functest, then you need to give the functest testcase that you want to use. If the type is +yardstick, then a yardstick testcase is needed. The 'sub_testcase_list' lists the sub_testcases +that you put a high value on. Even though the whole testcase faild, we still think it passed +when all the sub_testcases pass. The sub_testcase_list is just available for functest. When +for yardstick testcase, retain 'sub_testcase_list' section and keep the content empty. + +:: + + sub_testcase_list: + - + +This is the definition of the dovetail testcases. They can just be tested by adding into +scenarios such as ``cert/basic.yml``. diff --git a/docs/installationprocedure/index.rst b/docs/installationprocedure/index.rst new file mode 100644 index 00000000..9a755800 --- /dev/null +++ b/docs/installationprocedure/index.rst @@ -0,0 +1,14 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +================== +Dovetail Overview +================== + +.. toctree:: + :maxdepth: 2 + + installation.instruction + feature.configuation diff --git a/docs/installationprocedure/installation.instruction.rst b/docs/installationprocedure/installation.instruction.rst new file mode 100644 index 00000000..aa1b436a --- /dev/null +++ b/docs/installationprocedure/installation.instruction.rst @@ -0,0 +1,142 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +===================== +Dovetail Installation +===================== + +Abstract +======== + +Dovetail currently supports installation on Ubuntu 14.04 or by using a docker image. +Detailed steps about installing dovetail can be found below. + +To use dovetail you should have access to an OpenStack environment, +with at least Nova, Neutron, Glance, Keystone and Heat installed. + + +Run dovetail on local machine +============================= + +The steps needed to run dovetail on Ubuntu are: + +1. Download source code and prepare environment. +2. Create certification configuration file. +3. Run certification. + + +Download dovetail and prepare environment +----------------------------------------- + +The source code of dovetail can be got from Gerrit: + +:: + + git clone https://gerrit.opnfv.org/gerrit/dovetail + +After that you need to prepare the environment which means install some tools, +modules and anything else dovetail needs. A file named ``prepare_env.py`` can +do all of these for you. + +:: + + cd dovetail/dovetail/ + python prepare_env.py + +Now the environment should be prepared for running the certification. + +Besides, you may want to install it in a python virtualenv, that is beyond the scope +of this installation instruction. + +Create certification configuration file +--------------------------------------- + +You can wirte a certification configuration file with the format of yaml, +which can be defined to meet your own requirements. The yaml file should +be located in the folder ``dovetail/dovetail/cert``. There provide a sample +file named ``basic.yml``. + +:: + + certification_basic: + name: certification_basic + testcase_list: + - dovetail.ipv6.tc001 + +As you can see, the configuration file combines the testcases you want to run +together, and all of them should already been defined in ``dovetail/dovetail/testcase``. +The name of the configuration file should be certification_%s, where %s (here is basic) +can be used to run this certification. + + +Run the certification +--------------------- + +When you get the dovetail source code, prepare the environment and define the +certification configuration file, you can then run the certification. Take +certification_basic as the example. You can run it using the file ``run.py`` with +basic (scenario name) as its argument. + +:: + + python run.py --scenario basic + +If you did not give an argument, it will be default set as basic. + + +Run dovetail on Docker container +================================ + +steps: +1. pull image +2. run container +3. config +4. run testcase + +pull image +---------- + +Get the latest docker image from docker hub. + +:: + sudo docker pull opnfv/dovetail:latest + +run container +------------- + +Start a container from the image that you just pulled. + +:: + sudo docker run -it --privileged=true -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results + -v /var/run/docker.sock:/var/run/docker.sock opnfv/dovetail:latest "/bin/bash" + +config +------ + +For now, we need only change value of ``INSTALLER_TYPE`` and ``INSTALLER_IP`` +of the variable ``envs`` to your own environment. + +:: + sudo docker exec -it $(CONTAINER_ID) "/bin/bash" + vi /home/opnfv/dovetail/dovetail/conf/functest_config.yml + vi /home/opnfv/dovetail/dovetail/conf/yardstick_config.yml + +run testcase +------------ + +In docker, you just run following cmds to execute the testcase. + +:: + cd /home/opnfv/dovetail/dovetail + python run.py + + + + +Output +====== + +The running log is stored in ``/home/opnfv/dovetail/results/dovetail.log``. +The report of certification is stored in ``/home/opnfv/dovetail/results/dovetail_report.txt``. diff --git a/docs/userguide/01-introduction.rst b/docs/userguide/01-introduction.rst index f3d57be8..dbe33d5d 100644 --- a/docs/userguide/01-introduction.rst +++ b/docs/userguide/01-introduction.rst @@ -2,7 +2,7 @@ Dovetail Introduction ====================== -**Welcome to Dovetail' documentation !** +**Welcome to Dovetail's documentation !** .. _Dovetail: https://wiki.opnfv.org/dovetail @@ -11,7 +11,7 @@ Dovetail_ is an OPNFV Project. This project intends to define and provide a set of OPNFV related validation criteria that will provide input for the evaluation of the use of OPNFV trademarks. The dovetail project will be executed with the guidance and oversight of the Compliance -and Certification committee and work to secure the goals of the C&C committee for each release +and Certification committee and work to secure the goals of the C&C committee for each release. This project intends to incrementally define qualification criteria that establish the foundations of how we are able to measure the ability diff --git a/docs/userguide/04-dovetail_config_template.rst b/docs/userguide/04-dovetail_config_template.rst new file mode 100644 index 00000000..fce99d21 --- /dev/null +++ b/docs/userguide/04-dovetail_config_template.rst @@ -0,0 +1,134 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International +.. License. +.. http://creativecommons.org/licenses/by/4.0 +.. (c) OPNFV, Huawei Technologies Co.,Ltd and others. + +Config Template Syntax +====================== + +Dovetail uses Functest/Yardstick Docker container to run its testcases. So you need to give +configurations for building the container and the commands it needs to do. In dovetail, +all of these are defined in config yaml files ``dovetail/dovetail/conf/functest_config.yml`` +and ``dovetail/dovetail/conf/yardstick_config.yml``. + +Functest template syntax +------------------------ + +For example, you can define your ``functest_config.yml`` as: + +:: + + functest: + image_name: opnfv/functest + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal' + opts: '-id --privileged=true' + pre_condition: + cmds: + - 'echo test for precondition' + testcase: + cmds: + - 'python /home/opnfv/repos/functest/ci/prepare_env.py start' + - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r' + post_condition: + cmds: + - '' + result: + dir: '/home/opnfv/functest/results' + store_type: 'file' + file_path: 'tempest/tempest.log' + db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' + +First, you need to give the image that you want to use for building functest/yardstick container. +Besides, there also need some envirnment parameters such as ``INSTALLER_TYPE`` and ``INSTALLER_IP`` +and the options for you container. Then the functest/yardstick container can be build with your +settings. + +Second, there need three kinds of commands, ``pre_condition``, ``testcase`` and ``post_condition``. +If you want to do some cleanups or preparations, the commands can be put into ``pre_condition`` +section orderly. All commands in this section will just be executed once in the begining. +The ``testcase`` section does the main jobs of the testing. All functest testcases will use the +container to execute these commands one by one. After finishing that, the test is accomplished +and the results are stored in files or uploaded to database. The ``post_condition`` section +does some work such as clean Docker images or something else after all testcases finished. +All commands in this section will just execute once. + +Besides, there need a ``result`` section and it gives the directory of the functest/yardstick +results. The ``store_type`` should be the same with the cmds in ``testcase``. That means if the +test results are stored in files, then store_type need to be file and the file_path is also +needed. If the test results are uploaded to database, then a db_url is needed for acquiring the results. + +Yardstick template syntax +------------------------- + +The framework of ``yardstick_config.yml`` is almost the same as ``functest_config.yml``. + +:: + + yardstick: + image_name: opnfv/yardstick + docker_tag: latest + envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2 + -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn + -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal + -e EXTERNAL_NETWORK=ext-net' + opts: '-id --privileged=true' + pre_condition: + cmds: + - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && + source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup' + - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh && + cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh' + testcase: + cmds: + - 'mkdir -p /home/opnfv/yardstick/results/' + - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh && + yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml + --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &> + /home/opnfv/yardstick/results/yardstick.log' + post_condition: + cmds: + - '' + result: + dir: '/home/opnfv/yardstick/results' + store_type: 'file' + file_path: 'yardstick.log' + db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1' + +The main differences between ``yardstick_config.yml`` and ``functest_config.yml`` are the commands. + +Jinja2 template syntax +---------------------- + +Note that you can use jinja2 template for your parameters such as ``{{script_testcase}}``. The +parameters are defined in ``dovetail/dovetail/conf/dovetail_config.yml``: + +:: + + parameters: + - name: testcase + path: '("name",)' + - name: script_testcase + path: '("scripts", "testcase")' + +Here ``path`` is the path in testcase config files that you can find the value of parameters. Take +``script_testcase`` as the example. For testcase dovetail.ipv6.tc001: + +:: + + dovetail.ipv6.tc001: + name: dovetail.ipv6.tc001 + objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation + scripts: + type: functest + testcase: tempest_smoke_serial + sub_testcase_list: + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port + - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet + +The path ("scripts", "testcase") means 'testcase' subsection of 'scripts' section. So follow +the path ("scripts", "testcase") we can get the value of ``{{script_testcase}}`` that is +'tempest_smoke_serial'. diff --git a/docs/userguide/index.rst b/docs/userguide/index.rst index 03246e7b..0dcf55ce 100644 --- a/docs/userguide/index.rst +++ b/docs/userguide/index.rst @@ -14,3 +14,4 @@ Dovetail - user guide ./01-introduction.rst ./02-certification_criteria.rst ./03-certification_progress.rst + ./04-dovetail_config_template.rst -- cgit n357'>357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879