diff options
| -rw-r--r-- | design_docs/main.rst | 1 | ||||
| -rw-r--r-- | docs/images/ci_infra.png | bin | 0 -> 123943 bytes | |||
| -rw-r--r-- | docs/images/daily_job.png | bin | 0 -> 11968 bytes | |||
| -rw-r--r-- | docs/images/merge_job.png | bin | 0 -> 28138 bytes | |||
| -rw-r--r-- | docs/images/pipeline_overview.png | bin | 0 -> 26210 bytes | |||
| -rw-r--r-- | docs/images/verify_job.png | bin | 0 -> 26956 bytes | |||
| -rw-r--r-- | docs/octopus_info.rst (renamed from docs/main.rst) | 5 | ||||
| -rw-r--r-- | docs/opnfv-artifact-repository.rst | 254 | ||||
| -rw-r--r-- | docs/opnfv-ci-infrastructure.rst | 66 | ||||
| -rw-r--r-- | docs/opnfv-ci-pipelines.rst | 89 | ||||
| -rw-r--r-- | docs/opnfv-jenkins-slave-connection.rst | 258 | ||||
| -rw-r--r-- | docs/opnfv-jjb-usage.rst | 232 | ||||
| -rw-r--r-- | docs/release/main.rst | 1 | ||||
| -rw-r--r-- | requirements/main.rst | 1 |
14 files changed, 563 insertions, 344 deletions
diff --git a/design_docs/main.rst b/design_docs/main.rst deleted file mode 100644 index 7bfa94b..0000000 --- a/design_docs/main.rst +++ /dev/null @@ -1 +0,0 @@ -Just a dummy file diff --git a/docs/images/ci_infra.png b/docs/images/ci_infra.png Binary files differnew file mode 100644 index 0000000..53c329c --- /dev/null +++ b/docs/images/ci_infra.png diff --git a/docs/images/daily_job.png b/docs/images/daily_job.png Binary files differnew file mode 100644 index 0000000..7386ec6 --- /dev/null +++ b/docs/images/daily_job.png diff --git a/docs/images/merge_job.png b/docs/images/merge_job.png Binary files differnew file mode 100644 index 0000000..e48ecde --- /dev/null +++ b/docs/images/merge_job.png diff --git a/docs/images/pipeline_overview.png b/docs/images/pipeline_overview.png Binary files differnew file mode 100644 index 0000000..de1d4bf --- /dev/null +++ b/docs/images/pipeline_overview.png diff --git a/docs/images/verify_job.png b/docs/images/verify_job.png Binary files differnew file mode 100644 index 0000000..e5ab9db --- /dev/null +++ b/docs/images/verify_job.png diff --git a/docs/main.rst b/docs/octopus_info.rst index b854b1d..8686d32 100644 --- a/docs/main.rst +++ b/docs/octopus_info.rst @@ -23,3 +23,8 @@ The CI project provides the starting point for all OPNFV development activities. * **Execute the continuous automated builds and basic verification** +**Documentation tracking** + +Revision: _sha1_ + +Build date: _date_ diff --git a/docs/opnfv-artifact-repository.rst b/docs/opnfv-artifact-repository.rst index a0f4be1..38efb5c 100644 --- a/docs/opnfv-artifact-repository.rst +++ b/docs/opnfv-artifact-repository.rst @@ -1,80 +1,174 @@ -TL;DR
------
-
-Don't use git to store binary files that are built from source such as PDF files or Virtual Machine Images.
-
-What is Artifact Repository
----------------------------
-
-An artifact repository is akin to what Subversion is to source code, i.e. it is a way of versioning code binary artifacts. In the Java world these artifacts could be jars, wars, ears, fully fledged applications, libraries or a collections of libraries that are packaged. (Read the rest from `the reference <http://blogs.collab.net/subversion/why-you-should-be-using-an-artifact-repository-part-1>`_.)
-
-What will be used as Artifact Repository for OPNFV
---------------------------------------------------
-
-It has been decided to use Google Storage as OPNFV Artifact Repository.
-
-This is due to not having enough storage in OPNFV infrastructure and installing/managing a robust Artifact Repository is not straightforward.
-
-Alternatives
-------------
-
-There are number of different alternatives that can serve as Artifact Repository Manager (ARM).
-
-Well known ones are
-
-* `JFrog Artifactory <http://www.jfrog.com/open-source/>`_
-* `Sonatype Nexus <http://www.sonatype.org/nexus/>`_
-* `Apache Archiva <http://archiva.apache.org/index.cgi>`_
-* `Swift <https://wiki.openstack.org/wiki/Swift>`_
-* `Google Cloud Storage <https://cloud.google.com/storage/>`_
-
-What other OSS Projects Use
----------------------------
-
-Different OSS Projects use different ARMs as listed below.
-
-* OpenDaylight: Sonatype Nexus. (provided by Linux Foundation.) `https://wiki.opendaylight.org/view/Infrastructure:Nexus <https://wiki.opendaylight.org/view/Infrastructure:Nexus>`_.
-* OpenStack: Uses tarball site. `http://tarballs.openstack.org/ <http://tarballs.openstack.org/>`_.
-* Apache: Sonatype Nexus. `https://repository.apache.org/ <https://repository.apache.org/>`_.
-
-Use of ARM in CI
-----------------
-
-Binaries/packages that are produced by CI can be deployed/uploaded to Artifact Repository making it possible to reuse artifacts during later stages of CI. They can be consumed by individual developers/organizations as well.
-
-Whatever is produced by CI can always be tracked by doing this when needed.
-
-Uploading artifacts using REST
-------------------------------
-
-Both Artifactory and Nexus provide REST APIs to upload/manipulate artifacts. Below are the links to some examples.
-
-* `How can I programatically upload an artifact into Nexus? <https://support.sonatype.com/entries/22189106-How-can-I-programatically-upload-an-artifact-into-Nexus->`_
-* `Deploy Artifact to Artifactory <http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-DeployArtifact>`_
-
-Uploading artifacts using Jenkins Plugins
------------------------------------------
-
-Jenkins has number of plugins that offer integration of different ARMs. One of the examples is `Artifactory Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Artifactory+Plugin>`_.
-
-Longer version
---------------
-
-This is a summary of various email discussions on the topic of where to store large binary image files. Typical examples include videos, virtual machine images, software installers, ISOs for Operating Systems.
-
-Since many developers check their source code into the GIT repository it may seem natural to just place the files you've built into the repo too. This can work okay for a single developer working on a project over the weekends but with a team working on many components that need to be tested and integrated this won't scale.
-
-The way git works, no revision of any file is ever lost. So if you ever check in a big file, the repository will always contain it, and a git clone will be that much slower for every clone from that point onward.
-
-The golden rule of revision control systems applies: check in your build scripts, not your build products.
-
-Unfortunately, it only takes one person to start doing this and we end up with huge repositories. Please don't do this. It will make your computers sad. Thankfully, gerrit and code review systems are a massive disincentive to doing this.
-
-You definitely need to avoid storing binary images in git. This is what artifact repositories are for, like `Nexus or Artifactory <https://twiki.auscope.org/wiki/Grid/NexusVsArtifactory>`_
-
-A “centralized image repository” is needed that can store multiple versions of various virtual machines and have something like /latest pointing to the newest uploaded image. It could be a simple nginx server that stores the output images from any jenkins job if it's successful, for instance. Any “client” would be able to fetch the latest “autorelease” using `https://server/image/latest <https://server/image/latest>`_ to get the latest image or `https://server/stcv/1.1 <https://server/stcv/1.1>`_ and so on for any older image.
-
-For the specific project “vswitch performance characterization”, we have a directory as place holder to store the VM images. The VM images are not going to be in git repo. We use the same working git repository inside VM to run initial setup, and save the VM image in the host.
-
-TAG: GIT repository file download section
-
+=========================== + OPNFV Artifact Repository +=========================== + +Artifact Repository +=================== + +What is Artifact Repository +--------------------------- + +An Artifact Repository is akin to what Subversion is to source code, i.e. it is a way of versioning artifacts produced by build systems, CI, and so on. [1] + +Why Artifact Repository is Needed +--------------------------------- + +Since many developers check their source code into the GIT repository it may seem natural to just place the files you've built into the repo too. This can work okay for a single developer working on a project over the weekends but with a team working on many components that need to be tested and integrated, this won't scale. + +The way git works, no revision of any file is ever lost. So if you ever check in a big file, the repository will always contain it, and a git clone will be that much slower for every clone from that point onward. + +The golden rule of revision control systems applies: *check in your build scripts, not your build products*. + +Unfortunately, it only takes one person to start doing this and we end up with huge repositories. Please don't do this. It will make your computers sad. Thankfully, Gerrit and code review systems are a massive disincentive to doing this. + +You definitely need to avoid storing binary images in git. This is what artifact repositories are for. [2] + +A “centralized image repository” is needed that can store multiple versions of various virtual machines and have something like /latest pointing to the newest uploaded image. It could be a simple nginx server that stores the output images from any jenkins job if it's successful, for instance. + +OPNFV Artifact Repository +========================= + +What is used as Artifact Repository for OPNFV +--------------------------------------------- + +Setting up, hosting, and operating an artifact repository on OPNFV Infrastructure in Linux Foundation (LF) environment requires too much storage space. It is also not a straightforward undertaking to have robust Artifact Repository and provide 24/7 support. + +OPNFV Project decided to use **Google Cloud Storage** as OPNFV Artifact Repository due to reasons summarized above. [3] + +Usage of Artifact Repository in OPNFV CI +---------------------------------------- + +Binaries/packages that are produced by OPNFV Continuous Integration (CI) are deployed/uploaded to Artifact Repository making it possible to reuse artifacts during later stages of OPNFV CI. Stored artifacts can be consumed by individual developers/organizations as well. + +In OPNFV, we generally produce PDF, ISO and store them on OPNFV Artifact Repository. + +OPNFV Artifact Repository Web Interface +---------------------------------------- + +OPNFV Artifact Repository is accessible via link http://artifacts.opnfv.org/. + +A proxy has been set up by LF for the community members located in countries with access restrictions to Google http://build.opnfv.org/artifacts/. + +Access Rights to OPNFV Artifact Repository +========================================== + +As summarized in previous sections, OPNFV uses Google Cloud Storage as Artifact Repository. By default, everyone has read access to it and artifacts can be fetched/downloaded using browser, a curl-like command line HTTP client, or gsutil. + +Write access to Artifact Repository is given per request basis and all the requests must go through `LF Helpdesk <opnfv-helpdesk@rt.linuxfoundation.org>`_ with an explanation regarding the purpose of write access. Once you are given write access, you can read corresponding section to store artifacts on OPNFV Artifact Repository. + +How to Use OPNFV Artifact Repository +==================================== + +There are 3 basic scenarios to use OPNFV Artifact repository. + +* browsing artifacts +* downloading artifacts +* uploading artifacts + +Please see corresponding sections regarding how to do these. + +How to Browse Artifacts Stored on OPNFV Artifact Repository +----------------------------------------------------------- + +You can browse stored artifacts using + +* **Web Browser**: By navigating to the address `OPNFV Artifact Storage <http://artifacts.opnfv.org/>`_. + +* **Command Line HTTP-client** + ``curl -o <output_filename> http://artifacts.opnfv.org`` + + Example: + + ``curl -o opnfv-artifact-repo.html http://artifacts.opnfv.org`` + +* **Google Storage Util (gsutil)** + ``gsutil ls gs://artifacts.opnfv.org/<path_to_bucket>`` + + Example: + + ``gsutil ls gs://artifacts.opnfv.org/octopus`` + +How to Download Artifacts from OPNFV Artifact Repository +-------------------------------------------------------- + +You can download stored artifacts using + +* **Web Browser**: By navigating to the address `OPNFV Artifact Storage <http://artifacts.opnfv.org/>`_ and clicking the link of the artifact you want to download. + +* **Command Line HTTP-client** + ``curl -o <output_filename> http://artifacts.opnfv.org/<path/to/artifact>`` + + Example: + + ``curl -o main.pdf http://artifacts.opnfv.org/octopus/docs/release/main.pdf`` + +* **Google Storage Util (gsutil)** + ``gsutil cp gs://artifacts.opnfv.org/<path/to/artifact> <output_filename>`` + + Example: + + ``gsutil cp gs://artifacts.opnfv.org/octopus/docs/release/main.pdf main.pdf`` + +How to Upload Artifacts to OPNFV Artifact Repository +---------------------------------------------------- + +As explained in previous sections, you need to get write access for OPNFV Artifact Repository in order to upload artifacts. + +Apart from write access, you also need to have Google account and have the Google Cloud Storage utility, **gsutil**, installed on your computer. + +Install and Configure gsutil +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Please follow steps listed below. + +1. Install gsutil + + Please follow steps listed on `this link <https://cloud.google.com/storage/docs/gsutil_install>`_ to install gsutil to your computer. + +2. Configure gsutil + + Issue below command and follow the instructions. You will be asked for the project-id. The project-id is **linux-foundation-collab**. + + ``gsutil config`` + +3. Request write access for OPNFV Artifact Repository + + Send an email to `LF Helpdesk <opnfv-helpdesk@rt.linuxfoundation.org>`_ and list the reasons for the request. Do not forget to include gmail mail address. + +Upload Artifacts +~~~~~~~~~~~~~~~~ + +Once you installed and configured gsutil and got write access from LF Helpdesk, you should be able to upload artifacts to OPNFV Artifact Repository. + +The command to upload artifacts is + + ``gsutil cp <file_to_upload> gs://artifacts.opnfv.org/<path/to/bucket>`` + + Example: + + ``gsutil cp README gs://artifacts.opnfv.org/octopus`` + +Once the upload operation is completed, you can do the listing and check to see if the artifact is where it is expected to be. + + ``gsutil ls gs://artifacts.opnfv.org/<path/to/bucket>`` + + Example: + + ``gsutil ls gs://artifacts.opnfv.org/octopus`` + +Getting Help +============ + +Send an email to `LF Helpdesk <opnfv-helpdesk@rt.linuxfoundation.org>`_ or join the channel **#opnfv-octopus** on IRC. + +References +---------- +1. `Why you should be using an Artifact Repository <http://blogs.collab.net/subversion/why-you-should-be-using-an-artifact-repository-part-1>`_ +2. `Regarding VM image and Git repo <http://lists.opnfv.org/pipermail/opnfv-tech-discuss/2015-January/000591.html>`_ +3. `Google Cloud Storage <https://cloud.google.com/storage/>`_ + +**Documentation tracking** + +Revision: _sha1_ + +Build date: _date_ diff --git a/docs/opnfv-ci-infrastructure.rst b/docs/opnfv-ci-infrastructure.rst new file mode 100644 index 0000000..6038fa1 --- /dev/null +++ b/docs/opnfv-ci-infrastructure.rst @@ -0,0 +1,66 @@ +=========================================== +OPNFV Continuous Integration Infrastructure +=========================================== + +This document covers the Continuous Integration(CI) infrastructure used in the day to day operation of the OPNFV, which may be of interest to people who want to help develop this infrastructure or integrate their tools into it. + +Infra-overview +=============== + +The OPNFV CI infrastructure includes hardware/tools to check, build, deploy and test, etc, in pipeline. Below diagram shows the hardware/tools resources used in Octopus project, + +.. image:: images/ci_infra.png + :scale: 150% + :alt: OPNFV CI Infrastructure + +The table below lists the tools/resources that are used in OPNFV CI, + ++-----------------------------+-----------------------+ +| Tools/Resources | Name | ++=============================+=======================+ +| CI Engine | Jenkins | ++-----------------------------+-----------------------+ +| Source Code Management(SCM) | Git | ++-----------------------------+-----------------------+ +| Code Review | Gerrit | ++-----------------------------+-----------------------+ +| Artifact Repository | Google Storage | ++-----------------------------+-----------------------+ +| Hardware Resources | servers supplied by LF| ++-----------------------------+-----------------------+ + +The hardware resources are located in Linux Foundation(LF) lab and community labs, for more details, please read the Pharos project, which is shown in https://wiki.opnfv.org/pharos, a guide is provided to describe how to connect your hosting to LF Jenkins, shown in https://wiki.opnfv.org/octopus/jenkins_slave_connection. + +Major Infrastructures +====================== + +Git/Gerrit +----------- + +Git is the famous open source distributed version control software, it was initially designed and developed for Linux Kernel development, and has become the most widely adopted version control system for software development. Git puts emphasis on speed, data integrity, and support for distributed, non-linear workflows. + +Jenkins git plugin is used as the Source Code Manager(SCM), in some way, the Git features make it stand out apart from nearly every other SCM, for more details, please refer to http://git-scm.com/about/. For developers, you can refer to http://git-scm.com/docs for its usage. + +Gerrit is used here for facilitating online code reviews for our git version control system. As a reviewer, you can see the changes are shown in a side-by-side style and add some inline comments. + +Jenkins and JJB +---------------- + +Jenkins is a Continuous Integration system that runs tests and automates some parts of project operations. Jenkins mainly focuses on building/testing software projects continuously and monitoring executions of externally-run jobs. Upstream documentation is available at https://jenkins-ci.org/. + +Jenkins supports plugins, which allows to be extended to meet specific requirements. Numbers of plugins have been installed, and new ones can be installed when requirements arise. + +The Jenkins jobs are defined by Jenkins Job Builder(JJB) in human readable YAML format. The jobs defined are the key points of CI pipeline. To make clear how the jobs in pipeline run to complete the build, deploy, test works, you can refer to https://wiki.opnfv.org/octopus/pipelines. Moreover, to start your own job in Jenkins, you can write a JJB under the guide of https://wiki.opnfv.org/octopus/jenkins_wow. + +Artifact Repository +------------------- + +An artifact repository is a collection of binary software artifacts and metadata stored in a defined directory structure, it is a kin to what subversion is to source code, i.e., it is a way of versioning artifacts produced by build systems, CI, and so on. At this moment, since there is not enough storage space of LF environment, Google Cloud Storage is used as the OPNFV artifact repository temporarily. + +If you want to further find out what the artifact repository is and how to use OPNFV artifact repository, the wiki link https://wiki.opnfv.org/octopus/artifact_repository provides a good reference. + +**Documentation tracking** + +Revision: _sha1_ + +Build date: _date_ diff --git a/docs/opnfv-ci-pipelines.rst b/docs/opnfv-ci-pipelines.rst new file mode 100644 index 0000000..599d96d --- /dev/null +++ b/docs/opnfv-ci-pipelines.rst @@ -0,0 +1,89 @@ +================= +OPNFV CI PIPELINE +================= + +OPNFV CI +======== + +OPNFV Continuous Integration (CI) project provides the starting point for all OPNFV development activities. It creates a full development environment for OPNFV including automatic build, deployment, and testing. + +In order to provide fast and continuous feedback to OPNFV community, CI pipeline utilizes different tools, runs different type of verification activities in different phases depending on the needs of different OPNFV projects and the needs of the OPNFV community. + +This document aims to provide information regarding OPNFV CI Pipeline which is currently being enabled for the projects. + +CI Pipeline Overview +==================== + +OPNFV CI Pipeline starts with a change (commit) and stages in the pipeline are triggered based on events that happen while the change travels through the pipeline, producing feedback based on different verification activities. + +Below diagram shows overview of the OPNFV CI pipeline. + +.. image:: images/pipeline_overview.png + :scale: 150% + :alt: OPNFV CI Pipeline Overview + +Please note that the daily job is neglected on above diagram as the daily job is currently triggered once during night time based on timer, not based on Gerrit events. + + +Jenkins Jobs in CI Pipeline +=========================== + +Verify Jenkins Job +------------------ + +OPNFV CI Pipeline has **verify** jobs for all OPNFV Projects in order to run quick verification activities for each and every patchset sent to Gerrit for review. + +The main purpose of this job is to keep the quality of codebase on certain level so whoever clones the repo at any given time can get *stable* version of the software. It also provides feedback regarding the quality of the patchset to developer who submitted the patchset for review, reviewer(s) who are requested to do review(s) and the rest of the OPNFV community, as early as possible. + +This job is triggered automatically when developers issue **git review** command to publish their changes to Gerrit. Gerrit then publishes **patchset created** event under normal circumstances, triggering the job. If the job fails to trigger or fails during execution for some reason that is not related to patchset itself, developers can retrigger it by adding a new comment to change on Gerrit and include either one of the keywords **recheck** or **reverify**. + +The result of this job will be verified/failed vote (+1 or -1) on Gerrit. Depending on reviews, the commit can later be submitted to master and merged. + +This job does not produce any artifact (document, ISO, etc.). + +Please check the diagram below to see how the flow looks. + +.. image:: images/verify_job.png + :scale: 150% + :alt: Verify Jenkins Job + +Merge Jenkins Job +----------------- + +OPNFV CI Pipeline has **merge** jobs for all OPNFV Projects in order to run verification activities for each and every change that gets merged to master. + +The main purpose of this job is to give feedback regarding the quality of the master branch once a certain change gets merged to master and the current scope of the job is same as verify job. + +This job is triggered automatically by Gerrit **change merged** event under normal circumstances. If the job fails to trigger or fails during execution for some reason that is not related to patchset itself, developers can retrigger it by adding a new comment to change on Gerrit and include the keyword **remerge**. + +This job currently produces documents and publishes them on `OPNFV Artifact Repository <http://artifacts.opnfv.org/>`_. + +Please check the diagram below to see how the flow looks. + + +.. image:: images/merge_job.png + :scale: 150% + :alt: Merge Jenkins Job + +Daily Jenkins Job +----------------- + +OPNFV CI Pipeline has **daily** jobs for all OPNFV Projects in order to run more extensive verification activities that can take long time to finish. + +The main purpose of this job is to run full build, produce artifacts (installer ISOs, etc.), store artifacts in OPNFV Artifact Repository so they can be downloaded to target environment, do the deployment using build artifacts, run tests (Tempest, Robot, etc.) and give feedback regarding the quality of the master branch. + +This job is triggered automatically every night 00:00UTC based on **timer** under normal circumstances. If the job fails to trigger or fails during execution for some reason that is not related to software itself, it can only be retriggered by LF aor Octopus team members. + +This job produces build artifacts and documents and publishes them on `OPNFV Artifact Repository <http://artifacts.opnfv.org/>`_. + +Please check the diagram below to see how the flow looks. + +.. image:: images/daily_job.png + :scale: 150% + :alt: Daily Jenkins Job + +**Documentation tracking** + +Revision: _sha1_ + +Build date: _date_ diff --git a/docs/opnfv-jenkins-slave-connection.rst b/docs/opnfv-jenkins-slave-connection.rst index be105de..2adea18 100644 --- a/docs/opnfv-jenkins-slave-connection.rst +++ b/docs/opnfv-jenkins-slave-connection.rst @@ -1,145 +1,113 @@ -Jenkins
---------
-
-Jenkins is an extensible open source continuous integration server. See the details from `here <http://jenkins-ci.org/>`_.
-
-Linux Foundation provides Jenkins for OPNFV. You can access it from `this link <https://build.opnfv.org/ci/>`_.
-
-How to Connect Servers from Labs to OPNFV Jenkins
---------------------------------------------------
-
-The method that is normally used for connecting servers(slaves) to Jenkins requires SSH access to servers from outside. This may not be something to fix quickly for all the labs so we use the alternative way to connect slaves to Jenkins.
-
-This alternative uses JNLP. Via JNLP, slaves open connection towards Jenkins instead of Jenkins accessing to them.
-
-The server you want to connect to OPNFV Jenkins must have access to internet obviously.
-
-Please follow below steps to enable this.
-
-* Please create a ticket by sending mail to OPNFV LF Helpdesk first, opnfv-helpdesk@rt.linuxfoundation.org.
-* Ensure DNS is setup for your public IP addresses: DNS A records and PTR records need to be matching
-* Create a local user on server you want to connect to OPNFV Jenkins. (user could be named jenkins for example)
-* Download jenkins war using this link: `http://mirrors.jenkins-ci.org/war/1.599/jenkins.war <http://mirrors.jenkins-ci.org/war/1.599/jenkins.war>`_
-* Extract contents, find the file named slave.jar and copy it to somewhere which jenkins user created in first step can access.
-* Create a directory /home/jenkins/opnfv_slave_root
-* Ping/contact LF(Aric) via chat as getting the server connected requires Aric's help.
-* Ask Aric to create a slave on OPNFV Jenkins. The slave should use JNLP as method. (You can mention Ericsson slave so he can copy that slave and do small modifications on it.)
-* Provide needed information to Aric such as the IP of the server, name for your slave, slave root ( /home/jenkins/opnfv_slave_root for example) and so on.
-* Aric should provide you the key/token you need to use.
-* Have a quick try to see if you can establish connection towards OPNFV Jenkins by using below command.
-* java -jar slave.jar -jnlpUrl `https://build.opnfv.org/ci/computer/ <https://build.opnfv.org/ci/computer/>`_ <slave_name>/slave-agent.jnlp -secret <token_you_get_from_Aric>
-* Navigate to OPNFV Jenkins and look for your slave. It should have some executors in “Idle” state if the connection is successful. (`https://build.opnfv.org/ci/ <https://build.opnfv.org/ci/>`_)
-* Aric will probably create a “HelloWorld” job and execute it on the slave to verify basic functionality.
-* Once you reach this step, you have the server connection to OPNFV Jenkins completed. You can script the command you used above so the connection between slave and Jenkins can be kept open.
-
-Jenkins Slaves
----------------
-
-The slaves connected to OPNFV Jenkins using the method explained above are listed below.
-
-`https://build.opnfv.org/ci/computer/ <https://build.opnfv.org/ci/computer/>`_
-
-The ones that don't have red cross next to computer icon are fully functional.
-
-Basic Jenkins Setup
---------------------
-
-Certain things need to be set up in order to use Jenkins in CI. Below list contains some of the obvious ones.
-
-------------------------------------------------------
-Creating Service/Functional Account and Setting it up
-------------------------------------------------------
-
-This is used for connecting slaves to Jenkins. Whatever account is used for connecting slave(s) then used for executing jobs/build.
-
-This account can also be used for running verification jobs on Git commits and giving +1/-1 if certain commit fails passing certain checks such as commit message format, PEP8, and so on. (Account name could be octopus.)
-
-----------------------------------------
-Setting up Credentials/Creating Domains
-----------------------------------------
-
-This defines who can do what and which credential is valid for which domain and so on.
-
---------------
-Adding Slaves
---------------
-
-These are the servers where the jobs/builds are executed.
-
---------------
-Creating Jobs
---------------
-
-These are the jobs that build software, test it and so on. See notes on Creating/Configuring/Verifying Jenkins Jobs
-
--------------------
-Installing Plugins
--------------------
-
-With plugins new functionality can be added to Jenkins in order to handle SCM stuff, create pipelines, throttle builds, logging/graphing, and so on.
-
-Plugins
---------
-
-Jenkins has hundreds of different plugins that can be installed in order to extend its functionality. One thing to keep in mind is installing huge number of plugins increases the load on the Jenkins master.
-
-Full list of plugins can be seen from `https://wiki.jenkins-ci.org/display/JENKINS/Plugins <https://wiki.jenkins-ci.org/display/JENKINS/Plugins>`_.
-
----------------
-Slave Handling
----------------
-
-* Connecting slaves to Jenkins with SSH: `SSH Slaves Plugin <https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin>`_
-
----------------
-SCM/Git/Gerrit
----------------
-
-* Use Git as SCM: `Git Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin>`_
-* Integrate Gerrit with Jenkins: `Gerrit Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Plugin>`_
-* Integrate Gerrit with Jenkins to trigger jobs: `Gerrit Trigger Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger>`_
-
-------------
-Jobs/Builds
-------------
-
-* Create jobs that can trigger other jobs and take action depending on result: `Parameterized Trigger Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin>`_
-* Create complex/hierarchical jobs: `Multijob Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin>`_
-* Create jobs programmatically: `Job DSL Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin>`_
-* Create workflows: `Workflow Plugin <https://github.com/jenkinsci/workflow-plugin>`_
-* Throttle concurrent builds: `Throttle Concurrent Builds Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin>`_
-
-------------------
-Artifact Handling
-------------------
-
-* Integrate Jenkins with Artifactory: `Artifactory Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Artifactory+Plugin>`_
-
----------------------------------
-Notification/Email/Visualization
----------------------------------
-
-* Send customized emails: `Email-ext Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin>`_
-* Make Jenkins talk: `Instant Messaging Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Instant+Messaging+Plugin>`_, `Jabber Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Jabber+Plugin>`_
-* Show quick build status: `eXtreme Feedback Plugin <https://wiki.jenkins-ci.org/display/JENKINS/eXtreme+Feedback+Panel+Plugin>`_
-* Create and visualize pipelines: `Build Pipeline Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin>`_
-* Use Green Balls for successful builds: `Green Balls <https://wiki.jenkins-ci.org/display/JENKINS/Green+Balls>`_
-
----------------------
-Logging and Graphing
----------------------
-
-An easy option to get started with graphing performance and event logs from test runs is to just use Jenkins with some plugins.
-
-The advantage to having Jenkins do this is the data is readily available for each job. As the number of jobs increased and mining, trends, and more sophisticated numerical data analysis and logging investigation is required the data can also be pushed to external databases such as `splunk <https://wiki.opnfv.org/wiki/splunk>`_ and `logstash <https://wiki.opnfv.org/wiki/logstash>`_.
-
-This plugin provides generic plotting (or graphing) capabilities in Jenkins.
-
-This plugin will plot one or more single values variations across builds in one or more plots. Plots for a particular job (or project) are configured in the job configuration screen, where each field has additional help information. Each plot can have one or more lines (called data series). After each build completes the plots' data series latest values are pulled from Java properties file(s), CSV file(s), or XML file(s) via an XPath (which you should have generated during the build) somewhere below you workspace. Data for each plot is stored in a CSV file within the job's root project directory.
-
-It can generate various kind of plots, including Area, Bar, Line, Stacked Bar, Waterfall…
-
-Here is an example of the plots generated by this plugin:
-.. image:: images/exampleplot.png
-
-See more details here: `https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin <https://wiki.jenkins-ci.org/display/JENKINS/Plot+Plugin>`_
+:Authors: Fatih Degirmenci (Ericsson AB) +:Version: 0.1.0 + +================================ +Connecting Labs to OPNFV Jenkins +================================ + +Abstract +======== + +This document describes how to connect resources (servers) located in Linux Foundation (LF) lab and labs provided by the OPNFV Community to OPNFV Jenkins. + +License +======= +Connecting Labs to OPNFV Jenkins (c) by Fatih Degirmenci (Ericsson AB) + +Connecting Labs to OPNFV Jenkins document is licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>. + +**Contents** + +1 Version history + +2 Jenkins + +3 Jenkins Slaves + +4 Connecting Slaves to OPNFV Jenkins + +5 References + + +1 Version History +=================== + ++--------------------+--------------------+--------------------+--------------------+ +| **Date** | **Ver.** | **Author** | **Comment** | +| | | | | ++--------------------+--------------------+--------------------+--------------------+ +| 2015-05-05 | 0.1.0 | Fatih Degirmenci | First draft | +| | | | | ++--------------------+--------------------+--------------------+--------------------+ + +2 Jenkins +=========== + +Jenkins is an extensible open source Continuous Integration (CI) server. [1] + +Linux Foundation (LF) hosts and operates `OPNFV Jenkins <https://build.opnfv.org/ci/>`_. + +3 Jenkins Slaves +================== + +**Slaves** are computers that are set up to build projects for a **Jenkins Master**. [2] + +Jenkins runs a separate program called "**slave agent**" on slaves. When slaves are registered to a master, the master starts distributing loads to slaves. [2] + +Term **Node** is used to refer to all machines that are part of Jenkins grid, slaves and master. [2] + +Two types of slaves are currently connected to OPNFV Jenkins and handling different tasks depending on the purpose of connecting the slave. + +* Slaves hosted in `LF Lab <https://wiki.opnfv.org/get_started/lflab_hosting#hardware_setup>`_ +* Slaves hosted in `Community Test Labs <https://wiki.opnfv.org/pharos#community_test_labs>`_ + +The slaves connected to OPNFV Jenkins can be seen using this link: https://build.opnfv.org/ci/computer/ +Slaves without red cross next to computer icon are fully functional. + +4 Connecting Slaves to OPNFV Jenkins +====================================== + +The method that is normally used for connecting slaves to Jenkins requires direct SSH access to servers. [3] This is the method that is used for connecting slaves hosted in LF Lab. + +Connecting slaves using direct SSH access can become a challenge given that OPNFV Project has number of different labs provided by community as mentioned in previous section. All these labs have different security requirements which can increase the effort and the time needed for connecting slaves to Jenkins. In order to reduce the effort and the time needed for connecting slaves and streamline the process, it has been decided to connect slaves using `Java Network Launch Protocol (JNLP) <https://docs.oracle.com/javase/tutorial/deployment/deploymentInDepth/jnlp.html>`_. + +4.1 Connecting Slaves from LF Lab to OPNFV Jenkins +-------------------------------------------------- + +Slaves hosted in LF Lab are handled by LF. All the requests and questions regarding these slaves should be submitted to `OPNFV LF Helpdesk <opnfv-helpdesk@rt.linuxfoundation.org>`_. + +4.2 Connecting Slaves from Community Labs to OPNFV Jenkins +---------------------------------------------------------- + +As noted in corresponding section, slaves from Community Labs are connected using JNLP. Via JNLP, slaves open connection towards Jenkins Master instead of Jenkins Master accessing to them directly. + +Servers connecting to OPNFV Jenkins using this method must have access to internet. + +Please follow below steps to connect a slave to OPNFV Jenkins. + +1. Create a ticket by sending mail to OPNFV LF Helpdesk first, opnfv-helpdesk@rt.linuxfoundation.org. +2. Ensure DNS is setup for your public IP addresses: DNS A records and PTR records need to be matching. +3. Create a local user on server you want to connect to OPNFV Jenkins. (named **jenkins** for example) +4. Download slave.jar using https://build.opnfv.org/ci/jnlpJars/slave.jar and place it to somewhere so jenkins user created in previous step can access. +5. Create a directory /home/jenkins/opnfv_slave_root. +6. Contact LF via chat as getting the server connected requires help from LF. (aricg on freenode) +7. Provide needed information to LF such as the IP of the server, name of the slave, and slave root. (/home/jenkins/opnfv_slave_root for example) +8. LF will provide you the key/token you need to use. +9. Try to see if you can establish connection towards OPNFV Jenkins by using below command. + +``java -jar slave.jar -jnlpUrl https://build.opnfv.org/ci/computer/<slave_name>/slave-agent.jnlp -secret <token>`` + +10. Navigate to OPNFV Jenkins and look for your slave. It should have some executors in “Idle” state if the connection is successful. +11. Once you reach this step, you have the server connection to OPNFV Jenkins completed. You can script the command you used above so the connection between slave and Jenkins can be kept open. + +5 References +============== +* `What is Jenkins <https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins>`_ +* `Jenkins Terminology <https://wiki.jenkins-ci.org/display/JENKINS/Terminology>`_ +* `Jenkins SSH Slaves Plugin <https://wiki.jenkins-ci.org/display/JENKINS/SSH+Slaves+plugin>`_ + +**Documentation tracking** + +Revision: _sha1_ + +Build date: _date_ diff --git a/docs/opnfv-jjb-usage.rst b/docs/opnfv-jjb-usage.rst index b7e0a49..71511ab 100644 --- a/docs/opnfv-jjb-usage.rst +++ b/docs/opnfv-jjb-usage.rst @@ -1,164 +1,164 @@ -Creating/Configuring/Verifying Jenkins Jobs +How to write and use JJB? ============================================ -Clone the repo:: - - git clone ssh://YOU@gerrit.opnfv.org:29418/releng +What is Jenkins Job Builder +---------------------------- -make changes:: +Jenkins Job Builder(JJB) takes simple descriptions of Jenkins jobs in YAML format, and uses them to configure Jenkins jobs. JJB keeps your job descriptions in human readable format and job template system simplifies the configuration of Jenkins jobs. Upstream documentation is available at http://ci.openstack.org/jenkins-job-builder/. - git commit -sv - git review - remote: Resolving deltas: 100% (3/3) - remote: Processing changes: new: 1, refs: 1, done - remote: - remote: New Changes: - remote: https://gerrit.opnfv.org/gerrit/51 - remote: - To ssh://agardner@gerrit.opnfv.org:29418/releng.git - * [new branch] HEAD -> refs/publish/master +How to Write/Use Jenkins Job Builder +------------------------------------ -Follow the link to gerrit https://gerrit.opnfv.org/gerrit/51 in a few moments the verify job will have completed and you will see Verified +1 jenkins-ci in the gerrit ui. +Job template is widely used in JJBs and makes the configuration of Jenkins jobs simple, if you need to define several jobs which are nearly identical, except perhaps in their names, SCP targets, etc., then you may use a job template to specify the particulars of the job, and then use a project to realize the job with appropriate variable substitution. -If the changes pass the verify job https://build.opnfv.org/ci/view/builder/job/builder-verify-jjb/ The patch can be submitited by a committer. +To illustrate how to configure Jenkins jobs by using job template, we can start with a simple example used in our OPNFV releng project octopus directory, which is just used to print "Hello world from Octopus", shown as below:: -Job Types + -job-template: + name: octopus-test -* Verify Job + node: master - * Trigger: **recheck** or **reverify** + project-type: freestyle -* Merge Job - * Trigger: **remerge** + logrotate: + daysToKeep: 30 + numToKeep: 10 + artifactDaysToKeep: -1 + artifactNumToKeep: -1 -The verify and merge jobs are retriggerable in Gerrit by simply leaving a comment with one of the keywords listed above. This is useful in case you need to re-run one of those jobs in case if build issues or something changed with the environment. + builders: + - shell: | + echo "Hello world from octopus" -You can add below persons as reviewers to your patch in order to get it reviewed and submitted. +the value "-1" here means keep forever, you can add this job template into the project jobs to run in jenkins:: -* Ulrich Kleber (Ulrich.Kleber@huawei.com) -* Fatih Degirmenci (fatih.degirmenci@ericsson.com) -* Xinyu Zhao(Jerry) (zhaoxinyu@huawei.com) + - project: + name: octopus + jobs: + - 'octopus-test' -Or just email a request for submission to opnfv-helpdesk@rt.linuxfoundation.org +then this job works! -The Current merge and verify jobs for jenkins job builder as pulled from the repo:: +Further, if you are a developer who wants to set up a job template to be used in jenkins, we should dive into much more, taking one job template in releng project used for octopus project, the 'octopus-daily-{stream}', as an example:: - - project: - name: builder-jobs - jobs: - - 'builder-verify-jjb' - - 'builder-merge' + -job-template: + name: 'octopus-daily-{stream}' - project: 'releng' + node: master - - job-template: - name: builder-verify-jjb + # Job template for daily builders + # + # Required Variables: + # stream: branch with - in place of / (eg. stable) + # branch: branch (eg. stable) project-type: freestyle + varsetabove: '{somevar}' logrotate: - daysToKeep: 30 - numToKeep: 10 - artifactDaysToKeep: -1 - artifactNumToKeep: -1 + daysToKeep: '{build-days-to-keep}' + numToKeep: '{build-num-to-keep}' + artifactDaysToKeep: '{build-artifact-days-to-keep}' + artifactNumToKeep: '{build-artifact-num-to-keep}' parameters: - project-parameter: project: '{project}' - - gerrit-parameter: - branch: 'master' + scm: - - gerrit-trigger-scm: + - git-scm: credentials-id: '{ssh-credentials}' - refspec: '$GERRIT_REFSPEC' - choosing-strategy: 'gerrit' + refspec: '' + branch: '{branch}' wrappers: - ssh-agent-credentials: user: '{ssh-credentials}' triggers: - - gerrit: - trigger-on: - - patchset-created-event: - exclude-drafts: 'false' - exclude-trivial-rebase: 'false' - exclude-no-code-change: 'false' - - draft-published-event - - comment-added-contains-event: - comment-contains-value: 'recheck' - - comment-added-contains-event: - comment-contains-value: 'reverify' - projects: - - project-compare-type: 'ANT' - project-pattern: 'releng' - branches: - - branch-compare-type: 'ANT' - branch-pattern: '**/master' - file-paths: - - compare-type: ANT - pattern: jjb/** - - compare-type: ANT - pattern: jjb-templates/** + - timed: 'H H * * *' + prebuilders: + - test-macro builders: - - shell: | - source /opt/virtualenv/jenkins-job-builder/bin/activate - jenkins-jobs test /opt/jenkins-ci/builder/ + - shell: + !include-raw build-upload-docu.sh - - job-template: - name: 'builder-merge' + postbulders: + - test-macro - # builder-merge job to run JJB update - # - # This job's purpose is to update all the JJB +the {stream} here means when you add this into jobs, you can replace {stream} with what you want, such as:: - project-type: freestyle + stream: + - master: + branch: 'master' - logrotate: - daysToKeep: 30 - numToKeep: 40 - artifactDaysToKeep: -1 - artifactNumToKeep: 5 +the:: - parameters: - - project-parameter: - project: '{project}' - - gerrit-parameter: - branch: 'master' + node: master - scm: - - gerrit-trigger-scm: - credentials-id: '{ssh-credentials}' - refspec: '' - choosing-strategy: 'default' +means to restrict this job to run in Jenkins master node. Next, several important procedures are illustrated here, - wrappers: - - ssh-agent-credentials: - user: '{ssh-credentials}' +- scm, this mudule allows you to specify the source code location for the project, and it allows referencing multiple repositories in a Jenkins job. +- triggers, this module defines what causes a Jenkins job to start building. +- prebuilders and postbuilders, which define job need done pre and post the builders. +- builders, which defines actions that the Jenkins job should execute, usually the shell scripts or maven targets are existed there, e.g., build-upload-docu.sh used in our example. - triggers: - - gerrit: - trigger-on: - - change-merged-event - - comment-added-contains-event: - comment-contains-value: 'remerge' - projects: - - project-compare-type: 'ANT' - project-pattern: 'releng' - branches: - - branch-compare-type: 'ANT' - branch-pattern: '**/master' - file-paths: - - compare-type: ANT - pattern: jjb/** +Generally, the modules used in a job template is sequenced as - builders: - - shell: | - source /opt/virtualenv/jenkins-job-builder/bin/activate - cd /opt/jenkins-ci/releng - git pull - jenkins-jobs update --delete-old jjb/ +1. parameters, properties +2. scm +3. triggers +4. wrappers +5. prebuilders +6. builders +7. postbuilders +8. publishers, reporters, notifications + +Working with OPNFV Jenkins Jobs +------------------------------- + +By now, the releng project of OPNFV is the release engineering project for JJBs, you can clone the repo:: + + git clone ssh://YOU@gerrit.opnfv.org:29418/releng + +make changes:: + + git commit -sv + git review + remote: Resolving deltas: 100% (3/3) + remote: Processing changes: new: 1, refs: 1, done + remote: + remote: New Changes: + remote: https://gerrit.opnfv.org/gerrit/51 + remote: + To ssh://agardner@gerrit.opnfv.org:29418/releng.git + "* [new branch] HEAD -> refs/publish/master + +Follow the link to gerrit https://gerrit.opnfv.org/gerrit/51 in a few moments the verify job will have completed and you will see Verified +1 jenkins-ci in the gerrit ui. + +If the changes pass the verify job https://build.opnfv.org/ci/view/builder/job/builder-verify-jjb/ The patch can be submitted by a committer. + +The verify and merge jobs are retriggerable in Gerrit by simply leaving a comment with one of the keywords listed below. This is useful in case you need to re-run one of those jobs in case if build issues or something changed with the environment. + +* Verify Job: Trigger: **recheck** or **reverify** + +* Merge Job: Trigger: **remerge** + +You can add below persons as reviewers to your patch in order to get it reviewed and submitted. + +* Ulrich Kleber (Ulrich.Kleber@huawei.com) +* Fatih Degirmenci (fatih.degirmenci@ericsson.com) +* Xinyu Zhao(Jerry) (zhaoxinyu@huawei.com) +* Aric Gardner (agardner@linuxfoundation.org) + +The Current merge and verify jobs for jenkins job builder for releng project, shown in https://git.opnfv.org/cgit/releng/tree/jjb. + +Assuming that you have set up some job templates and put them into a project, then the question is that how they work? Taking the jobs 'builder-verify-jjb', 'builder-merge' used in releng project as examples, 'builder-verify-jjb' is to verify jobs you commited, you will see verified '+1' jenkins-ci in gerrit if it succeed, 'builder-merge' is to set up a merge job and update all the JJBs. If you have some new jobs need to be run, you can set up your own job templates and add them into the project. + +**Documentation tracking** +Revision: _sha1_ +Build date: _date_ diff --git a/docs/release/main.rst b/docs/release/main.rst deleted file mode 100644 index 7bfa94b..0000000 --- a/docs/release/main.rst +++ /dev/null @@ -1 +0,0 @@ -Just a dummy file diff --git a/requirements/main.rst b/requirements/main.rst deleted file mode 100644 index 7bfa94b..0000000 --- a/requirements/main.rst +++ /dev/null @@ -1 +0,0 @@ -Just a dummy file |
