summaryrefslogtreecommitdiffstats
path: root/config/utils
AgeCommit message (Collapse)AuthorFilesLines
2018-08-21Fail to set the value in multiple linesJulien1-2/+2
refer to the log of Jenkins: https://build.opnfv.org/ci/view/pharos/job/ validate-pdf-jinja2-templates-master/326/console /home/jenkins-ci/opnfv/slave_root/workspace/ validate-pdf-jinja2-templates-master/config/utils/check-jinja2.sh: line 54: -i ./config/installers/apex: No such file or directory [GENERATE] [OK] [YAMLLINT] [OK] yamllint -s ./config/installers/apex/pod_config.yaml Change-Id: I13c220e7fba5355124ae2b41e7399140c5d92523 Signed-off-by: Julien <zhang.jun3g@zte.com.cn>
2018-06-04[fuel] IA: Add PXE/admin static IPsAlexandru Avadanii1-1/+2
- check-jinja2.sh: Add template dir to include dirs; - add param definitions for PXE/admin static IP for each machine, moving MaaS DHCP range start after the new addresses, similar to public network; - drop 'opnfv_' prefix for compute parameters; - drop 'opnfv_jump_bridge_*' variables (deprecated); - drop 'opnfv_infra_maas_*' variables, handled directly in Fuel now; - add 'opnfv_net_*_mask' for all used networks in idf.net_config; - move all internal j2 variables to net_map.j2, so we can reuse them without pod_config; - refactor maas variables into a full `maas.region.machines` section when baremetal nodes are defined; - drop 'openstack_compute_*' reclass params in favor of direct expansion in reclass.storage definitions via j2 in Fuel@OPNFV; - add `net_macros.j2` for reclass linux.network reclass generation; - move all non-sensitive reclass parameters to Fuel@OPNFV; Change-Id: I495c6072d845b273b6607d45dc5152ec4ee767cf Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-04-30[PDF] Preparations for vPDF supportAlexandru Avadanii2-5/+6
- extend check-{jinja2,schema}.sh to scan vPDFs and construct proper IDF path for vPDFs; - add support for '/' in PDF fields, by default filtered by j2 template rendering; SPEC changes: - extend remote_management 'type' enum with 'libvirt'; SCHEMA changes: - add new enum value 'libvirt' according to SPEC update; - add separate handling of 'remote_management' for baremetal/virtual; - remote_management 'versions' and 'mac_address' are mandatory for all but 'libvirt' (used to be mandatory for all); Change-Id: Ib3f711cae807c3ab27920391402ba74844f155c3 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-05generate_config: minor code formatting updatesAlexandru Avadanii1-2/+4
Change-Id: If0ee12287f4257f2de315bd6ad82edb709e24492 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-02generate_config: New `-b` arg for batch processingAlexandru Avadanii1-14/+32
Sometimes, parsing multiple j2 templates based on the same PDF+IDF is needed, in which case we'd have to call generate_config once for each template. Add a new argument, `-b`, which allows batch processing of multiple templates in one go: - files ending in '.j2' (e.g. 'template.yml.j2') will be expanded to a file without the '.j2' suffix in the same directory as the source template (e.g. 'template.yml'); - templates not ending in '.j2' are skipped in batch mode; - in order to pass multiple templates, `-j` argument can now be passed multiple times, similar to `-i`; - although relative filepaths still work, it is highly recommended to use the full path (abspath) for j2 templates; - to avoid filename collisions, j2 parser will open '/' (as well as any directories specified via `-i`) and lookup each j2 template by its full path; Add a new argument, `-v` for verbose logging. While at it, import only needed functions from 'os.path' to make our code more compact and easier to read. Change-Id: I2291935d42e4196813cd75154d5b674bcc330183 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-02Merge "config/utils: YAML: Use C bindings if available"Alexandru Avadanii2-11/+16
2018-03-01generate_config: New `-i` arg for j2 includes dirAlexandru Avadanii1-1/+2
jinja2 does not support imports from parent directories, so in order for j2 to be able to use '{% from <some>.j2 import ... %}' constructs with included j2 residing on a separate dirpath than current j2, add support for explicitly adding directories to the templater search path. This also allows calling generate_config with only file basename as '-j' arg value, as long as the dirname is passed via '-i' (can be extended later to parsing multiple j2 files at once). Change-Id: I943fda9262a6f7de5ee6fcc3e7bc9828c339651a Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-01generate_config: Enable 'do' j2 extensionAlexandru Avadanii1-1/+4
'do' is useful to avoid hacks like {% set _ = x.update(...) %}. Instead: {% do x.update(...) %}. [1] http://jinja.pocoo.org/docs/2.10/extensions/ Change-Id: I886beb97e0383ba11b81aa21cf12b11ebc4b05b2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-03-01config/utils: YAML: Use C bindings if availableAlexandru Avadanii2-11/+16
Time diff before/after this change for the same template via generate_config expansion: -user 0m0.144s +user 0m0.096s Change-Id: Id574afcd8a74a530d791ebed8b72ccae53703611 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-22gen_config_lib: Add storage_size_num filterAlexandru Avadanii1-0/+19
New filter for converting PDF disk_capacity strings into a numeric only string (to bypass scientific notation), parsable as float. Change-Id: I779bec7db03ef1b3ce1bfd6ca239a3e7b4021eb2 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-02-20generate_config: Move filters to external libraryGuillermo Herrero2-43/+212
Clean and isolate main script logic from custom filters by moving all them to a external library file. Library has a function to load all the existing filters at once. New enhanced IP handling custom filters: - ipnet_hostaddr - ipnet_hostmin - ipnet_hostmax - ipnet_broadcast - ipnet_netmask - ipnet_contains_ip - ipnet_contains_iprange - ipnet_range_size ipnet filters work with proper IP network with prefixlen mask, ensuring consistent IP calculations and error handling. Previous IP handling filters should be deprecated. Change-Id: I83c41d7ad3c6bd1d9df1deca6cc5b9d2481ecf52 Signed-off-by: Guillermo Herrero <guillermo.herrero@enea.com>
2018-01-28[IDF] Add optional 'idf.installer' arrayAlexandru Avadanii1-15/+7
There is no easy way to determine automatically if a PDF/IDF set is compatible with a specific installer adapter. Most common incompatibilities are caused by: - installer adapters hardcode more network ports than nodes have defined in PDF; - installer adapters hardcode more nodes than PDF defines; Previous attempt at handling this dynamically (cd54edf) fails for recently added PODs, so let's handle this explicitly per-pod via a new optional IDF property, 'idf.installer'. If 'idf.installer' is defined, it will be used by `check-jinja2.sh` as a filter-in list of installer adapters to check against. The default (no 'idf.installer' present) is to check all IAs. Leverage the new property for: - arm-pod5: not enough network ports on computes for Compass, Joid; - arm-pod7: same as arm-pod5; - bii-pod1: no 'storage' network defined in IDF for Daisy; - zte-pod9: 3 cluster nodes are only supported by Daisy for now; SPEC changes: - net_config (previously mandatory for IDF version '0.1') is now mandatory only if 'idf.fuel' or 'idf.daisy' is defined; Fixes: cd54edf Change-Id: I706c05519e5491ad631069d7cc090e9c8bfd3011 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-26generate_config: Check yaml arg is really a fileAlexandru Avadanii1-1/+1
In order to support bash process substition, e.g. $ ./generate_config.py -j /path/to/template.j2 -y <(echo stuff); do not consume the contents of ARGS.yaml unless it is a file. Change-Id: I89053cfaadc158f6a15f85836892b21980081772 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-24[IDF] Add IDF schema validation based on versionAlexandru Avadanii3-5/+20
- add basic structure for IDF schema validation; - fill in Fuel section; - add stub for Daisy section; - add stub for Compass section; - add stub for net_config; - enable IDF schema validation via `check-schema.sh`; - prepare for new IDF versions with different schemas, while allowing subsection schema reuse from IDF v0.1; - extend validate_schema.py for IDF; Change-Id: I59b1c9d857e9dee9e6ff9956e81b2ae6dc27a767 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-22[PDF] [SPEC] Add 'version: 1.0'Alexandru Avadanii2-1/+14
- add 'version: 1.0' to PDF spec; - add 'version: 1.0' to all existing lab PDFs; - extend schema with new property; - add workaround for value-based decision-making in schema version selection via `validate-template.py`; - add support for multiple schema versions; - add versions for all schema blocks defined so far; - fix PDF schema pattern for disk size decimals (e.g. '1.8T'); Change-Id: Ie8f768803ec19f1f9a7982fe5ca59df80764fc4a Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-22Merge "[PDF] check-jinja: Filter-out incompatible PDF/IA"Alexandru Avadanii1-0/+18
2018-01-22Merge "generate_config: Add netmask jinja filter"Alexandru Avadanii1-0/+10
2018-01-22Merge "check-jinja: Add optional filtering of test matrix"Alexandru Avadanii1-2/+11
2018-01-22Merge "[PDF] config: Add/update license headers"Aric Gardner4-4/+4
2018-01-19check-jinja: Add optional filtering of test matrixAlexandru Avadanii1-2/+11
When adding new PDFs, it is sometimes useful to check only the newly added PDF (or all the PDFs for a specific lab) against all or only some installer templates. Change-Id: I1d02943527597df1e1908a3ff4398bc398c887e7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-18generate_config: Add netmask jinja filterGuillermo Herrero1-0/+10
- Add custom filter to convert integer prefix network mask to IP format network mask string Change-Id: I13d6a2725b93132f858b64d7f93de3a4e0dc5961 Signed-off-by: Guillermo Herrero <guillermo.herrero@enea.com>
2018-01-16[PDF] check-jinja: Filter-out incompatible PDF/IAAlexandru Avadanii1-0/+18
While checking all PDFs against all installers, some mandatory requirements are sometimes not met: - minimum cluster nodes (e.g. Apex requires 5 nodes, 1 PDF has 3); - minimum interface count (e.g. Compass requires min 3 NICs, some PDFs only have two interfaces or less); The added filter is far from ideal: - node requirements are based on the latest (not highest) index used by the installer adapter to accomodate special handling of 3-node cluster in Daisy templates; - interface req is based on the average interfaces/node, with quite some margin - this is a very loose mechanism; With this in, `check-jinja` has a return code of 0 (note that yamllint failures of output files are not affecting the return code). Change-Id: I43ade3567bf3026069ff93eca17abc212fab211c Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-14[PDF] Add schema validationAlexandru Avadanii2-0/+64
- create new YAML schema for PDF validation; - add basic python script for checking a PDF against the schema; - add bash wrapper for checking all PDFs in Pharos, to be leveraged later via a new verify CI job; Change-Id: I47e02642756b7a231138dec3d5258b100b4db72b Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-14[PDF] config: Add/update license headersAlexandru Avadanii4-4/+4
This only covers the `config` subdir. Change-Id: I1fb9470e216cc2de9a4ac91de44d99e7044e708e Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-14[PDF] check-jinja: cleanup, compact outputAlexandru Avadanii1-16/+16
The Pharos git repo already has a CI verify job for `yamllint`-ing the PDF files (check-jinja input yaml), so drop that check and keep linting only output YAML files. While at it, slightly rework the output log to make it easier to read. Change-Id: I2e47902d71514709eb48432f87d6ce68fb795d73 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-14[PDF] check-jinja: Replace encrypted str w/ dummyAlexandru Avadanii1-1/+3
During PDF validation using `check-jinja.sh`, most if not all encrypted strings will fail to decrypt due to missing keys on the build server. The templater will fallback to using raw plaintext, leading to `line too long` yamllint violations. Since we don't care about the actual value of the unencrypted string, replace the encrypted one with a dummy (shorter) string. Change-Id: I0f96db0e055bf84ad43a1df4a0f2bd86cc50fd22 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-14[PDF] check-jinja: Disable check against templateAlexandru Avadanii1-1/+1
config/pdf/pod1*yaml files are not valid PDFs, but only templates. Trying to expand said files against installer adapters would lead to funky results, since most fields are expressed as '{val1|val2}' in the template files. Change-Id: Ieec4d596d0b5246aa506239183a31463951403e7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2018-01-14[PDF] generate_config: eyaml only encrypted PDFsAlexandru Avadanii2-5/+9
Silence misleading warnings about failed decription for PDFs that do not use this feature. While at it, print yamllint version used by check-jinja.sh. Change-Id: Ica1ff90abaee8c9bb20996899c8f0a7527133618 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-12-20validate-pdf-jinja2-templates-master failingagardner2-4/+4
If keys do not exist, the skipping decryption python stuff doesn't work, see: https://build.opnfv.org/ci/job/validate-pdf-jinja2-templates-master/16/consoleFull Still does not fallback to plain in the case of a 'bad decryption', but the current code does not either, bad decryption is logged in any case. Change-Id: I8da2c7cf8568a6e77ca24cc9ce665410d93b7ba9 Signed-off-by: agardner <agardner@linuxfoundation.org>
2017-12-15Update the example config filename in eyaml READMEAlex Yang1-1/+1
The example configuration file in the directory pharos/config/utils/ is config.example.yaml. Change-Id: I493274e4ddb689eb729ed642b5f495b5be0b75b0 Signed-off-by: Alex Yang <yangyang1@zte.com.cn>
2017-12-13Update eyaml READMETrevor Bramwell1-1/+2
On CentOS7 systems the gem command is contained in the 'rubygems' package, not 'ruby-gems'. After the keys have been moved, there is not reason to keep around the 'keys' directory. Change-Id: I5cb9f474276de75b650ddf323a279e139fb62d37 Signed-off-by: Trevor Bramwell <tbramwell@linuxfoundation.org>
2017-12-09generate_config.py: Fix running without eyamlAlexandru Avadanii1-3/+7
If eyaml is not installed or misconfigured, we should output raw encrypted data and throw a warn/error accordingly, instead of bailing out completely. Needed for Fuel deploys on PODs that do not use PDF encryption, e.g. lf-pod2. Change-Id: I2905df11a3cf2f2eb1ab9bfd8bd88619af2cab04 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-11-24Jenkins does not have /usr/local/bin/ in pathagardner1-0/+1
Stupid jenkins Change-Id: Ibfd83e7dadd674cfe7d7a9c2a712ef069fb8d5de Signed-off-by: agardner <agardner@linuxfoundation.org>
2017-11-23Moving check-jinja2 to pharos.agardner1-0/+66
Securedlab will be going away now that we can encrypt strings (eyaml) we need to check jinja2 PDF templates that will be moving into pharos This adds one pod yaml files and fixes paths in check-jinja2.sh After this is merged I can create a jenkins job for pharos. Then I can start moving pdf files into the pharos repo Change-Id: I2f2f2925275de49035d7d0160249d75b25ca0d20 Signed-off-by: agardner <agardner@linuxfoundation.org>
2017-11-17generate_config: Use eyaml to decrypt secret valuesAlexandru Avadanii3-4/+101
Note: IDF data encryption is not supported. Supporting that is trivial, but it leads to slightly more complicated code, plus it breaks support for multiline scalar encrypted data in the PDF ('>'), forcing us to define each encrypted value as inline string. While at it, fix silly limitation of jinja2 path residing in a subdir of CWD. Change-Id: I441ec754d8b6e4aad2ed73aba0b9b18ed65f05f4 Signed-off-by: agardner <agardner@linuxfoundation.org> Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-26PDF: generate_config: Add IDF parsing supportAlexandru Avadanii1-0/+8
For a given 'podN.yaml' file, check whether a file named 'idf-podN.yaml' exists in the same directory, and if it does, pass its contents under the 'idf' dict key to the j2 template. This assumes the contents of IDF yaml have a root 'idf' key. Change-Id: I6c6f1b9f28c38989f8a6ed4a389c9a1da423d76d Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>
2017-09-18config/utils: Add generate_config.pyAlexandru Avadanii1-0/+53
Previously, generate_config.py resided in securedlab git repo, but since we want to be able to use it in both securedlab (for validation of new PDF files during verify jobs) and installer projects (for actually parsing the PDF file into usable installer inputs), we decided to move it to a common location, also available for regular users. This change merely replicates the file from securedlab git repo. Change-Id: I9ff7889e408338d3911853fe01b752b013de1db7 Signed-off-by: Alexandru Avadanii <Alexandru.Avadanii@enea.com>