User Tools

Site Tools


maintenance:releases:7.1.1_20190924
LDAP: couldn't connect to LDAP server

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
maintenance:releases:7.1.1_20190924 [2019/12/23 16:28] – ↷ Links adapted because of a move operation yspeertemaintenance:releases:7.1.1_20190924 [2020/01/15 13:55] (current) – ↷ Links adapted because of a move operation bdorlandt
Line 1: Line 1:
 +{{indexmenu_n>20190924}}
 +
 +===== NetYCE 7.1.1 Build_20190924 =====
 +===== Release notes =====
 +Date: 2019-09-24
 +
 +
 +\\ 
 +<WRAP widths 60% box safety>
 +==== Enhancement ====
 +</WRAP>
 +
 +=== Custom reports scheduling ===
 +<WRAP indent>
 +Custom reports can be scheduled to be executed periodically. The scheduling allowed
 +the time-of-day, date and day-of-the-week to be specified. Options for multiple 
 +times per day, dates or weekdays were not available.
 +
 +Since the report scheduler deploys the standard unix 'crontab' functions, these 
 +advanced scheduling functions could be made available for the reports too. The
 +crontab options are powerful but require some familiarity.
 +
 +The Custom reports tool now includes a text box with the resulting crontab time
 +specification after the (basic) choices are made. This text box can then be modified
 +to include the advanced options as desired. This information is then used to
 +schedule the report. Incorrect settings are rejected.
 +
 +The crontab format uses five fields:
 +<code>
 + .---------------- minute (0 - 59)
 +  .------------- hour (0 - 23)
 +  |  .---------- day of month (1 - 31)
 +  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
 +  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
 +  |  |  |  |
 +  *  *  *  * 
 +
 +Examples:
 + 00 12 * * *                   = daily at noon
 + 05 04 * * mon                 = mondays at 04:05 in the morning
 +</code>
 +
 +The advanced options allow values to be divided, enumerated and ranged:
 +<code>
 +Examples:
 + */10 * * * *                   = every 10 minutes
 + 00 12 * * mon,tue,wed,thu,fri  = noon on weekdays
 + */15 7-18 * * mon-fri          = every quarter hour during office hours
 +</code>
 +</WRAP>
 +
 +=== Ansible support ===
 +<WRAP indent>
 +Scenario's now have the ability to support a (local) **Ansible** installation. 
 +
 +The new "Ansible_exec" command for scenarios allow the launching of either an 
 +existing Playbook, or the launching of a NetYCE Scenariobook by Ansible. This
 +effective extends the use of NetYCE with the vendors supported by Ansible, while
 +at the same time extending Ansible with the scenarios, parameterisation and
 +modelling of NetYCE.
 +
 +Where the Playbook option refers to a pre-existing playbook-file that can be 
 +used as-is, the Scenariobook refers to the NetYCE template-based generated Playbook
 +that can be parameterised using the NetYCE modelling and database.
 +
 +The NetYCE GUI supporting these Scenariobooks will soon be extended with dedicated XML and
 +YAML template types.
 +</WRAP>
 +
 +=== Scenario 'signal_json' ===
 +<WRAP indent>
 +Scenarios have long had the ability to signal external systems or users using SNMP traps and
 +emails. Now the ability to signal an event using a RESTFUL JSON transaction has been added.
 +
 +The scenario command "Signal_json" is used to issue a JSON transaction to targets of choice.
 +Details for call (like the authentication to be used) is defined in a dedicated configuration
 +file. 
 +
 +See [[menu:operate:scenarios:commands#signal_json|Scenario commands - Signal_json]] for details 
 +on the use and configuration of this new command.
 +</WRAP>
 +
 +=== Context-template syntax ===
 +<WRAP indent>
 +Context templates are used to include sub-templates indirectly based on values in a relation. The
 +relation has for instance the names of the port-templates of a device. Then the context-template
 +will be substituted with the parsed port-templates for all interfaces in the relation.
 +
 +Common line for a context-template is:
 +<code>
 +-- include loopback interfaces
 +{<Port_template@Loopbacks_node>, Loopbacks_node}
 +-- include physical interfaces
 +{<Port_template@Interfaces>, Interfaces}
 +</code>
 +
 +The syntax for these context-templates has been extended to support multiple conditions. These
 +conditions can be used to filter the appropriate rows of a relation before including the remaining
 +templates. 
 +
 +These conditions use the syntax:
 +<code>
 +{tpl_reference, relation, column=value, column=value, ...}
 +
 +example:
 +{<Port_template@Port_map>, Port_map, Port_class='Lo'}
 +{<Port_template@Port_map>, Port_map, Port_class='Gi', Port_id != '*2*'}
 +</code>
 +
 +All conditions need to be 'true' to be included in the output. Conditions can use the operands '=' 
 +and '!=' and the values support wildcards. Conditions using non-existing column names are
 +ignored. There is no limit on the number of conditions.
 +</WRAP>
 +
 +=== Scenario 'send-email' ===
 +<WRAP indent>
 +At customer suggestion the 'send_email' command for scenarios has been extended with options
 +for setting the To: and From: email addresses of the issued email.
 +
 +The implementation allows for multiple To: addresses. By default are the user mail-address and
 +optionally (by tweak) the user-group mail-addresses used for the To: field.
 +from the operator 
 +</WRAP>
 +
 +=== Scenario 'script_exec' ===
 +<WRAP indent>
 +For scenario's a new command has been added that allows scenarios to launch scripts. This 
 +facilitates the use of network automation scripts that engineers build to execute specific
 +tasks. By integrating these scripts, the effort that originally went into the development
 +can be reused.
 +
 +The 'script_exec' command will execute any file one the NetYCE server for which the interpreter
 +is installed or is a compiled binary. These interpreters include python, perl, bash and sh by default, 
 +but can be extended. The python support is version 2.6 for CentOS 6.x and version 2.7 and 3.x for
 +CentOS 7.x.
 +
 +The 'script_exec' command includes an option for overriding the interpreter and a timeout to
 +forcibly kill the script after an interval. Script arguments can be passed in using multiple options.
 +</WRAP>
 +
 +=== Scenario 'grep' ===
 +<WRAP indent>
 +When creating scenarios involving list variables, the 'grep' function is a useful command to 
 +extract desired items. This function was reworked to accept filter strings in two distinct
 +formats: wildcards ('*' and '?') and regular expressions (regex).
 +
 +The wildcard format is widely used and will satisfy most demands. Wildcard filters are case-insensitive:
 +<code>
 +# filter the entries using WILDCARDS in the -c argument
 +<nie_names> := grep -l <all_names> -c "m*nie"
 +</code>
 +
 +For more advanced filters the more complex and versatile regex format is available. To distinguish
 +the condition argument must be placed between slashes ('/.../') and accepts modifiers like 'i'
 +to compare case-insensitive:
 +<code>
 +# filter the entries using REGEX
 +<nie_names> := grep -l <all_names> -c "/m.*nie$/i"
 +</code>
 +
 +For details and examples, please consult the wiki page on 
 +[[menu:operate:scenarios:commands#grep| Scenario Command details - grep]]
 +
 +</WRAP>
 +
 +=== Scenario 'match' ===
 +<WRAP indent>
 +In addition to the 'grep' function, the 'match' function was added to the scenarios.
 +
 +The behaviour of 'match' is similar to the 'grep' function but with an essential difference: where 
 +the 'grep' returns the actual element of the matching condition, the 'match' returns only 
 +the matching string. In essence, the 'match' is a grep function and a substring function 
 +rolled into one.
 +
 +When using the regex condition a distinct feature can be used: multiple substring matches. 
 +Regular expressions have the capability to 'mark' matched strings using '()' and refer to 
 +them by position ($1, $2 and so on). The 'match' function allows you to use this feature 
 +to use these marks in the condition regex and have these marked strings returned as a 
 +separate entry in the result.
 +
 +For details and examples, please consult the wiki page on 
 +[[menu:operate:scenarios:commands#match| Scenario Command details - match]]
 +</WRAP>
 +
 +=== Supernets for Sites ===
 +<WRAP indent>
 +Automating network designs including IP-address allocations in NetYCE requires the use of IP-plans.
 +IP-plans (ipv4 and ipv6) provide an abstraction for segmenting and standardising ip-spaces that are
 +to be used in the design. They function as a 'blueprint' for allocating required subnets needed
 +to 'build' the various components of the design.
 +
 +For the implementation an ip-space (or ip-block) in the form of a Supernet is assigned to a 
 +NetYCE Client along with the IP-plan that will be governing it. Many large or small IP-plans can
 +be made available for the networking design of the Client.
 +
 +Because not all designs allow for a 'Client' based approach, customers have requested an option
 +to manage IP-plans and Supernets per Site. Starting this release, Client Supernets can now be 
 +'reserved' to be exclusively used by a Site. The Site will from then on be 'owning' the ip-space
 +of that Supernet.
 +
 +The assignment to a Site can be done using the GUI by editing a new or existing Supernet in 
 +the Client details form. The Site reservation is made by entering a valid SiteCode within the 
 +Client. Alternatively the Service-type 'ASSIGN-SUPERNET-SITE_CODE' can be used to further 
 +automate this task.
 +
 +When a subnet for a requested Net_name is to be added, the Net_name is initially searched for in
 +all Supernets 'owned' by the Site. When available in multiple Supernets (regardless of ip-plan), the 
 +new subnet is allocated from the Supernet that has the least free subnets of that type (thus filling
 +up used supernets before newer). Should the subnet not be available in 'own' Supernets, the search
 +will continue in Supernets that are not 'owned' by other Sites but are 'shared' within the Client.
 +The same logic in selecting the Supernet then applies when multiple provide the Net_name.
 +
 +For existing designs set up using Client-'shared' Supernets nothing will change, the implementation 
 +of the Site-supernets is fuly downwards compatible.
 +</WRAP>
 +
 +=== Infoblox DNS IPv6 ===
 +<WRAP indent>
 +For customers deploying the Infoblox integration another step in integrating IPv6 was implemented.
 +
 +The module for Infoblox DNS management now allows fo the creation of 'host' records with IPv6
 +addresses using the NetYCE 'add_host' API call and the scenario function 'dns_add_host'
 +
 +This initial implementation is limited 'host' records and the use of a specified IPV6 address. 
 +Support for AAAA-records or searching for a 'free' IPv6 address is planned for a subsequent phase.
 +
 +See the NetYCE Wiki for details in the pages 
 +[[menu:operate:scenarios:commands#dns_add_host|Scenario commands - dns_add_host]]
 +and
 +[[guides:reference:infoblox:plugin_infoblox_dns#add_host|Infoblox API plugin - add_host]]
 +</WRAP>
 +
 +=== Menu defaults ===
 +<WRAP indent>
 +The NetYCE front-end menus 'Design', 'Build', 'Operate' and 'Admin' each have a list of forms and tools
 +for the end-user to choose from. It is now possible to configure the system to define which of these
 +items wil be automatically selected when its menu is opened. 
 +
 +The selection what form or tool(set) to open for each menu is controlled using the Lookup "Tweaks"
 +category. The Variables "Default_menu_design", "Default_menu_build" and so on can be set to specify
 +the item. The values to be used are listed in the Wiki article:
 +[[guides:reference:lookup_tweaks#default-menu_tweaks|Default-menu tweaks]].
 +
 +These Default-menu tweaks can also be used in combination with the 
 +[[guides:reference:lookup_tweaks#default-tool_tweaks|Default-tool tweaks]]
 +that specify which tool from a tool set is selected by default.
 +</WRAP>
 +
 +
 +\\ 
 +<WRAP widths 60% box safety>
 +==== Change ====
 +</WRAP>
 +
 +=== Scenario 'config_save' ===
 +<WRAP indent>
 +The scenario command 'config_save' will push the devices' startup-configuration to
 +the NetYCE server. This worked properly but uploaded the existing startup file.
 +
 +The 'config-save' procedure has been changed such that it will now first create a new
 +startup file and then upload it to the NetYCE server.
 +</WRAP>
 +
 +=== Domain permissions ===
 +<WRAP indent>
 +The required privileges to view and edit the Domain form have been modified. Users with Engineer or
 +Modeller permissions were denied access to the various 'password' type fields. These levels have
 +been loosened.
 +
 +The following privileges now apply by default:
 +  * Operators can view the non-password fields but all password fields are starred out and disabled.
 +  * Engineers can edit the non-password fields but all password fields are read-only. The password fields appear starred out until it clicked, then the value is displayed.
 +  * Modellers and Managers have full access to all password and non-password fields. Again the passwords
 +appear starred out until the field becomes active, then it can be edited.
 +
 +This default behaviour can be modified using the Auth_permissions table where for each of the fields and
 +for all user levels the edit permissions can be modified using the "Admin - Custom data" tool for
 +the "Auth_permissions" table. 
 +
 +<code>
 +By filtering on "Controller: 'domains'" and "Type: 'field'", the value
 +for the desired role can be modified in one of 3, 4, or 5: 
 +  '3' denotes 'disabled', 
 +  '4' results in 'read-only'
 +  '5' in full access.
 +</code>
 +</WRAP>
 +
 +=== Help menu items ===
 +<WRAP indent>
 +The 'Help' menu item of all four menus has been removed. The link to the NetYCE Wiki is moved
 +to the upper right hand corner of the page. 
 +
 +This change was made to reduce the number of items in the various menus.
 +</WRAP>
 +
 +=== Longer Net_names ===
 +<WRAP indent>
 +At customer request, the 'Net_name' of ipv4 and ipv6 subnets can now be 50 characters long. 
 +Previously the maximum length was 20 characters.
 +</WRAP>
 +
 +=== Scenario arguments ===
 +<WRAP indent>
 +Scenario functions use arguments like '-a' and '-p' and so on. Some of these functions allow
 +for many (repeated) arguments resulting in very long function calls. Although this offers no
 +functional issues, these long lines of can arguments become unwieldy and look untidy.
 +
 +Therefore the scenario arguments can now be split over several lines. All lines starting with 
 +something that looks like an argument flag (like '-a ' or '-p ') will be taken as an argument
 +belonging to the previous function. This allows for a better overview of the arguments and
 +makes editing a lot easier.
 +
 +Be aware that the list of arguments cannot contain comments (starting with '#').
 +</WRAP>
 +
 +=== Relation test ===
 +<WRAP indent>
 +After the completion of a NetYCE software update or the updating of a system configuration many
 +configuration files are regenerated and processes requiring them are restarted. Because these
 +software updates also install patches on the NetYCE database, the database could be changed in
 +a fashion that potentially renders customer-created Relations non-functional.
 +
 +Therefore, after all changes were mede and processes are restarted, all existing Relations are 
 +checked for errors and the results listed in the Task-log. This process of error-checking of
 +Relations has been modified in two ways:
 +
 +First, the relation will now use relevant data fields for its substitutions (where possible). 
 +Where previously all variable substitutions used '0', it now uses the values from one of the 
 +nodes in the database. This results in more relevant queries and catches errors that otherwise
 +would be overlooked.
 +
 +Secondly, since the queries now run on the database using actual values, the queries will likely
 +respond in a more representative fashion. For all relations the required execution time of the query
 +is measures and reported. For those queries where this exceeds one second, the relation is marked
 +as "Slow" and should be examined for optimisation.
 +</WRAP>
 +
 +=== Operator permissions ===
 +<WRAP indent>
 +Users with Operator-level permissions will find they have now read-only access to Templates where
 +they previously could not access this sesion of the menu. 
 +
 +The same change was made to the Mpls-vrfs forms.
 +</WRAP>
 +
 +
 +\\ 
 +<WRAP widths 60% box safety>
 +==== Fix ====
 +</WRAP>
 +
 +=== Scenario 'db_query' ===
 +<WRAP indent>
 +The scenario command 'db_query' did not return any resulting values from the
 +requested database table. It turned out that the result was returned in an
 +incorrect format.
 +The db_query command now returns a single value or list of values (array) correctly.
 +</WRAP>
 +
 +=== Telnet Vendor modules ===
 +<WRAP indent>
 +Several vendor modules exhibited some shortcomings when using Telnet instead of SSH. If a 
 +username or password failed, the job did not report the origin of the login failure but just
 +aborted.
 +
 +The issue has been resolved for most vendors, but testing could not be completed for all 
 +vendors at this time. This testing is in progress.
 +</WRAP>
 +
 +=== HP VPN warning messages ===
 +<WRAP indent>
 +The HP C5 and C7 devices will issue error messages about already existing VPN's. These messages
 +were incorrectly ignored by NetYCE jobs.
 +
 +These VPN messags are now added to these modules, and depending on the type, marked as an
 +error (stopping additional commands) or a warning (logging it, but continuing).
 +</WRAP>
 +
 +=== Export download ===
 +<WRAP indent>
 +Under some circumstances the download of a model export (Service-type, Node-type, Node-groups, 
 +Region, Domain, Relations, Scenarios) would result in an empty file. The use of vpn tunnels or
 +firewall settings could cause this.
 +
 +The issue was caused by the disallowed http data-type 'xml'. By changing it to the the 'octet-stream'
 +data-type, the problem was resolved.
 +</WRAP>
 +
 +=== Huawei command parse ===
 +<WRAP indent>
 +When using command parsing to extract information from a Huawei device, the command failed because
 +NetYCE incorrectly enabled the privileged mode where show commands are not allowed.
 +
 +By modifying the Huawei State_actions table entries for this command, the issue was resolved.
 +</WRAP>
 +
 +=== Template conditions ===
 +<WRAP indent>
 +Over the years the supported template syntax has been extended with many features. These enhancements
 +also affected the capabilities for template conditionals. As a consequence some valid conditional 
 +syntaxes of the past suddenly became ambiguous or invalid.
 +
 +To prevent these situations, the NetYCE configuration generator would automatically detect these
 +older style conditionals and convert them on the fly. The kind of corrections are mostly like these:
 +<code>
 +original
 +|variable|
 +|variable = value|
 +|!variable@relation = value1,value2|
 +
 +corrected
 +|<variable>|
 +|<variable> = 'value'|
 +|<variable@relation> != ('value1','value2')|
 +</code>
 +
 +Recently the syntax again allowed for more advanced options but were found working fine for 
 +data substitution situations (template text), but produced unpredictable results in conditionals.
 +These were found to be the result of the implicit conditional corrections outlined above. The
 +existing conversion was not suited to deal with these syntaxes and would break an otherwise
 +valid conditional.
 +
 +A complete new conditional converter was created that will still provide the required conversions
 +for the old-style formats while also recognising the sometimes very complicated syntaxes like:
 +<code>
 +|<variable@relation:column=<variable2>> != (<variable3>,'value')|
 +|[function(<variable1@relation1:'value1')] = <variable2@relation2>|
 +</code>
 +
 +> Update: some valid conditionals in existing templates were incorrectly malformed by the converter.
 +These cases were identified and fixed. After extensive retesing the resulting converter should now
 +be fully downward compatible.
 +</WRAP>
 +
 +===  Scenario variables ===
 +<WRAP indent>
 +Variables defined in the [parameter] section of a scenario are shared with the configuration
 +generator and, though it, the relations. These variables are made available to the generator using an 
 +intermediate format. 
 +
 +A malformed variable caused the generator to abort when reading the intermediate format resulting
 +in empty configurations. Extending the validation of external variables was added to resolve this issue.
 +</WRAP>
 +
 +=== Scenario 'dns_clear_host' ===
 +<WRAP indent>
 +A problem was found with the 'dns_clear_host' call for Infoblox DNS manipulations. It was found that 
 +some hosts addresses could not be found to be removed while the Infoblox front-end clearly showed
 +it existed in the IPAM views.
 +
 +The issue could be resolved when it became understood that the ip-address was not a regular IPAM
 +address, but consisted of an Infoblox constructed object "FixedAddr" that is used for DHCP 
 +address-to-attribute associations like MAC-addresses. 
 +
 +The scenario 'dns_clear_host' and API 'clear_host" function now locate and remove the "FixedAddr"
 +like a regular ip-address.
 +</WRAP>
 +
  
maintenance/releases/7.1.1_20190924.txt · Last modified: 2020/01/15 13:55 by bdorlandt