Differences

This shows you the differences between two versions of the page.

--- maintenance:general:tools:yce_setup.pl [2020/12/08 09:44] – yspeerte
+++ maintenance:general:tools:yce_setup.pl [2024/07/03 12:31] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
+{{indexmenu_n>5}}
+====== yce_setup.pl ======
+The ''**yce_setup.pl**'' script, located in ''/opt/yce/system/'', is used for setting up the YCE application. The script is used at initial system setup, when changes to the system setup must be made, when the networking environment has changed, and when a software update is completed.
+The **yce_setup** script is tasked to create configuration files for all components the NetYCE application is composed of and activating these configurations. These components include many background processes (often referred to as daemons) like http-, tftp-, ftp- and scp-servers, but also backend api's, process monitoring, database replication and maintenance.
+In its interactive mode the user can add and delete NetYCE servers to its environment and define its roles. The resulting configuration setup file (''/opt/yce/etc/yce_setup.conf'') is then used in the non-interactive mode of **yce_setup** to re-generate the various configurations and restart their processes.
+> <color orange>Note</color>: Each NetYCE server has it unique ''yce_setup.conf'' file. When making changes to the NetYCE environment (like adding a server or enabling SSL), it needs to be done for **//all servers//** successively.
+==== net_setup dependency ===
+The networking addresses for the local server are taken from the ''net_setup.conf'' file (also located in ''/opt/yce/etc''). It is managed by running the **net_setup** script (see [[maintenance:general:tools:net_setup.pl|net_setup.pl]]) and can be automatically updated by the **yce_netmon** daemon when using dynamic ip-addresses using DHCP.
+To setup a new NetYCE server, executing **net_setup** (as 'root') prior to executing **yce_setup** is the most logical, but not essential. When completing the net-setup, the yce-setup is automatically initiated in its non-interactive mode using common defaults.
+==== Roles and settings ====
+The yce_setup will prompt the user in a dialog to set NetYCE server parameters in several categories. The settings apply primarily to the local server, but because of the communication and replication with the remote NetYCE servers, the setup must include the full environment.
+  * Add new or delete existing NetYCE servers (depending on architecture license)
+  * Edit primary IPv4 and IPv6 addresses of remote servers
+  * IPv4 and IPv6 addresses of the local server are retrieved from the net_setup.conf
+  * Assign any server the role of Front-end server (Web GUI, API's, device management)
+  * Assign any server the role of Database server (up to two per environment)
+  * Set up the server to run in a non-DNS (ip-addresses only) network
+  * Enable the Web GUI to use SSL certificates (https)
+  * Limit SSL support to most hardened protocols
+  * Assign another than the default port 8080 to the GUI backend
+  * Assign the unique database id for replication
+  * Assign the primary and secondary database server for each server to use
+  * Define the single-sign-on parameters (dns-)domain and validity duration
+  * Select the use of the public NetYCE wiki server or a local private copy
+Changes or additions to the setup must be repeated identically to the other servers in the environment.
+<color red>NOTES</color>:
+> The yce_setup script should to be started as 'yce' user.
+> In the dialog with yce_setup the (default) values and the user entries are shown using the color '<color green>green</color>' for easy legibility. This colour provides a good contrast on both white and black terminal backgrounds.
+> Default values are shown between square brackets (''['' and '']''). An <enter> suffices to accept de default value. The use of these defaults is to permit the user to enter as few as possible values and re-use existing values where available.
+> At each prompt a help message is available by entering the ''**?**'' as value.
+> Abort the script by typing ''quit'' to any prompt or hitting ''<control>C''.
+==== Invocation ====
+The **yce_setup** script is executed by the 'yce' user (not as root) from a SSH session or console (default credentials: user 'yce', password 'NetYCE01'). The interactive mode is started without any argument:
+<code>
+$ yce_setup.pl
+-- ----------------------------------------
+-- ----------------------------------------
+-- Starting 'yce_setup' interactive
+-- System release
+   identified CentOS - 7.9.2009
+   using setup for Redhat V7
+-- Connected to database at '172.17.10.24' using version '10.2.36-MariaDB-log'
+NOTE:
+  When prompted for input help on the question is available by entering '?'.
+  Incorrect responses result in a message on the expect input.
+  Just hitting <enter> will accept the existing or default value '[...]'.
+  The proces can be aborted at any prompt by entering 'quit'.
+YCE servers currently in setup:
+) devel7a.netyce.org (*)
+   | IPv4-address          | IPv6-address
+   |  172.17.10.24         |  3001::24
+) devel7b.netyce.org
+   | IPv4-address          | IPv6-address
+   |  172.17.10.25         |  3001::25
+  local server is marked with (*)
+  Select the server-number to Edit/Remove, or 'A' to add, 'C' to continue: [C]
+</code>
+The non-interactive mode is normally invoked automatically when needed. It uses the ''**-r**'' option (for "regenerate"):
+<code>
+$ yce_setup7.pl -r
+-- ----------------------------------------
+-- Starting 'yce_setup' regenerate
+-- System release
+   identified CentOS - 7.9.2009
+   using setup for Redhat V7
+-- Connected to database at '172.17.10.24' using version '10.2.36-MariaDB-log'
+Current setup:
+devel7a.netyce.org (*)
+  | IP-address  | IPv4             | IPv6
+  |  users      |  172.17.10.24    |  3001::24
+  | Database    | Primary          | Secondary
+  |  id=1       |  devel7a (*)     |  devel7b
+devel7b.netyce.org
+  | IP-address  | IPv4             | IPv6
+  |  users      |  172.17.10.25    |  3001::25
+  | Database    | Primary          | Secondary
+  |  id=2       |  devel7b         |  devel7a (*)
+  local server is marked with (*)
+-- Create configs for server 'devel7a'
+-- Yce: /opt/yce/etc/devel7a_yce.conf
+-- Retrieving file-transfer configurations...
+     can support 'sftp'
+     can support 'scp'
+     can support 'ftp'
+     can support 'tftp'
+-- Mojo: /opt/yce/htdocs/angular/app/host.js
+     mojo url set to 'https://devel7a.netyce.org:8080/'
+     wiki url set to 'http://wiki.netyce.com/'
+-- Yce_psmon: /opt/yce/etc/devel7a_psmon.conf
+-- Crontab: /opt/yce/etc/devel7a_crontab.conf
+-- Httpd: /opt/yce/etc/devel7a_httpd.conf
+-- Mysql: /opt/yce/etc/devel7a_mysql.conf
+   ::
+-- mojo: 6348 10365 17550 17551 20759 20763 20764
+     mojo hot-deploy on pid 10365
+     running 'mojo': 6348 10365 17550 17551 20759 20763 20764
+-- yce_xch: 10426
+     stop: /opt/yce/system/init/yce_xch stop
+     wait stop 'yce_xch':
+     start: /opt/yce/system/init/yce_xch start
+     wait start 'yce_xch': 23348
+-- Completed
+</code>
+The yce_setup "regenerate" option is also available as a front-end function using the "Admin - System - System status" tool. The tool also allows the regenerate command to be executed on remote servers, provided of-course that the local setup included these servers.
+{{:maintenance:general:tools:yce_setup_regenerate_option.png?600|}}
+==== Adding servers ====
+After starting **yce_setup**, the user is presented with a concise overview of the servers in the NetYCE environment. The example shows two servers, the top server is the local server.
+{{:maintenance:general:tools:yce_setup_servers.png?600|}}
+The local server reads the IP-addresses from the net_setup and can therefore not be changed. Choosing a remote server permits the server to be removed or its addresses to be updated. Servers are identified by a number as displayed in the overview (in the example, 1 and 2)
+<code>
+  Select the server-number to Edit/Remove, or 'A' to add, 'C' to continue: [C] 1
+    cannot edit or remove the local server.
+    use 'net_setup.pl' to change its ip-address
+  Select the server-number to Edit/Remove, or 'A' to add, 'C' to continue: [C] 2
+    Delete server 'devel7b' from setup?       [no]
+    Enter IPv4-address for 'devel7b'          [172.17.10.25]
+    Enter IPv6-address for 'devel7b'          [3001::25]
+</code>
+After choosing the 'A' option a new server can be added:
+<code>
+  Select the server-number to Edit/Remove, or 'A' to add, 'C' to continue: [C] a
+  Add new server
+    Hostname for new server (without domain):  devel7c
+    Domain name for new server (fqdn without hostname):  [netyce.org]
+    IPv4-address for new server:              172.17.10.26
+    IPv6-address for new server:              3001::26
+YCE servers currently in setup:
+) devel7a.netyce.org (*)
+   | IPv4-address          | IPv6-address
+   |  172.17.10.24         |  3001::24
+) devel7b.netyce.org
+   | IPv4-address          | IPv6-address
+   |  172.17.10.25         |  3001::25
+) devel7c.netyce.org
+   | IPv4-address          | IPv6-address
+   |  172.17.10.26         |  3001::26
+  local server is marked with (*)
+  Select the server-number to Edit/Remove, or 'A' to add, 'C' to continue: [C]
+</code>
+To change the name or domain of a remote server first remove it before re-adding it.
+To finish this dialog type 'C' for continue.
+==== Assigning roles ====
+A NetYCE server can have two roles, that of a Front-end server and that of a Database server, or both. As with the server setup, an overview of the current roles is displayed first.
+{{:maintenance:general:tools:yce_setup_roles.png?600|}}
+Select the number of the server to edit and a series of prompts will collect the required values. Remember that a ''?'' will offer help.
+=== Front-end role ===
+The term "Front-end" as a role is a bit of a misnomer as this role will enable all NetYCE tasks but that of a database server. These tasks include the user Web GUI, the support of the NetYCE API's and the different type of jobs when communicating with the network devices. And these 'jobs' range from executing configuration changes, fetching and restoring configurations (NCCM), and performing Compliance assessments on them.
+Every front-end needs at least one database connection which can be local or remote. A basic (evaluation) NetYCE environment has one (virtual) server which has both the front-end and the database role. More production oriented setups have two of these servers. When environments scale up, more servers can be added to separate tasks into user, Command jobs, NCCM and Compliance.
+As many as seven front-end servers and two database servers can combined into a single environment. For more information see the reference guide on the [[guides:reference:connection_matrix|connection matrix]].
+A sample session to select the options available for front-end servers:
+<code>
+  Select the server-number to change, 'C' to continue: [1]
+    'devel7a' is (also) a Front-end server?   [yes]
+    'devel7a' is DNS resolvable (y/n)?        [yes] ?
+  NetYCE servers are expected to be registered in a DNS to resolve their fqdn into an ip-address.
+  However, for non-production environments this might not be the case. By selecting non-DNS the
+  system will be setup to function using ip-addresses only.
+  When setting up a NetYCE server for 'ip-only', the user MUST access the front-end GUI by
+  entering the server IP-address in the browser, not its DNS name. This is a technical restriction
+  enforced by the browser to prevent 'cross-domain' security issues.
+    'devel7a' is DNS resolvable (y/n)?        [yes]
+    'devel7a' uses SSL (y/n)?                 [yes]
+    'devel7a' uses SSL-hardening (y/n)?       [yes] ?
+  SSL can be setup to accept older (weaker) levels of TLS (transport-layer-security)
+  as well as the newer (hardened) level of TLS1.2. When selecting 'SLL-hardening'
+  the http server will only accept connections supporting TLS1.2 and reject older
+  levels.
+    'devel7a' uses SSL-hardening (y/n)?       [yes]
+    'devel7a' portnumber of backend server?   [8080]
+    'devel7a' is (also) a Database server?    [yes]
+</code>
+The selection for 'SSL' enables https in the communication with the users. This applies to the Web GUI channel and the backend channel. To use https SSL certificates need to be generated by a trusted Certificate Authorization (CA) and installed.
+Usually a new server is installed and made operational with the SSL certificate to be installed later. Please consult the article on the [[maintenance:general:tools:mk_ssl_cert.pl|SSL certificate]] tool.
+The option for SSL hardening will result in only accepting connections supporting TLS1.2 and reject older TLS levels.
+=== Database role ===
+The second example illustrates that only two database servers can be defined. These will automatically be configured for master/master replication to allow redundancy, failover and load balancing.
+Each database server must have its unique id, either '1' or '2'. Make sure the database ids assigned are consistently configured in all servers or replication setup will fail.
+<code>
+  Select the server-number to change, 'C' to continue: [1] 2
+    'devel7b' is (also) a Front-end server?   [yes]
+    'devel7b' is DNS resolvable (y/n)?        [yes]
+    'devel7b' uses SSL (y/n)?                 [yes]
+    'devel7b' uses SSL-hardening (y/n)?       [yes]
+    'devel7b' portnumber of backend server?   [8080]
+    'devel7b' is (also) a Database server?    [yes]
+    'devel7b' uses database-id value (1/2)?   [2] ?
+	  NetYCE servers running a database must be assigned a unique database-id to setup
+	  replication. As there are only two databases permitted, one server is must use the
+	  id '1' whereas the other must use '2'.
+    'devel7b' uses database-id value (1/2)?   [2]
+YCE server roles:
+</code>
+<code>
+  Select the server-number to change, 'C' to continue: [2] 3
+    'devel7c' is (also) a Front-end server?   [yes]
+    'devel7c' is DNS resolvable (y/n)?        [yes]
+    'devel7c' uses SSL (y/n)?                 [no]
+    'devel7c' portnumber of backend server?   [8080]
+    'devel7c' cannot be a database, two servers max
+YCE server roles:
+</code>
+==== Database mapping ====
+Every NetYCE server needs at least one connection to a database. When two database servers are present, one will be assigned as the 'primary', the other as 'secondary'. With the replication in place, the primary database will be used by the server when it is available. The moment the primary connection fails, all database requests will switch to the secondary. When the primary becomes active again, the databases will re-synchronize and when completed the connections switch back to the primary database.
+It is customary to use the local database as the primary database if available. So in a standard two-server setup each server is the failover/standby database of the other. Additional servers without a database must be assigned a primary and secondary based on their proximity for best performance.
+In situations where the replication is prone to fail frequently due to unstable connections between the databases, it is advisable to select one database as the primary for //all// servers. This will result in the secondary becoming a 'standby' only ensuring that the primary is leading.
+{{:maintenance:general:tools:yce_setup_dbmapping.png?600|}}
+To assign a database server select its number and you are prompted to answer if the first of the database servers will be its primary. Answer 'yes' and this databse becomes the primary, the other the secondary. Answer no and the assignment reverses.
+<code>
+  Select the server-number to change, 'C' to continue: [1] 3
+    'devel7c' connects to 'devel7a' as primary database? [yes] no
+YCE server database mapping:
+) devel7a.netyce.org (*)
+   | Db-id     | Primary           |  Secondary
+   |  1        |  devel7a (*)      |  devel7b
+) devel7b.netyce.org
+   | Db-id     | Primary           |  Secondary
+   |  2        |  devel7b          |  devel7a (*)
+) devel7c.netyce.org
+   | Db-id     | Primary           |  Secondary
+   |           |  devel7b          |  devel7a (*)
+  local server is marked with (*)
+  Select the server-number to change, 'C' to continue: [2]
+</code>
+==== Login setup ====
+The NetYCE web GUI uses session cookies to keep you signed at all servers in an environment for a limited duration. To allow multiple servers to use the session cookies as a means for single-sign-on (SSO) the domain name used in the cookie must match.
+The session cookies are stored in memory by the browser. While the browser was not quit and the cookie is valid, no logins are required for that user. Switching user names on the same browser will replace the cookie.
+<code>
+Login setup:
+    Domain name for login (single-sign-on cookie)? [netyce.org] ?
+  NetYCE uses a session cookie to allow single-sign-on for the servers
+  that share the domain-name (or some trailing sections) specified here.
+  The default is the domain name of this server.
+    Domain name for login (single-sign-on cookie)? [netyce.org]
+    Hours until Login session expiry (single-sign-on cookie)? [12] ?
+  The single-sign-on session cookie will be valid for the duration specified
+  here. The session cookie will be lost should the browser be restarted.
+    Hours until Login session expiry (single-sign-on cookie)? [12]
+Login setup:
+  | Single-sign-on domain           | Expire (hrs)
+  |  netyce.org
+</code>
+If all servers use the exact same DNS domain the SSO domain is the same. But if different DNS domains are used, the SSO domain should be modified to use the common part of the DNS domain
+Example:
+<code>
+server fqdn               DNS domain         SSO domain
+-----------               ----------         ----------
+netyce01.ams.acme.com     ams.acme.com       acme.com
+netyce02.ldn.acme.com     ldn.acme.com       acme.com
+</code>
+Only if all servers use the same SSO domain will the cookies be accepted and validated.
+==== Wiki setup ====
+The NetYCE public wiki is the default target for the 'Help' function in the GUI. As this requires Internet access which might not be permitted, the NetYCE wiki can also be installed locally on any of the NetYCE servers.
+The procedure is described in the [[maintenance:downloads:wiki_updates|WIKI installation]] page. The corresponding configuration for the hosting server and the redirection must be entered here.
+<code>
+Wiki setup:
+    'devel7a' will use the NetYCE public Wiki server? [yes] ?
+  By default the public NetYCE Wiki server will be assigned to the 'Help' button of
+  the menu. When the internet is not accessible from the browser, the administrator
+  can install a local copy of the NetYCE Wiki on this server.
+  The installation instructions located at:
+  'https://wiki.netyce.com/doku.php/maintenance:downloads:wiki_updates'
+    'devel7a' will use the NetYCE public Wiki server? [yes] n
+    The (alias) name of the local Wiki server? [wiki] netyce-wiki
+    the domain name of the local Wiki server? [netyce.org]
+Wiki setup:
+  Local    Protocol Hostname             Domain                                   Ip-address
+  yes      http     netyce-wiki          netyce.org
+  Is this Wiki setup correct?                 [yes]
+</code>
+==== Config generation and activation ====
+After all sections are completed, a summary of the new setup is shown along with the prompt to activate it. Activation will perform many steps that assesses the server capabilities and resources, then generating the different configuration files for tis components and copying to the target locations. Finally, many of the background processes (daemons) are restarted to load the new configurations.
+{{:maintenance:general:tools:yce_setup_new.png?600|}}
+When invoking the ''yce_setup.pl'' script with the ''-r'' option to "regenerate", the config generation and restarting is the only action it performs.
+<code>
+   Create configuration for local server 'devel7a'? [yes]
+-- Creating configs for server 'devel7a'
+-- Yce: /opt/yce/etc/devel7a_yce.conf
+-- Retrieving file-transfer configurations...
+     can support 'sftp'
+     can support 'scp'
+     can support 'ftp'
+     can support 'tftp'
+-- Mojo: /opt/yce/htdocs/angular/app/host.js
+     mojo url set to 'https://devel7a.netyce.org:8080/'
+     wiki url set to 'http://wiki.netyce.com/'
+-- Yce_psmon: /opt/yce/etc/devel7a_psmon.conf
+-- Crontab: /opt/yce/etc/devel7a_crontab.conf
+-- Httpd: /opt/yce/etc/devel7a_httpd.conf
+-- Mysql: /opt/yce/etc/devel7a_mysql.conf
+     mysql version is '10.2.36'
+     mysql key_buffer set to '376M'
+     mysql tmpdir set to '/var/tmp'
+-- SSL certificate found. Run 'mk_ssl_cert.pl' to re-create when desired
+-- Updating 'devel7a' menu-tree (C)
+     Creating menus for the role(s): "frontend","database"
+     Renewed the menu tree using the default
+     Updating 'devel7a' encryption keys
+     Updating scenario syntax highlighting file
+-- Renewing NMS table permissions
+-- Checking database replication
+     replication local: 172.17.10.24, remote: 172.17.10.25
+-- Updating config-sync setup
+     located '55' config-files in '6' groups
+     updated config_sync.conf has '28' entries
+-- Relaunching NetYCE daemons...
+-- yce_psmon: 28227
+     stop: /usr/bin/sudo /usr/bin/systemctl stop yce_psmon.service
+     wait stop 'yce_psmon':
+     start: /usr/bin/sudo /usr/bin/systemctl start yce_psmon.service
+     wait start 'yce_psmon': 10766
+-- yce_netmon: 2220
+     stop: /opt/yce/system/init/yce_netmon stop
+     wait stop 'yce_netmon':
+     start: /opt/yce/system/init/yce_netmon start
+     wait start 'yce_netmon': 10846
+-- yce_cramer:
+     # disabled
+-- yce_tftpd: 28351
+     stop: /usr/bin/sudo /opt/yce/system/init/yce_tftpd stop
+     wait stop 'yce_tftpd':
+     start: /usr/bin/sudo /opt/yce/system/init/yce_tftpd start
+     wait start 'yce_tftpd': 10890
+-- yce_skulker: 28374
+     stop: /opt/yce/system/init/yce_skulker stop
+     wait stop 'yce_skulker': 28374
+     wait stop 'yce_skulker':
+     start: /opt/yce/system/init/yce_skulker start
+     wait start 'yce_skulker': 10913
+-- yce_sched: 28406
+     stop: /opt/yce/system/init/yce_sched stop
+     wait stop 'yce_sched':
+     start: /opt/yce/system/init/yce_sched start
+     wait start 'yce_sched': 10943
+-- yce_nccmd: 28430
+     stop: /opt/yce/system/init/yce_nccmd stop
+     wait stop 'yce_nccmd':
+     start: /opt/yce/system/init/yce_nccmd start
+     wait start 'yce_nccmd': 10967
+-- yce_ibd:
+     # disabled
+-- morbo:
+     # disabled
+-- mojo: 5798 8363 8364 28440 28472 28473 28475
+     mojo hot-deploy on pid 28440
+     running 'mojo': 5798 8363 8364 28440 28472 28473 28475
+-- yce_xch: 28500
+     stop: /opt/yce/system/init/yce_xch stop
+     wait stop 'yce_xch':
+     start: /opt/yce/system/init/yce_xch start
+     wait start 'yce_xch': 11039
+-- Completed
+</code>