Table of Contents
Debugging
The 'Admin - System - Debugging' page allows users with Manager of System privileges access to the log files of the NetYCE server. It also provides the user to enable or disable the 'debug' mode which creates additional and extensive log files.
Log files can be viewed using a pop-up window from this tool. The 'follow' option shows the log file in a scrolling window as information is added to it.
Additionally, the standard and debug log files combined with server configuration files can be converted into a 'NetYCE support file' to provide the support team with the relevant information to a reported problem. To ensure confidentially, each support file is encrypted on creation to allow secure email of the data.
Debug mode
Every NetYCE server can be configured dynamically to run in debug mode. This mode, when enabled, will create extensive debug files used for troubleshooting and development purposes.
The debug mode can be enabled and disabled using the front-end tool. Be mindful that running in debug mode will have some impact on the systems performance and disk space usage. For this reason is the debug mode automatically disabled after 24 hours. This period can be altered by modifying the “Debug_max_days” Tweak using the Lookup tool. It defines the number of days the debug can be active.
When changing the debug setting, the system will be unresponsive for a short period (5-10 seconds) to restart some processes that need to switch to debug/development mode. Currently the front-end serving 'mojo' processes are restarted, but others might need to be restarted manually (using the 'Systems' page) if troubleshooting requires it. Those might involve the yce_skulker, yce_xch, yce_nccmd and yce_sched.
To enable the debug mode simple click the “Start debug” button. The default action is to first clear any existing debug files. This behaviour can be omitted by clearing the “Clear debug files” checkbox.
Not all systems will show the same log and debug files. Depending on installed licensed functions and used NMS integrations will some log files be present or not.
While the debug mode is enabled and the 'Debug logs' page is visited, the username and timestamps of the start and scheduled end of the debug period is displayed. Because it is desirable to collect not more data than the relevant period, the current running duration of the debug mode is also included.
To monitor the disk usage of the logging and debug files the current free disk space on the underlying filesystem of the logging directory is shown. The “Refresh” button needs to be clicked to update any information on the page.
View or Follow log
The regular and debug log files can be viewed using the front-end. Two options exist: The “View” opens a pop-up window and displays the full file, the “Follow” opens a pop-up window to display the last 100 lines of the file and then keeps displaying new lines as they are created.
When using the “Follow” the window automatically scrolls to the bottom with every line that is written to it. However, the browser will keep this window active for only so long. Depending on the browser, the window hets appended for 5 to 15 minutes.
The content and structure of the data differs per log file. Some are structured to include the software subroutines calling sequence in data header lines. These lines are highlighted in teal when detected.
Note: Be aware that the log files opened for viewing can be quite sizeable or that a file opened for following can receive a lot of data in a short time. This information is sent to the browser regardless of available resources on the local PC. In some cases the browser may not be able to accommodate the request or consume all memory causing the PC or browser to crash.
Job debug
For the debugging of command jobs, additional debug files will be created for each running job. The filenames of these job debug files start with the job-id of the corresponding job. They will contain a very detailed log of the scenario commands and all intermediate parsing results of the received data from the device. These files cannot be cleared, only deleted.
The job debug files are of prime interest to our developers investigating a communication or parsing problem.
Support file
To assist NetYCE support engineers and developers in investigating and resolving experienced issues, a NetYCE support file can be created. It will take a snapshot of the all relevant (debug) log files and system configuration files and combine these in a single downloadable file.
The files included in the support file:
-- regular and debug log files: /var/opt/yce/logs/*log -- the two last regular log rollover files /var_opt_yce/logs/*log.[01] -- the job definition and execution files for jobs having -- /var/opt/yce/logs/<job_id>_debug.log files /var/opt/yce/jobs/<job_id>/* -- the NetYCE configuration directory /opt/yce/etc/* -- the angular and mojo local setup file /opt/yce/htdocs/angular/app/host.js
All NetYCE support files are first compressed and then encrypted for safe communication with NetYCE support. The encryption key is the same key that is used to encrypt the database archives and is based on the customer specific license file.
This key is available to NetYCE support and need not be communicated. To obtain this key, login to the NetYCE cli as yce user and execute the command ck_license -y
If it is desirable to review the contents of the NetYCE support file, follow the steps below. Replace the appropriate file names and key values. It is essential to login/execute these commands as yce user, never as root.
-- find the support file $ go logs -- goto logs -> /var/opt/yce/logs yce@genesis /var/opt/yce/logs $ l *.bin -rw-r--r-- 1 yce nms 243712 Mar 4 10:08 netYCE_support_20190304_1008.bin yce@genesis /var/opt/yce/logs -- obtain the encryption key $ ck_license -y 6CEA-5C24-EC27-B699-0B4C yce@genesis /var/opt/yce/logs -- this step decrypts the support file -- replace the filename and key values to the appropriate values -- the decrypted tar-gzip file is created in /var/tmp $ openssl des3 -d -salt -in 'netYCE_support_20190304_1008.bin' -pass pass:6CEA-5C24-EC27-B699-0B4C -out '/var/tmp/netYCE_support_20190304_1008.tgz' $ go tmp -- goto tmp -> /var/tmp yce@genesis /var/tmp $ l total 244 -rw-rw-r-- 1 yce nms 243692 Mar 4 12:00 netYCE_support_20190304_1008.tgz drwxr-xr-x 2 yce nms 4096 Mar 1 14:44 yce_install/ yce@genesis /var/tmp -- extract the tar-gzip file $ gtar xvzf netYCE_support_20190304_1008.tgz logs/0304_0001_debug.log jobs/0304_0001/ jobs/0304_0001/0304_0001.scn jobs/0304_0001/0304_0001.log jobs/0304_0001/0304_0001.tsk jobs/0304_0001/0304_0001.dat jobs/0304_0001/0304_0001.job jobs/0304_0001/cmdb1_mgmt.sess.log jobs/0304_0001/0304_0001_yce.log jobs/0304_0001/0304_0001.cmd jobs/0304_0001/cmdb1.cmd logs/apache_access_log logs/apache_error_log logs/c3_vpn.log logs/net_update.log logs/patch_install.log ::: opt/yce/etc/yce_setup.xml opt/yce/etc/devel6_mysql.conf opt/yce/etc/ssl_cert.conf opt/yce/etc/devel6.netyce.org.crt opt/yce/etc/xch_ib_dns2.ini opt/yce/etc/cr_fz_hfc.conf opt/yce/etc/NetworkInventory-GetServiceConfigurationDetails.wsdl opt/yce/etc/yce_license_Genesis_v7_20190801 opt/yce/htdocs/angular/app/host.js $ l total 256 drwxrwxr-x 3 yce nms 4096 Mar 4 12:01 jobs/ drwxrwxr-x 2 yce nms 4096 Mar 4 12:01 logs/ -rw-rw-r-- 1 yce nms 243692 Mar 4 12:00 netYCE_support_20190304_1008.tgz drwxrwxr-x 3 yce nms 4096 Mar 4 12:01 opt/ drwxr-xr-x 2 yce nms 4096 Mar 1 14:44 yce_install/
The config, log and debug files are now extracted to the jobs/, logs/ and opt/ directories here.