Differences

This shows you the differences between two versions of the page.

--- guides:user:os_repo_file_organization [2022/04/11 16:10] – yspeerte
+++ guides:user:os_repo_file_organization [2024/07/03 12:31] (current) – external edit 127.0.0.1
@@ Line 1: / Line 1: @@
+====== OS-Repository file organization ======
+The OS-Repository uses a directory tree to store all vendor device OS-specific files. This directory tree resides under **''/var/opt/yce/shared/public/os''**.
+When the OS-Repository is 'enabled' in the yce-setup configuration, this tree can be shared as a NFS (Network File System) volume by the "OS-repo host' and mounted as the OS-repo on other NetYCE servers or Unix systems.
+The result is a single, but shared, location where these OS-files are stored, but can be transparently de-centrally managed and accessed.
+For details on the configuration setup, see the [[maintenance:general:os-repository|OS-Repository setup]] page.
+This page describes the file-organization in the OS-repository and the working of the management and monitoring of the Repository when a NetYCE server is the host of the repository (as outlined in the setup guide, non-NetYCE unix servers can be made the host too, but offer no monitoring).
+Network vendors commonly provide os-image files for a specific type of hardware which are available in various versions to accommodate a varying number of ports and types or are otherwise equipped with specific feature. Many variations exist for the same base hardware all using the same os-image file.
+At the same time, variations on the os-image could exist that provide different feature-sets. And then there are of course the different versions, some you use in production for one specific role and another for for general use and the latest might still be in test.
+The end-result is that there is an overwhelming number of dependencies what os-image to use where, some hardware related, other on costs, procedures or device role. This is where the OS-Repository is intended to bring some structure and aims to assist in automating os-upgrades.
+===== File tree =====
+The OS-Repository allows the end-user to create 'image-sets' that links a set of os-files to a vendor, the intended device-type, its usage status, and some meta data together. An operator can thus quickly filter for a device-type to find which set to use for the general production version.
+See the article on [[menu:build:osversions|OS-Repository form]]
+To prevent the OS-Repository from creating multiple copies of the same (large) os-image-file for each variant of the same base device-type, the image-sets just link to the image file(s) so only a single copy need to exist.
+The file-organization of the repository reflects this: All files from the same vendor-family are stored together in the same directory. For every Vendor-type NetYCE supports a unique directory is created in the ''public/os'' folder by default. Additional vendor-type directories are created for vendors named in the CMDB table, but can be created manually too.
+All vendor-type directories will be created using lowercase characters only. Any vendor-type directories using uppercase characters are still shared transparently but will be ignored by the OS-repo file monitoring. See the section below on File Monitoring.
+A typical set of directories in the ''shared/public'' tree.
+<code>
+/var/opt/shared/public
+|-- OS -> /var/opt/shared/public/os
+|-- os
+|   |-- alcatel_timos
+|   |-- arista_eos
+|   |-- aruba_mc
+|   |-- aruba_mm
+|   |-- avaya_ers
+|   |-- avaya_vsp
+|   |-- checkpoint_mgmt
+|   |-- ciena_ci6
+|   |-- ciena_ci8
+|   |-- cisco_aci
+|   |-- cisco_asa
+|   |-- cisco_ios
+|   |-- cisco_nexus
+|   |-- cisco_ucs
+|   |-- cisco_wism
+|   |-- cisco_wlc
+|   |-- cisco_xe
+|   |-- cisco_xr
+|   |-- corvil_cne
+|   |-- f5_bigip
+|   |-- falcon_mrotek
+|   |-- firewall
+|   |-- fortinet_fortigate
+|   |-- gigamon
+|   |-- hp_c5
+|   |-- hp_c7
+|   |-- huawei_ce
+|   |-- huawei_s
+|   |-- infoblox
+|   |-- junos
+|   |-- paloalto_panos
+|   |-- perle_iolan
+|   |-- riverbed
+|   |-- server
+|   |-- unknown
+|   `-- windows
+|-- previous
+`-- users
+</code>
+This allows for relatively short and predictable paths when up- or down-loading image files to the devices.
+For example to pull an image for a Cisco IOS device:
+<code>
+scp ycicle@netyceA:/public/os/cisco_ios/cat4500-ipbasek9-mz.150-2.SG81.bin .
+</code>
+===== Repository monitor =====
+Part of the OS-Repository is a file monitor. It is active on the NetYCE 'host' server of the repository and will monitor all file manipulations to keep de repository up-to-date. Any file that is copied, moved, deleted, renamed, uploaded or created is tracked and will be reflected in the repository database tables.
+Since the repository 'file-sets' have only links to the repository files, the monitor can maintain file names, the number of links to a file and inform on files currently missing from the set.
+The OS-repo file monitor is part of the NetYCE process 'yce_skulker'. If this process is not running the monitor will not be active and file changes will not be incorporated. The monitor is also inactive when the OS-repository status is set 'disabled' in the yce-setup.
+The monitor will recognize new (vendor) directories automatically and will start adding files it finds to the repository immediately. Note however, that any (vendor) directory in ''public/os'' must be **all lowercase**, otherwise the monitor will ignore the directory.
+Subdirectories in the vendor directories are also ignored by the monitor.
+If needed, the activities of the file monitor can be observed by first enabling the system debug mode and then restarting the yce_skulker. This enables the detailed logging of the skulker to the file ''/var/opt/yce/logs/yce_skulker_debug.log''.
+This can be done using the frontend ('Admin - System - Debug logs' and 'System status') or ssh cli. The latter uses the commands below.
+<code>
+-- enable debug mode
+go set dev
+-- restart the skulker process
+go restart skulker
+-- read logs
+go logs
+less yce_skulker_debug.log
+-- stop debug mode
+go clr dev
+go restart skulker
+</code>
+Example debug logs when adding a file to cisco_ios:
+<code>
+os-repo Event: fullname [/var/opt/shared/public/os/cisco_ios/c2960-ipbasek9-mz.150-2.SG83.bin] file [Cisco_IOS][c2960-ipbasek9-mz.150-2.SG83.bin] watcher [/var/opt/shared/public/os/cisco_ios]
+os-repo NewFile IN_CREATE - /var/opt/shared/public/os/cisco_ios/c2960-ipbasek9-mz.150-2.SG83.bin
+os-repo Event: fullname [/var/opt/shared/public/os/cisco_ios/c2960-ipbasek9-mz.150-2.SG83.bin] file [Cisco_IOS][c2960-ipbasek9-mz.150-2.SG83.bin] watcher [/var/opt/shared/public/os/cisco_ios]
+os-repo UpdateFile IN_CLOSE_WRITE - /var/opt/shared/public/os/cisco_ios/c2960-ipbasek9-mz.150-2.SG83.bin
+-04-12 10:13:42 os-repo event actions
+(os-repo event actions)
+Action $VAR1 = {
+  'event' => 'AddFile',
+  'file' => 'c2960-ipbasek9-mz.150-2.SG83.bin',
+  'fullname' => '/var/opt/shared/public/os/cisco_ios/c2960-ipbasek9-mz.150-2.SG83.bin',
+  'mask' => 8,
+  'vendor' => 'Cisco_IOS',
+  'watcher' => '/var/opt/shared/public/os/cisco_ios'
+};
+os-repo AddFile [Cisco_IOS][c2960-ipbasek9-mz.150-2.SG83.bin] -> [Cisco_IOS][c2960-ipbasek9-mz.150-2.SG83.bin] [3516] [2022-04-12] [cac7cca621fe8fcac93e5cfd6ea29b18]
+os-repo AddFile - AddFile [Cisco_IOS][c2960-ipbasek9-mz.150-2.SG83.bin]
+</code>