Technical documentation

User Tools

Site Tools

Trace:
maintenance:general:mariadb-10.6-upgrade

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
maintenance:general:mariadb-10.6-upgrade [2022/06/14 09:59] – [Restoring replication] yspeertemaintenance:general:mariadb-10.6-upgrade [2024/07/03 12:31] (current) – external edit 127.0.0.1
Line 1: Line 1:
 +====== MariaDB 10.6 upgrade ======
  
 +> <color orange>This procedure is work-in-progress. Support for MariaDB 10.6 is not released yet (look for 8.0.0 builds later than 20220610)</color>
 +
 +This article describes the procedure to upgrade the installed Mariadb version to 10.6.x as there is no (semi) automatic process available. , the removal and initial install requires manual steps. 
 +
 +This procedure covers the MariaDB upgrade to version 10.6.x (specifically 10.6.8) from either \\ version **10.2.x** or **10.3.x**. \\
 +You can verify the current version in the System status tool or using the ''ck_setup.pl'' command.
 +
 +Two install options are available:
 +  * semi automatic using [[#yum install|YUM install]] when an internet connection is available
 +  * manually using [[#rpm install|RPM install]] when no internet connection is available.
 +
 +As part of both procedures, the **yum** configuration will be modified to allow yum to __update__ Mariadb installation to the latest 10.6.x release.
 +
 +====== YUM install ======
 +
 +This procedure can only be used when a NetYCE server is connected to the internet.
 +
 +> <color blue>The required Linux permissions needed include ''root'' access. However, all steps can be performed as the ''yce'' functional user. Where root privileges are needed the command includes a ''sudo'' prefix.</color>
 +
 +===== Prepare upgrade =====
 +
 +==== check db ====
 +
 +Prepare the upgrade.
 +Ensure the database is free from defects:
 +<code>
 +$ log_maint.pl
 +</code>
 +
 +==== backup YCE db ====
 +
 +Go the the **YCE database archives** tool under "Admin -> System". Create a manual archive with a description indicating it is using the older version of the MariaDB, eg "Mariadb 10.3 pre-upgrade"
 +
 +==== backup CMDB ====
 +
 +Do the same for the NCCM database using the **Backups database archives**.
 +
 +
 +==== stop replication ====
 +
 +If the NetYCE server is part of a redundant master/master pair, the replication must be halted on **BOTH** systems.
 +Go the the 'Admin -> System -> System status' tool and select a "Full report". Scroll down to the "Database status" section and hit the "**Stop replication slave**" button.
 +
 +Do the same for the other NetYCE database server.
 +
 +==== yum update ====
 +
 +It is recommended to update the Linux OS using ''yum'' before the upgrade. It allows the system to update any packages to their latest revisions. Some of them will be required for the MariaDB upgrade. 
 +<code>
 +$ sudo yum update
 +
 +Loaded plugins: fastestmirror
 +Loading mirror speeds from cached hostfile
 +epel/x86_64/metalink                                                                   |  21 kB  00:00:00
 + * base: mirror.1000mbps.com
 + * epel: mirrors.xtom.de
 + * extras: ftp.nluug.nl
 + * updates: nl.mirrors.clouvider.net
 +base                                                                                   | 3.6 kB  00:00:00
 +epel                                                                                   | 4.7 kB  00:00:00
 +extras                                                                                 | 2.9 kB  00:00:00
 +mariadb                                                                                | 3.4 kB  00:00:00
 +mysecureshell                                                                          |  951 B  00:00:00
 +updates                                                                                | 2.9 kB  00:00:00
 +:::
 +Total download size: 9.9 M
 +Is this ok [y/d/N]: y
 +</code>
 +
 +
 +===== Uninstall existing MariaDB =====
 +
 +First, halt the NetYCE application, including database
 +<code>
 +$ go down
 +</code>
 +
 +Then remove the old packages
 +<code>
 +$ rpm -qa | egrep -i 'mariadb|galera'
 +
 +galera-25.3.35-1.el7.centos.x86_64
 +MariaDB-client-10.2.44-1.el7.centos.x86_64
 +MariaDB-devel-10.2.44-1.el7.centos.x86_64
 +MariaDB-shared-10.2.44-1.el7.centos.x86_64
 +MariaDB-server-10.2.44-1.el7.centos.x86_64
 +MariaDB-common-10.2.44-1.el7.centos.x86_64
 +MariaDB-compat-10.2.44-1.el7.centos.x86_64
 +
 +$ --nodeps -ev <all the above packages>
 +
 +Or do this all at once:
 +$ rpm -qa | egrep -i 'mariadb|galera' | xargs -i sudo rpm --nodeps -ev {}
 +</code>
 +
 +===== yum install MariaDB 10.6 =====
 +
 +To update the Centos7 or RedHat7 system, ''yum'' is used. It uses external "repositories" to locate and update the Linux packages installed on the server. By configuring yum to use the MariaDB 10.6 repository, it will also update the MariaDB to the latest release of version 10.6.x.
 +
 +Create or edit the file ''/etc/yum.repos.d/MariaDB.repo''. It should already exist but could have a slightly different name (eg lowercase or added version). Make sure only one file with a mariadb-like name exists as yum will simply execute all files in this directory, even on conflicting versions.
 +
 +The content of the file should be updated as below. On existing files only the 10.6 reference in the "baseurl" will need updating.
 +<code>
 +[MariaDB]
 +baseurl = http://yum.mariadb.org/10.6/centos7-amd64
 +enabled = 1
 +gpgcheck = 1
 +name = http://downloads.mariadb.org/mariadb/repositories/
 +gpgkey = https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
 +</code>
 +
 +With this yum configuration file in place, the MariaDB 10.6 installation can also be executed using ''yum'':
 +<code>
 +$ sudo yum install mariadb-server
 +
 +Loaded plugins: fastestmirror
 +Loading mirror speeds from cached hostfile
 + * base: mirror.1000mbps.com
 + * epel: mirrors.xtom.de
 + * extras: ftp.nluug.nl
 + * updates: nl.mirrors.clouvider.net
 +Package mariadb-server is obsoleted by MariaDB-server, trying to install MariaDB-server-10.6.8-1.el7.centos.x86_64 instead
 +Resolving Dependencies
 +--> Running transaction check
 +---> Package MariaDB-server.x86_64 0:10.6.8-1.el7.centos will be installed
 +--> Processing Dependency: MariaDB-common >= 10.6.1 for package: MariaDB-server-10.6.8-1.el7.centos.x86_64
 +:::
 +--> Finished Dependency Resolution
 +
 +Dependencies Resolved
 +
 +==============================================================================================================
 + Package                     Arch                Version                           Repository            Size
 +==============================================================================================================
 +Installing:
 + MariaDB-server              x86_64              10.6.8-1.el7.centos               mariadb               25 M
 +Installing for dependencies:
 + MariaDB-client              x86_64              10.6.8-1.el7.centos               mariadb               14 M
 + MariaDB-common              x86_64              10.6.8-1.el7.centos               mariadb               81 k
 + MariaDB-compat              x86_64              10.6.8-1.el7.centos               mariadb              2.2 M
 + galera-4                    x86_64              26.4.11-1.el7.centos              mariadb              9.9 M
 + libpmem                     x86_64              1.5.1-2.1.el7                     base                  59 k
 + pcre2                       x86_64              10.23-2.el7                       base                 201 k
 + socat                       x86_64              1.7.3.2-2.el7                     base                 290 k
 +
 +Transaction Summary
 +==============================================================================================================
 +Install  1 Package (+7 Dependent packages)
 +
 +Total download size: 52 M
 +Installed size: 222 M
 +Is this ok [y/d/N]: y
 +:::
 +Installed:
 +  MariaDB-server.x86_64 0:10.6.8-1.el7.centos
 +
 +Dependency Installed:
 +  MariaDB-client.x86_64 0:10.6.8-1.el7.centos           MariaDB-common.x86_64 0:10.6.8-1.el7.centos
 +  MariaDB-compat.x86_64 0:10.6.8-1.el7.centos           galera-4.x86_64 0:26.4.11-1.el7.centos
 +  libpmem.x86_64 0:1.5.1-2.1.el7                        pcre2.x86_64 0:10.23-2.el7
 +  socat.x86_64 0:1.7.3.2-2.el7
 +
 +Complete!
 +</code>
 +
 +The new MariaDB is now installed but will fail to run as its default configuration file will need to be replaced. 
 +
 +Continue the installation using the section [[#MariaDB configuration]] section below.
 +
 +
 +====== RPM install ======
 +
 +Us this procedure when the NetYCE server cannot connect to the internet. The required packages need to be downloaded and installed manually.
 +
 +> <color blue>The required Linux permissions needed include ''root'' access. However, all steps can be performed as the ''yce'' functional user. Where root privileges are needed the command includes a ''sudo'' prefix.</color>
 +
 +===== Downloads =====
 +
 +==== MariaDB 10.6.x ====
 +
 +Retrieve latest release of MariaDB 10.6 for CentOS7/RedHat7 on x86_64 
 +
 +point browser to:
 +https://mirror.mariadb.org/yum/10.6/centos7-amd64/rpms/
 +
 +find the latests versions (now 10.6.8) of the rpms for:
 +  * server -> MariaDB-server-10.6.8-1.el7.centos.x86_64.rpm
 +  * client -> MariaDB-client-10.6.8-1.el7.centos.x86_64.rpm
 +  * common -> MariaDB-common-10.6.8-1.el7.centos.x86_64.rpm
 +  * compat -> MariaDB-compat-10.6.8-1.el7.centos.x86_64.rpm
 +  * shared -> MariaDB-shared-10.6.8-1.el7.centos.x86_64.rpm
 +  * galera -> galera-4-26.4.11-1.el7.centos.x86_64.rpm
 +
 +Note that the 'devel' package is optional for customer systems. It is only required for NetYCE development servers
 +  * devel -> MariaDB-devel-10.6.8-1.el7.centos.x86_64.rpm
 +
 +Using these filenames, retrieve them and download or copy them to the NetYCE server:
 +<code> 
 +$ mkdir /var/tmp/mariadb-10.6.8
 +$ cd /var/tmp/mariadb-10.6.8
 +
 +$ wget https://mirror.mariadb.org/yum/10.6/centos7-amd64/rpms/MariaDB-server-10.6.8-1.el7.centos.x86_64.rpm
 +$ wget https://mirror.mariadb.org/yum/10.6/centos7-amd64/rpms/MariaDB-client-10.6.8-1.el7.centos.x86_64.rpm
 +$ wget https://mirror.mariadb.org/yum/10.6/centos7-amd64/rpms/galera-4-26.4.11-1.el7.centos.x86_64.rpm
 +.. etc
 +</code>
 +
 +==== dependencies ===
 +
 +The MariaDb 10.6 installation depends on some other packages that need to be installed at the same time, or installed earlier.
 +
 +point browser to:
 +http://mirror.centos.org/centos-7/7/os/x86_64/Packages/
 +
 +find the latests X68_64 versions of the rpms for:
 +  * libpmem -> libpmem-1.5.1-2.1.el7.x86_64.rpm
 +  * pcre2 -> pcre2-10.23-2.el7.x86_64.rpm
 +  * socat -> socat-1.7.3.2-2.el7.x86_64.rpm
 +
 +<code>
 +$ cd /var/tmp/mariadb-10.6.8
 +
 +$ wget http://mirror.centos.org/centos-7/7/os/x86_64/Packages/libpmem-1.5.1-2.1.el7.x86_64.rpm
 +$ wget http://mirror.centos.org/centos-7/7/os/x86_64/Packages/pcre2-10.23-2.el7.x86_64.rpm
 +$ wget http://mirror.centos.org/centos-7/7/os/x86_64/Packages/socat-1.7.3.2-2.el7.x86_64.rpm
 +</code>
 +
 +Additionally, **glibc** must be installed and must be of a specific version. The current (for MariaDB 10.6.8) glibc version is 2.17-326.el7_9 for x86_64. 
 +To check which glibc version is installed:
 +<code>
 +$ rpm -qa | grep -i glibc
 +
 +glibc-headers-2.17-326.el7_9.x86_64
 +glibc-2.17-326.el7_9.x86_64
 +glibc-common-2.17-326.el7_9.x86_64
 +glibc-devel-2.17-326.el7_9.x86_64
 +</code>
 +
 +If your current version is lower than the 2.17.326, download these packages from:
 +http://centos.mirror.transip.nl/7/updates/x86_64/Packages
 +
 +  * glibc -> glibc-2.17-326.el7_9.x86_64.rpm
 +  * glibc-devel -> glibc-devel-2.17-326.el7_9.x86_64.rpm
 +  * glibc-common -> glibc-common-2.17-326.el7_9.x86_64.rpm
 +  * glibc-headers -> glibc-headers-2.17-326.el7_9.x86_64.rpm
 +
 +<code>
 +$ cd /var/tmp/mariadb-10.6.8
 +
 +$ wget http://centos.mirror.transip.nl/7/updates/x86_64/Packages/glibc-2.17-326.el7_9.x86_64.rpm
 +$ wget http://centos.mirror.transip.nl/7/updates/x86_64/Packages/glibc-common-2.17-326.el7_9.x86_64.rpm
 +$ wget http://centos.mirror.transip.nl/7/updates/x86_64/Packages/glibc-devel-2.17-326.el7_9.x86_64.rpm
 +$ wget http://centos.mirror.transip.nl/7/updates/x86_64/Packages/glibc-headers-2.17-326.el7_9.x86_64.rpm
 +</code>
 +
 +===== Prepare upgrade =====
 +
 +==== check db ====
 +
 +Prepare the upgrade.
 +Ensure the database is free from defects:
 +<code>
 +$ log_maint.pl
 +</code>
 +
 +==== backup YCE db ====
 +
 +Go the the **YCE database archives** tool under "Admin -> System". Create a manual archive with a description indicating it is using the older version of the MariaDB, eg "Mariadb 10.3 pre-upgrade"
 +
 +==== backup CMDB ====
 +
 +Do the same for the NCCM database using the **Backups database archives**.
 +
 +
 +==== stop replication ====
 +
 +If the NetYCE server is part of a redundant master/master pair, the replication must be halted on **BOTH** systems.
 +Go the the 'Admin -> System -> System status' tool and select a "Full report". Scroll down to the "Database status" section and hit the "**Stop replication slave**" button.
 +
 +Do the same for the other NetYCE database server.
 +
 +
 +===== Uninstall existing MariaDB =====
 +
 +First, halt the NetYCE application, including database
 +<code>
 +$ go down
 +</code>
 +
 +Then remove the old packages
 +<code>
 +$ rpm -qa | egrep -i 'mariadb|galera'
 +
 +galera-25.3.35-1.el7.centos.x86_64
 +MariaDB-client-10.2.44-1.el7.centos.x86_64
 +MariaDB-devel-10.2.44-1.el7.centos.x86_64
 +MariaDB-shared-10.2.44-1.el7.centos.x86_64
 +MariaDB-server-10.2.44-1.el7.centos.x86_64
 +MariaDB-common-10.2.44-1.el7.centos.x86_64
 +MariaDB-compat-10.2.44-1.el7.centos.x86_64
 +
 +$ --nodeps -ev <all the above packages>
 +
 +Or do this all at once:
 +$ rpm -qa | egrep -i 'mariadb|galera' | xargs -i sudo rpm --nodeps -ev {}
 +</code>
 +
 +===== YUM configuration =====
 +
 +To update the Centos7 or RedHat7 system, ''yum'' is used. It uses external "repositories" to locate and update the Linux packages installed on the server. By configuring yum to use the MariaDB 10.6 repository, it will also update the MariaDB to the latest release of version 10.6.x.
 +
 +Create or edit the file ''/etc/yum.repos.d/MariaDB.repo''. It should already exist but could have a slightly different name (eg lowercase or added version). Make sure only one file with a mariadb-like name exists as yum will simply execute all files in this directory, even on conflicting versions.
 +
 +The content of the file should be updated as below. On existing files only the 10.6 reference in the "baseurl" will need updating.
 +<code>
 +[MariaDB]
 +baseurl = http://yum.mariadb.org/10.6/centos7-amd64
 +enabled = 1
 +gpgcheck = 1
 +name = http://downloads.mariadb.org/mariadb/repositories/
 +gpgkey = https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
 +</code>
 +
 +
 +===== rpm install MariaDB 10.6 =====
 +
 +From the location of the downloaded rpm files
 +<code>
 +$ cd /var/tmp/mariadb-10.6.8
 +</code>
 +
 +Install all packages at once. Trying to do these one-by-one will result in circular dependencies.
 +<code>
 +$ sudo rpm -Uvh *.rpm
 +
 +Preparing...                          ################################# [100%]
 +Updating / installing...
 +   1:MariaDB-compat-10.6.8-1.el7.cento################################# [ 10%]
 +   2:MariaDB-common-10.6.8-1.el7.cento################################# [ 20%]
 +   3:libpmem-1.5.1-2.1.el7            ################################# [ 30%]
 +   4:pcre2-10.23-2.el7                ################################# [ 40%]
 +   5:MariaDB-client-10.6.8-1.el7.cento################################# [ 50%]
 +   6:MariaDB-shared-10.6.8-1.el7.cento################################# [ 60%]
 +   7:socat-1.7.3.2-2.el7              ################################# [ 70%]
 +   8:galera-4-26.4.9-1.el7.centos     ################################# [ 80%]
 +   9:MariaDB-server-10.6.8-1.el7.cento################################# [ 90%]
 +  10:MariaDB-devel-10.6.8-1.el7.centos################################# [100%]
 +</code>
 +
 +The new MariaDB is now installed but will fail to run as its default configuration file will need to be replaced. 
 +
 +Continue the installation using the section [[#MariaDB configuration]] section below.
 +
 +
 +====== MariaDB configuration ======
 +
 +First the NetYCE process monitor needs to be started:
 +<code>
 +$ go start yce_psmon
 +-- starting Daemon 'yce_psmon'
 +yce_psmon:
 +  start: /usr/bin/sudo /usr/bin/systemctl start yce_psmon.service
 +  wait start 'yce_psmon': 3816
 +done
 +</code>
 +
 +Then, to re-create the configuration file for the MariaDB, run the ''yce_setup'' command with the ''-r'' (regenerate) option:
 +<code>
 +$ yce_setup.pl -r
 +</code>
 +
 +This command will regenerate the configuration files for all NetYCE components and will (re)start the relevant processes. It will also start the MaraDB server.
 +
 +During this initial run of ''yce_setup.pl'' some warnings will be issued due to the missing mysql/mariadb.
 +
 +To complete the configuration, re-run the ''yce_setup.pl -r'' command. As the database is now reachable, the warnings should disappear.
 +<code>
 +$ yce_setup.pl - r
 +
 +</code>
 +
 +
 +===== Verifications =====
 +
 +Verify the NetYCE setup acknowledges the MariaDB dependencies are met:
 +<code>
 +$ ck_setup.pl
 +
 +-- OK. Have a 'x86_64' architecture
 +-- OK. Have a supported EL7 distribution: CentOS EL7 7.9.2009
 +-- OK. Have YCEperl version '8.0.0'
 +-- OK. Found current '5.32.0' perl CORE/libperl.so
 +-- OK. Found link to previous '5.24.0' perl CORE/libperl.so
 +-- OK. Mariadb version '10.6' is supported for EL7
 +-- OK. Found required mysql library 'libmariadbd.so.19'
 +</code>
 +
 +Next, the upgrade will require some updates to the database tables. This command will check and optimize all tables after the initial 'mysql_upgrade' that is part of the 10.6 distribution is completed:
 +<code>
 +$ mysql_repair.sh
 +</code>
 +
 +When done, these commands are part of the daily maintenance and should produce no errors:
 +
 +<code>
 +$ log_maint.pl
 +
 +$ logs_daily.pl
 +
 +$ nccm_daily.pl
 +</code>
 +
 +To check if the database can be properly accessed, use the ''ck_license'' command:
 +<code>
 +$ ck_license -X
 +This will output license details like:
 +
 +OK: Version '8.0.0' matches license for '8'
 +OK: License will not expire until '20221231'
 +OK: Architecture license Full. Found '1' servers
 +OK: Users license 'Unlim' sufficient for '0' users
 +OK: Scheduled jobs license 'Unlim' sufficient for '0' jobs
 +OK: Client-types license 'Unlim' sufficient for '5' client-types
 +OK: Service-types license 'Unlim' sufficient for '11' service-types
 +OK: Node license 'Unlim' sufficient for '414' nodes
 +OK: Port license 'Unlim' sufficient for '747' ports
 +OK: Service license 'Unlim' sufficient for '43' services
 +OK: Node-compliance license 'Unlim' sufficient for '0' nodes
 +OK: Mpls_vrf license 'Unlim' sufficient for '177' mpls_vrfs
 +</code>
 +
 +===== Restoring replication =====
 +
 +After - and only after - **both** NetYCE servers are upgraded to use MariaDB 10.6 can the master/master replication between the servers be restored.
 +
 +This involves the (re)configuration of the replication between the servers using the information provided in an earlier ''yce_setup.pl'' session.
 +
 +But as both systems will have been unable to communicate for a while, the databases are almost certainly out-of-sync. To restore synchronization, use the 'ck_dbsync'' tool to verify and synchronize the databases. Do not allow users to login or API requests while this synchronization is in progress. Depending on database size, this can take half an hour to several hours to complete.
 +
 +<code>
 +$ ck_dbsync.pl -R -x
 +</code>
 +
 +When done, the synchronization status should look like this:
 +<code>
 +$ ck_dbsync.pl
 +2022-06-08 17:49:23 === Started ===
 +2022-06-08 17:49:23 Defaulting to replication status (try -h for help)
 +2022-06-08 17:49:23 Using local PRI and remote SEC setup:
 +2022-06-08 17:49:23   PRImary:   172.17.0.24
 +2022-06-08 17:49:23   SECondary: 172.17.0.25
 +2022-06-08 17:49:23 Connected to both databases
 +2022-06-08 17:49:23 1/1 Sync status
 +2022-06-08 17:49:23 1/1 Replication status
 +2022-06-08 17:49:23 1/1   PRI slave IO : Yes
 +2022-06-08 17:49:23 1/1   PRI slave SQL: Yes
 +2022-06-08 17:49:23 1/1   SEC slave IO : Yes
 +2022-06-08 17:49:23 1/1   SEC slave SQL: Yes
 +2022-06-08 17:49:23 === Completed ===
 +</code>
 +
 +===== Create 10.6 archives ======
 +
 +To complete the upgrade save the upgraded 10.6 databases using the archiving tools.
 +
 +Go the the YCE database archives tool under “Admin → System”. Create a manual archive with a description indicating it is using the new version of the MariaDB, eg “Mariadb 10.6 post-upgrade”
 +
 +Do the same for the Backups archive.
 +
 +//
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki