Recently, we were at a cross-roads. We did not know whether we could leave an old GitLab Server version 10.2.1-ee running on RHEL 6 and begin afresh in a new GitLab server or we could migrate all repositories to the new instance. After going through articles and documentation, we found out that GitLab had its own CLI tools that could backup and restore content out of the box and this was a big relief. Having succeeded we thought it wise to document it here for anyone interested to follow along.
First and foremost for migration to take place, the versions of GitLab need to match on both servers, so we decided to upgrade the old GitLab version 10.2.1-ee in RHEL 6 step by step until version 13.5.4-ee, the last available for RHEL 6 in GitLab’s repositories. Thanks to GitLab team for maintaining the repositories past EOL of RHEL 6. The upgrade process was painful but good thing is that they were working. Just a few issues here and there especially the leap from version 12 to 13.
Something to note in the whole experience is the change of one of GitLab’s core programs from version 12 to 13 as it has been mentioned above. The previous GitLab’s webserver was known as “unicorn” gets replaced by a new one known as Puma. This led to a couple of errors because Puma is not enabled during the upgrade process which will be discussed.
Step 1: Step-wise upgrade
In the first step, we decided to upgrade GitLab from version 10.2.1-ee to version 13.5.4-ee. This is the roadmap of the versions we went through in RHEL 6. You can choose a different route.
10.2.1-> 10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.5.4Sample commands for we used for the step by step upgrade are as follows all the way to gitlab-ee-13.5.4-ee.0.el6.x86_64.
sudo yum install gitlab-ee-10.2.1-ee.0.el6.x86_64 -y
sudo yum install gitlab-ee-10.8.7-ee.0.el6.x86_64 -y
sudo yum install gitlab-ee-11.11.8-ee.0.el6.x86_64 -y
sudo yum install gitlab-ee-12.0.12-ee.0.el6.x86_64 -y
When upgrading from 12.10.14-ee to 13.0.14-ee GitLab’s webserver “unicorn” changes to “Puma” and this caused lots of errors once the upgrade was complete. What we did was enable “puma” and disable “unicorn” in GitLab as follows:
$ sudo vim /etc/gitlab/gitlab.rb
puma['enable'] = true #Enable puma
unicorn['enable'] = false #Look for this setting or add if it is no thereIn case there are still problems later, check if “unicorn” is still running by looking at the application listening at port 8080. If unicorn is running thereat, kill it.
sudo kill -9 PID of unicornLater, restart all services thus:
sudo gitlab-ctl restartOnce we are done with the upgrades, time came to migrate the server to a new CentOS 7 with GitLab server version 13.5.4-ee installed. Remember we mentioned that for migration to happen, the versions of GitLab need to be the same. So now we have version 13.5.4-ee in RHEL 6 and the same version in RHEL 7/CentOS 7.
Step 2: Backup GitLab content
Since it is a RHEL 6 server, we could not update or upgrade anything after version 13.5.4. So we just had to backup this version of GitLab as follows:
sudo gitlab-rake gitlab:backup:createAn output similar to the one below was witnessed.
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
* jkibet/peoject-Social-Lending ... [DONE]
* jkibet/project-Social-Lending.wiki ... [SKIPPED]
* jkibet/peoject-Broom ... [DONE]
* jkibet/peoject-Broom.wiki ... [SKIPPED]
* jkibet/QmicaCS-Qmicash ... [DONE]
* jkibet/QmicaCS-Qmicash.wiki ... [SKIPPED]
* jkibet/Qmica-phoneloan ... [DONE]
.......
Creating backup archive: 1628175183_2021_08_05_13.5.4-ee_gitlab_backup.tar ... done
Uploading backup archive to remote storage ... skipped
Deleting tmp directories ... done
The moment the backup process is done, look at “/var/opt/gitlab/backups/” directory where we hope you will find your backup file in a tar archive. As you can guess, this file should be copied to the new GitLab instance next
Step 3: Copy Backup Archive and Files to the new server
Depending on your needs or preference, there are several ways the file can be copied over to the new server. You can use rsync, scp or other GUI-based tools out there. For this example, we will keep it simple and use a tool you will get already installed in Linux. And that is scp (Secure Copy). Run the command as follows. Remember to replace “user” with your server user.
sudo scp /var/opt/gitlab/backups/1628175183_2021_08_05_13.5.4-ee_gitlab_backup.tar user@new-server:/home/userOther files we need to copy over to the new server include the following: “/etc/gitlab/gitlab.rb” and “/etc/gitlab/gitlab-secrets.json“. These files contain sensitive data and GitLab lets you back them up manually.
sudo scp /etc/gitlab/gitlab.rb /etc/gitlab/gitlab-secrets.json user@new-server:/home/user
Step 4: Prepare the new server for restoration
We assume that you have a new GitLab server already installed with the same version as your old RHEL 6 server. If not, the following guides will be pretty beneficial for you. Once you are ready, run the commands below for a new installation in CentOS 7.
curl -s https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh | sudo bash
sudo yum update -y
sudo yum install pygpgme yum-utils policycoreutils-python openssh-server perl postfix -y
sudo systemctl enable postfix
sudo systemctl start postfix
sudo EXTERNAL_URL="//gitlab.geeksforgeeks.org" yum install -y gitlab-ee-13.5.4-ee.0.el7.x86_64Give it sometime to install. You can later find out if all services are running smoothly as follows:
sudo gitlab-ctl statusOnce you have GitLab running successfully, stop the following services before we can restore the files we have copied over in it.
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiqYou can again check if the services are running by running thus:
sudo gitlab-ctl statusAnd you can start and restart the components as follows:
# Start all GitLab components
sudo gitlab-ctl start
# Stop all GitLab components
sudo gitlab-ctl stopStep 5: Copy and restore the backup
Now in this step, we are going to move the backup we have copied in this new server to the default backup folder where GitLab will read from.
sudo mv /home/user/1628175183_2021_08_05_13.5.4-ee_gitlab_backup.tar /var/opt/gitlab/backups/Then give “git” user and group rights to the /var/opt/gitlab/backups/ directory
sudo chown git:git /var/opt/gitlab/backups/ -RIf you can remember, we copied some files to this new server, that is “gitlab.rb” and “gitlab-secrets.json“. We should replace the ones that came with the installation with these ones. Back up the original files then replace them with the ones we copied from the old server.
cd /etc/gitlab
#Back up the original files
sudo mv gitlab.rb gitlab.rb-backup
sudo mv gitlab-secrets.json gitlab-secrets.json-backup
#Copy the ones from the old server
sudo cp /home/user/gitlab.rb /home/user/gitlab-secrets.json /etc/gitlabOnce that is done, we are now ready to restore our GitLab. Fingers crossed, let us now restore everything and hope that it works without any issues. Run the command below to restore your repositories and database. Notice that the backup file has been truncated as follows. It was “1628175183_2021_08_05_13.5.4-ee_gitlab_backup.tar” but in the command, it is “1628175183_2021_08_05_13.5.4-ee” which is its timestamp. Basically, remove the “_gitlab_backup.tar” end part and then run the command with the timestamp only.
$ sudo gitlab-rake gitlab:backup:restore BACKUP=1628175183_2021_08_05_13.5.4-ee
Unpacking backup ...During the restoration process, you will see the message below as a prompt. Simply acknowledge by typing “yes” then let GitLab do its thing.
Be sure to stop Puma, Sidekiq, and any other process that
connects to the database before proceeding. For Omnibus
installs, see the following link for more information:
https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-for-omnibus-gitlab-installations
Before restoring the database, we will remove all existing
tables to avoid future upgrade problems. Be aware that if you have
custom tables in the GitLab database these tables and all data will be
removed.
Do you want to continue (yes/no)? yesAt the tail end of the back up, another prompt will appear as shared below. Simply type yes and the restore should end successfully.
This task will now rebuild the authorized_keys file.
You will lose any data stored in the authorized_keys file.
Do you want to continue (yes/no)? yesOnce done, run reconfigure command and then restart GitLab services.
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
ok: run: alertmanager: (pid 8132) 0s
ok: run: gitaly: (pid 8143) 1s
ok: run: gitlab-exporter: (pid 8164) 1s
ok: run: gitlab-workhorse: (pid 8168) 0s
ok: run: grafana: (pid 8177) 0s
ok: run: logrotate: (pid 8188) 0s
ok: run: nginx: (pid 8195) 0s
ok: run: node-exporter: (pid 8206) 1s
ok: run: postgres-exporter: (pid 8216) 0s
ok: run: postgresql: (pid 8228) 0s
ok: run: prometheus: (pid 8232) 0s
ok: run: puma: (pid 8247) 0s
ok: run: redis: (pid 8255) 0s
ok: run: redis-exporter: (pid 8261) 0s
ok: run: sidekiq: (pid 8268) 1sLater, after everything is successfully restored and working, the only thing remaining is to upgrade GitLab from 13.5.4-ee all the way to the latest at the time of writing, which is 14.1.2-ee. Below is the roadmap we used. Be aware that the commands were not ran at once, they were done step by step checking that GitLab is okay every update.
13.9.2 -> 13.12.6 -> 14.0.5 -> 14.1.2Sample commands that we ran include the following. Just like previously, we did not run the commands at once but step by step checking that GitLab services are working at each update we made.
sudo yum install gitlab-ee-13.9.2-ee.0.el6.x86_64 -y
sudo yum install gitlab-ee-13.12.6-ee.0.el6.x86_64 -y
sudo yum install gitlab-ee-14.0.5-ee.0.el6.x86_64 -y
sudo yum install gitlab-ee-14.1.2-ee.0.el6.x86_64 -y
sudo gitlab-ctl statusIf everything worked, login with an admin user account, then click on “Menu” > “Admin“. You should see that your GitLab is wonderfully up to date as illustrated below:
Final Happy Musings
The long journey finally ended with a fully working GitLab together with the previous users, their credential and projects intact. We hope you had the same experience in case you were coming all the way from an old legacy server. You may experience other errors and maybe they may not be covered here. So feel free to share them.
Otherwise, we than you for staying to the end and we appreciate the tremendous support and comments we receive. You make it all happen.
Other guides you can look at include:
How To Install GitLab on Rocky Linux 8 With Let’s Encrypt
Install and Configure GitLab CE on Debian 10 (Buster)
Install and Configure GitLab on CentOS 8 / RHEL 8
Configure GitLab FreeIPA LDAP Authentication
