Disaster Recovery

Ensure Disaster Recovery by replicating Automatic Backups across Regions in Oracle Cloud

Introduction

Before we start, I want to mention that Oracle (Active) Data Guard provides the best disaster recovery solution for Oracle Databases with zero data loss and minimal recovery time. Restoring from backup usually comes with a much higher RPO and RTO than what Data Guard provides. However, if restoring from backup satisfies your application’s requirements, then go for it, and have at least a backup copy in a remote region.

Oracle Base Database and Exadata Database Service provide automatic backups to an oracle-managed Object Storage bucket in the same OCI region. An automated way to replicate these backups across regions is not yet provided via Cloud Tooling. However, mv2bucket – Oracle Managed Bucket Content Manager (Doc ID 2723911.1) enables you to access the oracle-managed bucket and replicate the backups to a customer-managed bucket across regions.

This blog post provides a step-by-step guide on using the mv2bucket utility to replicate backups and restore them into a new database across regions.

The Environment

  • VM DB System in Frankfurt region using Oracle Grid Infrastructure, Oracle Database version 19.16, and automatic backups enabled. The database’s unique name is CDBPROD_fra.
  • Object Storage bucket in London region named Backup_CDBPROD_LHR using the standard storage tier.
  • VM DB System in London region using Oracle Grid Infrastructure and Oracle Database version 19.16. This VM will be used to restore the database from backup.

Preparing

Step 1: Install and Configure OCI CLI on Local Host

In a previous blog post, we discussed different ways how to install and configure OCI CLI. Again, for VM DB Systems in Oracle Cloud, it’s highly recommended to follow the manual installation procedure instead of installing OCI CLI via RPM. This ensures that OCI CLI is run in a virtual environment and will not conflict with the RPMs on the database machine, which could lead to issues while updating the operating system at a later point in time.

Check the OCI CLI installation by querying the version:

[opc@hostfra bin]$ oci -v
3.15.2

You should only get the version number as a result, without any error or warning messages, e.g., about the depreciation of python version:

[opc@hostfra bin]$ oci -v
/home/opc/lib/oracle-cli/lib64/python3.6/site-packages/oci/_vendor/httpsig_cffi/sign.py:10: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend  # noqa: F401
3.15.2

The mv2bucket utility uses OCI CLI to make calls to Object Storage. It expects a response in JSON format. If the response contains warning messages like above, mv2bucket will not be able to proceed. So, make sure OCI CLI is working properly by following Step 1: Install Python in this blog post.

If you don’t want to install OCI CLI, you can provide additional parameters “–nocli –ociregion <region_name> –ocitenancy <tenancy_name> –ociid <user_name>” when using mv2bucket. In this case, you will be prompted to provide your Auth Token interactively, e.g.:

#use the tenancy name and user name, not the OCIDs
... --nocli --ociregion eu-frankfurt-1 --ocitenancy ocidb --ociid sinan.petrus.toma@oracle.com

#use your Auth Token (similar to 8wjyYTheD9+cD[Bm<1mQ), not your user login password
Please enter the 'sinan.petrus.toma@oracle.com' Auth Tokens password:
Please re-enter the 'sinan.petrus.toma@oracle.com' Auth Tokens password:

Step 2: Download and Install MV2Bucket on Local Host

Download the latest version of mv2bucket (mv2bucket-2.0.1-11.el7.x86_64.rpm) from the Attachments section of Doc ID 2723911.1 ((OCI) mv2bucket – Oracle Managed Bucket Content Manager):

I uploaded it via SCP to the /home/opc folder on the local database host. From there, install mv2bucket using the RPM (RedHat Package Manager) command as follows:

[opc@hostfra ~]$ sudo rpm -i mv2bucket-2.0.1-11.el7.x86_64.rpm

mv2bucket-2.0.1.11 binary has been installed on /opt/mv2bucket succesfully!

The mv2bucket.bin binary is now available in the /opt/mv2bucket folder:

[opc@hostfra ~]$ ls -l /opt/mv2bucket
total 10180
-rwxr-xr-x 1 root root 10421551 Aug 24 12:23 mv2bucket.bin

Copy the Backups to the Remote Region

Step 3: Get the Backup to the Local File System

Use the mv2bucket get command to get the automatic backups stored in an oracle-managed bucket to a local stage area. As user root, or using sudo: 

[opc@hostfra ~]$ sudo mkdir -p /u01/stage_area/
[opc@hostfra ~]$ sudo /opt/mv2bucket/mv2bucket.bin get --dbuniquename CDBPROD_fra --stage /u01/stage_area/ --incremental
...
INFO: 2022-09-06 17:30:10: Database 'CDBPROD_fra' existence check
...
WARNING: 2022-09-06 17:30:18: Target stage '/u01/stage_area' looks empty, skipping incremental copy
...
INFO: 2022-09-06 17:30:18: Getting '181' Oracle Managed Bucket files, it will take time, please wait...
SUCCESS: 2022-09-06 17:30:37: Files downloaded from Oracle Managed Bucket successfully

Note 1: mv2bucket copies all existing files in the bucket, including full, incremental, archive log, control file backups, etc. The –incremental option means that mv2bucket should only copy the missing files (and not the entire bucket content) to the target.

Note 2: You can copy only backup files specific to a standalone backup by specifying the RMAN TAG in the –filter option which supports wildcards. You also can query the database for a specific backup, extract the file names and copy only those files if needed.

Check the content of the stage area:

[opc@hostfra ~]$ sudo ls -l /u01/stage_area
total 2
drwxr-x--- 1 root root 0 Jan  1  1970 dbSyszf3lhp4q
drwxr-x--- 1 root root 0 Jan  1  1970 file_chunk
drwxr-x--- 1 root root 0 Jan  1  1970 hostfra
drwxr-x--- 1 root root 0 Jan  1  1970 sbt_catalog

The TDE wallet file is also included in the backup. Check the folder /<stage_area>/<host_name>/tdewallet/<db_unique_name>/<db_id>. In this case /u01/stage_area/hostfra/tdewallet/CDBPROD_fra/4203835916

[opc@hostfra ~]$ sudo ls -l /u01/stage_area/hostfra/tdewallet/CDBPROD_fra/4203835916
total 12
-rw-rw---- 1 root root 12121 Sep  6 17:30 TDEWallet.tar.gz

[opc@hostfra ~]$ sudo cp -p /u01/stage_area/hostfra/tdewallet/CDBPROD_fra/4203835916/TDEWallet.tar.gz .

[opc@hostfra ~]$ sudo tar -xf TDEWallet.tar.gz

[opc@hostfra ~]$ ls -l opt/oracle/dcs/commonstore/wallets/CDBPROD_fra/tde/
total 16
-rw------- 1 oracle asmadmin 2555 Sep  5 17:36 ewallet_2022090517365325.p12
-rw------- 1 oracle asmadmin 3995 Sep  5 17:41 ewallet_2022090517411328.p12
-rw------- 1 oracle asmadmin 5467 Sep  5 17:41 ewallet.p12

Step 4: Copy the Stage Area Content to the Remote Bucket

Change the region value in the OCI CLI config file ‘~/.oci/config’ to reflect your remote region. In this case, uk-london-1:

[opc@hostfra ~]$ vi .oci/config
...
region=uk-london-1

Use the mv2bucket load command to copy the content in the stage area on the local file system to a customer-managed Object Storage bucket in a remote region. As user root, or using sudo:

[opc@hostfra ~]$ sudo /opt/mv2bucket/mv2bucket.bin load --bucketname Backup_CDBPROD_LHR --stage /u01/stage_area/ --incremental
...
INFO: 2022-09-06 18:27:52: Target bucket 'Backup_CDBPROD_LHR' existence check
...

Note 3: you can execute both get and load at the same time by using the mv2bucket copy command:

sudo /opt/mv2bucket/mv2bucket.bin copy --dbuniquename CDBPROD_fra --bucketname Backup_CDBPROD_LHR --stage /u01/stage_area/ --incremental

Note 4: you can mount the target Object Storage bucket as a file system using s3fs-fuse as described in this blog post and use the mv2bucket get command to put the backups on that file system. The files are then automatically in the target customer-managed Object Storage bucket.

Check the content of the target bucket Backup_CDBPROD_LHR:

These are the same folders as in /u01/stage_area.

Restore the Database in the Remote Region

Step 5: Restore the Database

In a previous blog post, we described how to restore a database from Automatic Backups across Availability Domains. Follow the same procedure to restore the database in the target region while using the Backup_CDBPROD_LHR bucket this time, which contains a copy of the automatic backup of the source database.

Conclusion

Automatic Backups are configured with a click of a button. The backups are stored in an oracle-managed bucket in the same region as the database. Replication of an oracle-managed bucket across regions is not yet provided by Cloud Tooling. However, you can use the mv2bucket utility to copy the automatic backups to a customer-managed bucket in the same or a remote region. If needed, you can recover the database or create a new database from backups in the remote region.

You benefit from automatic backups but need to copy the backups manually across regions.

Further Reading

Would you like to get notified when the next post is published?