How to use Customer-Managed TDE Encryption Keys in Oracle Exadata Cloud Service

Introduction

Oracle Databases in Oracle Cloud use Transparent Data Encryption (TDE) by default to protect data at rest. The default configuration “Oracle-Managed Keys” stores the master encryption keys in a wallet file locally on the database servers. Now, Oracle also supports the integration with the OCI Vault Service, where you are able to create and control your TDE master encryption keys or import your existing ones.

OCI Vault provides a highly available, durable, and managed service to create and control all your keys from a central location.

Do not confuse OCI Vault with Oracle Key Vault (OKV):

In this blog post, well will see how to create a database that uses customer-managed keys, and how to change the key management for an existing database from oracle-managed keys (using local wallet) to customer-managed keys (using OCI Vault).

The Environment

  • Exadata Cloud Service (ExaCS) X7-2 quarter rack
  • Oracle Database version 19.11 using multitenant architecture

Preparation

Step 1: Create a Vault and an Encryption Key

From the Cloud Console, search for “Vault” and click on the “Vault” service in “Identity & Security”.

Click on “Create Vault”, provide a name for your Vault, and create the Vault. When the Vault becomes active, click on the name of your Vault to see the Vault’s details.

Click on “Create Key”, provide a name for your key, and choose AES 256 as this is the only supported encryption algorithm for TDE customer-managed keys.

Create the key. The key will be available within few seconds.

Step 2: Allow your Exadata Database resources to access the keys in the OCI Vault

Get the Exadata VM Cluster OCID from the VM Cluster details page.

Create a Dynamic Group providing the VM Cluster OCID as a resource.

Any { resource.id = 'ocid1.cloudvmcluster.oc1.phx.abyhqljrxrwolxhj6cqu7ymberiakbj3hj2fwt7hfwlfppeapreaajv3jqna' }

Create a Policy to allow the members of the Dynamic Group to access the keys in OCI Vault.

allow dynamic-group Vault_DG to manage keys in tenancy

Step 3: Allow Access to Oracle Services Network

Configure your Exadata client subnet security list to allow egress traffic to Oracle Services Network. The route table must have the rule to forward the traffic to Oracle Services Network via a Service Gateway.

Step 4: Update the dbaastools to the latest version

Log in to your Exadata virtual machine as user root and execute the following commands to update the dbaastools to the latest version.

sudo -s
rpm -qa | grep dbaas
dbaascli patch tools list
dbaascli patch tools apply --patchid LATEST

Creating a new Database

To use customer-managed keys for a new database, just scroll down to the Advanced Options, click on the “Encryption” tab, choose “Use customer-managed keys” and select the Vault and the Key you created previously.

Ok, that was very simple! Now, let’s have a look at how the TDE configuration does look like when using oracle-managed keys, change the key management to customer-managed keys, and see the changes applied to the TDE configuration.

Change the Key Management Type for an existing Database

Before the Change

Before we change the key management type, let’s have a look at the current configuration when the database uses oracle-managed keys (local wallet file).

-- formatting
set lines 300
set pages 100
col name for a20
col wrl_type for a10
col status for a15
col wallet_order for a15
col key_id for a60
col keystore_type for a20
col origin for a20
col encryptionalg for a15
col encryptedts for a15
col inst_id for 999
col value for a60

-- status of the wallet and the wallet location
SQL> select p.con_id, p.name, p.open_mode, ew.wrl_type, ew.wallet_type, ew.status, ew.wallet_order 
from v$pdbs p join v$encryption_wallet ew on (ew.con_id = p.con_id)
order by p.con_id;

    CON_ID NAME            OPEN_MODE  WRL_TYPE   WALLET_TYPE          STATUS          WALLET_ORDER
---------- --------------- ---------- ---------- -------------------- --------------- ---------------
         2 PDB$SEED        READ ONLY  FILE       AUTOLOGIN            OPEN            SINGLE
         3 PDB1            READ WRITE FILE       AUTOLOGIN            OPEN            SINGLE

-- master encryption key description attributes
SQL> select con_id, key_id, keystore_type, origin from v$encryption_keys;

    CON_ID KEY_ID                                                       KEYSTORE_TYPE        ORIGIN
---------- ------------------------------------------------------------ -------------------- --------------------
         1 ASFX5KQeDE9mvwJzKyhBtD8AAAAAAAAAAAAAAAAAAAAAAAAAAAAA         SOFTWARE KEYSTORE    LOCAL
         3 AfuItHCV5k/Fv1BsWpeHJgoAAAAAAAAAAAAAAAAAAAAAAAAAAAAA         SOFTWARE KEYSTORE    LOCAL

-- encrypted tablespaces
SQL> select t.con_id, t.name, et.encryptionalg, et.encryptedts, et.encryptedkey from v$tablespace t join v$encrypted_tablespaces et on (t.ts# = et.ts#) order by t.con_id, name;

    CON_ID NAME            ENCRYPTIONALG   ENCRYPTEDTS     ENCRYPTEDKEY
---------- --------------- --------------- --------------- ----------------------------------------------------------------
         1 SYSAUX          AES128          YES             0D9F8565ACC8FA55F9622697FE362D4E00000000000000000000000000000000
         1 SYSTEM          AES128          YES             9E1C0695053E874930B89C69681807DB00000000000000000000000000000000
         . ......

-- wallet path and configuration
SQL> select name, value from v$parameter where name in ('wallet_root', 'tde_configuration');

NAME                 VALUE
-------------------- ------------------------------------------------------------
wallet_root          /var/opt/oracle/dbaas_acfs/CDB1911/wallet_root
tde_configuration    keystore_configuration=FILE

-- content of sqlnet.ora
cat $ORACLE_HOME/network/admin/CDB1911/sqlnet.ora
ENCRYPTION_WALLET_LOCATION =
 (SOURCE=
  (METHOD=FILE)
   (METHOD_DATA=
    (DIRECTORY=/var/opt/oracle/dbaas_acfs/CDB1911/wallet_root/tde)))

-- content of the wallet directory
ls -ltr /var/opt/oracle/dbaas_acfs/CDB1911/wallet_root/tde
-rw------- 1 oracle oinstall 5467 Jul 15 07:14 ewallet.p12 --password wallet
-rw------- 1 oracle oinstall 5512 Jul 15 07:14 cwallet.sso --auto login wallet

Changing the Key Management Type to use Customer-Managed Keys

From the database details page, click on “More Actions”, then “Administer Encryption Keys”.

Click on “Change Key Management Type”, select “Use customer-manged keys”, and select the Vault and the Key you created previously.

Click on “Apply”. You we be prompted to confirm your change by typing the database name and clicking on “Change Key Management Type”.

The database goes into the “Updating” status in the UI until the operation completes.

Meanwhile, if you are curious, you can monitor the database alert log in the background.

tail -f /u02/app/oracle/diag/rdbms/cdb1911_phx1qn/CDB19111/trace/alert_CDB19111.log

The conversion to customer-managed keys takes about 15 minutes in total. Changing key management causes the database to become unavailable, as all database instances will be restarted. After the operation completes successfully, the UI will reflect the change. The UI takes further few seconds to show the change.

After the Change

Now, let’s have a look at the new TDE configuration and see what has been changed.

Status of the wallet and the wallet location:

We have additional “OKV” wallet, which is the primary wallet.

However, this is a bit confusing for me, as we are not using OKV (Oracle Key Vault), but the OCI Vault service. These are two very different things.

On another database, which is a non-CDB version 12.1, the wallet type is showing “HSM”, which is the more correct value for OCI Vault in my opinion.

Master encryption key description attributes:

The KEY_ID has been changed as the master encryption key has been rotated. You’ll see a new key version being automatically created for the OCI Vault Key on the key’s details page.

Encrypted tablespaces:

The tablespace encryption keys have been changed, but the encryption algorithm remains AES128. So this is independent of the master encryption key being AES256.

Wallet path and configuration

The wallet_root remains the same, while tde_configuration gets “OKV|FILE” instead of just “FILE”.

sqlnet.ora

METHOD in sqlnet.ora also becomes “OKV”. Here again, for the non-CDB 12.1 database it was “HSM”.

Content of the wallet directory

The password wallet (ewallet.p12) and the auto-login wallet (cwallet.sso) are both still there. The ewallet.p12 still contains the keys.

cd /var/opt/oracle/dbaas_acfs/CDB1911/wallet_root/tde
orapki wallet display -wallet ./ewallet.p12
...
ORACLE.SECURITY.DB.ENCRYPTION.AfuItHCV5k/Fv1BsWpeHJgoAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.ASFX5KQeDE9mvwJzKyhBtD8AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
...

So, how to check whether the keys in OCI vault are being used? This is quite simple. Rename the ewallet.p12 password wallet and restart the database. The restart will be successful as the auto-login wallet is used.

Creating a new tablespace will also be successful:

Creating further Pluggable Databases

When creating a new pluggable database from PDB$SEED, the new PDB will not have any master encryption keys.

SQL> create pluggable database PDB2 admin user pdbadmin identified by MyAdminPW__22;

Pluggable database created.

SQL> alter pluggable database PDB2 open instances=all;

Pluggable database altered.

SQL> alter session set container = PDB2;

Session altered.

SQL> select key_id, keystore_type, origin from v$encryption_keys;

no rows selected

Create a new master encryption key.

SQL> ADMINISTER KEY MANAGEMENT SET KEY FORCE KEYSTORE IDENTIFIED BY MyWalletPW__11 WITH BACKUP;

keystore altered.

SQL> select key_id, keystore_type, origin from v$encryption_keys;

KEY_ID                                   KEYSTORE_TYPE        ORIGIN
---------------------------------------- -------------------- --------------------
06AD09A1F0E7014F21BF9F2150E312A6D5       OKV                  LOCAL

While the command is executing, you’ll see a new key version being created in the UI.

Local PDB Cloning works the same way:

SQL> create pluggable database PDB3 from PDB1 keystore identified by "MyWalletPW__11";

Pluggable database created.

For remote PDB Cloning, the target CDB must also use OCI Vault for key management and use the same OCI Vault key. When you change the key management for the target CDB, you have to choose the same key being used by the source CDB. During the operation, a new key version will be created.

-- on source CDB
SQL> grant create session, sysoper to C##SYSOPER identified by MySYSoperWP__11 container=all;

Grant succeeded.

-- on target CDB
SQL> create database link DBLINK_TO_B connect to C##SYSOPER identified by MySYSoperWP__11 using '(DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.0.2.2)(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = CDB1911B_phx1n8.exacsdns.exavcn.oraclevcn.com)))';

Database link created.

SQL> create pluggable database PDB1RC from PDB1@DBLINK_TO_B keystore identified by "MyWalletPW__11";

Pluggable database created.

Further Considerations

Create a new database from backup

Usually, you are able to create a new database from backup using Cloud Tooling. However, if the database is using OCI Vault for key management, then backups encrypted using customer-manged keys can only be used for the in-place restore.

Export Keys using ADMINISTER KEY MANAGEMENT SQL Command

The ADMINISTER KEY MANAGEMENT SQL command can only be used to export keys from a wallet, but not from OKV or OCI Vault.

SQL> administer key management export keys with secret "MySecretPW__11" to '/home/oracle/cdb.key' force keystore identified by "MyWalletPW__11";

ORA-46650: cannot retrieve information for a master key identifier

If you want to do so, you need to migrate back from the external keystore first.

Migrating Back from an External Keystore

Migrating from customer-managed keys back to oracle-managed keys functionality is not provided via Cloud Tooling. However, you can migrate out of OCI Vault to a local wallet again by following this documentation. The database remains available during this operation.

-- edit the keystore directory location on all nodes, set METHOD=FILE
vi $ORACLE_HOME/network/admin/CDB1911/sqlnet.ora
ENCRYPTION_WALLET_LOCATION =
  (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = /var/opt/oracle/dbaas_acfs/CDB1911/wallet_root/tde/)
    )
  )

SQL> alter system set tde_configuration='KEYSTORE_CONFIGURATION=FILE' scope=both sid='*';

System altered.

-- reverse migrate the keystore, execute on one node
SQL> ADMINISTER KEY MANAGEMENT SET ENCRYPTION KEY IDENTIFIED BY "MyWalletPW__11" REVERSE MIGRATE USING "SYS:MyWalletPW__11" WITH BACKUP;

keystore altered.

-- restart the database instances on all other nodes to reflect the change
srvctl stop instance -db CDB1911_phx1qn -instance CDB19112
srvctl start instance -db CDB1911_phx1qn -instance CDB19112

SQL> select p.con_id, p.name, p.open_mode, ew.wrl_type, ew.wallet_type, ew.status, ew.wallet_order
from v$pdbs p join v$encryption_wallet ew on (ew.con_id = p.con_id)
order by p.con_id; 

    CON_ID NAME            OPEN_MODE  WRL_TYPE   WALLET_TYPE          STATUS     WALLET_ORDER
---------- --------------- ---------- ---------- -------------------- ---------- ---------------
         2 PDB$SEED        READ ONLY  FILE       PASSWORD             OPEN       SINGLE
         3 PDB1            READ WRITE OKV        OKV                  OPEN       SECONDARY
         3 PDB1            READ WRITE FILE       PASSWORD             OPEN       PRIMARY
-- password wallet is the primary now

-- change the keystore password if required (do not change the password if you want to "reverse reverse migrate", check next topic below)
SQL> ADMINISTER KEY MANAGEMENT ALTER KEYSTORE PASSWORD IDENTIFIED BY "MyWalletPW__11" SET "MyWalletPW__22" WITH BACKUP USING 'pwd_change';

keystore altered.

-- export the master encryption key
SQL> administer key management export keys with secret "MySecretPW__22" to '/home/oracle/cdb.key' force keystore identified by "MyWalletPW__22";

keystore altered.

Keep in mind that this is a manual operation outside of the Cloud Tooling, so the changes will not be synced back to the cloud control plane, and the UI would still show “customer-managed keys”.

“Reverse Reverse Migrate”

What if you want to get back to OCI Vault again? There is no option in the Cloud Console for that, but you can use the dbaascli command line tool to achieve this. And remember, from the cloud control plane point’s of view, the database IS using OCI Vault.

The database becomes unavailable during this operation, as all database instances will be restarted. It takes around 5-7 minutes to complete.

Note: The database name must be in uppercase for the db wallet (cwallet.sso) to be found in /var/opt/oracle/dbaas_acfs/CDB1911/db_wallet/, which contains the TDE password. This is why you don’t need to provide the TDE password in the dbaascli command.

-- as root user
KEY_OCID=ocid1.key.oc1.phx.bbqccvs3aafqw.abyhqljr7pca4v7beztarxj2kd6ekhwp5h3nco65wl6n5csp6emluegjvdbq
DBNAME=CDB1911
dbaascli tde file_to_hsm --kmsKeyOCID $KEY_OCID --dbname $DBNAME --precheckOnly yes
dbaascli tde file_to_hsm --kmsKeyOCID $KEY_OCID --dbname $DBNAME --precheckOnly no

# if you changed the password during reverse migrate, then:
# set it back to the original value using "dbaascli tde changePassword --dbname $DBNAME", this will update the db wallet as well.
# or revert it back in the database: ADMINISTER KEY MANAGEMENT ALTER KEYSTORE PASSWORD IDENTIFIED BY MyWalletPW__22 SET MyWalletPW__11 WITH BACKUP;

And here we are, OCI Vault is primary again:

select p.con_id, p.name, p.open_mode, ew.wrl_type, ew.wallet_type, ew.status, ew.wallet_order
from v$pdbs p join v$encryption_wallet ew on (ew.con_id = p.con_id)
order by p.con_id;

    CON_ID NAME            OPEN_MODE  WRL_TYPE   WALLET_TYPE          STATUS          WALLET_ORDER
---------- --------------- ---------- ---------- -------------------- --------------- ---------------
         2 PDB$SEED        READ ONLY  FILE       AUTOLOGIN            OPEN            SINGLE
         2 PDB$SEED        READ ONLY  OKV        OKV                  OPEN            SINGLE
         3 PDB1            READ WRITE FILE       AUTOLOGIN            OPEN            SECONDARY
         3 PDB1            READ WRITE OKV        OKV                  OPEN            PRIMARY

METHOD is set to OKV in the sqlnet.ora automatically on all nodes.

Conclusion

Using OCI Vault for managing your TDE encryption keys is quite simple, whether immediately at database creation time or changing the key management afterwards. Just create or import your own key in the Vault service and select it at database creation time or afterwards when changing the key management.

Once you are using OCI Vault (customer-managed keys), you cannot switch back to use a local wallet (oracle-managed keys) using cloud tooling. However, reserving this operation is possible using SQL commands. It’s even possible to switch back to OCI Vault again using the dbaascli command line tool.

Last but not least, do NOT confuse OCI Vault with Oracle Key Vault (OKV).

Further Reading

Would you like to get notified when the next post is published?