To overcome these difficulties and improve the availability of our single-instance databases Oracle introduced Oracle Restart. This feature appeared with 11.2.0.1 and can be used to start any database, ASM instance, Net listener and more. On top of this Oracle Restart keeps an eye on all components that it started. Should any of the components fail, Oracle Restart tries to restart it and bring it back to life.

Oracle Restart runs out of the Grid infrastructure home. When you install Oracle Grid Infrastructure for a standalone server and then install and create your database, the database will automatically be added to the Oracle Restart configuration.

Configuring Oracle Restart for existing database

Adding Oracle Restart to an already installed database requires a little bit more effort. You should start by installing Oracle Grid Infrastructure. In this tutorial I will add Oracle Restart to an existing 11gR2 database. The host is running Oracle Linux 5 and the database is installed in /u01/app/oracle/product/11.2.0/db_orcl.

You shouldn't forget that installing Oracle Grid Infrastructure (even for a standalone installation) requires us to provide space on Oracle ASM for clusterware files.

When installing Oracle Grid Infrastructure select the "Configure Oracle Grid Infrastructure for a Standalone Server" option. During the installation process you will be asked to run the root.sh script. It will put stuff like the oraenv script in place, perform some root actions related to network and most important - it will make the initial configuration of Oracle Clusterware.

[root@el5 ~]# /u01/app/oracle/product/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/oracle/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'oracle', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node el5 successfully pinned.
Adding Clusterware entries to inittab

el5     2012/02/18 06:56:51     /u01/app/oracle/product/11.2.0/grid/cdata/el5/backup_20120218_065651.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
[root@el5 ~]# 

If you want to confirm that Oracle Restart is running, go to the bin directory inside the Grid installation and use the crsctl utility. The check has command will display the current Oracle Restart status (in case you wonder, has stands for High Availability Services).

[oracle@el5 bin]$ ./crsctl check has
CRS-4638: Oracle High Availability Services is online
[oracle@el5 bin]$ 

If you want to confirm that Oracle Restart will start automatically with the next reboot, use the config has command.

[oracle@el5 bin]$ ./crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
[oracle@el5 bin]$

The following table summarizes the crsctl commands relevant to Oracle Restart.

Registering a database with Oracle Restart

When performing a new database installation and Oracle Grid Infrastructure is already in place, there is no need for manual components registration. There are however three cases where you have to manually add the database and listener to Oracle Restart:

• when installing the Grid Infrastructure after the Oracle Database (our case)
• when creating databases outside DBCA (for example - by running CREATE DATABASE)
• when creating database services outside SRVCTL (by executing DBMS_SERVICE.CREATE_SERVICE)

To start, stop and configure components with Oracle Restart we use the srvctl command. There is something very important that you should note - there is more than one srvctl executable. Which one to use depends on the component that you want to deal with. The next table summarizes the different options.

The OS User column points out which user you should use when changing the component's configuration. If you simply want to start or stop the component via Oracle Restart than any user who can run srvctl would do.

Let's start with adding the database. In my installation the user who owns the database is called oracle. I will call srvctl with the add database command in order to register my existing database with Oracle Restart.

[oracle@el5 ~]$ cd /u01/app/oracle/product/11.2.0/db_orcl/bin/
[oracle@el5 bin]$ ./srvctl add database -d orcl -o /u01/app/oracle/product/11.2.0/db_orcl
[oracle@el5 bin]$ 

If you want to confirm that the database was registered you can call srvctl with the config option. This will show a list of components that are configured with Oracle Restart.

[oracle@el5 bin]$ ./srvctl config
orcl
[oracle@el5 bin]$

Providing a database name to the config command displays more details about the configuration of this specific database.

[oracle@el5 bin]$ ./srvctl config database -d orcl
Database unique name: orcl
Database name:
Oracle home: /u01/app/oracle/product/11.2.0/db_orcl
Oracle user: oracle
Spfile:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Disk Groups:
Services:
[oracle@el5 bin]$ 

Testing if Oracle Restart actually works

For testing if Oracle Restart works correctly we could restart the host and see if the database comes up automatically, but there is a quicker and more interesting way to do the test. We said in the beginning of this article that Oracle Restart is not only responsible for starting and stopping the database, but it also constantly monitors the registered components for a failure and restarts them if necessary. To demonstrate this I will kill on of the background database processes (Oracle Recovery Process - RECO) and see if Oracle Restart reacts as expected.

Let's start by finding the process id.

[root@el5 ~]# ps U oracle |grep reco
oracle    6710  0.0  0.9 790876 19968 ?        Ss   07:21   0:00 ora_reco_orcl
[root@el5 ~]#

In my case the ora_reco_orcl process has the id of 6710.

[root@el5 ~]# kill -9 6710
[root@el5 ~]# ps U oracle |grep reco
[root@el5 ~]#

You can see that the process is no longer available. If everything is configured correctly, the process should quickly come back (took around 8 seconds on my test installation). Calling ps again shows:

[root@el5 ~]# ps U oracle |grep reco
10771 ?        Ss     0:00 ora_reco_orcl
[root@el5 ~]#

Adding a listener to Oracle Restart

When Oracle Grid Infrastructure is installed it configures a default listener on port 1521. The ASM+ instance registers with this listener and if there is no static registration configured for the database it will register with this listener as well.

If you want to configure another listener you have to edit the $ORACLE_HOME/networ/listener.ora and add a record for the new listener there. Let's say I want to configure another listener on port 1522. I have to add the following lines to my listener.ora file:

LISTENER_ORCL =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = el5)(PORT = 1522))
    )
  )

My next step is to add the listener to Oracle Restart via srvctl.

[oracle@el5 bin]$ ./srvctl add listener -l LISTENER_ORCL -o /u01/app/oracle/product/11.2.0/db_orcl -p "TCP:1522/IPC:ORCL" 
[oracle@el5 bin]$ 

After that I can configure a static database registration with the newly created listener. First I need a record about the listener in tnsnames.ora.

LISTENER_ORCL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = el5)(PORT = 1522))
  )

Then I have to change the LOCAL_LISTENER parameter of the database, setting its value to the name of my new listener (LISTENER_ORCL).

[oracle@el5 bin]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Feb 21 07:03:35 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter system set LOCAL_LISTENER="LISTENER_ORCL" scope=spfile;

System altered.

SQL>

I have to restart the database for the changes to take effect. I will also start LISTENER_ORCL.

[oracle@el5 bin]$ ./srvctl stop database -d orcl
[oracle@el5 bin]$ ./srvctl start listener -l LISTENER_ORCL
[oracle@el5 bin]$ ./srvctl start database -d orcl
[oracle@el5 bin]$ 

Running lsnrctl shows that LISTENER_ORCL runs on port 1522 and the orcl instance is correctly registered with it.

[oracle@el5 bin]$ ./lsnrctl status LISTENER_ORCL

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 21-FEB-2012 07:21:42

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=el5)(PORT=1522)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_ORCL
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                21-FEB-2012 07:21:22
Uptime                    0 days 0 hr. 0 min. 19 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/db_orcl/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/11.2.0/db_orcl/log/diag/tnslsnr/el5/listener_orcl/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=el5)(PORT=1522)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=ORCL)))
Services Summary...
Service "orcl" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@el5 bin]$ 

Starting and stopping components via srvctl

Starting and stopping components is done via the srvctl command. Although you can still start different components in the conventional way (lsnrctl for listeners, SQL*Plus for databases etc.), using srvctl is the recommended approach. There are two important reasons for this. First, Oracle Restart will not monitor any components that were not started with srvctl. Second, if you start a component via srvctl, Oracle Restart will automatically start any components on which the specified component depends.

The syntax for starting and stopping components is as follows.

srvctl [start|stop] [asm|database|diskgroup|home|listener|ons|service] options

Starting the orcl database via srvctl looks like this.

srvctl start database -d orcl

You can pass various options with the -o switch. For example, starting the database in MOUNT mode is done like this:

srvctl start database -d orcl -o mount

Starting an ASM instance is similar.

srvctl start asm

Or if you want it in NOMOUNT mode:

srvctl start asm -o nomount

For stopping components we use the stop command. Here are two examples.

srvctl stop database -d orcl -o immediate

srvctl stop asm -o abort -f

The -f option enables srvctl to stop (unmount) the diskgroups before stopping ASM.

You can also start and stop all components managed by Oracle Restart in a given Oracle home.

srvctl stop home -o oracle_home -s state_file [-t stop_options] [-f]

Here is an example of stopping everything in /u01/app/oracle/product/11.2.0/db_orcl

[oracle@el5 ~]$ srvctl stop home -o /u01/app/oracle/product/11.2.0/db_orcl -s /home/oracle/orcl_state
[oracle@el5 ~]$ 

The state file (orcl_state) is created by srvctl. It contains a list of the components stopped by srvctl and is used for identifying which components should be started when calling srvctl start home.

[oracle@el5 ~]$ srvctl start home -o /u01/app/oracle/product/11.2.0/db_orcl -s /home/oracle/orcl_state
[oracle@el5 ~]$ 

For a complete description of all options for srvctl you should check the SRVCTL Command Reference for Oracle Restart.

Environment variables

If you have to set some environment variables (like ORACLE_SID, ORACLE_HOME etc.) before starting your database, you should know that Oracle Restart can automatically perform this task for you. In fact it can store different sets of variables for any listener, database or ASM instance.

Setting and unsetting variables is done via srvctl, as it is part of the component's configuration. Using srvctl getenv database lists all variables that are currently configured for the database.

[oracle@el5 ~]$ srvctl getenv database -d orcl
orcl:
[oracle@el5 ~]$ 

Adding new variables is done with the setenv command. Let's add ORACLE_SID and ORACLE_HOME to the configuration of orcl.

[oracle@el5 ~]$ srvctl setenv database -d orcl -t "ORACLE_SID=orcl,ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_orcl"
[oracle@el5 ~]$ 

Running srvctl getenv again shows the newly configured variables.

[oracle@el5 ~]$ srvctl getenv database -d orcl
orcl:
ORACLE_SID=orcl
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_orcl
[oracle@el5 ~]$ 

Configuration maintenance

Oracle Restart's configuration is stored into the Oracle Local Registry (OLR). OLR was introduced with 11gR2 and it is a repository similar to OCR, but is not shared among other RAC nodes. It is instead located on a local storage and holds information specific to the individual host.

You can check the status of OLR using the ocrcheck utility (you can find it in /bin in the Grid Infrastructure home).

[root@el5 bin]# ./ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2384
         Available space (kbytes) :     259736
         ID                       : 1621070741
         Device/File Name         : /u01/app/oracle/product/11.2.0/grid/cdata/localhost/el5.olr
                                    Device/File integrity check succeeded

         Local registry integrity check succeeded

         Logical corruption check succeeded

[root@el5 bin]# 

If you are interested in what's in the OLR you can dump it to a file (or bring in on screen by using -stdout instead of a file name, but be careful - my OLR for this example was around 2300 lines).

[root@el5 bin]# ./ocrdump -local olr_export
[root@el5 bin]# cat olr_export
[SYSTEM]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall}

[SYSTEM.crs]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall}

…

[CRS]
UNDEF :
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall}

[root@el5 bin]# 

There is no automated backup for the Oracle Local Registry. Backups are taken only during install and upgrade processes. You can see a list of the OLR backups by using the ocrconfig command.

[root@el5 bin]# ./ocrconfig -local -showbackup

el5     2012/02/18 06:56:51     /u01/app/oracle/product/11.2.0/grid/cdata/el5/backup_20120218_065651.olr
[root@el5 bin]# 

You can take a manual backup of the repository at any time. This is done with the following command:

[root@el5 bin]# ./ocrconfig -local -manualbackup

el5     2012/03/04 07:49:06     /u01/app/oracle/product/11.2.0/grid/cdata/el5/backup_20120304_074906.olr

el5     2012/02/18 06:56:51     /u01/app/oracle/product/11.2.0/grid/cdata/el5/backup_20120218_065651.olr
[root@el5 bin]# 

Keep in mind that before restoring an OLR backup you should turn off the High Availability Services.

# crsctl stop crs
# ocrconfig -local -restore

# ocrcheck -local
# crsctl start crs

There is no command to remove an entry for the backup of the OLR. The default behavior for OLR backups is to maintain only the 5 most recent backups. Although the older backups are not reported by the ocrconfig utility, they are still kept as an OS files. It is your responsibility to schedule some clean up for these files.

Final remarks

You will be pleased to know Oracle Restart is integrated with Oracle Data Guard and the Data Guard Broker. When a database shutdown and restart is required in response to a role change request, Oracle Restart shuts down and restarts the database in an orderly fashion. Oracle Restart also ensures that, following a Data Guard role transition, all database services configured to run in the new database role are active and all services not configured to run in the new role are stopped.
Another feature of Oracle Restart is that it uses Oracle Notification Services (ONS) and Oracle Advanced Queues to publish Fast Application Notification high availability events. This allows the clients to automate the failover to a standby database, when a service or instance goes down.

For more information you should check the "Oracle Restart Integration with Oracle Data Guard" and "Fast Application Notification with Oracle Restart" sections of the Oracle Database Administrator's Guide 11g Release 2 (11.2).

At the end, keep in mind that Oracle development has confirmed that Oracle Restart is only intended for managing of single-instance (non-RAC) 11.2 Oracle products only. This is described in Metalink note ID 954552.1.

If we want to keep an exact replica of a production database we have to take care for three things. First, we have to ship the changes (archive logs) to the failover database. Second, we have to keep track of what was shipped, so we know what needs to be recovered if something goes wrong. Third, we have to apply the changes at the failover database.

Before 11gR2 came out, detecting and shipping log files between hosts had to be done outside of the database. In GNU/Linux environments most people use rsync or a similar program to do the job. On the other hand I always prefer to do such tasks within the database and relaying on the OS is not always an option (what if one of the databases runs on Windows?). In this tutorial I will show you how to pickup archivelogs by using File Watchers (introduced in 11gR2) and transfer them via FTP to a remote host.

Demonstration scenario

I will be using two hosts, both running Oracle Linux 5.5. The one with the production database is called el5-prd and the one that will host the failover database is called el5-backup. The production host has Database 11gR2 installed. There is a default database configured and it includes the sample schemas. The el5-backup has only the Oracle software installed. Both installations reside in /u01/app/oracle/product/11.2.0/db_orcl and the software owner is user oracle.

We should perform the following steps to build the failover configuration:

1. Set the production database in archivelog mode.
2. Perform full database backup and create copies of control and parameter files.
3. Prepare the failover server for restore - setup database directories, copy the backup, control, parameter and password file. Create a listener file.
4. Restore the backup on el5-backup.
5. Install ftpd on el5-backup.
6. Setup ACLs for FTP transfer and install FTP packages on el5-prd.
7. Setup archivelog directory and test FTP transfer from the production host.
8. Setup a file watcher on el5-prd.
9. Test that archivelogs are shipped.
10. Setup a mechanism to apply & delete the shipped logs on el5-backup.

The list is quite long, so let's begin.

Set the production database in archivelog mode

First we have to create an OS directory, where the database should write the archivelogs.

Login to the production hosts as the oracle software owner and create an empty directory. I will create a directory named archivelog in my FRA.

[oracle@el5-prd ~]$ mkdir /u01/app/oracle/fast_recovery_area/ORCL/archivelog
[oracle@el5-prd ~]$

Next, put the database in archivelog mode.

[oracle@el5-prd]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Wed Dec 14 07:24:02 2011
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Release 11.2.0.3.0 - Production

SQL> alter system set log_archive_dest_1='LOCATION=/u01/app/oracle/fast_recovery_area/ORCL/archivelog/' scope=spfile;

System altered.

SQL> alter system set log_archive_format='%t_%s_%r.arc' scope=spfile;

System altered.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area  422670336 bytes
Fixed Size                  1345380 bytes
Variable Size             264243356 bytes
Database Buffers          150994944 bytes
Redo Buffers                6086656 bytes
Database mounted.
SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

SQL> alter database force logging;

Database altered.

SQL> exit
Disconnected from Oracle Database 11g Release 11.2.0.3.0 - Production
[oracle@el5-prd]$

Perform full database backup and create copies of control and parameter files

We perform a full backup of the production database by using RMAN.

[oracle@el5-prd ~]$ rman target=/

Recovery Manager: Release 11.2.0.3.0 - Production on Sat Dec 10 14:24:16 2011

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1297199097)

RMAN> backup database plus archivelog;

Starting backup at 10-DEC-11
current log archived
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=40 device type=DISK
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=6 RECID=1 STAMP=769530270
channel ORA_DISK_1: starting piece 1 at 10-DEC-11
channel ORA_DISK_1: finished piece 1 at 10-DEC-11
piece handle=/u01/app/oracle/fast_recovery_area/ORCL/backupset/2011_12_10/o1_mf_annnn_TAG20111210T142431_7g6mw03l_.bkp tag=TAG20111210T142431 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 10-DEC-11

Starting backup at 10-DEC-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/u01/app/oracle/oradata/orcl/system01.dbf
input datafile file number=00002 name=/u01/app/oracle/oradata/orcl/sysaux01.dbf
input datafile file number=00005 name=/u01/app/oracle/oradata/orcl/example01.dbf
input datafile file number=00003 name=/u01/app/oracle/oradata/orcl/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/orcl/users01.dbf
channel ORA_DISK_1: starting piece 1 at 10-DEC-11
channel ORA_DISK_1: finished piece 1 at 10-DEC-11
piece handle=/u01/app/oracle/fast_recovery_area/ORCL/backupset/2011_12_10/o1_mf_nnndf_TAG20111210T142433_7g6mw1lw_.bkp tag=TAG20111210T142433 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:26
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current control file in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 10-DEC-11
channel ORA_DISK_1: finished piece 1 at 10-DEC-11
piece handle=/u01/app/oracle/fast_recovery_area/ORCL/backupset/2011_12_10/o1_mf_ncsnf_TAG20111210T142433_7g6mytqr_.bkp tag=TAG20111210T142433 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 10-DEC-11

Starting backup at 10-DEC-11
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=7 RECID=2 STAMP=769530364
channel ORA_DISK_1: starting piece 1 at 10-DEC-11
channel ORA_DISK_1: finished piece 1 at 10-DEC-11
piece handle=/u01/app/oracle/fast_recovery_area/ORCL/backupset/2011_12_10/o1_mf_annnn_TAG20111210T142604_7g6myw91_.bkp tag=TAG20111210T142604 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 10-DEC-11

RMAN> exit

Recovery Manager complete.
[oracle@el5-prd ~]$

Next we create copies of the control and parameter file, placing them in the oracle user home directory.

[oracle@el5-prd]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Sat Dec 10 14:28:45 2011

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 - Production

SQL> alter database create standby controlfile as '/home/oracle/orcl-backup.ctl'; 
Database altered.

SQL> create pfile='/home/oracle/initORCL-backup.ora' from spfile;

File created.

SQL>

Prepare the failover server for restore

After you have completed a "software only" installation of Database 11gR2 you have to create the following directories that are needed for successfully restoring the database backup:

[oracle@el5-backup ~]$ mkdir -p /u01/app/oracle/oradata/ORCL
[oracle@el5-backup ~]$ mkdir -p /u01/app/oracle/fast_recovery_area/ORCL
[oracle@el5-backup ~]$ mkdir -p /u01/app/oracle/admin/ORCL/adump

Next we take the control file copy from the production server.

[oracle@el5-backup ~]$ scp oracle@el5-prd:/home/oracle/orcl-backup.ctl /u01/app/oracle/oradata/ORCL/control01.ctl
oracle@el5-prd's password:
orcl-backup.ctl                               100% 9520KB   9.3MB/s   00:01
[oracle@el5-backup ~]$

Another copy of the control file goes to the FRA.

[oracle@el5-backup ~]$ cp /u01/app/oracle/oradata/ORCL/control01.ctl /u01/app/oracle/fast_recovery_area/ORCL/control02.ctl
[oracle@el5-backup ~]$

We also need the parameter and the password file.

[oracle@el5-backup ~]$ scp oracle@el5-prd:/home/oracle/initORCL-backup.ora /home/oracle/
oracle@el5-prd's password:
initORCL-backup.ora                           100%  945     0.9KB/s   00:00
[oracle@el5-backup ~]$ scp -r oracle@el5-prd:/u01/app/oracle/product/11.2.0/db_orcl/dbs/orapwORCL /u01/app/oracle/product/11.2.0/db_orcl/dbs/orapwORCL
oracle@el5-prd's password:
orapwORCL                                     100% 1536     1.5KB/s   00:00
[oracle@el5-backup ~]$

Let's copy the archivelogs and the backup as well.

[oracle@el5-backup ~]$ scp -r oracle@el5-prd:/u01/app/oracle/fast_recovery_area/ORCL/archivelog /u01/app/oracle/fast_recovery_area/ORCL/
oracle@el5-prd's password:
o1_mf_1_7_7g7hwtfw_.arc                       100%   23KB  22.5KB/s   00:00
o1_mf_1_6_7g7hs8tx_.arc                       100% 4085KB   4.0MB/s   00:00
[oracle@el5-backup ~]$
[oracle@el5-backup ~]$ scp -r oracle@el5-prd:/u01/app/oracle/fast_recovery_area/ORCL/backupset /u01/app/oracle/fast_recovery_area/ORCL/
oracle@el5-prd's password:
o1_mf_annnn_TAG20111210T222058_7g7hsbnq_.bkp  100% 4086KB   4.0MB/s   00:00
o1_mf_annnn_TAG20111210T222250_7g7hwv01_.bkp  100%   24KB  24.0KB/s   00:00
o1_mf_ncsnf_TAG20111210T222059_7g7hws7f_.bkp  100% 9600KB   9.4MB/s   00:00
o1_mf_nnndf_TAG20111210T222059_7g7hsdkp_.bkp  100% 1172MB  13.6MB/s   01:26
[oracle@el5-backup ~]$

The final set of files are the redo log files.

[oracle@el5-backup ~]$ scp oracle@el5-prd:/u01/app/oracle/oradata/ORCL/redo* /u01/app/oracle/oradata/ORCL
oracle@el5-prd's password:
redo01.log                                    100%   50MB  16.7MB/s   00:03
redo02.log                                    100%   50MB  25.0MB/s   00:02
redo03.log                                    100%   50MB  10.0MB/s   00:05
[oracle@el5-backup ~]$

The last thing we have to do is to create a listener.ora file.

[oracle@el5-backup ~]$ cat >> /u01/app/oracle/product/11.2.0/db_orcl/network/admin/listener.ora << EOF > LISTENER = > (DESCRIPTION_LIST = > (DESCRIPTION = > (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521)) > (ADDRESS = (PROTOCOL = TCP)(HOST = el5-backup)(PORT = 1521)) > ) > ) > > ADR_BASE_LISTENER = /u01/app/oracle > EOF
[oracle@el5-backup ~]$

As you can see, the listener for our failover database will use the default 1521 port. Time to restore from the backup.

Restore the backup on el5-backup

Before running the restore we have to start up and bring the failover database to a mount state. The first step is to start the listener on el5-backup.

[oracle@el5-backup ~]$ lsnrctl start

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 10-DEC-2011 22:45:13

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/11.2.0/db_orcl/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.3.0 - Production
System parameter file is /u01/app/oracle/product/11.2.0/db_orcl/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/el5-backup/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=el5-backup)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                10-DEC-2011 22:45:15
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/db_orcl/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/el5-backup/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=el5-backup)(PORT=1521)))
The listener supports no services
The command completed successfully
[oracle@el5-backup ~]$

Next we have to set the SID and create a SPFILE from the paramater file that we have in our home directory.

[oracle@el5-backup ~]$ export ORACLE_SID=ORCL
[oracle@el5-backup ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Sat Dec 10 22:45:52 2011

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> create spfile from pfile='/home/oracle/initORCL-backup.ora';

File created.

SQL>

We can now restore the database by using RMAN.

[oracle@el5-backup ~]$ rman target=/

Recovery Manager: Release 11.2.0.3.0 - Production on Sat Dec 10 22:47:11 2011

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database (not started)

RMAN> startup mount;

Oracle instance started
database mounted

Total System Global Area     422670336 bytes

Fixed Size                     1345380 bytes
Variable Size                268437660 bytes
Database Buffers             146800640 bytes
Redo Buffers                   6086656 bytes

RMAN> restore database;

Starting restore at 10-DEC-11
Starting implicit crosscheck backup at 10-DEC-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 device type=DISK
Crosschecked 4 objects
Finished implicit crosscheck backup at 10-DEC-11

Starting implicit crosscheck copy at 10-DEC-11
using channel ORA_DISK_1
Finished implicit crosscheck copy at 10-DEC-11

searching for all files in the recovery area
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /u01/app/oracle/fast_recovery_area/ORCL/archivelog/2011_12_10/o1_mf_1_7_7g7hwtfw_.arc
File Name: /u01/app/oracle/fast_recovery_area/ORCL/archivelog/2011_12_10/o1_mf_1_6_7g7hs8tx_.arc

using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/ORCL/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /u01/app/oracle/oradata/ORCL/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /u01/app/oracle/oradata/ORCL/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /u01/app/oracle/oradata/ORCL/users01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /u01/app/oracle/oradata/ORCL/example01.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/fast_recovery_area/ORCL/backupset/2011_12_10/o1_mf_nnndf_TAG20111210T222059_7g7hsdkp_.bkp
channel ORA_DISK_1: piece handle=/u01/app/oracle/fast_recovery_area/ORCL/backupset/2011_12_10/o1_mf_nnndf_TAG20111210T222059_7g7hsdkp_.bkp tag=TAG20111210T222059
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:01:36
Finished restore at 10-DEC-11

RMAN> exit

Recovery Manager complete.
[oracle@el5-backup ~]$

We successfully created an identical copy of the production database.

Install ftpd on el5-backup

Installing the FTP daemon on Oracle Linux is pretty straightforward.

[root@el5-backup ~]# yum install vsftpd
Loaded plugins: security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package vsftpd.i386 0:2.0.5-16.el5_4.1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package        Arch         Version                  Repository           Size
================================================================================
Installing:
 vsftpd         i386         2.0.5-16.el5_4.1         el5_u5_base         140 k

Transaction Summary
================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)

Total download size: 140 k
Is this ok [y/N]: Y
Downloading Packages:
vsftpd-2.0.5-16.el5_4.1.i386.rpm                         | 140 kB     00:01
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : vsftpd                                                   1/1

Installed:
  vsftpd.i386 0:2.0.5-16.el5_4.1

Complete!
[root@el5-backup ~]#

You should not forget to reconfigure the firewall on the failover server to allow FTP communication. First add ip_conntrack_ftp to the IPTABLES_MODULES line in /etc/sysconfig/iptables-config. The line in iptables-config should look like this:

IPTABLES_MODULES="ip_conntrack_netbios_ns ip_conntrack_ftp"

Next, edit /etc/sysconfig/iptables and add a rule for the FTP traffic (be sure to put the line before the REJECT rule). The line you have to add in iptables looks like this:

-A RH-Firewall-1-INPUT -m state --state NEW -p tcp --dport 21 -j ACCEPT

Bounce iptables and set the FTP service to autostart with the server.

[root@el5-backup ~]# service iptables restart
Flushing firewall rules:                                   [  OK  ]
Setting chains to policy ACCEPT: filter                    [  OK  ]
Unloading iptables modules:                                [  OK  ]
Applying iptables firewall rules:                          [  OK  ]
Loading additional iptables modules: ip_conntrack_netbios_n[  OK  ]
[root@el5-backup ~]# chkconfig vsftpd on
[root@el5-backup ~]# service vsftpd start
Starting vsftpd for vsftpd:                                [  OK  ]
[root@el5-backup ~]#

You might want to test the access from el5-prd to the failover server.

[oracle@el5-prd ~]$ ftp el5-backup
Connected to el5-backup.
220 (vsFTPd 2.0.5)
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (el5-backup:oracle): oracle
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> bye
221 Goodbye.
[oracle@el5-prd ~]$

Setup ACLs for FTP transfer and install FTP packages on el5-prd

Our next task is to prepare the production server for communicating with el5-backup over FTP. We start by creating a dedicated database user that will be used for shipping and tracking the archivelog files. I will name it logship.

[oracle@el5-prd ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Wed Dec 14 07:24:02 2011

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 - Production

SQL> create user logship identified by logship;

User created.

SQL> grant connect, resource to logship;

Grant succeeded.

SQL>

Next we should configure an Access Control List (ACL) that will allow FTP connections to el5-backup for user logship. We have to use the CREATE_ACL, ADD_PRIVILEGE and ASSIGN_ACL procedures from the DBMS_NETWORK_ACL_ADMIN package. We will call the procedures with the following parameters:

DBMS_NETWORK_ACL_ADMIN.CREATE_ACL (
    acl          => 'ftp.xml',
    description  => 'Allow FTP connections',
    principal    => 'SYSTEM',
    is_grant     => TRUE,
    privilege    => 'connect',
    start_date   => SYSTIMESTAMP,
    end_date     => NULL);

 DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE (
    acl         => 'ftp.xml',
    principal   => 'LOGSHIP',
    is_grant    => FALSE,
    privilege   => 'connect',
    position    => NULL,
    start_date  => NULL,
    end_date    => NULL);

  DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL (
    acl         => 'ftp.xml',
    host        => 'el5-backup',
    lower_port  => NULL,
    upper_port  => NULL);

Here is an output of their execution:

SQL> exec dbms_network_acl_admin.create_acl (acl => 'ftp.xml', description => 'Allow FTP connections', principal => 'LOGSHIP', is_grant => TRUE, privilege => 'connect', start_date => SYSTIMESTAMP,end_date => NULL);

PL/SQL procedure successfully completed.

SQL> exec dbms_network_acl_admin.add_privilege (acl => 'ftp.xml', principal => 'LOGSHIP', is_grant => FALSE, privilege => 'connect', position => NULL, start_date => NULL, end_date => NULL);

PL/SQL procedure successfully completed.

SQL> exec dbms_network_acl_admin.assign_acl (acl => 'ftp.xml', host => 'el5-backup', lower_port => NULL,upper_port => NULL);

PL/SQL procedure successfully completed.

SQL>

For connecting to el5-backup from the production database we will be using the FTP API developed by Tim Hall. You need to download the FTP package and the package body creation scripts and run them as user logship.

SQL> conn logship/logship;
Connected.
SQL> @ftp.pks;

Package created.

No errors.
SQL> @ftp.pkb;

Package body created.

No errors.
SQL>

Setup archivelog directory and test FTP transfer

We move on by creating a directory object within the production database that points to the location of the archivelog files.

[oracle@el5-prd ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Sun Dec 11 08:44:57 2011

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 - Production

SQL> create directory arc_dir as '/u01/app/oracle/fast_recovery_area/ORCL/archivelog';

Directory created.

SQL> grant read on directory arc_dir to logship;

Grant succeeded.

SQL>

It is a good idea to test the FTP communication from within the database. You can create a dummy test file in the archivelog dir:

[oracle@el5-prd ~]$ cat >> /u01/app/oracle/fast_recovery_area/ORCL/archivelog/testfile.txt << EOF > FTP test file > EOF
[oracle@el5-prd ~]$

You can then connect as user logship and run the following PL/SQL block:

declare l_conn utl_tcp.connection;
begin
  l_conn := ftp.login('el5-backup','21','oracle','welcome1');
  ftp.put(p_conn => l_conn, p_from_dir  => 'ARC_DIR', p_from_file => 'testfile.txt', p_to_file => '/u01/app/oracle/fast_recovery_area/ORCL/archivelog/testfile.txt');
  ftp.logout(l_conn);
end;
/

If everything goes fine the testfile.txt will appear at el5-backup.

[oracle@el5-backup ~]$ cd /u01/app/oracle/fast_recovery_area/ORCL/archivelog/
[oracle@el5-backup archivelog]$ cat testfile.txt
FTP test file
[oracle@el5-backup archivelog]$

Setup a file watcher on el5-prd

We will be using a database File Watcher for detecting new archivelog files and triggering FTP transfer.
First we login as user logship and create a table for storing detected archivelog files and the date of attempted transfer. This table is needed only for our own convinience - it can be used to check for missing log files if anything goes wrong.

SQL> conn logship/logship;
Connected.
SQL> create sequence transfered_logs_seq start with 1 increment by 1 cache 20 nocycle;

Sequence created.

SQL> create table transfered_logs (id number, transfer_date date, file_name varchar2(4000), error char(1));

Table created.

SQL>

Next we set the file detection interval to 1 minute. Of course, you can tune this to match closer you archivelog generation interval.

SQL> conn / as sysdba
Connected.
SQL> exec dbms_scheduler.set_attribute('file_watcher_schedule', 'repeat_interval', 'freq=minutely; interval=1');

PL/SQL procedure successfully completed.

SQL>

In order to have access to the archivelog directory, the file watcher needs an OS user account. We will create a credentials that the watcher can use and provide them with the oracle's username and password. For my demo install the oracle's password is welcome1.

SQL> exec dbms_scheduler.create_credential(credential_name => 'local_credential', username => 'oracle', password => 'welcome1');

PL/SQL procedure successfully completed.

SQL>

The final preparation is to create a PL/SQL procedure that the file watcher will call upon detecting a new archivelog file. The procedure I am using looks like this:

create or replace procedure trasnfer_arc_log(p_sched_result SYS.SCHEDULER_FILEWATCHER_RESULT) as
  v_transfer_id		number;
  v_file_name		varchar2(4000);
  v_ftp_conn		utl_tcp.connection;
begin
  v_transfer_id := transfered_logs_seq.nextval;
  v_file_name := p_sched_result.actual_file_name;
  v_ftp_conn := ftp.login('el5-backup','21','oracle','welcome1');
  ftp.put(p_conn => v_ftp_conn, p_from_dir  => 'ARC_DIR', p_from_file => v_file_name, p_to_file => '/u01/app/oracle/fast_recovery_area/ORCL/archivelog/'||v_file_name);
  ftp.logout(v_ftp_conn);
  insert into transfered_logs values (v_transfer_id, sysdate, v_file_name, null);
  commit;
exception when others then
  insert into transfered_logs values (v_transfer_id, sysdate, v_file_name, 'Y');
  commit;
end;
/

This procedure will try to ftp the file that the watcher will pass to it. If the operation is successful the procedure will insert a record in the TRANSFERED_LOGS table with the file name and date and time of its transfer. If an error occurs the procedure will set the ERROR column for the record to "Y".

Let's create this procedure in the logship schema.

SQL> conn logship/logship
Connected.
SQL> create or replace procedure trasnfer_arc_log(p_sched_result SYS.SCHEDULER_FILEWATCHER_RESULT) as 2 v_transfer_id number; 3 v_file_name varchar2(4000); 4 v_ftp_conn utl_tcp.connection; 5 begin 6 v_transfer_id := transfered_logs_seq.nextval; 7 v_file_name := p_sched_result.actual_file_name; 8 v_ftp_conn := ftp.login('el5-backup','21','oracle','welcome1'); 9 ftp.put(p_conn => v_ftp_conn, p_from_dir => 'ARC_DIR', p_from_file => v_file_name, p_to_file => '/u01/app/oracle/fast_recovery_area/ORCL/archivelog/'||v_file_name); 10 ftp.logout(v_ftp_conn); 11 insert into transfered_logs values (v_transfer_id, sysdate, v_file_name, null); 12 commit; 13 exception when others then 14 insert into transfered_logs values (v_transfer_id, sysdate, v_file_name, 'Y'); 15 commit; 16 end; 17 /

Procedure created.

SQL> show errors
No errors.
SQL>

Time to create the file watcher. This is done by calling the CREATE_FILE_WATCHER procedure from the DBMS_SCHEDULER package. I call the procedure with the following parameters.

BEGIN
  DBMS_SCHEDULER.create_file_watcher(
    file_watcher_name => 'arc_watcher',
    directory_path    => '/u01/app/oracle/fast_recovery_area/ORCL/archivelog',
    file_name         => '*.arc',
    credential_name   => 'local_credential',
    destination       => NULL,
    enabled           => FALSE);
END;
/

Here is the execution:

SQL> conn / as sysdba
Connected.
SQL> exec dbms_scheduler.create_file_watcher(file_watcher_name => 'arc_watcher', directory_path => '/u01/app/oracle/fast_recovery_area/ORCL/archivelog', file_name => '*.arc', credential_name => 'local_credential', destination => NULL, enabled => FALSE);

PL/SQL procedure successfully completed.

SQL>

Next we create a program that will bind the file watcher and the TRANSFER_ARC_LOG PL/SQL procedure.

SQL> exec dbms_scheduler.create_program(program_name => 'arc_watcher_prog', program_type => 'stored_procedure', program_action => 'logship.trasnfer_arc_log', number_of_arguments => 1, enabled => FALSE);

PL/SQL procedure successfully completed.

SQL> exec dbms_scheduler.define_metadata_argument(program_name => 'arc_watcher_prog', metadata_attribute => 'event_message', argument_position => 1);

PL/SQL procedure successfully completed.

SQL>

The final touch is creating a job for the ARC_WATCHER_PROG.

SQL> exec dbms_scheduler.create_job(job_name => 'arc_watcher_job', program_name => 'arc_watcher_prog', event_condition => NULL, queue_spec => 'arc_watcher', auto_drop => FALSE, enabled => FALSE);

PL/SQL procedure successfully completed.

SQL>

An important step is to set a value for the PARALLEL_INSTANCES attribute for our job. We will set this to TRUE to let the scheduler run multiple instances of our job. If you omit this step the system will process archivelogs one a time and if it's busy with a file it will just ignore any new archivelogs that appear in this period. You definitely do not want this to happen.

SQL> exec dbms_scheduler.set_attribute('arc_watcher_job','parallel_instances',TRUE);

PL/SQL procedure successfully completed.

SQL>

As finally everything is in place, we can enable the watcher, its program and the job. This is done by executing the DBMS_SCHEDULER.ENABLE procedure.

SQL> exec dbms_scheduler.enable('arc_watcher');

PL/SQL procedure successfully completed.

SQL> exec dbms_scheduler.enable('arc_watcher_prog');

PL/SQL procedure successfully completed.

SQL> exec dbms_scheduler.enable('arc_watcher_job');

PL/SQL procedure successfully completed.

SQL>

Test that archivelogs are shipped

For testing if archivelog transfers are happening we will take a look at the archivelog directory on failover server.

[oracle@el5-backup ~]$ ls -la /u01/app/oracle/fast_recovery_area/ORCL/archivelog/
total 5664
drwxr-xr-x 2 oracle oinstall    4096 Dec 27 07:56 .
drwxr-xr-x 4 oracle oinstall    4096 Dec 27 07:22 ..
-rw-r----- 1 oracle oinstall 1043968 Dec 27 07:52 1_10_769951554.arc
-rw-r----- 1 oracle oinstall 1701888 Dec 27 07:52 1_7_769951554.arc
-rw-r----- 1 oracle oinstall  544256 Dec 27 07:52 1_8_769951554.arc
-rw-r----- 1 oracle oinstall 2481152 Dec 27 07:52 1_9_769951554.arc
[oracle@el5-backup ~]$

We then execute ALTER SYSTEM SWITCH LOGFILE on el5-prd.

SQL> alter system switch logfile;

System altered.

SQL>

We connect with the logship user and check the contents of TRANSFERED_LOGS.

SQL> conn logship/logship;
Connected.
SQL> select count(*) from transfered_logs;

  COUNT(*)
----------
         0

SQL>

OK, the archivelog directory is checked in 60 seconds interval, so you might have to wait some more. After one minute tops the new file should be detected and transferred.

SQL> select count(*) from transfered_logs;

  COUNT(*)
----------
         1

SQL>

The new log is detected in a transfer attempt was made. Check the archivelog directory on el5-backup again to see if the file is there.

[oracle@el5-backup ~]$ ls -la /u01/app/oracle/fast_recovery_area/ORCL/archivelog/
total 6920
drwxr-xr-x 2 oracle oinstall    4096 Dec 27 08:00 .
drwxr-xr-x 4 oracle oinstall    4096 Dec 27 07:22 ..
-rw-r----- 1 oracle oinstall 1043968 Dec 27 07:52 1_10_769951554.arc
-rw-r--r-- 1 oracle oinstall 1282048 Dec 27 08:00 1_11_769951554.arc
-rw-r----- 1 oracle oinstall 1701888 Dec 27 07:52 1_7_769951554.arc
-rw-r----- 1 oracle oinstall  544256 Dec 27 07:52 1_8_769951554.arc
-rw-r----- 1 oracle oinstall 2481152 Dec 27 07:52 1_9_769951554.arc
[oracle@el5-backup ~]$

The logfile appears as expected. This concludes the detect and transfer part of our configuration.

A mechanism to apply and delete the shipped logs

Having the log files transferred to a failover server is not enough. If you really want to have an identical copy that is ready to take over the primary role you should take care to apply the database changes described in the logs. The easiest way is to simply start RMAN and apply the log files manually.

[oracle@el5-prd ~]$ rman target=/ 
Recovery Manager: Release 11.2.0.3.0 - Production on Sat Dec 27 11:24:16 2011

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1297199097)

RMAN> recover database noredo;

Starting recover at 17-DEC-11
using channel ORA_DISK_1

Starting recover at 17-DEC-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=18 device type=DISK

starting media recovery

archived log for thread 1 with sequence 8 is already on disk as file /u01/app/oracle/fast_recovery_area/ORCL/archivelog/1_8_769951554.arc
archived log file name=/u01/app/oracle/fast_recovery_area/ORCL/archivelog/1_8_769951554.arc thread=1 sequence=8
archived log file name=/u01/app/oracle/fast_recovery_area/ORCL/archivelog/1_9_769951554.arc thread=1 sequence=9
archived log file name=/u01/app/oracle/fast_recovery_area/ORCL/archivelog/1_10_769951554.arc thread=1 sequence=10
archived log file name=/u01/app/oracle/fast_recovery_area/ORCL/archivelog/1_11_769951554.arc thread=1 sequence=11
unable to find archived log
archived log thread=1 sequence=12

Finished recover at 17-DEC-11

You can then open the failover database and use in place of the production by executing

alter database open resetlogs

The thing is that you probably want to automate the process. This automation can not happen in the failover database as it is not really operational (it's not in open state). You will probably go with some kind of OS level automation, but this will be platform dependent.

For GNU/Linux environments, what you can do is to create a simple shell script that looks like this:

rman target / nocatalog << EOF
run {
  recover database noredo;
  delete noprompt force archivelog until time 'SYSDATE-7';
}
exit
EOF

This script will call RMAN, apply the received archivelogs and delete all log files that are older than 7 days (I keep the others just in case). You can then setup a cron job to run the script at an appropriate interval and you should not worry for managing the archivelogs manually.

Final remarks

In this tutorial I showed you how to build a platform independant archivelogs shipping mecahnism, that does all the work from within the database. This approach has its limitations and it's not in anyway a substitution of Data Guard and the other recovery features of Enterprise Edition. It's just a simple workaround when you are forced to use Database SE and you are looking for a simple way to be more protected from failiures.

There are several areas for improvement in this mechanism, especially when it comes to security. You should keep in mind that FTP is not really secure, so if you're dealing with sensitive data you might want to consider using SFTP or something else that provides encryption. Another issue is keeping plaintext passwords in TRANSFER_ARC_LOG procedure (you might want to wrap this one) and the database dictionary.

This list is built for my personal use and I do not guarantee that it is error free. I used almost exclusively the official Oracle documentation and some Metalink notes, where the official docs where not clear enough. This list is also available for download as a PDF file.

Deprecated Parameters in Database 11g

Obsolete Parameters in Database 11g

Thinking about protection and disk drives, the first thing that comes to mind is RAID (Redundant Array of Inexpensive Disks). This technology provides redundancy by combining multiple disk drives and distributing data across. There are different ways of data distribution called "RAID levels" (1 to 6), as RAID 1 and RAID 5 being the most common.

In RAID 1 two or more drives keep mirrored copies of the data. No matter how many disk drives fail, if there is at least one operational drive, the whole information is accessible. This availability comes at a price of disk capacity.

With RAID 5 the data is distributed over three disk drives at least. This configuration uses block-level striping with parity data distributed across all member disks. If any of the drives fails, the data is still available (the parity data is used to recover the missing information). The advantage of using RAID 5 is that the total capacity is not reduced by half (as with RAID 1). It is reduced only by the capacity of a single drive. For example if you have four 1 TB disk drives you can configure two RAID 1 arrays, each with a capacity of 1 TB. This makes a total of 2 TB of available disk capacity. If you configure a RAID 5 array instead, you will loose only 1 TB of capacity totaling in 3 TB of total storage being available.

This increased capacity, however, comes at a price of reduced availability. RAID 5 configurations can tolerate only a single drive failure. If two or more disk drives become unavailable, the entire array fails.

To illustrate the usage of RAID within LVM we will change the LVM configuration that we built in the previous tutorial, adding RAID 1 protection to it.

We will put /dev/sdb1 and /dev/sdc1 in RAID 1 (mirroring without parity or striping) and mount it under /dev/md0. In a similar way we will combine /dev/sdd1 and /dev/sde1 and configure them to be accessible under /dev/md1.

To build this configuration we will first evict all data from /dev/sdc1 and /dev/sde1. After that we will remove the /dev/sdc1 and /dev/sde1 partitions from the volume group. We will next create /dev/md0, but we will make it use a single disk for the moment (/dev/sdc1). We will create /dev/md1 in a similar fashion (using only /dev/sde1). We will then move away all data from /dev/sdb1 and /dev/sde1. After these two partitions are freed we will add them as mirrored disks to /dev/md0 and /dev/md1. This will let us build our RAID configurations step by step, without losing any data during the process.

Let's start by freeing /dev/sdc1. We use the pvmove command to evict its data.

[root@el5 ~]# pvmove /dev/sdc1
  /dev/sdc1: Moved: 26.5%
  /dev/sdc1: Moved: 52.2%
  /dev/sdc1: Moved: 77.8%
  /dev/sdc1: Moved: 92.7%
  /dev/sdc1: Moved: 100.0%
[root@el5 ~]# 

Removing /dev/sdc1 from the database group is done via the vgreduce command.

[root@el5 ~]# vgreduce database /dev/sdc1
  Removed "/dev/sdc1" from volume group "database"
[root@el5 ~]# 

Next we delete the physical volume by using pvremove.

[root@el5 ~]# pvremove /dev/sdc1
  Labels on physical volume "/dev/sdc1" successfully wiped
[root@el5 ~]# 

Our next step should be to change the partition type for /dev/sdc1. Its type should be "fd" (Linux raid autodetect), if it is going to be used in a RAID configuration.

[root@el5 ~]# fdisk /dev/sdc

The number of cylinders for this disk is set to 6527.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@el5 ~]# 

Building the RAID array is done via the mdadm command. The name is derived from Multiple Device (MD) and it is the main Linux command for managing RAID configurations.

[root@el5 ~]# mdadm --create /dev/md0 --auto=yes -l 1 -n 2 /dev/sdc1 missing
mdadm: array /dev/md0 started.
[root@el5 ~]# 

After /dev/md0 is in place, we can build a physical volume on top of it.

[root@el5 ~]# pvcreate /dev/md0
  Physical volume "/dev/md0" successfully created
[root@el5 ~]# 

After the PV is created we can extend the database group using vgextend.

[root@el5 ~]# vgextend database /dev/md0 
  Volume group "database" successfully extended
[root@el5 ~]# 

Our RAID 1 configuration is using a single disk for the time being and it does not provide any level of protection. Let's move the data out of /dev/sdb1 and add it as a second device for /dev/md0.

Evicting the data is done by running the pvmove command.

[root@el5 ~]# pvmove /dev/sdb1
  /dev/sdb1: Moved: 7.3%
  /dev/sdb1: Moved: 19.1%
  /dev/sdb1: Moved: 30.9%
  /dev/sdb1: Moved: 42.7%
  /dev/sdb1: Moved: 54.5%
  /dev/sdb1: Moved: 66.3%
  /dev/sdb1: Moved: 78.1%
  /dev/sdb1: Moved: 89.9%
  /dev/sdb1: Moved: 100.0%
[root@el5 ~]# 

Before changing the partition's type we have to take it out of the database group and remove the PV that is built on top of it.

[root@el5 ~]# vgreduce database /dev/sdb1
  Removed "/dev/sdb1" from volume group "database"
[root@el5 ~]# pvremove /dev/sdb1
  Labels on physical volume "/dev/sdb1" successfully wiped
[root@el5 ~]# 

We can now safely change the partition type to "fd".

[root@el5 ~]# fdisk /dev/sdb

The number of cylinders for this disk is set to 6527.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@el5 ~]# 

Adding /dev/sdb1 to /dev/md0 is done by running mdadm.

[root@el5 ~]# mdadm --manage /dev/md0 --add /dev/sdb1
mdadm: added /dev/sdb1
[root@el5 ~]# 

Executing the mdam command starts a synchronization process. Its task is to mirror the data from /dev/sdc1 to /dev/sdb1. We can check the process's progress by looking at /proc/mdstat.

[root@el5 ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[2] sdc1[0]
      52428032 blocks [2/1] [U_]
      [>....................]  recovery =  3.0% (1600000/52428032) finish=3.7min speed=228571K/sec

unused devices: 
[root@el5 ~]

The synchronization process mirrored about 3% of the data so far. When it completes, the output from /proc/mdstat will look like this.

[root@el5 ~]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sdc1[0]
      52428032 blocks [2/2] [UU]

unused devices: 
[root@el5 ~]# 

Our first RAID 1 configuration /dev/md0 is now operational. Let's move on with building the second one - /dev/md1.

We start by evicting the data from /dev/sde1

[root@el5 ~]# pvmove /dev/sde1
  /dev/sde1: M	oved: 6.0%
  /dev/sde1: Moved: 17.8%
  /dev/sde1: Moved: 29.6%
  /dev/sde1: Moved: 41.4%
  /dev/sde1: Moved: 53.2%
  /dev/sde1: Moved: 65.0%
  /dev/sde1: Moved: 76.8%
  /dev/sde1: Moved: 88.6%
  /dev/sde1: Moved: 100.0%
[root@el5 ~]# 

Next we take /dev/sde1 out of the group and remove the PV.

[root@el5 ~]# vgreduce database /dev/sde1
  Removed "/dev/sde1" from volume group "database"
[root@el5 ~]# pvremove /dev/sde1
  Labels on physical volume "/dev/sde1" successfully wiped
[root@el5 ~]# 

Changing the partition type to "fd" is done via fdisk.

[root@el5 ~]# fdisk /dev/sde

The number of cylinders for this disk is set to 6527.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@el5 ~]# 

As with /dev/md0 we build the /dev/md1 RAID configuration with /dev/sde1 and a missing second disk.

[root@el5 ~]# mdadm --create /dev/md1 --auto=yes -l 1 -n 2 /dev/sde1 missing
mdadm: array /dev/md1 started.
[root@el5 ~]# 

We create a PV on top of /dev/md1 and extend the database group with it.

[root@el5 ~]# pvcreate /dev/md1
  Physical volume "/dev/md1" successfully created
[root@el5 ~]# vgextend database /dev/md1
  Volume group "database" successfully extended
[root@el5 ~]# 

Let's inspect the PV's, just to be sure that everything is correct.

[root@el5 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdd1
  VG Name               database
  PV Size               50.00 GB / not usable 3.31 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              12799
  Free PE               5119
  Allocated PE          7680
  PV UUID               RF37f3-SfBR-aozU-7l01-Rqh9-4YV2-1o7GdK

  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               database
  PV Size               50.00 GB / not usable 3.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              12799
  Free PE               7679
  Allocated PE          5120
  PV UUID               PAEniI-bCth-w430-3gsV-ToeS-JARL-wsJ2g6

  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               database
  PV Size               50.00 GB / not usable 3.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              12799
  Free PE               12799
  Allocated PE          0
  PV UUID               I9Hrbo-ehjq-vn9b-PgNa-PhAY-57tj-j8f8Vx

[root@el5 ~]# 

Our final tasks is to move the data out of /dev/sdd1 and add it as a second disk for /dev/md1.

[root@el5 ~]# pvmove /dev/sdd1
  /dev/sde1: M	oved: 6.0%
  /dev/sde1: Moved: 17.8%
  /dev/sde1: Moved: 29.6%
  /dev/sde1: Moved: 41.4%
  /dev/sde1: Moved: 53.2%
  /dev/sde1: Moved: 65.0%
  /dev/sde1: Moved: 76.8%
  /dev/sde1: Moved: 88.6%
  /dev/sde1: Moved: 100.0%
[root@el5 ~]# vgreduce database /dev/sdd1
  Removed "/dev/sdd1" from volume group "database"
[root@el5 ~]# pvremove /dev/sdd1
  Labels on physical volume "/dev/sdd1" successfully wiped
[root@el5 ~]# 

We change the partition type to Linux raid autodetect (fd).

[root@el5 ~]# fdisk /dev/sdd

The number of cylinders for this disk is set to 6527.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@el5 ~]# 

Adding /dev/sdd1 to /dev/md1 is done via mdadm.

[root@el5 ~]# mdadm --manage /dev/md1 --add /dev/sdd1
mdadm: added /dev/sdd1

This starts the mirroring process that we already saw for /dev/md0.

[root@el5 ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdd1[2] sde1[0]
      52428032 blocks [2/1] [U_]
      [=>...................]  recovery =  5.3% (2800000/52428032) finish=3.8min speed=215384K/sec

md0 : active raid1 sdb1[1] sdc1[0]
      52428032 blocks [2/2] [UU]

unused devices: 
[root@el5 ~]# 

When the process completes we will notice that /proc/mdstat shows two RAID 1 groups (md0 and md1).

[root@el5 ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdd1[1] sde1[0]
      52428032 blocks [2/2] [UU]

md0 : active raid1 sdb1[1] sdc1[0]
      52428032 blocks [2/2] [UU]

unused devices: 
[root@el5 ~]# 

Looking at the information about our physical volumes we can notice that from LVM's perspective there are only two PV's - /dev/md0 and /dev/md1.

[root@el5 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               database
  PV Size               50.00 GB / not usable 3.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              12799
  Free PE               7679
  Allocated PE          5120
  PV UUID               PAEniI-bCth-w430-3gsV-ToeS-JARL-wsJ2g6

  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               database
  PV Size               50.00 GB / not usable 3.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              12799
  Free PE               5119
  Allocated PE          7680
  PV UUID               I9Hrbo-ehjq-vn9b-PgNa-PhAY-57tj-j8f8Vx

[root@el5 ~]# 

Let's take a look at the database group as well.

[root@el5 ~]# vgdisplay
  --- Volume group ---
  VG Name               database
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  31
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               99.99 GB
  PE Size               4.00 MB
  Total PE              25598
  Alloc PE / Size       12800 / 50.00 GB
  Free  PE / Size       12798 / 49.99 GB
  VG UUID               DcuO3X-HFRV-ZVyx-MVlD-gY3c-jyo0-FwR3uO

[root@el5 ~]# 

The group consists of two PV's and its capacity is reduced to 100 GB. This is the price we have to pay for having mirrored copies of each of the volumes.

You probably noticed that all changes that we did for configuring the RAID groups were accomplished with mounted devices. This is one of the key advantages provided by LVM - the ability to manipulate physical disks without disrupting the system's availability.

Increasing the RAID's capacity

The available free space always depletes, sooner or later. There are two ways for increasing the capacity for a RAID configuration like the one we built. The first approach is to add more physical disks of the same size. The md0 group for instance consists of two drives, 50 GB each, and provides a total of 50 GB of disk space. If add two more 50 GB disks, we will increase the group's capacity to 100 GB (4x50 GB divided by two, because of the mirroring).

Another approach would be to take out a single disk from md0 and md1 and replace them with disks, having bigger capacity. We can then use these new disks and built a third, bigger RAID 1 group (md2). We can then move all data from md0 and md1 to the newly created md2 group. After no data is left in the old RAID 1 groups, we can destroy them, replace the remaining two disks with bigger ones and built another RAID 1 group. At the end we will have two RAID 1 groups again, but they will have much bigger capacity, as they will use bigger disk drives.

We are already familiar with the process of adding new disks for increasing the capacity, so there is nothing new for us in the first approach. It will be more interesting to see how we can implement the second approach, changing the current drives with bigger ones.

Let's start by inspecting the PV's that are in use (we have two RAID 1 groups at the moment).

[root@el5 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               database
  PV Size               50.00 GB / not usable 3.06 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              12799
  Free PE               7679
  Allocated PE          5120
  PV UUID               PAEniI-bCth-w430-3gsV-ToeS-JARL-wsJ2g6

  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               database
  PV Size               50.00 GB / not usable 3.06 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              12799
  Free PE               5119
  Allocated PE          7680
  PV UUID               I9Hrbo-ehjq-vn9b-PgNa-PhAY-57tj-j8f8Vx

[root@el5 ~]# 

Let's remove the second and the fourth disks of the machine. We have to mark /dev/sdb1 and /dev/sdd1 as failed and remove them from their RAID groups (md0 and md1).

[root@el5 ~]# mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
[root@el5 ~]# mdadm --manage /dev/md0 --remove /dev/sdb1
mdadm: hot removed /dev/sdb1
[root@el5 ~]# mdadm --manage /dev/md1 --fail /dev/sdd1
mdadm: set /dev/sdd1 faulty in /dev/md1
[root@el5 ~]# mdadm --manage /dev/md1 --remove /dev/sdd1
mdadm: hot removed /dev/sdd1
[root@el5 ~]# 

We turn of the virtual machine, remove the disks and replace them with two new drives, 200 GB each. After the change is completed, the VM's configuration looks like this:

Our first task after the system comes up is to build "fd" partitions on the two new disks (/dev/sdb and /dev/sdd).

[root@el5 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 26108.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-26108, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-26108, default 26108):
Using default value 26108

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@el5 ~]# fdisk /dev/sdd
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 26108.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-26108, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-26108, default 26108):
Using default value 26108

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@el5 ~]# 

Next, we use mdadm to build the third RAID 1 group (md2). This time there will be no missing disks in the initial setup.

[root@el5 ~]# mdadm --create /dev/md2 --auto=yes -l 1 -n 2 /dev/sdb1 /dev/sdd1
mdadm: array /dev/md2 started.
[root@el5 ~]# 

Looking at /proc/mdstat will show us the three RAID groups.

[root@el5 ~]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdd1[1] sdb1[0]
      209712384 blocks [2/2] [UU]

md1 : active raid1 sde1[0]
      52428032 blocks [2/1] [U_]

md0 : active raid1 sdc1[0]
      52428032 blocks [2/1] [U_]

unused devices: 
[root@el5 ~]# 

Moving the data from md0 to md2 is done by running pvmove.

[root@el5 ~]# pvmove /dev/md0 /dev/md2
  /dev/md0: Moved: 10.3%
  /dev/md0: Moved: 20.6%
  /dev/md0: Moved: 30.9%
  /dev/md0: Moved: 41.5%
  /dev/md0: Moved: 51.9%
  /dev/md0: Moved: 61.7%
  /dev/md0: Moved: 72.2%
  /dev/md0: Moved: 82.7%
  /dev/md0: Moved: 93.1%
  /dev/md0: Moved: 100.0%
[root@el5 ~]# 

We do the same for md1.

[root@el5 ~]# pvmove /dev/md1 /dev/md2
  /dev/md1: Moved: 3.5%
  /dev/md1: Moved: 21.0%
  /dev/md1: Moved: 35.0%
  /dev/md1: Moved: 45.5%
  /dev/md1: Moved: 59.4%
  /dev/md1: Moved: 73.3%
  /dev/md1: Moved: 87.2%
  /dev/md1: Moved: 100.0%
[root@el5 ~]# 

After md0 and md1 are freed, we can safely remove them from the database group. This is done by running vgreduce.

[root@el5 ~]# vgreduce database /dev/md0
  Removed "/dev/md0" from volume group "database"
[root@el5 ~]# vgreduce database /dev/md1
  Removed "/dev/md1" from volume group "database"
[root@el5 ~]# 

Let's remove the PV's as well.

[root@el5 ~]# pvremove /dev/md0
  Labels on physical volume "/dev/md0" successfully wiped
[root@el5 ~]# pvremove /dev/md1
  Labels on physical volume "/dev/md1" successfully wiped
[root@el5 ~]# 

If we run pvdisplay, we will see that the only available physical volume is the newly created /dev/md2 and that its capacity totals to 200 GB.

[root@el5 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               database
  PV Size               200.00 GB / not usable 1.25 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              51199
  Free PE               38399
  Allocated PE          12800
  PV UUID               hmTRBJ-gjMJ-Dqnm-QhFL-6187-Zhby-vHL9ko

[root@el5 ~]# 

This is also the only physical volume used by the database group.

[root@el5 ~]# vgdisplay
  --- Volume group ---
  VG Name               database
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  44
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               200.00 GB
  PE Size               4.00 MB
  Total PE              51199
  Alloc PE / Size       12800 / 50.00 GB
  Free  PE / Size       38399 / 150.00 GB
  VG UUID               DcuO3X-HFRV-ZVyx-MVlD-gY3c-jyo0-FwR3uO

[root@el5 ~]# 

We just have to remove /dev/md0 and /dev/md1. After that we should replace the two remaining disks with 200 GB ones. We can then build a new RAID 1 group and extend database with it. Performing these steps should already be trivial for you, so let's finish this tutorial here.