Contributing Factors

In order to understand how to improve availability in a non-clustered environment, it is first necessary to understand what limitations are imposed by the hardware and operating system. Given the basic knowledge of "how it works", the techniques to improve availability approach mere "common-sense".

ORACLE's architecture under the Windows NT operating system

Under NT, an ORACLE instance is composed of the same background and user processes that would be found in an ORACLE instance running under UNIX. However, the ORACLE processes are, from the NT operating system perspective, multiple threads under a single O/S process. This design has a number of implications:

1) The multiple threads share a single segment of memory, therefore there is no IPC (inter-process communication) required. Under strictly theoretical terms, a multi-threaded process such as this offers a performance advantage compared to a multi-process, single-threaded design due to the lack of IPC.

2) Since multiple threads share a single memory segment, it is much easier for one thread to corrupt the memory in use by another thread. In fact, the only factor preventing this corruption is the precision with which the application developer wrote the application code. It is close to impossible for one process to corrupt the memory of a different process under any reasonably stable commercial operating system, but thread-safety is a major concern under multi-threaded processes.

3) Most operating systems impose some maximum limit on the amount of memory that a process can access. Therefore, an application constructed from multiple processes can utilize a much larger amount of memory than can an application constructed as a single process with multiple threads, all other architectural concerns held the same.

Microsoft's Component Object Model

Microsoft provides an application development framework around which MS Windows-based applications can (and theoretically, should) be developed. The two major concepts mandated by this framework are the use of shared component objects typically implemented as DLL's (dynamic-link libraries) and the MS Windows Registry. Objects and their interfaces are implemented within various DLL's in the typical object-oriented fashion. The Registry completes the model by providing an additional layer of abstraction such that applications (or even other components) do not need to know which DLL in which folder on which drive implements a particular object and it's interface; the application or component merely needs to know the object's name and interface. The Registry provides the resolution to the particular DLL in which this object is implemented. The Registry can be thought of as an "object speed-dial": the application can ask for the object by name, rather than having to know it's name and explicitly look-up it's "telephone number", so to speak.

This model also has it's own implications. Sharing common components is an architecturally sound concept, but in practice, versioning conflicts have a tendency to arise. The introduction of the Registry for object resolution further complicates this matter. Unless a vendor has specifically written their application to provide seperate registry entries for different versions of the application, then hosting multiple versions of the same application on the same operating system instance is disallowed. This characteristic is true of ORACLE7 on NT, although ORACLE8 now provides a means for seperate ORACLE_HOMES (meaning separate folders for the software components and seperate registry entries).

Also, information about many of the objects and interfaces are loaded at boot time rather than run time. This characteristic means that certain types of O/S-level modifications, regardless of how minor they may be, require a reboot before the changes will take effect. Modifications such as installing or uninstalling software, changing the IP address, changing NT's self-tuning objectives, etc., will typically require a reboot, as well as other unexpected occurrences such as process errors, process address conflicts, etc. In practice, the assumption should be made that any operating system modification will require a reboot.

Maximum Operating System and hardware Limits

CPU: Although CPU's themselves double in speed approximately once per year, the same is not necessarily true of multi-processor scaleability. Standard Intel-based SMP servers typically employ 2-8 processors that share a reasonably large segment of bulk memory, with each processor having it's own very small private cache memory. Typically, the bulk memory is around 10 times slower than the processor. The processor cache, though much smaller, is typically approximately equivalent in speed to the processor. This very expensive memory is used for instructions and other data that can be obtained in a read-ahead fashion, thus emulating the effect of having a large amount of fast, expensive memory rather than the slow but inexpensive bulk memory. For operations that are parallelized, coordinating the contents of each processor's private cache becomes a difficult task. This cache-coherence is maintained through the use of one or more I/O buses. Two processors must maintain cache-coherence with only one other processor each, meaning that essentially two streams of communication are occurring across the I/O bus used for processor communication, plus keeping the contents of bulk memory up to date. On the other hand, eight processors must maintain cache-coherence with seven other processors each, meaning that essentially fifty-six streams of bus communication are necessary, plus keeping the contents of bulk memory up to date. Given this brief explanation, it should now be apparent where limitations in SMP scaleability originate. The hardware is readily available, but the software (ie. operating systems or applications that are explicitly parallel in nature) must be written with extreme precision to provide the necessary processor communication in a fast and reliable manner. Within an SMP model, bus speed and processor cache size are as important to overall scaleability as is processor speed. In practice, every time the number of processors is doubled, the increase in performance grows exponentially smaller. Interestingly enough, the types of coordination required with multiple processors sharing bulk memory is very similar to clustered solutions where nodes with private memory share common disk (such as ORACLE Parallel Server). With NT, migrating from a uniprocessor box to a dual-processor box typically yields a 65-75% performance increase for database applications such as ORACLE, assuming the application is significantly processor-bound. Migrating from a dual processor box to a quad processor box typically yields a 35-45% performance improvement, again assuming the database instance is reasonably processor-bound.

Memory: The very expensive and fast memory used for processor caches would make Intel-based SMP server's very unaffordable if it were used throughout the shared physical memory. Luckily, using the expensive memory as processor cache and performing "read-ahead" into slower bulk memory provides a reasonably close emulation (typically 90% or greater). Bulk memory may be 10 times slower than the processor and processor caches, but it is also about 200-300 times faster than disk. Most hardware and software in the 32-bit realm is capable of addressing 4 Gb of physical memory, so high-end Intel-based SMP servers typically scale to this quantity. Under NT4.0, an operating system process can access a total of 2 Gb of memory. This 2 Gb limit has increased to 3Gb with the Enterprise version of NT4.0. Given ORACLE's multi-threaded architecture under NT, this O/S limit cannot be exceeded by the combination of the ORACLE O/S process requirements, the memory structures in the SGA, the background processes, and all of the user shadow processes.

Disk: Since NT follows the concept of mapping letters of the alphabet to logical disk drives, there is a finite number of logical disks that can be attached to an NT box (22-23, to be exact), without resorting to fairly complicated drive-mapping "tricks". A logical drive can, however, equal one physical disk or an entire array of physical disks. With the disk array approach, the limitation becomes the maximum number of array controller slots and drive bays that the NT box can have attached. With a very large number of multi-channel array controllers, there is still a risk that the I/O bus they are plugged into may become an I/O bottleneck. Typically under NT, physical disks tend to be grouped into disk arrays to provide increases in workload threshold and/or fault-tolerance. Typical RAID configurations under NT include RAID0, RAID1, and RAID5, with a few vendors starting to offer RAID10 as an additional alternative. (Please see the section on performance tuning for a detailed discussion of analyzing RAID performance characteristics.) Typical SCSI RAID adapters are capable of sustaining 20-40 Mb/sec of data transfer per controller channel, and their performance improves, up to a point, as their cache memory increases. Disk on any hardware and O/S combination will exhibit the typical characteristics of rotational and mechanical latency and other factors influencing workload and throughput limitations, so these specific physical traits of disk are essentially O/S independent.