Basic Components of the Apple Qmaster Distributed Processing System

Although the Apple Qmaster software includes a few different applications (see The Interfaces in the Apple Qmaster Distributed Processing System), as a whole it is part of a networked system that includes the following basic components.

Note: Within a distributed processing system, the Compressor, QuickTime, and Mac OS versions must all be identical.

The client computer, the service nodes, and the cluster controller are often on separate (but network-connected) computers, for the most rapid processing potential. However, the cluster controller could be on a client computer or a service node. See Using One Computer to Serve Two Distributed Processing Roles for more information on this scenario.

Following is a closer look at the part each component plays in the Apple Qmaster system.

Clients

Batches are submitted for distributed processing from the client computers. A client computer can be any computer that has Compressor installed and is on the same network (subnet) as the cluster controller. Multiple client computers can be on the same subnet, using the same cluster to do the processing for various applications.

You use Compressor or the Apple Qmaster application to submit batches to be processed for a client. See The Basic Transcoding Workflow and the Apple Qmaster User Manual for details on using these applications.

Clusters

When a client sends batches to the Apple Qmaster distributed processing system, all the processing and subsequent moving of any output files is performed by a group of Apple Qmaster–configured computers called a cluster. You can create one or more clusters of service nodes, with one cluster controller included in each cluster. Each computer in the cluster is connected to the other computers in the cluster through a network connection.

Figure. Diagram showing the Apple Qmaster cluster including one cluster controller and three service nodes.

Note: This illustration provides only one simple example of a cluster. Other possibilities are described in Examples of a Distributed Processing Network.

Service Nodes

The service nodes are where the processing work is done. When you assign a group of service nodes to a cluster, they function as one very powerful computer because all their resources are shared. If one service node is overloaded or otherwise inaccessible, another service node is used.

You make a computer available as a service node by configuring it in the Apple Qmaster Sharing window, accessible from Compressor by choosing Share This Computer from the Apple Qmaster menu.

The Minimum You Need to Know

The following are the basic rules for setting up a distributed processing network:

  • A cluster must contain one (and only one) computer acting as the cluster controller, and at least one computer acting as the service node. (These two can be the same computer, as shown in Using One Computer to Serve Two Distributed Processing Roles.)

  • The client computers and the computers in any cluster that supports them must be on the same network.

  • The network must support the Apple networking technology built in to Mac OS X.

  • All the computers in a cluster need read-and-write access to any computers (or storage devices) that will be specified as output destinations for files.

Examples of a Distributed Processing Network

A very small distributed processing setup could include as few as two computers:

  • One computer connected to the client and configured to act as both the service node and the cluster controller

  • One client computer

    Figure. Diagram showing the client computer and the computer acting as both the service node and the cluster controller.

Though simple, this setup is useful in a small-scale environment because it allows the client computer to offload a lot of processing work.

Using One Computer to Serve Two Distributed Processing Roles

To maximize your resources, you may want to consider using some computers for more than one distributed processing function.

  • Service node and cluster controller: In a small setup, one of the service nodes in a cluster can also act as the cluster controller so that it performs both functions. However, in a cluster of many service nodes, the processing load required for the cluster controller could be so high that it would not be efficient to use one computer as both a service node and a cluster controller.
  • Client computer and cluster controller or service node: You could also set up a client computer to act as a cluster controller or service node in a cluster, but again, keep in mind that the more available processing power a computer has, the faster it can manage or process jobs.

The following setup is for an environment that uses desktop computers. It is called “part-time” processing because each computer acts as someone’s workstation, but at the same time is also part of the distributed processing cluster. The bulk of the processing jobs can be submitted with Compressor or Apple Qmaster at the end of the day, so that the computers are busy processing a large queue of distributed processing batches after everyone has gone home. (See About Advanced Settings in the Apple Qmaster Sharing Window and Scheduling Service Availability in the Work Schedule Dialog for information on scheduling service node availability.)

Figure. Diagram showing a network of five computers in which each computer acts as both a client and a service node.

In the sample setup shown above, five computers act as both the clients (user workstations from which users submit jobs for distributed processing) and cluster computers (which do the processing). Each computer has an additional volume, such as a FireWire drive, that is used for media before and after it is rendered, and for the associated files.

For more powerful rendering, a network might include a number of client computers on a LAN, connected to a cluster using a high-speed switch. A rack of servers plus a shared storage device, acting as the cluster, would be an extremely strong rendering engine. The service nodes would each have a local copy of the relevant client application software so that they could process the rendering jobs.

Figure. Diagram showing client computers, high-speed switches, and a rack of servers containing the cluster controller and service nodes.

How the Apple Qmaster System Distributes Batches

The Apple Qmaster cluster controller determines the most efficient use of the cluster resources. It makes this determination based on the availability of each service node and the number of separable parts (described next) of the batch.

Because Apple Qmaster subdivides individual batches across different service nodes, the work is shared and completed more quickly. And, because this method uses all the service nodes as much as possible, you avoid under-utilizing your resources.

Batches can be distributed to a cluster by the cluster controller in one or both of the following ways. (Apple Qmaster determines which way is the most efficient for specific batches, depending on the circumstances.)

  • The batch is subdivided into data segments: For example, for a render batch, the cluster controller could divide the frames into groups (segments). Each segment would be processed in parallel on the service nodes in the cluster.
  • The batch is subdivided into tasks: For example, for a render batch, the cluster controller could subdivide the rendering work into different processing tasks. Different tasks would be run on different service nodes.

Rather than actually moving segments, Apple Qmaster tells the service nodes which segments to read via the network, where to find them, and what to do with them. Below is an example of how one batch could be processed in an Apple Qmaster system.

Figure. Diagram showing the batch, the cluster controller, instructions to individual service nodes, and the processed output file.

In distributing batches, Apple Qmaster uses the technology built in to Mac OS X to locate services in a cluster on the same IP subnet and to dynamically share and receive information. Because the computers can continually transmit their current processing availability status, Apple Qmaster can distribute (load-balance) the workload evenly across the cluster.

Other Possible Components of a Distributed Processing Network

A distributed processing network can consist of as few as one or two computers, whereas a high-volume network may include many computers, an Xserve system and Xserve cluster nodes in a rack, and high-speed networking infrastructures. You can scale up a distributed processing system as your workload demands by adding features and devices to the network that supports it.

There are many ways to expand the capacity of a distributed processing network. You could include any of the following.

  • High-speed switch and cables: A 100Base-T or Gigabit Ethernet switch and compatible cables to allow your data to move over the LAN at maximum speed.
  • Multiple clients: Multiple client computers can use the services of the same cluster. And, you can have multiple client applications on the same client computer, using the same cluster.
  • Multiple clusters: Depending on how extensive your network is and how many clients it needs to serve, you may want to divide up available computers and create more than one cluster to serve various clients. (Users select the cluster they want to send a batch to when they submit the batch.)
  • Multiple service nodes: In general, more service nodes mean more processing power. In deciding how many service nodes to have in a cluster, consider the ratio of data movement time to computing time. If the processing demand is greater than the network demand required to move job segments throughout the cluster, as is the case with rendering, more service nodes are a good idea. If the computing load, per job, is closer to the network load, having a smaller number of service nodes per cluster may be more efficient. If you are using the Apple Qmaster distributed processing system with applications other than Shake or Compressor, consult the application’s user manual on how to optimize the number of service nodes.
  • Storage device: A storage device, such as a remote disk or group of disk arrays, can be used as cluster scratch storage, which is a place for short-term storage of temporary data generated by the cluster controller, clients, and service nodes. (You set the scratch storage location in the Apple Qmaster Sharing window of Compressor. See About Advanced Settings in the Apple Qmaster Sharing Window and Using Cluster Storage for more information.) Alternatively, a storage device can be used as a final destination for the files after they are processed.

For more information, see Examples of a Distributed Processing Network.