edu.isi.pegasus.planner.transfer.refiner

Class Chain

java.lang.Object

edu.isi.pegasus.planner.transfer.AbstractRefiner

edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner

edu.isi.pegasus.planner.transfer.refiner.Basic

edu.isi.pegasus.planner.transfer.refiner.Chain

All Implemented Interfaces:

Refiner

public class Chain
extends Basic

This transfer refiner builds upon the Default Refiner. The defaul Refiner allows the transfer of multiple files in a single condor job. However, it adds the stage in transfer nodes in parallel leading to multiple invocation of the globus-url-copy at remote execution pools, while running huge workflows. This refiner, tries to circumvent this problem by chaining up the stagein jobs instead of scheduling in parallel. This works best only when the top level of the workflow requires stage in jobs. The correct way is that the traversal needs to be done breath first in the TransferEngine.java.

Version:

$Revision$

Author:

Karan Vahi, Gaurang Mehta

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	Chain.SiteTransfer A container to manage the transfer jobs that are needed to be done on a single site.
private static class	Chain.TransferChain A shallow container class, that contains the list of the names of the transfer jobs and can return the last job in the list.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_BUNDLE_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for the workflow.
static String	DESCRIPTION A short description of the transfer refinement.
private Map	mSIBundleMap The map containing the stage in bundle values indexed by the name of the site.
protected SiteStore	mSiteStore The handle to the Site Catalog.
private Map	mStageInMap A map indexed by execution sites.

Fields inherited from class edu.isi.pegasus.planner.transfer.refiner.Basic

mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS

Fields inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore

Fields inherited from interface edu.isi.pegasus.planner.transfer.Refiner

INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX

Fields inherited from interface edu.isi.pegasus.planner.refiner.Refiner

VERSION

Constructor Summary

Constructors

Constructor and Description
Chain(ADag dag, PegasusBag bag) The overloaded constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	addRelation(String parent, String child, String site, boolean parentNew) Adds a new relation to the workflow.
void	done() Prints out the bundles and chains that have been constructed.
String	getDescription() Returns a textual description of the transfer mode.
int	getSiteBundleValue(String site, String key) Determines the bundle factor for a particular site on the basis of the key associcated with the underlying transfer transformation in the transformation catalog.

Methods inherited from class edu.isi.pegasus.planner.transfer.refiner.Basic

addInterSiteTXNodes, addJob, addRelation, addStageInXFERNodes, addStageInXFERNodes, addStageOutXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, getJobPriority, logRefinerAction

Methods inherited from class edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner

loadImplementations

Methods inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_BUNDLE_FACTOR

public static final String DEFAULT_BUNDLE_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for the workflow.

See Also:

Constant Field Values

mSiteStore

protected SiteStore mSiteStore

The handle to the Site Catalog. It is instantiated in this class.

mSIBundleMap

private Map mSIBundleMap

The map containing the stage in bundle values indexed by the name of the site. If the bundle value is not specified, then null is stored.

mStageInMap

private Map mStageInMap

A map indexed by execution sites. Each value is a SiteTransfer object, that contains the Bundles of stagin transfer jobs.

See Also:

Chain.TransferChain

DESCRIPTION

public static final String DESCRIPTION

A short description of the transfer refinement.

See Also:

Constant Field Values

Constructor Detail

Chain

public Chain(ADag dag,
     PegasusBag bag)

The overloaded constructor.

Parameters:

dag - the workflow to which transfer nodes need to be added.

bag - the bag of initialization objects

Method Detail

addRelation

public void addRelation(String parent,
               String child,
               String site,
               boolean parentNew)

Adds a new relation to the workflow. In the case when the parent is a transfer job that is added, the parentNew should be set only the first time a relation is added. For subsequent compute jobs that maybe dependant on this, it needs to be set to false.

Overrides:

addRelation in class Basic

Parameters:

parent - the jobname of the parent node of the edge.

child - the jobname of the child node of the edge.

site - the execution site where the transfer node is to be run.

parentNew - the parent node being added, is the new transfer job and is being called for the first time.

getSiteBundleValue

public int getSiteBundleValue(String site,
                     String key)

Determines the bundle factor for a particular site on the basis of the key associcated with the underlying transfer transformation in the transformation catalog. If none specified in transformation catalog then one is picked up from the site catalog. If the key is not found in the site catalog too , then the global default is returned.

Parameters:

site - the site at which the transfer job is being run.

key - the bundle key whose value needs to be searched.

Returns:

the bundle factor.

See Also:

DEFAULT_BUNDLE_FACTOR

done

public void done()

Prints out the bundles and chains that have been constructed.

Overrides:

done in class Basic

getDescription

public String getDescription()

Returns a textual description of the transfer mode.

Overrides:

getDescription in class Basic

Returns:

a short textual description