edu.isi.pegasus.planner.transfer.refiner

Class Cluster

java.lang.Object

edu.isi.pegasus.planner.transfer.AbstractRefiner

edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner

edu.isi.pegasus.planner.transfer.refiner.Basic

edu.isi.pegasus.planner.transfer.refiner.Bundle

edu.isi.pegasus.planner.transfer.refiner.Cluster

All Implemented Interfaces:

Refiner

public class Cluster
extends Bundle

A cluster refiner that builds upon the Bundle Refiner. It clusters the stage-in jobs and stage-out jobs per level of the workflow. The difference from the Bundle refiner beings

        - stagein is also clustered/bundled per level. In Bundle it was for the 
          whole workflow.
        - keys that control the clustering ( old name bundling are ) 
          cluster.stagein and cluster.stageout
 

In order to use the transfer refiner implemented by this class,

        - the property pegasus.transfer.refiner  must be set to value Cluster
 

Version:

$Revision$

Author:

Karan Vahi

Nested Class Summary

Nested classes/interfaces inherited from class edu.isi.pegasus.planner.transfer.refiner.Bundle

Bundle.BundleValue, Bundle.PoolTransfer, Bundle.TransferContainer

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR The default clustering factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.
static String	DEFAULT_LOCAL_STAGE_OUT_CLUSTER_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.
static String	DEFAULT_REMOTE_STAGE_IN_CLUSTER_FACTOR The default clustering factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.
static String	DEFAULT_REMOTE_STAGE_OUT_CLUSTER_FACTOR The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.
static String	DESCRIPTION A short description of the transfer refinement.
private int	mCurrentSILevel The current level of the jobs being traversed.
protected Map<String,Bundle.PoolTransfer>	mStageInLocalMapPerLevel A map indexed by site name, that contains the pointer to the stage in PoolTransfer objects for that site.
protected Map<String,Bundle.PoolTransfer>	mStageInRemoteMapPerLevel A map indexed by site name, that contains the pointer to the symlink stage in PoolTransfer objects for that site.
private Map<String,Job>	mSyncJobMap Maps the site name to the current synch job

Fields inherited from class edu.isi.pegasus.planner.transfer.refiner.Bundle

DEFAULT_LOCAL_STAGE_IN_BUNDLE_FACTOR, DEFAULT_LOCAL_STAGE_OUT_BUNDLE_FACTOR, DEFAULT_REMOTE_STAGE_IN_BUNDLE_FACTOR, DEFAULT_REMOTE_STAGE_OUT_BUNDLE_FACTOR, mAddNodesForSettingXBit, mJobPrefix, mPegasusProfilesInProperties, mSetupMap, mSiteStore, mStageinLocalBundleValue, mStageInRemoteBundleValue, mStageOutLocalBundleValue, mStageOutRemoteBundleValue

Fields inherited from class edu.isi.pegasus.planner.transfer.refiner.Basic

mCreateRegistrationJobs, mFileTable, mLogMsg, mPPS

Fields inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore

Fields inherited from interface edu.isi.pegasus.planner.transfer.Refiner

INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX

Fields inherited from interface edu.isi.pegasus.planner.refiner.Refiner

VERSION

Constructor Summary

Constructors

Constructor and Description
Cluster(ADag dag, PegasusBag bag) The overloaded constructor.

Method Summary

Methods

Modifier and Type	Method and Description
void	addStageInXFERNodes(Job job, boolean localTransfer, Collection files, int jobType, Map<String,Bundle.PoolTransfer> stageInMap, Bundle.BundleValue cValue, Implementation implementation) Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.
void	addStageInXFERNodes(Job job, Collection<FileTransfer> files, Collection<FileTransfer> symlinkFiles) Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.
protected void	constructCondorKey(Job job, String key, String value) Constructs a condor variable in the condor profile namespace associated with the job.
private Job	createSyncJobBetweenLevels(String name) It creates a NoOP synch job that runs on the submit host.
void	done() Signals that the traversal of the workflow is done.
protected String	getComputeJobBundleValue(Job job) Returns the bundle value associated with a compute job as a String.
String	getDescription() Returns a textual description of the transfer mode.
protected int	getSISiteBundleValue(String site, String deflt) Determines the bundle factor for a particular site on the basis of the stage in bundle value associcated with the underlying transfer transformation in the transformation catalog.
protected Bundle.PoolTransfer	getStageInPoolTransfer(Map<String,Bundle.PoolTransfer> stageInMap, String site, int num, boolean localTransfer) Returns the appropriate stagein pool transfer for a particular site.
Job	getSyncJob(String site) Returns the current synch job for a site.
protected String	getSyncJobBetweenLevelsName(String site, int level) Returns the name of the job that acts as a synchronization node in between stage in jobs of different levels.
protected void	initializeBundleValues() Initializes the bundle value variables, that are responsible determining the bundle values.
protected Map<String,Bundle.PoolTransfer>	resetStageInMap(Map<String,Bundle.PoolTransfer> stageInMap, Implementation implementation, Map<String,Job> transientSynchJobMap, int jobType, boolean createChildSyncJob, boolean localTransfer) Resets the stage in map and adds the stage in jobs for each site per level.
protected void	resetStageInMaps() Resets the stage in map and adds the stage in jobs for each site per level.

Methods inherited from class edu.isi.pegasus.planner.transfer.refiner.Bundle

addStageOutXFERNodes, doneStageIn, getDefaultBundleValueFromProperties, getStageOutPoolTransfer, resetStageOutMap, resetStageOutMaps

Methods inherited from class edu.isi.pegasus.planner.transfer.refiner.Basic

addInterSiteTXNodes, addJob, addRelation, addRelation, addStageInXFERNodes, addStageOutXFERNodes, appendAttribute, constructFileKey, createRegistrationJob, getJobPriority, logRefinerAction

Methods inherited from class edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner

loadImplementations

Methods inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DESCRIPTION

public static final String DESCRIPTION

A short description of the transfer refinement.

See Also:

Constant Field Values

DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR

public static final String DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR

The default clustering factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.

See Also:

Constant Field Values

DEFAULT_REMOTE_STAGE_IN_CLUSTER_FACTOR

public static final String DEFAULT_REMOTE_STAGE_IN_CLUSTER_FACTOR

The default clustering factor that identifies the number of transfer jobs that are being created per execution pool for stageing in data for the workflow.

See Also:

Constant Field Values

DEFAULT_LOCAL_STAGE_OUT_CLUSTER_FACTOR

public static final String DEFAULT_LOCAL_STAGE_OUT_CLUSTER_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.

See Also:

Constant Field Values

DEFAULT_REMOTE_STAGE_OUT_CLUSTER_FACTOR

public static final String DEFAULT_REMOTE_STAGE_OUT_CLUSTER_FACTOR

The default bundling factor that identifies the number of transfer jobs that are being created per execution pool for stageing out data for the workflow.

See Also:

Constant Field Values

mStageInLocalMapPerLevel

protected Map<String,Bundle.PoolTransfer> mStageInLocalMapPerLevel

A map indexed by site name, that contains the pointer to the stage in PoolTransfer objects for that site. This is per level of the workflow.

mStageInRemoteMapPerLevel

protected Map<String,Bundle.PoolTransfer> mStageInRemoteMapPerLevel

A map indexed by site name, that contains the pointer to the symlink stage in PoolTransfer objects for that site. This is per level of the workflow.

mCurrentSILevel

private int mCurrentSILevel

The current level of the jobs being traversed.

mSyncJobMap

private Map<String,Job> mSyncJobMap

Maps the site name to the current synch job

Constructor Detail

Cluster

public Cluster(ADag dag,
       PegasusBag bag)

The overloaded constructor.

Parameters:

dag - the workflow to which transfer nodes need to be added.

bag - the bag of initialization objects

Method Detail

initializeBundleValues

protected void initializeBundleValues()

Initializes the bundle value variables, that are responsible determining the bundle values.

Overrides:

initializeBundleValues in class Bundle

addStageInXFERNodes

public void addStageInXFERNodes(Job job,
                       Collection<FileTransfer> files,
                       Collection<FileTransfer> symlinkFiles)

Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.

Overrides:

addStageInXFERNodes in class Bundle

Parameters:

job - Job object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

symlinkFiles - Collection of FileTransfer objects containing source and destination file url's for symbolic linking on compute site.

addStageInXFERNodes

public void addStageInXFERNodes(Job job,
                       boolean localTransfer,
                       Collection files,
                       int jobType,
                       Map<String,Bundle.PoolTransfer> stageInMap,
                       Bundle.BundleValue cValue,
                       Implementation implementation)

Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.

Overrides:

addStageInXFERNodes in class Bundle

Parameters:

job - Job object corresponding to the node to which the files are to be transferred to.

files - Collection of FileTransfer objects containing the information about source and destURL's.

jobType - the type of transfer job being created

stageInMap - Map indexed by site name that gives all the transfers for that site.

cValue - used to determine the bundling factor to employ for a job.

implementation - the transfer implementation to use.

localTransfer - boolean indicating whether transfer has to happen on local site.

resetStageInMaps

protected void resetStageInMaps()

Resets the stage in map and adds the stage in jobs for each site per level.

resetStageInMap

protected Map<String,Bundle.PoolTransfer> resetStageInMap(Map<String,Bundle.PoolTransfer> stageInMap,
                                              Implementation implementation,
                                              Map<String,Job> transientSynchJobMap,
                                              int jobType,
                                              boolean createChildSyncJob,
                                              boolean localTransfer)

Resets the stage in map and adds the stage in jobs for each site per level.

Parameters:

stageInMap - the stagein map per level to be reset

implementation - the transfer implementation corresponding to the map being reset

transientSynchJobMap - transient map to keep track of synch jobs per site.

jobType - the type of Transfer Job created

createChildSyncJob - boolean indicating whether to create a child synch job or not.

Returns:

the reset map

getComputeJobBundleValue

protected String getComputeJobBundleValue(Job job)

Returns the bundle value associated with a compute job as a String.

Overrides:

getComputeJobBundleValue in class Bundle

Parameters:

job -

Returns:

value as String or NULL

getStageInPoolTransfer

protected Bundle.PoolTransfer getStageInPoolTransfer(Map<String,Bundle.PoolTransfer> stageInMap,
                                         String site,
                                         int num,
                                         boolean localTransfer)

Returns the appropriate stagein pool transfer for a particular site.

Parameters:

stageInMap - map that indexes site to PoolTransfer objects

site - the site for which the PT is reqd.

num - the number of stage in jobs required for that Pool.

localTransfer - whether the transfer needs to run on local site or not.

Returns:

the PoolTransfer

done

public void done()

Signals that the traversal of the workflow is done. At this point the transfer nodes are actually constructed traversing through the transfer containers and the stdin of the transfer jobs written.

Overrides:

done in class Bundle

getDescription

public String getDescription()

Returns a textual description of the transfer mode.

Overrides:

getDescription in class Bundle

Returns:

a short textual description

getSISiteBundleValue

protected int getSISiteBundleValue(String site,
                       String deflt)

Determines the bundle factor for a particular site on the basis of the stage in bundle value associcated with the underlying transfer transformation in the transformation catalog. If the key is not found, then the default value is returned. In case of the default value being null the global default is returned.

Parameters:

site - the site at which the value is desired.

deflt - the default value.

Returns:

the bundle factor.

See Also:

DEFAULT_LOCAL_STAGE_IN_CLUSTER_FACTOR

getSyncJobBetweenLevelsName

protected String getSyncJobBetweenLevelsName(String site,
                                 int level)

Returns the name of the job that acts as a synchronization node in between stage in jobs of different levels.

Parameters:

site - the site of the transfer job.

level - the level of the job

Returns:

name of synce job

createSyncJobBetweenLevels

private Job createSyncJobBetweenLevels(String name)

It creates a NoOP synch job that runs on the submit host.

Parameters:

name - of the job

Returns:

the noop job.

constructCondorKey

protected void constructCondorKey(Job job,
                      String key,
                      String value)

Constructs a condor variable in the condor profile namespace associated with the job. Overrides any preexisting key values.

Parameters:

job - contains the job description.

key - the key of the profile.

value - the associated value.

getSyncJob

public Job getSyncJob(String site)

Returns the current synch job for a site.

Parameters:

site -

Returns:

synch job if exists else null