edu.isi.pegasus.planner.partitioner

Class Horizontal

java.lang.Object

edu.isi.pegasus.planner.partitioner.Partitioner

edu.isi.pegasus.planner.partitioner.BFS

edu.isi.pegasus.planner.partitioner.Horizontal

public class Horizontal
extends BFS

Horizontal based partitioning scheme, that allows the user to configure the number of partitions per transformation name per level. To set the size of the partition per transformation, the following properties need to be set

       pegasus.partitioner.horizontal.collapse.[txName]
       pegasus.partitioner.horizontal.bundle.[txName]
 

The bundle value designates the number of partitions per transformation per level. The collapse values designates the number of nodes in a partitioning referring to a particular transformation. If both are specified, then bundle value takes precedence.

Version:

$Revision$

Author:

Karan Vahi

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	Horizontal.GraphNodeComparator A GraphNode comparator, that allows us to compare nodes according to the transformation logical names.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_COLLAPSE_FACTOR The default collapse factor for collapsing jobs with same logical name scheduled onto the same execution pool.
static String	DESCRIPTION A short description about the partitioner.
private int	mIDCounter The global counter that is used to assign ID's to the partitions.
private Horizontal.GraphNodeComparator	mNodeComparator A static instance of GraphNode comparator.
private Map	mPartitionMap A map indexed by the partition ID.

Fields inherited from class edu.isi.pegasus.planner.partitioner.Partitioner

mGraph, mLogger, mProps, mRoot, PACKAGE_NAME, VERSION

Constructor Summary

Constructors

Constructor and Description
Horizontal(GraphNode root, Map graph, PegasusProperties properties) The overloaded constructor.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	constructLevelRelations(Callback c, int parent, int child) Calls out to the callback with appropriate relations between the partitions constructed for the levels.
protected void	constructPartitions(Callback c, List nodes, int level) Given a list of jobs, constructs (one or more) partitions out of it.
protected void	constructPartitions(Callback c, List nodes, int level, String name) Given a list of jobs, constructs (one or more) partitions out of it.
protected Partition	createPartition(List nodes) Creates a partition out of a list of nodes.
String	description() Returns a textual description of the partitioner implementation.
protected void	done(Callback c) Indicates that we are done with the traversal of the graph.
protected int[]	getCollapseFactor(String txName, int size) Returns the collapse factor, that is used to determine the number of nodes going in a partition.
private String	getPartitionID(int id) Constructs the id for the partition.
private int	idCounter() Returns the current value of the ID counter.
private void	incrementIDCounter() Increments the ID counter by 1.
private Comparator	nodeComparator() Singleton access to the job comparator.

Methods inherited from class edu.isi.pegasus.planner.partitioner.BFS

determinePartitions

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DESCRIPTION

public static final String DESCRIPTION

A short description about the partitioner.

See Also:

Constant Field Values

DEFAULT_COLLAPSE_FACTOR

public static final int DEFAULT_COLLAPSE_FACTOR

The default collapse factor for collapsing jobs with same logical name scheduled onto the same execution pool.

See Also:

Constant Field Values

mPartitionMap

private Map mPartitionMap

A map indexed by the partition ID. Each value is a partition object.

mNodeComparator

private Horizontal.GraphNodeComparator mNodeComparator

A static instance of GraphNode comparator.

mIDCounter

private int mIDCounter

The global counter that is used to assign ID's to the partitions.

Constructor Detail

Horizontal

public Horizontal(GraphNode root,
          Map graph,
          PegasusProperties properties)

The overloaded constructor.

Parameters:

root - the dummy root node of the graph.

graph - the map containing all the nodes of the graph keyed by the logical id of the nodes.

properties - the properties passed to the planner.

Method Detail

nodeComparator

private Comparator nodeComparator()

Singleton access to the job comparator.

Returns:

the job comparator.

description

public String description()

Returns a textual description of the partitioner implementation.

Overrides:

description in class BFS

Returns:

a short textual description

constructPartitions

protected void constructPartitions(Callback c,
                       List nodes,
                       int level)

Given a list of jobs, constructs (one or more) partitions out of it. Calls out to the partitioner callback, for each of the partitions constructed.

Overrides:

constructPartitions in class BFS

Parameters:

c - the parititoner callback

nodes - the list of GraphNode objects on a particular level.

level - the level as determined from the root of the workflow.

constructPartitions

protected void constructPartitions(Callback c,
                       List nodes,
                       int level,
                       String name)

Given a list of jobs, constructs (one or more) partitions out of it. Calls out to the partitioner callback, for each of the partitions constructed.

Parameters:

c - the parititoner callback

nodes - the list of GraphNode objects on a particular level, referring to the same transformation underneath.

level - the level as determined from the root of the workflow.

name - the transformation name

constructLevelRelations

protected void constructLevelRelations(Callback c,
                           int parent,
                           int child)

Calls out to the callback with appropriate relations between the partitions constructed for the levels. This is an empty implementation, as we do our own book-keeping in this partitioner to determine the relations between the partitions.

Overrides:

constructLevelRelations in class BFS

Parameters:

c - the parititoner callback

parent - the parent level

child - the child level.

See Also:

done( Callback )

done

protected void done(Callback c)

Indicates that we are done with the traversal of the graph. Determines the relations between the partitions constructed and calls out to the appropriate callback function

Overrides:

done in class BFS

Parameters:

c - the partitioner callback

getCollapseFactor

protected int[] getCollapseFactor(String txName,
                      int size)

Returns the collapse factor, that is used to determine the number of nodes going in a partition. The collapse factor is determined by getting the collapse and the bundle values specified for the transformations in the properties file. There are two orthogonal notions of bundling and collapsing. In case the bundle key is specified, it ends up overriding the collapse key, and the bundle value is used to generate the collapse values. If both are not specified or null, then collapseFactor is set to size.

Parameters:

txName - the logical transformation name

size - the number of jobs that refer to the same logical transformation and are scheduled on the same execution pool.

Returns:

int array of size 2 where :- int[0] is the the collapse factor (number of nodes in a partition) int[1] is the number of parititons for whom collapsing is int[0] + 1.

createPartition

protected Partition createPartition(List nodes)

Creates a partition out of a list of nodes. Also stores it in the internal partition map to track partitions later on. Associates the partition ID with each of the nodes making the partition also.

Parameters:

nodes - the list of GraphNodes making the partition.

Returns:

the partition out of those nodes.

incrementIDCounter

private void incrementIDCounter()

Increments the ID counter by 1.

idCounter

private int idCounter()

Returns the current value of the ID counter.

getPartitionID

private String getPartitionID(int id)

Constructs the id for the partition.

Parameters:

id - an integer ID.

Returns:

the ID for the Partition.