Amesos_Scalapack: A serial and parallel dense solver. For now, we implement only the unsymmetric ScaLAPACK solver. More...

#include <Amesos_Scalapack.h>

Inheritance diagram for Amesos_Scalapack:

[legend]

Collaboration diagram for Amesos_Scalapack:

[legend]

Public Member Functions
	Amesos_Scalapack (const Epetra_LinearProblem &LinearProblem)
	Amesos_Scalapack Constructor.
	~Amesos_Scalapack (void)
	Amesos_Scalapack Destructor.
int	SymbolicFactorization ()
	Performs SymbolicFactorization on the matrix A.
int	NumericFactorization ()
	Performs NumericFactorization on the matrix A.
int	Solve ()
	Solves A X = B (or A^T X = B).
const Epetra_LinearProblem *	GetProblem () const
	Get a pointer to the Problem.
bool	MatrixShapeOK () const
	Returns true if SCALAPACK can handle this matrix shape.
int	SetUseTranspose (bool UseTranspose)
	SetUseTranpose(true) is more efficient in Amesos_Scalapack.
bool	UseTranspose () const
	Returns the current UseTranspose setting.
const Epetra_Comm &	Comm () const
	Returns a pointer to the Epetra_Comm communicator associated with this matrix.
int	SetParameters (Teuchos::ParameterList &ParameterList)
	Updates internal variables.
int	NumSymbolicFact () const
	Returns the number of symbolic factorizations performed by this object.
int	NumNumericFact () const
	Returns the number of numeric factorizations performed by this object.
int	NumSolve () const
	Returns the number of solves performed by this object.
void	PrintTiming () const
	Print timing information.
void	PrintStatus () const
	Print information about the factorization and solution phases.
void	GetTiming (Teuchos::ParameterList &TimingParameterList) const
	Extracts timing information from the current solver and places it in the parameter list.
Public Member Functions inherited from Amesos_BaseSolver
virtual	~Amesos_BaseSolver ()
	Destructor.
virtual void	setParameterList (Teuchos::RCP< Teuchos::ParameterList > const &paramList)
	Redefined from Teuchos::ParameterListAcceptor (Does Not Work).
virtual Teuchos::RCP< Teuchos::ParameterList >	getNonconstParameterList ()
	This is an empty stub.
virtual Teuchos::RCP< Teuchos::ParameterList >	unsetParameterList ()
	This is an empty stub.

Protected Attributes
int	iam_
int	NumGlobalElements_
int	NumGlobalNonzeros_
int	nprow_
int	npcol_
int	ictxt_
int	m_per_p_
int	DescA_ [10]
Epetra_Map *	ScaLAPACK1DMap_
Epetra_CrsMatrix *	ScaLAPACK1DMatrix_
Epetra_Map *	VectorMap_
std::vector< double >	DenseA_
std::vector< int >	Ipiv_
int	NumOurRows_
int	NumOurColumns_
bool	UseTranspose_
const Epetra_LinearProblem *	Problem_
double	ConTime_
double	SymTime_
double	NumTime_
double	SolTime_
double	VecTime_
double	MatTime_
bool	TwoD_distribution_
int	grid_nb_
int	mypcol_
int	myprow_
Epetra_CrsMatrix *	FatOut_
int	nb_
int	lda_
Epetra_Time *	Time_

Detailed Description

Amesos_Scalapack: A serial and parallel dense solver. For now, we implement only the unsymmetric ScaLAPACK solver.

Amesos_Scalapack, an object-oriented wrapper for LAPACK and ScaLAPACK, will solve a linear systems of equations: A X = B using Epetra objects and the ScaLAPACK library, where A is an Epetra_RowMatrix and X and B are Epetra_MultiVector objects.

Amesos_Scalapack can be competitive for matrices that are not particularly sparse. ScaLAPACK solves matrices for which the fill-in is roughly 10% to 20% of the matrix size in time comparable to that achieve by other Amesos classes. Amesos_Scalapack scales well and hence its performance advantage will be largest when large number of processes are involved.

Amesos_Scalapack uses the ScaLAPACK functions PDGETRF and PDGETRS if more than one process is used. If only one process is used, Amesos_ScaLAPACK uses the LAPACK function PDGETRF and PDGETRS.

AmesosScaLAPACK uses full partial pivoting and will therefore provide answers that are at least as accurate as any direct sparse solver.

AmesosScalapack makes sense under the following circumstances:

There is sufficient memory to store the entrie dense matrix. 8*n^2/p bytes will be required on each process. -AND- one of the following
- The matrix is relatively small and dense. Amesos_Scalapack will solve matrices less than 100 by 100 faster than other Amesos classes unless the matrices are very sparse.
- The matrix is relatively dense and many processes are available. If a thousand processes are available, Amesos_Scalapack should be competetive with other sparse direct solvers even for matrices whose L and U factors contain only 5% non-zeros.
- The matrix is quite dense. Amesos_Scalapack will be well on any matrix whose L and U factors contain 20% or more non-zeros.
- Execution time is less important than robustness. Amesos_Scalapack is among the most robust parallel direct solvers.

Common control parameters :

Amesos_Scalapack supports the following parameters which are common to across multiple Amesos solvers:

ParamList.set("MaxProcs", int MaximumProcessesToUse );
By default, this is set to -1, which causes Amesos_Scalapack to use a heuristic to determine how many processes to use. If set to a postive value, MaximumProcessesToUse, Amesos_Scalapack will use MaximumProcessesToUse provided that there are that many processes available. Testing should be performed with MaximumProcessesToUse set to some value larger than one to force parallel execution.
ParamList.set("PrintTiming", bool );
ParamList.set("PrintStatus", bool );
ParamList.set("ComputeVectorNorms", bool );
ParamList.set("ComputeTrueResidual", bool );
ParamList.set("OutputLevel", int );
ParamList.set("DebugLevel", int );
ParamList.set("ComputeTrueResidual", bool );

Amesos_Scalapack supports the following parameters specific to Amesos_Scalapack.
Teuchos::ParameterList ScalapackParams = ParameterList.sublist("Scalapack") ;

ScalapackParams.set("2D distribution", bool );
By default this is set "true". In general, because a two dimensional data distribution generally produces faster results.
However, in some cases, a one dimensional data distribution may provide faster execution time. The code for the one dimensional data distribution uses a different data redistribution algorithm and uses the transpose of the matrix internally (all of which is transparent to the user).
ScalapackParams.set("grid_nb", bool );
By default this is set to 32. On some machines, it may be possible to improve performance by up to 10% by changing the value of grid_nb. (16,24,48,64 or 128) are reasonable values to try. For testing on small matrices, small values of grid_nb will (if "MaxProcs" is set to a value greater than 1) force the code to execute in parallel.

Limitations:

None of the following limitations would be particularly difficult to remove.

The present implementation limits the number of right hand sides to the number of rows assigned to each process. i.e. nrhs < n/p.

The present implementation does not take advantage of symmetric or symmetric positive definite matrices, although ScaLAPACK has separate routines to take advantages of such matrices.

Constructor & Destructor Documentation

◆ Amesos_Scalapack()

Amesos_Scalapack::Amesos_Scalapack ( const Epetra_LinearProblem & LinearProblem )

Amesos_Scalapack Constructor.

Creates an Amesos_Scalapack instance, using an Epetra_LinearProblem, passing in an already-defined Epetra_LinearProblem object.

Note: The operator in LinearProblem must be an Epetra_RowMatrix.

References SetParameters().

◆ ~Amesos_Scalapack()

Amesos_Scalapack::~Amesos_Scalapack ( void )

Amesos_Scalapack Destructor.

Completely deletes an Amesos_Scalapack object.

References PrintStatus(), Amesos_Status::PrintStatus_, PrintTiming(), Amesos_Status::PrintTiming_, and Amesos_Status::verbose_.

Member Function Documentation

◆ Comm()

const Epetra_Comm & Amesos_Scalapack::Comm ( ) const

inlinevirtual

Returns a pointer to the Epetra_Comm communicator associated with this matrix.

Implements Amesos_BaseSolver.

References Comm(), and GetProblem().

Referenced by Comm(), NumericFactorization(), Solve(), and SymbolicFactorization().

◆ GetProblem()

const Epetra_LinearProblem * Amesos_Scalapack::GetProblem ( ) const

inlinevirtual

Get a pointer to the Problem.

Implements Amesos_BaseSolver.

Referenced by Comm(), and MatrixShapeOK().

◆ GetTiming()

void Amesos_Scalapack::GetTiming ( Teuchos::ParameterList & TimingParameterList ) const

inlinevirtual

Extracts timing information from the current solver and places it in the parameter list.

Reimplemented from Amesos_BaseSolver.

References Amesos_Time::GetTiming().

◆ MatrixShapeOK()

bool Amesos_Scalapack::MatrixShapeOK ( ) const

virtual

Returns true if SCALAPACK can handle this matrix shape.

Returns true if the matrix shape is one that SCALAPACK can handle. SCALAPACK only works with square matrices.

Implements Amesos_BaseSolver.

References GetProblem().

◆ NumericFactorization()

int Amesos_Scalapack::NumericFactorization ( )

virtual

Performs NumericFactorization on the matrix A.

In addition to performing numeric factorization on the matrix A, the call to NumericFactorization() implies that no change will be made to the underlying matrix without a subsequent call to NumericFactorization().

preconditions:

GetProblem().GetOperator() != 0 (return -1)
MatrixShapeOk(GetProblem().GetOperator()) == true (return -6) NOT IMPLEMENTED
The non-zero structure of the matrix should not have changed since the last call to SymbolicFactorization(). Irrelevant for Amesos_Scalapack.
The distribution of the matrix should not have changed since the last call to SymbolicFactorization(). Irrelevant for Amesos_Scalapack.

postconditions:

nprow_, npcol_, DescA_
DenseA will be factored
Ipiv_ contains the pivots

Returns: Integer error code, set to 0 if successful.

Implements Amesos_BaseSolver.

References Comm(), Amesos_Status::debug_, and Amesos_Status::NumNumericFact_.

◆ NumNumericFact()

int Amesos_Scalapack::NumNumericFact ( ) const

inlinevirtual

Returns the number of numeric factorizations performed by this object.

Implements Amesos_BaseSolver.

References Amesos_Status::NumNumericFact_.

◆ NumSolve()

int Amesos_Scalapack::NumSolve ( ) const

inlinevirtual

Returns the number of solves performed by this object.

Implements Amesos_BaseSolver.

References Amesos_Status::NumSolve_.

◆ NumSymbolicFact()

int Amesos_Scalapack::NumSymbolicFact ( ) const

inlinevirtual

Returns the number of symbolic factorizations performed by this object.

Implements Amesos_BaseSolver.

References Amesos_Status::NumSymbolicFact_.

◆ PrintStatus()

void Amesos_Scalapack::PrintStatus ( ) const

virtual

Print information about the factorization and solution phases.

Implements Amesos_BaseSolver.

Referenced by ~Amesos_Scalapack().

◆ PrintTiming()

void Amesos_Scalapack::PrintTiming ( ) const

virtual

Print timing information.

Implements Amesos_BaseSolver.

References Amesos_Status::NumNumericFact_, Amesos_Status::NumSolve_, and Amesos_Status::NumSymbolicFact_.

Referenced by ~Amesos_Scalapack().

◆ SetParameters()

int Amesos_Scalapack::SetParameters ( Teuchos::ParameterList & ParameterList )

virtual

Updates internal variables.

  <br \>Preconditions:<ul>
  <li>None.</li>
  </ul>

  <br \>Postconditions:<ul> 
  <li>Internal variables controlling the factorization and solve will
  be updated and take effect on all subsequent calls to NumericFactorization() 
  and Solve().</li>
  <li>All parameters whose value are to differ from the default values must

be included in ParameterList. Parameters not specified in ParameterList revert to their default values.

Returns: Integer error code, set to 0 if successful.

Implements Amesos_BaseSolver.

References Amesos_Status::debug_.

Referenced by Amesos_Scalapack().

◆ SetUseTranspose()

int Amesos_Scalapack::SetUseTranspose ( bool UseTranspose )

inlinevirtual

SetUseTranpose(true) is more efficient in Amesos_Scalapack.

If SetUseTranspose() is set to true,
- A^T X = B
  
  is computed
else
- A X = B
  
  is computed

Implements Amesos_BaseSolver.

References UseTranspose().

◆ Solve()

int Amesos_Scalapack::Solve ( )

virtual

Solves A X = B (or A^T X = B).

preconditions:

GetProblem().GetOperator() != 0 (return -1)
MatrixShapeOk(GetProblem().GetOperator()) == true (return -6) NOT IMPLEMENTED
X and B must have the same shape (NOT CHECKED)
X and B must have fewer than nb right hand sides. EPETRA_CHK_ERR(-2)
GetProblem()->CheckInput (see Epetra_LinearProblem::CheckInput() for return values)
The matrix should not have changed since the last call to NumericFactorization().

postconditions:

X will be set such that A X = B (or A^T X = B), within the limits of the accuracy of the the Scalapack solver.

Returns: Integer error code, set to 0 if successful.

Implements Amesos_BaseSolver.

References Comm(), Amesos_Status::ComputeTrueResidual_, Amesos_Status::ComputeVectorNorms_, Amesos_Status::debug_, Amesos_Status::NumSolve_, UseTranspose(), and Amesos_Status::verbose_.

◆ SymbolicFactorization()

int Amesos_Scalapack::SymbolicFactorization ( )

virtual

Performs SymbolicFactorization on the matrix A.

There is no symbolic factorization phase in ScaLAPACK, as it operates only on dense matrices. Hence, Amesos_Scalapack::SymbolicFactorization() takes no action.

Returns: Integer error code, set to 0 if successful.

Implements Amesos_BaseSolver.

References Comm(), Amesos_Status::debug_, and Amesos_Status::NumSymbolicFact_.

◆ UseTranspose()

bool Amesos_Scalapack::UseTranspose ( ) const

inlinevirtual

Returns the current UseTranspose setting.

Implements Amesos_BaseSolver.

Referenced by SetUseTranspose(), and Solve().

The documentation for this class was generated from the following files:

Amesos_Scalapack.h
Amesos_Scalapack.cpp

Public Member Functions

Protected Attributes

Detailed Description

Common control parameters :

Limitations:

Constructor & Destructor Documentation

◆ Amesos_Scalapack()

◆ ~Amesos_Scalapack()

Member Function Documentation

◆ Comm()

◆ GetProblem()

◆ GetTiming()

◆ MatrixShapeOK()

◆ NumericFactorization()

◆ NumNumericFact()

◆ NumSolve()

◆ NumSymbolicFact()

◆ PrintStatus()

◆ PrintTiming()

◆ SetParameters()

◆ SetUseTranspose()

◆ Solve()

◆ SymbolicFactorization()

◆ UseTranspose()