Olympus- scalable unstructured finite element solver

Prometheus is a multigrid solver for finite element matrices on unstructured meshes in solid mechanics. Prometheus uses maximal independent sets on modified matrix graphs; these node sets are meshed with a Delaunay mesher to provide coarse "grids", which are used with standard finite element shape functions to provide interpolation operators for multigrid. Coarse grid operators are formed with these interpolation operators and the finer grid matrix in a standard Galerkin way.  Prometheus is built on PETSc from  Argonne National Laboratory, and uses ParMetis from the University of Minnesota and FEAP from the University of California at Berkeley.

Prometheus has solved problems of over 76 million degrees of freedom on 1024 PowerPC processors at LLNL.

We have recently add a new multigrid algorithm to our library: Atlas, an implementation of "smoothed aggregation" multigrid by Vanek, Mandel, and Brezina.  Atlas shows promise, being as effective as Prometheus (an more effective for shell problems) while not requiring the rather complex construction of explicit coarse grids.


Olympus system architecture

Prometheus system architecture


Sample FEAP problem solved with Olympus


Sample fine grid and two coarse grids


We use a parameterized mesh of a "hard" sphere included in a "soft" material for scalability studies.  We test problems from 79,679 degrees of freedom to 39,160,959 degrees of freedom, on up to 960 processors of an IBM PowerPC cluster at  LLNL.  Each version of the problem is run with the number processors required to keep about 40,000 equations on each processor. The interior sphere is composed of 17 layers of  alternating "hard" and "soft" materials.  The hard material is steel-like  with a Poisson ratio of 0.3, and the soft material is rubber-like with a Poisson ratio of 0.49, and elastic modulus 10-4 that of the hard material.  The solver is Preconditioned Conjugate gradient (PCG), preconditioned with one "full" multigrid F-cycle, with Prometheus restriction operators.  The pre and post multigrid smoother is one applications of PCG,  preconditioned with a block Jacobi solver.



Times (sec) of solver on IBM PowerPC cluster with about 40,000 equations per processor
 


Times (sec) of full nonlinear solve 10 Newton iterations and about 3500 total multigrid iterations) solver on IBM PowerPC cluster with about 40,000 equations per processor
 



Scaled speedup (Efficiencies)


Scaled speedup for components of one linear solve

Efficiencies for first linear solve of Prometheus/FEAP/PETSc/Epimetheus on IBM PowerPC cluster  with about 40,000 equations per processor and upto 960 processors