Prometheus has solved problems of over 76 million degrees of freedom on 1024 PowerPC processors at LLNL.

We have recently add a new multigrid algorithm to our library: Atlas, an implementation of "smoothed aggregation" multigrid by Vanek, Mandel, and Brezina. Atlas shows promise, being as effective as Prometheus (an more effective for shell problems) while not requiring the rather complex construction of explicit coarse grids.

Prometheus system architecture

Sample FEAP problem solved with Olympus

Sample fine grid and two coarse grids

We use a parameterized mesh of a "hard" sphere included in a "soft"
material for scalability studies. We test problems from 79,679 degrees
of freedom to 39,160,959 degrees of freedom, on up to 960 processors of
an IBM PowerPC cluster at LLNL. Each version of the problem
is run with the number processors required to keep about 40,000 equations
on each processor. The interior sphere is composed of 17 layers of
alternating "hard" and "soft" materials. The hard material is steel-like
with a Poisson ratio of 0.3, and the soft material is rubber-like with
a Poisson ratio of 0.49, and elastic modulus 10^{-4} that of the
hard material. The solver is Preconditioned Conjugate gradient (PCG),
preconditioned with one "full" multigrid F-cycle, with Prometheus restriction
operators. The pre and post multigrid smoother is one applications
of PCG, preconditioned with a block Jacobi solver.

Times (sec) of solver on IBM PowerPC cluster with about 40,000 equations
per processor

Times (sec) of full nonlinear solve 10 Newton iterations and about
3500 total multigrid iterations) solver on IBM PowerPC cluster with about
40,000 equations per processor

Scaled speedup (Efficiencies)

Scaled speedup for components of one linear solve

Efficiencies for first linear solve of Prometheus/FEAP/PETSc/Epimetheus on IBM PowerPC cluster with about 40,000 equations per processor and upto 960 processors