Keywords: mesh multiplication, global refinement, parallel algorithm, petascale, hexahedral mesh, tube bundle.

The paper describes a mesh multiplication algorithm for the Code_Saturne software. The algorithm implements parallel global refinement of a mesh containing only hexahedral cells. Hexahedral meshes are often used for complex simulations that belong to a class of mathematical problems known as computational fluid dynamics (CFD). Using the algorithm, meshes with more than a billion cells are obtained that allows more accurate CFD simulations than those with the coarser mesh. The effectiveness of the implemented algorithm is demonstrated on practical examples and scalability results are also presented.

In this paper, we have considered the task of obtaining very large meshes for the purposes of the Code_Saturne computation. To be able to refine a fully defined simulation with initial boundary conditions, we have enhanced the algorithm for mesh multiplication, first described in [1], of meshes with hexahedral cells. We describe in detail the preprocessing of the mesh in the data format used in Code_Saturne. We explain how to: compute local and global numbering of new elements of the refined mesh; a strategy to avoid any unnecessary core-to-core communication during all steps of reminement; and independent single cell subdivision. This algorithm has been fully integrated into Code_Saturne and its functionality has been tested on the Cray XE6 (HECToR) and IBM Blue Gene/Q (Blue Joule) supercomputers. The tests prove good parallel scalability for the presented solution.

For testing purposes we use a problem from [2] starting with a coarse mesh (containing 13 millions of hexahedral cells) corresponding to a turbulent flow through a staggered tube bundle. Then we use mesh joining to create coarse meshes of 26M and 51M cells to fully test performance of the mesh multiplication algorithm. The results presented show the usefullness of algorithm to obtaining very fine solutions. We are able to prepare meshes containing billions of cells using the algorithm developed and solve such problems within a reasonable time. Refinement from millions to billions of cells took less than a minute, thus, using the algorithm presented is not a limitation of the whole computation. In our tests, we have tried more than three levels of refinement. The scalability is good on the IBM Blue Gene/Q (Blue Joule) supercomputer, but on Cray XE6 (HECToR) there are various conclusions about some of the results. More tests and performance analyses are proposed for further testing.

1

P. Kabelikova, A. Ronovsky, V. Vondrak, "Parallel Mesh Multiplication for Code_Saturne", PRACE project (FP7/2007-2013)

2

C. Moulinec, M.J.B.M. Pourquié, B.J. Boersma, T. Buchal, F.T.M. Nieuwstadt, "Direct Numerical Simulation on a Cartesian Mesh of the Flow through a Tube Bundle", International Journal of Computational Fluid Dynamics, Volume 18, Issue 1, pages 1-14, January 2004.

purchase the full-text of this paper (price £20)

go to the previous paper
go to the next paper
return to the table of contents
return to the book description
purchase this book (price £40 +P&P)