Spanish Pavillion

Shared Research Computing Policy Advisory Committee

The Shared Research Computing Policy Advisory Committee (SRCPAC) is a faculty-dominated group focused on a variety of policy issues related to shared high performance computing (HPC) for researchers on the Morningside and Manhattanville campuses. As an increasing number of disciplines engage in data-driven and/or computational methods in order to create and disseminate knowledge, there is a commensurate rise and duplication in the costs of individually establishing and maintaining the necessary resources. Sharing an independently-maintained HPC simultaneously reduces expense while providing access to a larger machine than individual groups of researchers can typically deploy.

SRCPAC is administratively supported by the Research Computing Services group within CU Information Technology, and the Office of Research Initiatives. The Committee additionally works in close partnership with Columbia University Libraries and the Data Science Institute. Please contact us at any time by emailing [email protected].

The Ginsburg high performance computing cluster, a $3.1M joint purchase by 33 research groups and units, went live in February 2021, expanded the following Spring of 2022, and was further expanded in Spring of 2023. When fully installed, the system will consist of 286 Nodes with a total of 9152 cores, including 39 GPU hardware-accelerated systems allowing certain highly-parallelized applications to achieve performance levels far beyond what would be possible on conventional hardware. In addition, a new GPU-only cluster of 15 nodes with a total of 1312 cores will also be installed in Spring/Summer of 2023.

Ginsburg Cluster Specifications

All servers are equipped with Dual Intel Xeon Gold 6226R processors (2.9 GHz):

  • 191 Standard Nodes (192 GB)
  • 56 High Memory Nodes (768 GB)
  • 18 RTX 8000 GPU nodes (2 GPUs modules per server)
  • 4 V100S GPU nodes (2 GPU modules per server)
  • 8 A100 GPU Nodes (2 GPU modules per server)
  • 9 A40 GPU Nodes (2 GPU modules per server)
  • 1PB of DDN ES7790 Lustre storage
  • HDR Infiniband
  • Red Hat Enterprise Linux 8
  • Slurm job scheduler

For more information, please visit Columbia's Shared High Performance Computing webpage.

Dear Colleagues,

I write to you enthusiastic over the Shared Research Computing Policy Advisory Committee’s (SRCPAC) advancement of the University’s high-performance computing resource. From its humble beginnings in 2011, research computing at Columbia is now something new under the sun.

As SRCPAC’s Chair, I am tasked with representing faculty interests in comprehensive governance of the shared research computing facility (SRCF). Our community – including rotating subcommittees and working groups devoted to multiple strategic initiatives – is comprised of over 150 faculty, postdocs, staff, and students, and meets semiannually to review topics of considerable range, including cloud computing, educational workshops, facility operations, and policy changes. All Columbia faculty are invited and strongly encouraged to attend SRCPAC meetings; a faculty designee can attend in the event of scheduling conflicts. I hope that you will join us for the many discussions that the future holds.

Formed in 2011, SRCPAC is the manifestation of a movement many years in the making. It is a unified effort in further developing the physical infrastructure, administrative network, and governance policies that are fundamental to innovative computational research and supporting corresponding grant-making activities. Columbia is a global leader in integrating data science methodologies across all domains and disciplines; this leadership is powerfully represented by our Data Science Institute, among many other academic units. The University is committed to furthering this integration and capitalizing upon new emergent opportunities in computationally-driven discovery. This is SRCPAC.

In Fall 2016, SRCPAC achieved yet another seminal milestone: the installation of Habanero, Columbia's third high performance computing cluster for shared use among Columbia’s researchers. At an initial cost of $1.5 Million and expanded the following year, Habanero consists of 44 discrete group purchases and contains 302 compute notes and 800 terabytes of storage. Terremoto, Columbia's fourth HPC cluster to enter production, went live in December 2018 and was expanded in 2019. Terremoto is a joint purchase of 35 research groups and departments and consists of 137 compute nodes and over half a petabyte of storage. And most recently, the Ginsburg cluster, a joint purchase by 33 research groups and departments, was launched in February 2021. Ginsburg consists of 139 nodes including 22 GPU hardware accelerated systems.

These significant strides were made possible in no small part by the tireless efforts and commitments of the Faculty of Arts and Sciences, The Fu Foundation School of Engineering & Applied Science, and CUIT, making Habanero and Terremoto collective achievements for which we should all be proud. The High Performance Operating Committee of users is chaired by my colleague, Dr. Kyle Mandli, Assistant Professor, Department of Applied Physics & Applied Mathematics.

We are at a pivotal time in research – both generally and especially so at Columbia and I encourage you to explore the wealth of information found below regarding SRCPAC’s mission, structure, and emergent themes.

Your inquiries and comments are welcome as we collectively decide how to navigate the future of research computing. As SRCPAC is a joint effort, there are two methods for communicating with staff resources:

  • For technical questions related to HPC use or for general research computing support, please contact CUIT Research Computing Services at [email protected];
  • For policy, governance, and faculty affairs questions, please contact the Office of Research Initiatives at [email protected].

Thank you again for joining us in this exciting endeavor – we look forward to working with you.

Chris Marianetti, PhD
Chair, Shared Research Computing Policy Advisory Committee
Associate Professor, Department of Applied Physics & Applied Mathematics

To provide researchers access to High Performance Computing (HPC) clusters larger than individual researchers can typically afford or wish to individually acquire and maintain, Columbia has created the Shared Research Computing Facility (SRCF) for Morningside, Lamont, and Manhattanville researchers to jointly acquire and use HPC clusters.

We hope the following information will be useful as you develop your research program:

  • If you wish to join the SRCPAC ListServ to keep informed of committee meetings and other important announcements pertaining to Columbia Shared HPC, please email [email protected].
  • There are three active Shared HPC clusters, including Terremoto, Habanero, and the newest system, Ginsburg. These clusters are governed by a faculty-led community, the Shared Research Computing Policy Advisory Committee (SRCPAC), and are administratively supported by full-time staff within the CUIT Research Computing Services team, providing maintenance, technical support, software installation and guidance on future computing needs.
  • Typically each Spring, to coincide with recruiting season, faculty are polled to see if there is interest in a joint expansion round or new system purchase. A good way to ensure you are aware of upcoming events is to join the SRCPAC ListServ by emailing [email protected].
  • Research Computing Services (RCS) within CUIT – the entity that administratively supports the SRCF – holds online zoom office hours for HPC users from 3:00 p.m. – 5:00 p.m. on the first Monday of each month. Please RSVP here if interested. The RCS team is happy to answer questions about the SRCF, Columbia’s agreement with Google Cloud Platform, Amazon Web Services, and access to external Government-supported resources (such as XSEDE).

If you have additional questions about the above broad overview, please feel free to email [email protected]. We very much hope to have you involved in governing and advancing the research computing infrastructure across Columbia University, and welcome!

Published research emerging out of computations run on our HPCs must recognize the grants that have made this service possible. We ask that all related publications include the following acknowledgement text:

We acknowledge computing resources from Columbia University's Shared Research Computing Facility project, which is supported by NIH Research Facility Improvement Grant 1G20RR030893-01, and associated funds from the New York State Empire State Development, Division of Science Technology and Innovation (NYSTAR) Contract C090171, both awarded April 15, 2010.

Version 4, 11.10.22

If you wish to include the purchase of nodes in the SRCPAC shared clusters in your proposal, you will want to include language in both the Facilities, Equipment and Other Resources (or equivalent) section of your proposal and in the Budget and Budget Justification.   Sample approaches and links to relevant information below.  You will need to adjust to the specific requests of your RFP, BAA or other. 

PLEASE NOTE THAT THE LINKS TO WEBSITES ARE FOR INFORMATION ONLY AND SHOULD NOT BE INCLUDED IN THE PROPOSAL ITSELF. 

 

Facilities, Equipment and Other Resources

Recommended Language If you wish to describe existing resources

[We/I/center name or other appropriate descriptor] will have access to the [insert name of relevant cluster], housed in the Columbia Shared Research Computing Facility, governed by the faculty-led Shared Research Computing Policy Advisory Committee (SRCPAC.)  Columbia’s Research Computing Services team administers the facility on a 24/7 basis.  [if relevant:  All members of the research team will have access to this resource.]    The [insert name of cluster] has total nodes of [get latest specs here: https://research.columbia.edu/content/srcpac]. 

Recommended Language if you wish to purchase nodes

To perform the research outlined, [we/I/center name] intend to purchase [X] [regular, high memory, GPU, …] nodes and [Y] storage as part of Columbia’s shared high-performance computing (HPC) cluster.  These clusters reside in the Columbia Shared Research Computing Facility, governed by the faculty-led Shared Research Computing Policy Advisory Committee (SRCPAC).  Columbia’s Research Computing Services team administers the facility on a 24/7 basis.  [if relevant:  All members of the research team will have access to this resource.]   

Every year, SRCPAC offers all researchers the chance to buy into the cluster through the purchase of nodes and storage. The purchase of a node includes peripherals (cabling, network, warranty) and support and administration for the life of the hardware (typically 5 years). The current[hh1]  cluster is:   [check this website for the specs if you wish to include specs https://research.columbia.edu/content/srcpac.]  We anticipate pricing close to the most recent purchase round [The most recent specific choices and costs can be found on the website under “Archive:  Price History by Cluster”  Or reach out to Research Computing Services at https://www.cuit.columbia.edu/about-research-computing-services.]

Budget Process Guidance

Although each sponsor may have different budgeting requirements, when budgeting for purchase of High Performance Computing (HPC) nodes and active storage in a SRCPAC shared cluster on a sponsored project, it is strongly encouraged to provide as much detail in the budget justification as possible. Examples at a minimum would include: 

  • how many nodes and how much active storage are needed?
  • What shared cluster will you be using? Or will you participate in purchase of a new cluster?  Typically SRCPAC annually alternates between adding to a current cluster and starting a new one.
  • At what price/node?  Note prices differ between regular, high-memory, GPU etc.

Vendor quotes may be required as backup documentation, should the sponsor request it. For purposes of the proposal, we recommend using the latest pricing which you can find.  You may wish to state explicitly that if selected for funding, a revised quote can be provided. The most recent specific choices and costs can be found on the website under “Archive:  Price History by Cluster.”  https://research.columbia.edu/content/srcpac  Note that the price history should be used to give you an estimate using the most recent information available on costs. If your proposal is awarded, the actual costs may differ. The sponsor may request an updated quote.

 

PLEASE NOTE THAT THIS IS ACTIVE STORAGE THAT IS USED WHILE COMPUTING.  IT DOES NOT FULFILL THE PURPOSE OF PERMANENT OR ARCHIVAL STORAGE.  For storage options, consult: https://research.columbia.edu/content/research-data-storage.

 

Example:

Several computer nodes will be purchased to exclusively support the code development and computation

associated with the research in this grant, and these nodes will be managed by Columbia within our Shared Research Computing Facility. Each node will approximately consist of a dual socket mother board containing two 16 core Intel Xeon processors. Two standard nodes, each having 192GB of RAM, will be purchased during the first year, with an estimated cost of approximately $7,850/node (see attached quote from previous purchase round), accounting for the $15,426 budgeted during year 1. A high memory node, containing 768 GB of RAM, will also be purchased, accounting for the $11,658 budgeted during year 2. Finally, approximately 11 TB of storage space will be purchased at an estimated cost of approximately $350/TB, accounting for the $4,341 budgeted during year 3.

The Shared Research Computing Facility (SRCF) consists of a dedicated portion of the university data center on the Morningside Campus.  It is dedicated to house shared computing resources managed by CUIT, such as Columbia's centrally-managed High Performance Computing (HPC) and the Secure Data Enclave (SDE). 

A project to upgrade the electrical infrastructure of the data center was completed in Summer 2013*.

In 2018**, cooling was expanded to increase capacity to accommodate shared computing into the foreseeable future.


*The Shared Research Computing Facility project is supported by NIH Research Facility Improvement Grant 1G20RR030893-01, and associated funds from the New York State Empire State Development, Division of Science Technology and Innovation (NYSTAR) Contract C090171, both awarded April 15, 2010.

**The 2018 Cooling expansion is supported by joint contributions from CUIT, the Office of the Executive Vice President for Research, Arts and Sciences, and Engineering and Applied Science. 

Excerpt from the SRCPAC Charter, November 9, 2011:

"The Shared Research Computing Policy Advisory Committee (SRCPAC) will be a faculty-dominated group focused on a variety of policy issues related to shared research computing on the Morningside campus. As the use of computational tools spreads to more disciplines to create, collaborate, and disseminate knowledge, there is a commensurate rise in the costs of establishing and maintaining these resources. Shared resources have proven to leverage those available to individuals or small groups, but require careful consideration of the policies governing the shared resource and the basis of the operating model.

While final authority and responsibility for such policies customarily rests with the senior administrators of the University, it is vital that the research faculty examine and recommend the policies and practices they deem best suited to accomplishing the research objectives."

For more information regarding shared research computing at Columbia University, or to register for the SRCPAC ListServ, please email [email protected].

SRCPAC Proposed Mandate

The University’s shared research computing clusters are not authorized to host HIPAA-protected data. Therefore, the collection, storage, or transmission of Sensitive Data, as defined within the Columbia University Data Classification Policy, is strictly prohibited on Columbia's shared HPC machines.

HPC Resources and Training

A number of no-cost internal and external resources exist to train new and existing users in computational methodologies, high performance computing, and data science. These computational research training resources are available to Columbia students, faculty, and staff.

Foundations for Research Computing provides informal training for Columbia University graduate students to develop fundamental skills for harnessing computation: core languages and libraries, software development tools, best practices, and computational problem-solving. Topics are covered from across the spectrum, from beginner to advanced. Beyond training, the Foundations program aims to create a computational community at Columbia, bringing disparate researchers together with the common thread of computation.

Habanero now includes an Education Tier for course instructors to use when educating students. Whereas previous shared high performance clusters offered capacity for classes deploying HPC, such use was always ranked below that of the researchers. Conversely, Habanero's current high-priority Education Tier was made possible through the generous commitments of Mary Boyce, Dean of The Fu Foundation School of Engineering and Applied Science, and David Madigan, Executive Vice President and Dean of the Faculty of Arts and Sciences.

This new resource is live and ready for course adoption. To begin utilizing the Habanero Education Tier, please contact CUIT’s Research Computing Services team at [email protected].

Archive

Insomnia - First Round (2023)

Ginsburg - Expansion (2022)

Ginsburg - Expansion (2021)

Ginsburg - First Round (2020)

Terremoto - Expansion (2019)

Terremoto - First Round (2018)

Habanero - Expansion (2017)

Habanero - First Round (2016)

SRCPAC meets every Fall and Spring semester for approximately 90 minutes, with select faculty, administrators, and leadership presenting updates pertaining to the University's shared research computing infrastructure. All Columbia faculty, research scientists, postdocs, students, and administrative staff are welcome to attend meetings.

Meetings are scheduled and announced via the SRCPAC ListServ. To be added to this ListServ, please contact [email protected].

SRCPAC Meetings

 

Research Computing Executive Committee (RCEC) Meetings

  • ​​​​​​Intercampus Subcommittee
  • Columbia Survey Working Group
  • Cloud Subcommittee
  • External Peer Survey Working Group
  • Hotfoot HPC Operations Committee
  • Manhattanville Liaison Working Group
  • Research Storage Working Group

Research conducted on our high performance computing machines has led to many peer-reviewed publications in top-tier research journals. To view citations for these publications please visit:

To report new publications utilizing one or more of these machines, please email [email protected].