Can Magic Castle beat our HPC facility?

[rsignell-usgs]

I'm excited to find out about Magic Castle because I'm in a group at USGS that does a lot of coastal ocean modeling and we've been looking for a way to do end-to-end workflows on the Cloud. We've been using AWS No-Tears-Cluster (which layers on AWS Parallel Cluster), but we were hoping to use a more cloud-agnostic simulation approach to pair with the cloud-agnostic analysis and visualization approach Qhub (Terraform/Ansible deployment of JupyterHub with Dask Gateway on Kubernetes) we are using.

I know that Magic Castle was created for HPC training but the framework can also deploy infrastructure to support real-world applications. We were inspired to try real-world HPC on AWS after a great talk by Danny Arevalo where he demonstrated beating the NAVY HPC using the NAVGEMS global weather model using c5.18xlarge instances with EFA (If curious, watch one minute starting here: https://www.youtube.com/watch?v=GTHWf0OVGrw&t=715s).

For our own applications on AWS, we have found that EFA and FSx for Lustre are both necessary to beat our on-premise HPC facilities, and to compete on price we need to use Spot pricing.

Looks like the ability to create faster networking, faster disk and preemptible nodes are already in the works so this should all be possible eventually with Magic Castle, right?

[ocaisa]

For our own applications on AWS, we have found that EFA and FSx for Lustre are both necessary to beat our on-premise HPC facilities, and to compete on price we need to use Spot pricing.

Looks like the ability to create faster networking, faster disk and preemptible nodes are already in the works so this should all be possible eventually with Magic Castle, right?

I would agree that this looks promising but there are is still some way to go before you get there

• Requesting EFA via terraform was not supported when I last looked into it a few months ago (Leveraging EFA fabric on AWS #133). EFA itself does work however, I tested it by adding EFA support after the compute nodes were up. Infiniband on Azure is effectively already supported, you only need to prepare a base image with the drivers installed.
• AFAIU @cmd-ntrf is currently working on putting the machinery in place that would allow SLURM to scale the cluster automatically depending on how full the queue is. This would also facilitate leveraging spot pricing. AWS has a nice feature that you can chose to have a minimum amount of time when you use spot pricing (I think 6 hours is the max for this minimum time).
• In theory you could support multiple options for parallel IO, but any of these would need the faster interconnect for performance. Your login and management nodes become expensive because you can't have those on spot pricing, and if you need EFA just to have access to IO those instances will cost you. I believe you should be able to use SoftROCE for those so you won't have pay that price for resources you don't really need but there is a fair bit of testing and tweaking to be done there to get that right.

[rsignell-usgs]

@ocaisa , thanks so much for this detailed response. We'll track progress and hopefully we will be also be able to contribute. cc @costrouc, @dharhas, @tarankalra-usgs