Skip to content

Google Cloud Batch

This guide assumes you have an existing Google Cloud account. Sign up for a free account here. Seqera Platform provides integration to Google Cloud via the Batch API.

The guide is split into two parts:

  1. How to configure your Google Cloud account to use the Batch API.
  2. How to create a Google Cloud Batch compute environment in Seqera.

Configure Google Cloud#

Create a project#

Go to the Google Project Selector page and select an existing project, or select Create project.

Enter a name for your new project, e.g., tower-nf.

If you are part of an organization, the location will default to your organization.

Enable billing#

See here to enable billing in your Google Cloud account.

Enable APIs#

See here to enable the following APIs for your project:

  • Batch API
  • Compute Engine API
  • Cloud Storage API

Select your project from the dropdown menu and select Enable.

Alternatively, you can enable each API manually by selecting your project in the navigation bar and visiting each API page:

IAM#

Seqera requires a service account with appropriate permissions to interact with your Google Cloud resources.

Create a service account

  1. In the navigation menu, select IAM & Admin > Service Accounts.

  2. Select the email address of the Compute Engine default service account.

  3. Select Keys > Add key > Create new key.

  4. Select JSON as the key type.

  5. Select Create.

A JSON file will be downloaded to your computer. This file contains the credential needed to configure the compute environment in Seqera.

You can manage your key from the Service Accounts page.

Cloud Storage bucket#

Google Cloud Storage is a type of object storage. To access files and store the results for your pipelines, create a Cloud bucket that your Seqera service account can access.

Create a Cloud Storage bucket

  1. In the hamburger menu (≡), select Cloud Storage > Create bucket.

  2. Enter a name for your bucket. You will reference this name when you create the compute environment in Seqera.

    Don't use underscores (_) in your bucket name. Use hyphens (-) instead.

  3. Select Region for the Location type and select the Location for your bucket. You'll reference this location when you create the compute environment in Seqera.

  4. Select Standard for the default storage class.

  5. Select Uniform for the Access control.

    The Batch API is available in a limited number of locations. These locations are only used to store metadata about the pipeline operations. The storage bucket and compute resources can be in any region.

  6. Select Create.

  7. Once the bucket is created, you'll be redirected to the Bucket details page.

  8. Select Permissions > + Add.

  9. Copy the email address of the Compute Engine default service account into New principals.

  10. Select the following roles:

    • Storage Admin
    • Storage Legacy Bucket Owner
    • Storage Legacy Object Owner
    • Storage Object Creator

You've created a project, enabled the necessary Google APIs, created a bucket, and created a service account JSON file with the required credentials. You are now ready to set up a new compute environment in Seqera.

Seqera compute environment#

After your Google Cloud resources have been created, create a new Seqera compute environment.

Create a Seqera Google Cloud Batch compute environment

  1. In a workspace, select Compute Environments > New Environment.

  2. Enter a descriptive name for this environment, e.g., Google Cloud Batch (europe-north1).

  3. Select Google Cloud Batch as the target platform.

  4. From the Credentials drop-down, select existing Google credentials, or select + to add new credentials. If you choose to use existing credentials, skip to step 7.

  5. Enter a name for the credentials, e.g., Google Cloud Credentials.

  6. Enter the Service account key created previously.

  7. Select the Location where you wish to execute pipelines.

  8. In the Pipeline work directory field, enter your storage bucket URL, e.g., gs://my-bucket. This bucket must be accessible in the location selected in the previous step.

  9. Select Enable Wave containers to facilitate access to private container repositories and provision containers in your pipelines using the Wave containers service. See Wave containers for more information.

  10. Select Enable Fusion v2 to allow access to your S3-hosted data via the Fusion v2 virtual distributed file system. This speeds up most data operations. The Fusion v2 file system requires Wave containers to be enabled (see above). See Fusion file system for configuration details.

  11. Enable Spot to use spot instances, which have significantly reduced cost compared to on-demand instances.

  12. Apply Resource labels to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.

  13. Expand Staging options to include optional pre- or post-run Bash scripts that execute before or after the Nextflow pipeline execution in your environment.

  14. Specify custom Environment variables for the Head job and/or Compute jobs.

  15. Configure any advanced options you need:

    • Enable Use Private Address to ensure that your Google Cloud VMs aren't accessible to the public internet.

    • Use Boot disk size to control the boot disk size of VMs.

    • Use Head Job CPUs and Head Job Memory to specify the CPUs and memory allocated for head jobs.

  16. Select Create to finalize the compute environment setup.

See Launch pipelines to start executing workflows in your Google Cloud Batch compute environment.

Back to top