Skip to main content

Config

Prerequisites​

Here's a checklist of things to have setup on AWS when getting started:

✅ Cloudformation permissions​

User must be able to run a cloudformation stack. WarpBuild provisions an IAM role and requires these permissions.

✅ VPC and subnets​

The VPC must have at least one public and private subnet. The subnets must have internet connectivity.

The private subnets must have a NAT gateway. Runners with static IPs will have the IPs of the NAT gateway as the external IP addresses.

Recommendations:

  1. Have three public and private subnets in different availability zones. This maximizes the availability of instance types and robustness.
  2. Each subnet must have enough IPs to accommodate the maximum number of runners you want to run concurrently. The minimum recommended number of IPs per subnet is 250.

✅ Security groups​

Security groups act as a virtual firewall for your EC2 instances. For WarpBuild:

  1. Create a security group that blocks all inbound traffic. This is to ensure that no unauthorized access to the runners is possible.
  2. Ensure outbound traffic is allowed to all destinations. This can be finetuned per your organization's security policy, but needs to allow runners to connect to the WarpBuild servers, Github API, and other locations (like package managers).

✅ Network Routes​

For optimal network routing in your WarpBuild setup:

  1. Configure your VPC route tables to direct internet-bound traffic through the Internet Gateway.
  2. Ensure proper routes are in place for outbound internet access for NAT Gateways in private subnets.
  3. Optionally, block all access between instances in the same subnet. Runner instances can be isolated and do not need to communicate with each other.
  4. Optionally, consider implementing VPC peering or AWS Transit Gateway. This is a simple and secure way to access services in other VPCs, accounts or private subnets.

✅ s3 bucket​

  1. Setup an s3 gateway endpoint for the VPC to allow the runners to connect to the s3 bucket without incurring data transfer charges.

  2. Ensure the s3 bucket is in the same region as the CI stack created.

  3. The s3 bucket is used for:

    • Cache: <bucket_name>/<org_id>/artifact_cache/<vcs_org>/<vcs_repo>/<vcs_ref>/<version>/<key>

    • Telemetry: <bucket_name>/runner/logs/<runner_id>

      Setup the s3 lifecycle policy for managing the cache and telemetry data according to your organization's policy. 7 days retention is recommended.

✅ Quotas​

Ensure that there are enough IPs, EBS volume capacity, and vCPUs available in the region for the selected instance types.

Cloud Connection​

Creating a cloud connection sets up an IAM role with the permissions required by WarpBuild Stacks and runners.

Permissions​

Expand to see the IAM role permissions
Policies:
- PolicyName: "FineGrainedEC2Permissions"
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - "ec2:DescribeInstanceTypes"
          - "ec2:DescribeInstanceTypeOfferings"
          - "ec2:DescribeInstances"
          - "ec2:RunInstances"
          - "ec2:CreateFleet"
          - "ec2:RequestSpotInstances"
          - "ec2:CancelSpotInstanceRequests"
          - "ec2:DescribeSpotInstanceRequests"
          - "ec2:DescribeSpotPriceHistory"
          - "ec2:CreateLaunchTemplate"
          - "ec2:DeleteLaunchTemplate"
          - "ec2:ModifyLaunchTemplate"
          - "ec2:TerminateInstances"
          - "ec2:CreateImage"
          - "ec2:DeregisterImage"
          - "ec2:DescribeImages"
          - "ec2:CreateTags"
          - "ec2:DeleteTags"
        Resource: "*"
- PolicyName: "SpotServiceLinkedRolePermissions"
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - "iam:CreateServiceLinkedRole"
          - "iam:DeleteServiceLinkedRole"
          - "iam:GetServiceLinkedRoleDeletionStatus"
          - "iam:AttachRolePolicy"
          - "iam:PutRolePolicy"
          - "iam:PassRole"
        Resource: "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
- PolicyName: "NetworkPermissions"
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - "ec2:DescribeRegions"
          - "ec2:DescribeVpcs"
          - "ec2:DescribeSubnets"
          - "ec2:DescribeRouteTables"
          - "ec2:DescribeSecurityGroups"
          - "ec2:DescribeInternetGateways"
          - "ec2:CreateVpcEndpoint"
          - "ec2:DeleteVpcEndpoints"
        Resource: "*"
- PolicyName: "StoragePermissions"
  PolicyDocument:
    Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - "ec2:CreateSnapshot"
          - "ec2:DeleteSnapshot"
          - "ec2:DescribeSnapshots"
          - "s3:ListBucket"
          - "s3:GetBucketLocation"
          - "s3:PutLifecycleConfiguration"
          - "s3:GetLifecycleConfiguration"
          - "s3:DeleteObject"
          - "s3:PutObject"
          - "s3:GetObject"
          - "s3:CreateBucket"
          - "s3:DeleteBucket"
          - "s3:ListBucketVersions"
          - "s3:ListBucketMultipartUploads"
          - "s3:AbortMultipartUpload"
          - "s3:PutObjectAcl"
          - "s3:GetObjectAcl"
          - "s3:PutBucketAcl"
          - "s3:GetBucketAcl"
          - "s3:PutBucketPolicy"
          - "s3:GetBucketPolicy"
          - "s3:DeleteBucketPolicy"
        Resource: "*"

Updates may be required to the IAM role if new permissions are required to the WarpBuild Stacks and runners for new features.

Stack​

Create Stack

Creating a stack imports the infra configuration provided and uses the s3 bucket for cache and telemetry.

The stack name, s3 bucket, and region cannot be changed after creation.

Tags​

Users can specify custom tags. Tags provided here are added to all resources created by the stack. These can be used for cost attribution and resource management.

By default, the following tags are added to all resources created by the stack:

keyvalue
warpbuild-managed-bywarpbuild
Name{runner-id}
warpbuild-github-org{github-org}
warpbuild-runner-labels{runner-label1}, {runner-label2}, ...
warpbuild-runner-id{runner-id}
warpbuild-stack-id{stack-id}
warpbuild-stack-name{stack-name}

Attach IAM roles​

You can specify IAM roles to attach to the runner instances. This is useful if you want to use a custom IAM role with specific permissions. This can be set at 2 levels:

  1. Stack level: All the runners created using this Stack use the IAM role provided, but can be overridden at the runner level.
  2. Runner level: inherits the stack level IAM role by default but can be overridden.

This is very useful when runners need to have fine-grained permissions to access specific AWS resources.

Custom Runners​

Custom Runners

  1. Spot instances are useful for short jobs that can be interrupted and can lead to significant (~70%) cost savings.
  2. One or more instance types in priority order can be chosen. The Github workflow uses a single runner label but picks the instance type based on availability.
  3. The minimum disk configurations are:
    • Size: 100GB
    • Throughput: 125MBps
    • IOPS: 3000

Best practices:​

  1. Use multiple instance types, especially when using spot instances to ensure availability and jobs aren't stuck in queue.
  2. When using multiple instance types, choose instance types that are similar in price and performance.
  3. Choose a minimum disk configuration of:
    • Size: 150GB
    • Throughput: 400MBps
    • IOPS: 4000

Coming Soon​

  • Instances with LocalSSD
  • Custom AMI