Configuration input

Input definition sources

ECS Files Composer aims to provide a lof of flexibility to user in how they want to configure the job definition. The inspiration of the files input schema comes from AWS CloudFormation ConfigSets.files which could be defined in JSON or YAML.

So in that spirit, so long as the file can be parsed down into an object that complies to the JSON Schema , the source can be varied.

From environment variable

The primary way to override configuration on the fly with containers is either change ENTRYPOINT/CMD or environment variables.

So in that spirit, you can define a specific environment variables or simply use the default one, ECS_CONFIG_CONTENT

You can do the JSON string encoding yourself, or more simply you could do that in docker-compose, as follows

version: "3.8"
services:
  files-sidecar:
    environment:
      AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"
      AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-eu-west-1}
      ECS_CONFIG_CONTENT: |

        files:
          /opt/files/test.txt:
            content: >-
              test from a yaml raw content
            owner: john
            group: root
            mode: 600
          /opt/files/aws.template:
            source:
              S3:
                BucketName: ${BUCKET_NAME:-sacrificial-lamb}
                Key: aws.yml

          /opt/files/ssm.txt:
            source:
              Ssm:
                ParameterName: /cicd/shared/kms/arn
            commands:
              post:
                - file /opt/files/ssm.txt

          /opt/files/secret.txt:
            source:
              Secret:
                SecretId: GHToken

From JSON or YAML file

If you prefer to use ECS Files Composer as a CLI tool, or simply to test (don’t forget about IAM permissions!) the configuration itself, you can write the configuration into a simple file.

So, you could have the following config file for the execution

test.yaml
files:
  /tmp/test.txt:
    content: >-
      test from a yaml raw content
    owner: john
    group: root
    mode: 600
  /tmp/aws.template:
    source:
      S3:
        BucketName: ${BUCKET_NAME:-sacrificial-lamb}
        Key: aws.yml

  /tmp/ssm.txt:
    source:
      Ssm:
        ParameterName: /cicd/shared/kms/arn
    commands:
      post:
        - file /tmp/ssm.txt

  /tmp/secret.txt:
    source:
      Secret:
        SecretId: GHToken

  /tmp/public.json:
    source:
      Url:
        Url: https://ifconfig.me/all.json

  /tmp/aws.png:
    source:
      Url:
        Url: https://dunhamconnect.com/wp-content/uploads/aws-migration-1200x675.jpg

And run

ecs_files_composer -f test.yaml

From AWS S3 / SSM / SecretsManager

This allows the ones who might need to generate the job instruction/input through CICD and store the execution file into AWS services.

Hint

If to retrieve the configuration file from another account, you can specify a Role ARN to use.

Hint

When running on ECS and storing the above configuration, you can use AWS ECS Task Definition Secrets which creates an environment variable for you. Therefore, you could just indicate to use that.

JSON Schema

The input for ECS Files Composer has to follow the JSON Schema below.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "typeName": "input",
  "description": "Configuration input definition for ECS Files Composer",
  "properties": {
    "files": {
      "type": "object",
      "uniqueItems": true,
      "patternProperties": {
        "^/[\\x00-\\x7F]+$": {
          "$ref": "#/definitions/FileDef"
        }
      }
    },
    "certificates": {
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "x509": {
          "patternProperties": {
            "^/[\\x00-\\x7F]+$": {
              "$ref": "#/definitions/X509CertDef"
            }
          }
        }
      }
    },
    "IamOverride": {
      "type": "object",
      "$ref": "#/definitions/IamOverrideDef"
    }
  },
  "definitions": {
    "S3Uri": {
      "type": "string",
      "description": "s3://bucket-name/path/to/file simple syntax. Does not support IamOverride",
      "pattern": "^s3://([a-zA-Z\\d\\-.]+)/([\\S]+)$"
    },
    "ComposeXUri": {
      "type": "string",
      "description": "bucket_name::path/to/file format used in other compose-x projects",
      "pattern": "([a-zA-Z\\-\\d.]+)::([\\S]+)$"
    },
    "FileDef": {
      "type": "object",
      "additionalProperties": true,
      "properties": {
        "path": {
          "type": "string"
        },
        "content": {
          "type": "string",
          "description": "The raw content of the file to use"
        },
        "source": {
          "$ref": "#/definitions/SourceDef"
        },
        "encoding": {
          "type": "string",
          "enum": [
            "base64",
            "plain"
          ],
          "default": "plain"
        },
        "group": {
          "type": "string",
          "description": "UNIX group name or GID owner of the file. Default to root(0)",
          "default": "root"
        },
        "owner": {
          "type": "string",
          "description": "UNIX user or UID owner of the file. Default to root(0)",
          "default": "root"
        },
        "mode": {
          "type": "string",
          "description": "UNIX file mode",
          "default": "0644"
        },
        "context": {
          "type": "string",
          "enum": [
            "plain",
            "jinja2"
          ],
          "default": "plain"
        },
        "ignore_failure": {
          "oneOf": [
            {
              "type": "object",
              "additionalProperties": false,
              "properties": {
                "commands": {
                  "type": "boolean",
                  "default": false,
                  "description": "Ignore if any of the commands failed"
                },
                "mode": {
                  "type": "boolean",
                  "default": false,
                  "description": "Ignore if `mode` (using chmod) failed."
                },
                "owner": {
                  "type": "boolean",
                  "default": false,
                  "description": "Ignore if `owner` (using chown) failed"
                },
                "source_download": {
                  "type": "boolean",
                  "default": false,
                  "description": "Ignore if a Source download failed. Any subsequent action is cancelled."
                }
              }
            },
            {
              "type": "boolean",
              "description": "Ignore if any step fails (download, transform, commands etc.)"
            }
          ]
        },
        "commands": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "post": {
              "$ref": "#/definitions/CommandsDef",
              "description": "Commands to run after the file was retrieved"
            },
            "pre": {
              "$ref": "#/definitions/CommandsDef",
              "description": "Commands executed prior to the file being fetched, after `depends_on` completed"
            }
          }
        }
      }
    },
    "SourceDef": {
      "type": "object",
      "properties": {
        "Url": {
          "$ref": "#/definitions/UrlDef"
        },
        "Ssm": {
          "$ref": "#/definitions/SsmDef"
        },
        "S3": {
          "$ref": "#/definitions/S3Def"
        },
        "Secret": {
          "$ref": "#/definitions/SecretDef"
        }
      }
    },
    "UrlDef": {
      "type": "object",
      "properties": {
        "Url": {
          "type": "string",
          "format": "uri"
        },
        "Username": {
          "type": "string"
        },
        "Password": {
          "type": "string"
        }
      }
    },
    "SsmDef": {
      "type": "object",
      "properties": {
        "ParameterName": {
          "type": "string"
        },
        "IamOverride": {
          "$ref": "#/definitions/IamOverrideDef"
        }
      }
    },
    "SecretDef": {
      "type": "object",
      "required": [
        "SecretId"
      ],
      "properties": {
        "SecretId": {
          "type": "string"
        },
        "VersionId": {
          "type": "string"
        },
        "VersionStage": {
          "type": "string"
        },
        "JsonKey": {
          "type": "string",
          "description": "If the SecretString is a valid JSON, use the Key to map to the value stored in secret"
        },
        "IamOverride": {
          "$ref": "#/definitions/IamOverrideDef"
        }
      }
    },
    "S3Def": {
      "type": "object",
      "oneOf": [
        {
          "required": [
            "BucketName",
            "Key"
          ]
        },
        {
          "required": [
            "S3Uri"
          ]
        },
        {
          "required": [
            "ComposeXUri"
          ]
        }
      ],
      "properties": {
        "S3Uri": {
          "$ref": "#/definitions/S3Uri"
        },
        "ComposeXUri": {
          "$ref": "#/definitions/ComposeXUri"
        },
        "BucketName": {
          "type": "string",
          "description": "Name of the S3 Bucket"
        },
        "BucketRegion": {
          "type": "string",
          "description": "S3 Region to use. Default will ignore or retrieve via s3:GetBucketLocation"
        },
        "Key": {
          "type": "string",
          "description": "Full path to the file to retrieve"
        },
        "IamOverride": {
          "$ref": "#/definitions/IamOverrideDef"
        }
      }
    },
    "IamOverrideDef": {
      "type": "object",
      "description": "When source points to AWS, allows to indicate if another role should be used",
      "properties": {
        "RoleArn": {
          "type": "string"
        },
        "SessionName": {
          "type": "string",
          "default": "S3File@EcsConfigComposer",
          "description": "Name of the IAM session"
        },
        "ExternalId": {
          "type": "string",
          "description": "The External ID to use when using sts:AssumeRole"
        },
        "RegionName": {
          "type": "string"
        },
        "AccessKeyId": {
          "type": "string",
          "description": "AWS Access Key Id to use for session"
        },
        "SecretAccessKey": {
          "type": "string",
          "description": "AWS Secret Key to use for session"
        },
        "SessionToken": {
          "type": "string"
        }
      }
    },
    "CommandsDef": {
      "type": "array",
      "description": "List of commands to run",
      "items": {
        "oneOf": [
          {
            "type": "string",
            "description": "Shell command to run"
          },
          {
            "type": "object",
            "description": "Command to run with options",
            "properties": {
              "command": {
                "type": "string"
              },
              "display_output": {
                "type": "boolean",
                "default": false,
                "description": "Displays the command output"
              },
              "ignore_error": {
                "type": "boolean",
                "description": "Ignore if command failed",
                "default": false
              }
            }
          }
        ]
      }
    },
    "X509CertDef": {
      "type": "object",
      "additionalProperties": true,
      "required": [
        "certFileName",
        "keyFileName"
      ],
      "properties": {
        "dir_path": {
          "type": "string"
        },
        "emailAddress": {
          "type": "string",
          "format": "idn-email",
          "default": "files-composer@compose-x.tld"
        },
        "commonName": {
          "type": "string",
          "format": "hostname"
        },
        "countryName": {
          "type": "string",
          "pattern": "^[A-Z]+$",
          "default": "ZZ"
        },
        "localityName": {
          "type": "string",
          "default": "Anywhere"
        },
        "stateOrProvinceName": {
          "type": "string",
          "default": "Shire"
        },
        "organizationName": {
          "type": "string",
          "default": "NoOne"
        },
        "organizationUnitName": {
          "type": "string",
          "default": "Automation"
        },
        "validityEndInSeconds": {
          "type": "number",
          "default": 8035200,
          "description": "Validity before cert expires, in seconds. Default 3*31*24*60*60=3Months"
        },
        "keyFileName": {
          "type": "string"
        },
        "certFileName": {
          "type": "string"
        },
        "group": {
          "type": "string",
          "description": "UNIX group name or GID owner of the file. Default to root(0)",
          "default": "root"
        },
        "owner": {
          "type": "string",
          "description": "UNIX user or UID owner of the file. Default to root(0)",
          "default": "root"
        }
      }
    }
  },
  "anyOf": [
    {
      "required": [
        "files"
      ]
    },
    {
      "required": [
        "certbot_store"
      ]
    },
    {
      "required": [
        "certificates"
      ]
    }
  ]
}

AWS IAM Override

ECS Files Composer uses AWS Boto3 as the SDK. So wherever you are running it, the SDK will follow the priority chain of credentials in order to figure out which to use.

In the case of running it on AWS ECS, your container will have an IAM Task Role associated with it. You are responsible for configuring the permissions you want to give to your service.

The IamOverride definition allows you to define whether the credentials used by the tool should be used to acquire temporary credential by assuming another role.

This can present a number of advantages, such as retrieving files from another AWS Account than the one you are currently using to run the application.

IAM Override Priority

When building the boto3 session to use, by default the boto3 SDK will pick the first valid set of credentials.

If you specify the IamOverride properties at the root level , as follows

files:
  /path/to/file1:
    source:
      S3:
        BucketName: some-bucket
        Key: file.txt
IamOverride:
  RoleArn: arn:aws:iam::012345678901:role/role-name

Then all subsequent API calls to AWS will be made by assuming this IAM role.

If however you needed to change the IamOverride for a single file, or use two different profiles for different files, then apply the IamOverride at that source level, as follows.

files:
  /path/to/file1:
    source:
      S3:
        BucketName: some-bucket
        Key: file.txt
        IamOverride:
          RoleArn: arn:aws:iam::012345678901:role/role-name
  /path/to/file2:
    source:
      Ssm:
        ParameterName: /path/to/parameter
        IamOverride:
          RoleArn: arn:aws:iam::012345678901:role/role-name-2

  /path/to/file3:
    source:
      S3:
        BucketName: some-other-other-bucket
        Key: file.txt

In the above example,

  • /path/to/file1, assume role and use arn:aws:iam::012345678901:role/role-name

  • /path/to/file2, assume role and use arn:aws:iam::012345678901:role/role-name-2

  • /path/to/file3, use the default credentials found by the SDK

Attention

If the SDK cannot find the credentials and an AWS Source is defined, the program will throw an exception.

Environment Variables substitution

ECS Files composer was created with the primary assumption that you might be running it in docker-compose or on AWS ECS. When you define environment variables in docker-compose or ECS Compose-X , the environment variables are by default interpolated.

docker compose allows to not interpolate environment variables, but it is all or nothing, which might not be flexible enough.

So to solve that, the environment files substitution has decided to use the AWS CFN !Sub function syntax to declare literal variables that shall not be interpolated.

So for example, if you have in docker-compose the following

services:
  connect-files:
    environment:
      ENV: stg
      ECS_CONFIG_CONTENT: |

        files:
          /opt/connect/truststore.jks:
            mode: 555
            source:
              S3:
                BucketName: ${CONNECT_BUCKET}
                Key: commercial/core/truststore.jks
          /opt/connect/core.jks:
            mode: 555
            source:
              S3:
                BucketName: ${CONNECT_BUCKET}
            Key: commercial/core/${ENV}.jks

docker-compose and compose-x would interpolate the value for ${ENV} and ${CONNECT_BUCKET} from the execution context. But here, you defined that ENV value should be stg , and it will create an environment variable that gets exposed to the container at runtime.

To avoid this situation and have the environment variable interpolated at runtime within the context of your container, not the context of docker-compose or ECS Compose-X, simply write it with ${!ENV_VAR_NAME} .

So this would give us the following as a result.

services:
  connect-files:
    environment:
      ENV: stg
      ECS_CONFIG_CONTENT: |

        files:
          /opt/connect/truststore.jks:
            mode: 555
            source:
              S3:
                BucketName: ${!CONNECT_BUCKET}
                Key: commercial/core/truststore.jks
          /opt/connect/core.jks:
            mode: 555
            source:
              S3:
                BucketName: ${!CONNECT_BUCKET}
                Key: commercial/core/${!ENV}.jks

When running, ECS Compose-X (or ECS Files Composer) will not interpolate the environment variable and remove the ! from the raw string and allow the environment variable name to remain intact once rendered.