ecs_files_composer package

Subpackages

• ecs_files_composer.jinja2_filters package
  • Module contents
    • base64decode()
    • base64encode()
    • env_override()
    • from_dict_to_simple_keys()
    • from_list_to_dict()
    • from_metadata_to_flat_keys()
    • get_property()
    • to_json()
    • to_yaml()
• ecs_files_composer.jinja2_functions package
  • Submodules
  • ecs_files_composer.jinja2_functions.aws module
    • define_ecs_metadata()
    • dump_ecs_details()
    • ec2_zone_id()
    • ecs_container_metadata()
    • ecs_task_metadata()
    • from_ssm()
    • from_ssm_json()
    • msk_bootstrap()
    • msk_cluster_zookeeper()
    • msk_endpoints()
    • using_resolve()
  • Module contents
    • env_var()
    • hostname()

Submodules

ecs_files_composer.aws_mgmt module

AWS module.

class ecs_files_composer.aws_mgmt. AwsResourceHandler ( RoleArn = None , ExternalId = None , region = None , iam_config_object = None , client_session_override = None )

Bases: object

Class to handle all AWS related credentials init.

class ecs_files_composer.aws_mgmt. S3Fetcher ( RoleArn = None , ExternalId = None , region = None , iam_config_object = None , client_session_override = None )

Bases: AwsResourceHandler

Class to handle S3 actions

bucket_re = re.compile('^s3://(?P<bucket>[a-zA-Z\\d\\-.]+)/(?P<key>[\\S]+)$')

property client

compose_x_re = re.compile('^(?P<bucket>[a-zA-Z\\d\\-.]+)::(?P<key>[\\S]+)$')

get_content ( s3_uri : Optional [ str ] = None , s3_bucket : Optional [ str ] = None , s3_key : Optional [ str ] = None , composex_uri : Optional [ str ] = None )

Retrieves a file in a temp dir and returns content

Returns :

The Stream Body for the file, allowing to do various things

class ecs_files_composer.aws_mgmt. SecretFetcher ( RoleArn = None , ExternalId = None , region = None , iam_config_object = None , client_session_override = None )

Bases: AwsResourceHandler

Class to handle Secret Manager actions

get_content ( secret )

Import the Content of a given parameter

Parameters :

secret ( input.SecretDef ) –

Returns :

class ecs_files_composer.aws_mgmt. SsmFetcher ( RoleArn = None , ExternalId = None , region = None , iam_config_object = None , client_session_override = None )

Bases: AwsResourceHandler

Class to handle SSM actions

arn_re = re.compile('^arn:aws(?:-[a-z]+)?:ssm:[\\S]+:[\\d]{12}:parameter(?P<name>/[\\S]+)$')

get_content ( parameter_name )

Import the Content of a given parameter If the parameter name is a valid ARN, parses and uses the name from ARN

Parameters :

parameter_name –

Returns :

ecs_files_composer.aws_mgmt. create_session_from_creds ( tmp_creds : dict , region : Optional [ str ] = None )

Function to easily convert the AssumeRole reply into a boto3 session :param tmp_creds: :return: :rtype boto3.session.Session

ecs_files_composer.aws_mgmt. set_session_from_iam_object ( iam_config_object , source_session : Optional [ Session ] = None )

Function to define the client session based on config input

Parameters :

• iam_config_object ( ecs_files_composer.input.IamOverrideDef ) –

• source_session –

Returns :

boto session

Return type :

boto3.session.Session

ecs_files_composer.certificates_mgmt module

class ecs_files_composer.certificates_mgmt. X509Certificate ( ** data : Any )

Bases: X509CertDef

Class to wrap actions around a new X509 certificate

certFileName : str

generate_cert ( )

generate_cert_content ( )

generate_key ( )

init_cert_paths ( )

keyFileName : str

set_cert_files ( )

set_common_name ( )

ecs_files_composer.certificates_mgmt. process_x509_certs ( job )

Processes x509 certificates

ecs_files_composer.cli module

Console script for ecs_files_composer.

ecs_files_composer.cli. main ( )

Console script for ecs_files_composer.

ecs_files_composer.common module

ecs_files_composer.common. setup_logging ( )

Function to setup logging for ECS ComposeX. In case this is used in a Lambda function, removes the AWS Lambda default log handler

Returns :

the_logger

Return type :

Logger

ecs_files_composer.ecs_files_composer module

Main module.

ecs_files_composer.ecs_files_composer. init_config ( raw = None , file_path = None , env_var = None , ssm_parameter = None , s3_config = None , secret_config = None , role_arn = None , external_id = None , decode_base64 = False , context = None , override_folder : Optional [ str ] = None , print_generated_config : bool = False )

Function to initialize the configuration as if it were a file itself

ecs_files_composer.ecs_files_composer. process_files ( job : Model , override_session = None ) → None

ecs_files_composer.ecs_files_composer. start_jobs ( config : dict , override_session = None )

Starting point to run the files job

ecs_files_composer.envsubst module

Module to do a better env variables handling.

ecs_files_composer.envsubst. expandvars ( path , default = None , skip_escaped = True , enable_litteral = True )

Expand environment variables of form $var and ${var}.

If parameter ‘skip_escaped’ is True, all escaped variable references (i.e. preceded by backslashes) are skipped. Unknown variables are set to ‘default’. If ‘default’ is None, they are left unchanged.

ecs_files_composer.files_mgmt module

Main module.

class ecs_files_composer.files_mgmt. File ( ** data : Any )

Bases: FileDef

Class to wrap common files actions around

property dir_path : str

exec_post_commands ( )

files_content_processing ( ) → None

handle_s3_source ( iam_override = None , session_override = None ) → bool

Handles retrieving the content from S3

Parameters :

• iam_override ( ecs_files_composer.input.IamOverrideDef ) –

• session_override ( boto3.session.Session ) –

Returns :

bool, result of the download from S3.

handle_secret_source ( iam_override = None , session_override = None ) → bool

Handles retrieving secrets from AWS Secrets Manager

Parameters :

• iam_override ( ecs_files_composer.input.IamOverrideDef ) –

• session_override ( boto3.session.Session ) –

Returns :

handle_sources ( iam_override = None , session_override = None ) → tuple [ bool , bool ]

Handles files from external sources

Parameters :

• iam_override ( ecs_files_composer.input.IamOverrideDef ) –

• session_override ( boto3.session.Session ) –

handle_ssm_source ( iam_override = None , session_override = None ) → bool

Handles retrieving the content from SSM Parameter

Parameters :

• iam_override ( ecs_files_composer.input.IamOverrideDef ) –

• session_override ( boto3.session.Session ) –

Returns :

handle_url_source ( ) → bool

Fetches the content from a provided URI

handler ( iam_override = None , session_override = None )

Main entrypoint for files to relate

Parameters :

• iam_override ( ecs_files_composer.input.IamOverrideDef ) –

• session_override ( boto3.session.Session ) –

post_processing ( )

render_jinja ( )

Allows to use the temp directory as environment base, the original file as source template, and render a final template.

set_unix_settings ( )

Applies UNIX settings to given file

write_content ( is_template = True , as_bytes = False , bytes_content = None )

Function to write the content retrieved to path.

Parameters :

• is_template ( bool ) – Whether the content should be considered to be a template.

• as_bytes –

• bytes_content –

Returns :

ecs_files_composer.input module

class ecs_files_composer.input. Certificates ( x509 : 'Optional[Dict[str, X509CertDef]]' = None )

Bases: object

x509 : Optional [ Dict [ str , X509CertDef ] ] = None

class ecs_files_composer.input. Commands ( post : 'Optional[CommandsDef]' = None , pre : 'Optional[CommandsDef]' = None )

Bases: object

post : Optional [ List [ Union [ str , CommandsDefItem ] ] ] = None

pre : Optional [ List [ Union [ str , CommandsDefItem ] ] ] = None

class ecs_files_composer.input. CommandsDefItem ( command : Optional [ str ] = None , display_output : Optional [ bool ] = False , ignore_error : Optional [ bool ] = False )

Bases: object

Command to run with options

command : Optional [ str ] = None

display_output : Optional [ bool ] = False

ignore_error : Optional [ bool ] = False

class ecs_files_composer.input. Context ( value )

Bases: str , Enum

An enumeration.

jinja2 = 'jinja2'

plain = 'plain'

class ecs_files_composer.input. Encoding ( value )

Bases: str , Enum

An enumeration.

base64 = 'base64'

plain = 'plain'

class ecs_files_composer.input. FileDef ( path : 'Optional[str]' = None , content : 'Optional[str]' = None , source : 'Optional[SourceDef]' = None , encoding : 'Optional[Encoding]' = 'plain' , group : 'Optional[str]' = 'root' , owner : 'Optional[str]' = 'root' , mode : 'Optional[str]' = '0644' , context : 'Optional[Context]' = 'plain' , ignore_failure : 'Optional[Union[IgnoreFailureItem, bool]]' = None , commands : 'Optional[Commands]' = None )

Bases: object

commands : Optional [ Commands ] = None

content : Optional [ str ] = None

context : Optional [ Context ] = 'plain'

encoding : Optional [ Encoding ] = 'plain'

group : Optional [ str ] = 'root'

ignore_failure : Optional [ Union [ IgnoreFailureItem , bool ] ] = None

mode : Optional [ str ] = '0644'

owner : Optional [ str ] = 'root'

path : Optional [ str ] = None

source : Optional [ SourceDef ] = None

class ecs_files_composer.input. IamOverrideDef ( RoleArn : Optional [ str ] = None , SessionName : Optional [ str ] = 'S3File@EcsConfigComposer' , ExternalId : Optional [ str ] = None , RegionName : Optional [ str ] = None , AccessKeyId : Optional [ str ] = None , SecretAccessKey : Optional [ str ] = None , SessionToken : Optional [ str ] = None )

Bases: object

When source points to AWS, allows to indicate if another role should be used

AccessKeyId : Optional [ str ] = None

ExternalId : Optional [ str ] = None

RegionName : Optional [ str ] = None

RoleArn : Optional [ str ] = None

SecretAccessKey : Optional [ str ] = None

SessionName : Optional [ str ] = 'S3File@EcsConfigComposer'

SessionToken : Optional [ str ] = None

class ecs_files_composer.input. IgnoreFailureItem ( commands : 'Optional[bool]' = False , mode : 'Optional[bool]' = False , owner : 'Optional[bool]' = False , source_download : 'Optional[bool]' = False )

Bases: object

commands : Optional [ bool ] = False

mode : Optional [ bool ] = False

owner : Optional [ bool ] = False

source_download : Optional [ bool ] = False

class ecs_files_composer.input. Model ( files : Optional [ Dict [ str , FileDef ] ] = None , certificates : Optional [ Certificates ] = None , IamOverride : Optional [ IamOverrideDef ] = None )

Bases: object

Configuration input definition for ECS Files Composer

IamOverride : Optional [ IamOverrideDef ] = None

certificates : Optional [ Certificates ] = None

files : Optional [ Dict [ str , FileDef ] ] = None

class ecs_files_composer.input. S3Def ( S3Uri : 'Optional[S3Uri]' = None , ComposeXUri : 'Optional[ComposeXUri]' = None , BucketName : 'Optional[str]' = None , BucketRegion : 'Optional[str]' = None , Key : 'Optional[str]' = None , IamOverride : 'Optional[IamOverrideDef]' = None )

Bases: object

BucketName : Optional [ str ] = None

BucketRegion : Optional [ str ] = None

ComposeXUri : Optional [ str ] = None

IamOverride : Optional [ IamOverrideDef ] = None

Key : Optional [ str ] = None

S3Uri : Optional [ str ] = None

class ecs_files_composer.input. SecretDef ( SecretId : 'str' , VersionId : 'Optional[str]' = None , VersionStage : 'Optional[str]' = None , JsonKey : 'Optional[str]' = None , IamOverride : 'Optional[IamOverrideDef]' = None )

Bases: object

IamOverride : Optional [ IamOverrideDef ] = None

JsonKey : Optional [ str ] = None

SecretId : str

VersionId : Optional [ str ] = None

VersionStage : Optional [ str ] = None

class ecs_files_composer.input. SourceDef ( Url : 'Optional[UrlDef]' = None , Ssm : 'Optional[SsmDef]' = None , S3 : 'Optional[S3Def]' = None , Secret : 'Optional[SecretDef]' = None )

Bases: object

S3 : Optional [ S3Def ] = None

Secret : Optional [ SecretDef ] = None

Ssm : Optional [ SsmDef ] = None

Url : Optional [ UrlDef ] = None

class ecs_files_composer.input. SsmDef ( ParameterName : 'Optional[str]' = None , IamOverride : 'Optional[IamOverrideDef]' = None )

Bases: object

IamOverride : Optional [ IamOverrideDef ] = None

ParameterName : Optional [ str ] = None

class ecs_files_composer.input. UrlDef ( Url : 'Optional[str]' = None , Username : 'Optional[str]' = None , Password : 'Optional[str]' = None )

Bases: object

Password : Optional [ str ] = None

Url : Optional [ str ] = None

Username : Optional [ str ] = None

class ecs_files_composer.input. X509CertDef ( keyFileName : 'str' , certFileName : 'str' , dir_path : 'Optional[str]' = None , emailAddress : 'Optional[str]' = 'files-composer@compose-x.tld' , commonName : 'Optional[str]' = None , countryName : 'Optional[str]' = 'ZZ' , localityName : 'Optional[str]' = 'Anywhere' , stateOrProvinceName : 'Optional[str]' = 'Shire' , organizationName : 'Optional[str]' = 'NoOne' , organizationUnitName : 'Optional[str]' = 'Automation' , validityEndInSeconds : 'Optional[float]' = 8035200 , group : 'Optional[str]' = 'root' , owner : 'Optional[str]' = 'root' )

Bases: object

certFileName : str

commonName : Optional [ str ] = None

countryName : Optional [ str ] = 'ZZ'

dir_path : Optional [ str ] = None

emailAddress : Optional [ str ] = 'files-composer@compose-x.tld'

group : Optional [ str ] = 'root'

keyFileName : str

localityName : Optional [ str ] = 'Anywhere'

organizationName : Optional [ str ] = 'NoOne'

organizationUnitName : Optional [ str ] = 'Automation'

owner : Optional [ str ] = 'root'

stateOrProvinceName : Optional [ str ] = 'Shire'

validityEndInSeconds : Optional [ float ] = 8035200

Module contents

Top-level package for ECS Files Composer.