Running LiveRamp’s Local Encoder in an AWS Fargate Environment

Local Encoder enables you to generate securely-encoded RampIDs for your consumer data files within your own cloud environment and then utilize that data for Activation or addressability use cases, depending on your needs. In this way your consumer data is never exposed to an external network, while still enabling full use of the LiveRamp solutions. The encoded RampIDs produced by the application cannot be decoded back to the original consumer identifiers.

Local Encoder can be run on any infrastructure that supports running Docker images. The Local Encoder Docker image is currently distributed via the Amazon Elastic Container Registry (ECR).

For more information on Local Encoder, including information on security, use cases, data you can send, and output options, see "LiveRamp Local Encoder".

For information on running Local Encoder in an AWS Fargate environment, see the sections below.

Overall Steps

Running the Local Encoder in an AWS Fargate environment involves the following overall steps:

You provide LiveRamp with your PGP public key.
LiveRamp provides you with credentials.
You decrypt the AWS IAM Secret Access Key.
You configure AWS IAM permissions.
You create an ECS Cluster with Fargate.
You create an ECS Task Definition.
You run the Local Encoder Task on Fargate.
You format your data files.
You upload your data files to the appropriate input location.
Local Encoder performs the following operations:
1. The data is normalized and hygiene is performed.
2. The identifiers in the data are converted into derived RampIDs.
3. If appropriate, the derived RampIDs for each record are encoded into secure RampID packets or identity envelopes.
4. The input identifiers are removed and replaced with the appropriate RampID output type (RampIDs, RampID packets, or identity envelopes).
For Activation use cases the following steps are performed:
1. The output containing RampID packets is delivered to LiveRamp.
2. LiveRamp decodes the RampID packets into their individual derived RampIDs.
3. LiveRamp matches those derived RampIDs to their associated maintained RampIDs.
4. LiveRamp creates the appropriate fields and segments from the segment data in your LiveRamp platform (such as Connect or Safe Haven).
For Addressability use cases, the following steps are performed:
1. The output containing RampIDs or identity envelopes is output to the destination of your choice.
2. You leverage the output to build a mapping table of your customer IDs to RampIDs or identity envelopes.

Prerequisites

Running the Local Encoder in an AWS Fargate environment requires that you have the following prerequisites:

A Docker installation
AWS Command Line Interface (CLI)
A PGP public key or Keybase username

Note

You will use the AWS command line tool to issue commands at your system's command line to perform Amazon ECR and other AWS tasks. We recommend that you have the latest version of the AWS CLI installed. For information about installing the AWS CLI or upgrading it to the latest version, see Installing the AWS Command Line Interface.
We need to maintain the same architecture, i.e., build the image to match the architecture used by the Runner.

Decrypt the AWS IAM Secret Access Key

In addition to providing the Local Encoder account ID and the AWS IAM Access Key ID, LiveRamp will provide the AWS IAM Secret Access Key in an encrypted format. The AWS IAM Secret Access Key must be decrypted for use in subsequent steps.

To decrypt the AWS IAM Secret Access Key:

Export your public and private key to files, name them “public.key” and “private.key”, and save them in the “/tmp” directory.
Save the encrypted AWS IAM Secret Access Key to a file named “secret.txt” and save it in the “/tmp” directory.
Note
If the “/tmp” directory has not already been created, you will need to create it.

From Docker, run the following command:

docker run -it --rm alpine apk add --no-cache wget gnupg && export GPG_TTY=$(tty) && gpg --import /tmp/public.key && gpg --import /tmp/private.key && base64 -d -i /tmp/secret.txt | gpg --decrypt > /tmp/output.txt

Open the “output.txt” file to access the decrypted AWS IAM Secret Access Key.

Set Up IAM Policies for ECR Image Pull and S3 Access

To set up IAM policies for the ECR image pull and S3 access, perform the following steps:

Grant the required permissions to pull Docker images from Amazon Elastic Container Registry (ECR).
See the example policy below for pulling Docker images (JSON):
```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
      ],
      "Resource": "arn:aws:ecr:REGION:ACCOUNT_ID:repository/REPOSITORY_NAME"
    }
  ]
}
```
Replace the placeholders shown above with the appropriate values:
- REGION: Your AWS region (such as "us-east-1").
- ACCOUNT_ID: Your AWS account ID.
- REPOSITORY_NAME: The name of your ECR repository.

Grant the required permissions for interacting with the S3 bucket, such as reading and writing objects.

See the example policy below for S3 bucket access (JSON):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET_NAME",
        "arn:aws:s3:::YOUR_BUCKET_NAME/*"
      ]
    }
  ]
}

Replace the placeholder YOUR_BUCKET_NAME with the name of your S3 bucket.

Create and attach IAM policies:
1. In the AWS Management Console, navigate to IAM > Policies.
2. Click Create Policy.
3. Select the JSON tab.
4. Paste the relevant policy JSON (for ECR or S3).
5. Review the policy, enter a name (such as "ECRImagePullPolicy"), and click Create Policy.
6. Navigate to IAM > Users/Groups/Roles.
7. Select the user, group, or role you want to attach the policy to and click Add Permissions.

Create an ECS Cluster with Fargate

To create (set up) an ECS cluster with Fargate:

Navigate to the ECS Console and click Create Cluster.
Enter a name for your cluster (such as “my-vaultapp-fargate-cluster”).
Under “Infrastructure”, check the AWS Fargate checkbox.
Click Create.

Create a Task Definition for Fargate

To create a task definition for Fargate in AWS Management Console, there are two options:

Create a new task definition
Create a new task definition with JSON

See the sections below for more information.

Create a New Task Definition

To create a new task definition (not with JSON):

In the Task Definitions tab, click Create new task definition.
Enter a task definition family name (such as “my-vaultapp-task-definition”).
Select AWS Fargate as the launch type.
Click Next Step to configure the task.
If your task needs access to AWS services like S3, you can attach an IAM role to the task to grant the necessary permissions.
Click Create to finalize the task definition.
Configure the container definition:
1. Click Add container to define the container settings.
2. Set the Container Name (such as "my-vaultapp-container").
3. In the Image field, enter the URL of the Docker image in ECR (such as “123456789012.dkr.ecr.us-east-1.amazonaws.com/my-repo:latest”).
4. Click Add to save the container configuration.
Set the Eenvironment variables by clicking Add environment variable (or Add Environment Variables from a File) and then adding the necessary environment variables as per the configuration requirements.
Add the necessary environment variables as per the configuration requirements
Click Create to finalize the task definition.

Create a New Task Definition with JSON

To create a new task definition with JSON:

From the dropdown, select Create a new task definition with JSON.
After replacing the variables in the sample shown below with the appropriate values, copy and paste the sample code into the code field and then click Create.
Note
For information on the configuration parameters to use when replacing the variables, see the “Configuration Parameters” section below.

{
    "taskDefinitionArn": "arn:aws:ecs:us-west-2:<AWS Account ID>:task-definition/vault-app-task:24",
    "containerDefinitions": [
        {
            "name": "vault-app-container",
            "image": "<image-name>",
            "cpu": 0,
            "memoryReservation": 512,
            "portMappings": [],
            "essential": true,
            "environment": [
                {
                    "name": "LR_VAULT_LR_AWS_REGION",
                    "value": "<LR_VAULT_LR_AWS_REGION>"
                },
                {
                    "name": "AWS_REGION",
                    "value": "<AWS_REGION>"
                },
                {
                    "name": "LR_VAULT_MODE",
                    "value": "<LR_VAULT_MODE>"
                },
                {
                    "name": "AWS_DEFAULT_REGION",
                    "value": "<AWS_DEFAULT_REGION>"
                },
                {
                    "name": "AWS_EXECUTION_ENV",
                    "value": "<AWS_EXECUTION_ENV>"
                },
                {
                    "name": "LR_VAULT_LR_AWS_SECRET_ACCESS_KEY",
                    "value": "<LR_VAULT_LR_AWS_SECRET_ACCESS_KEY>"
                },
                {
                    "name": "LR_VAULT_LR_AWS_ACCESS_KEY_ID",
                    "value": "<LR_VAULT_LR_AWS_ACCESS_KEY_ID>"
                },
                {
                    "name": "LR_VAULT_OUTPUT",
                    "value": "<LR_VAULT_OUTPUT>"
                },
                {
                    "name": "LR_VAULT_PACKET_TYPE",
                    "value": "unencoded"
                },
                {
                    "name": "LR_VAULT_ACCOUNT_TYPE",
                    "value": "awsiam"
                },
                {
                    "name": "LR_VAULT_ACCOUNT_ID",
                    "value": "<LR_VAULT_ACCOUNT_ID>"
                },
                {
                    "name": "LR_VAULT_LOCALE",
                    "value": "US"
                },
                {
                    "name": "LOGGING_LEVEL_SOFTWARE_AMAZON_AWSSDK_AUTH",
                    "value": "DEBUG"
                },
                {
                    "name": "AWS_LR_ACCOUNT_ID",
                    "value": "<AWS_LR_ACCOUNT_ID>"
                },
                {
                    "name": "LR_VAULT_INPUT",
                    "value": "<LR_VAULT_INPUT>"
                }
            ],
            "mountPoints": [],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/vault-app",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "systemControls": []
        }
    ],
    "family": "vault-app-task",
    "taskRoleArn": "arn:aws:iam::<AWS Account ID>:role/<Role Name>",
    "executionRoleArn": "arn:aws:iam::<AWS Account ID>:role/<Role Name>",
    "networkMode": "awsvpc",
    "revision": 24,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.ecr-auth"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.21"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "ecs.capability.execution-role-ecr-pull"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "512",
    "memory": "4096",
    "runtimePlatform": {
        "cpuArchitecture": "ARM64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2025-03-06T05:18:14.924Z",
    "registeredBy": "arn:aws:sts::<AWS Account ID>:assumed-role/atlantis-assumable-role/<Assumed Role Session ID>",
    "tags": []
}

Run the Fargate Task

To run the Fargate task:

From the ECS Console, navigate to the Clusters tab.
Select your cluster (such as “my-vaultapp-fargate-cluster”).
Click Run new Task.
Select Fargate as the launch type and select the task definition you created earlier.
Select a VPC and subnet for the task to run in. Ensure the subnet has internet access if needed.
Set the Security Group and configure inbound/outbound rules as needed.
Click Run Task.

Format the File

Input files must include identifier fields and (for Activation use cases where you're receiving RampID packets) can also include segment data fields if desired.

Before uploading a file to the input location, make sure to format the data according to these guidelines:

Include a header row in the first line of every file consistent with the contents of the file. Files cannot be processed without headers.
If you want to maintain the ability to sort the output file, or if you're utilizing one of the deconfliction options, you must include a column containing row IDs (“RID”) as the first column of the file.
Note
The row identifier column is only required to maintain sort order and should not contain any customer personally-identifiable data.
Make sure that the only identifiers included are the allowed identifier touchpoints listed below.
If you’re sending data for consumers in multiple countries or if you’re including phone numbers, you must include the appropriate country code column (depending on the method used) to identify the country of each record. For more information, see the “Optional Configuration Parameters” section below.
Include a maximum of 500 segment data fields in a single file (for Activation use cases where you're receiving RampID packets).
Segment data field types can be in the form of a string, numeral, enum, etc.
The application supports three file formats: CSV, PSV, and TSV.
Make sure to name your files in a format that includes info on the file part, such as "filename-part-0001-of-0200".
Note
You must make file names unique, as the application will not process a file that has the same name as a previous file (unless you restart the application to clear the memory).
Files must be rectangular (have the same number of columns for every row).
If any values contain the file’s delimiter character (for example, a comma in a .csv file), make sure that your values are contained within quotes.
The recommended maximum file size is 20GB.

Allowed Identifier Touchpoints

You can include any of the following allowed identifier touchpoints for translation to RampIDs in both Activation and Addressability use cases:

Plaintext email address (maximum of three per record)
SHA-256 hashed email address (maximum of three per record)
Plaintext mobile phone number (maximum of two per record)
SHA-256 hashed mobile phone number (maximum of two per record)
Plaintext landline phone number (maximum of one per record)
SHA-256 hashed landline phone number (maximum of one per record)

Additional Allowed Touchpoints for Activation Use Cases

For Activation use cases (where you're receiving RampID packets, the following additional identifier touchpoints are also allowed for translation to RampIDs:

Name and postcode, which consists of first name, last name, and postcode (maximum of one per record) (European countries only)
AAID (maximum of one per record)
IDFA (maximum of one per record)
IMEI (maximum of one per record)

Example Header

See the header shown below for an example of what the header might look like when sending data in a pipe-separated file (psv) for an Activation use case, where segment data fields are included:

RID|EMAIL1|EMAIL2|EMAIL3|SHA256EMAIL1|SHA256EMAIL2|SHA256EMAIL3|MOBILE1|MOBILE2|SHA256MOBILE1|SHA256MOBILE2|LANDLINE1|SHA256LANDLINE1|FIRSTNAME|LASTNAME|POSTCODE|AAID|IDFA|IMEI|ATTRIBUTE_1|...|ATTRIBUTE_N

Replace ATTRIBUTE_1 … N in the example header with the name of your CRM attributes.

Example Output Files

For more information on the output options and the format of the output files, see "Output Options and Examples".

Upload the File to the Input Bucket

Uploading a file to your Local Encoder services input bucket kicks off the encoding operation. To upload your file, run a command similar to the example below (this example shows using an AWS S3, but this could be any local directory):

aws s3 cp [your_file].csv s3://com-liveramp-vault-[your-vpc-id]-input

Once the file has been processed, you’ll get a confirmation message that includes the number of records processed.

For RampID packet output, all consumer identifier data in a row is transformed into derived RampIDs, packaged into one data structure, and encrypted again, yielding a RampID packet.
For RampID output, all consumer identifier data in a row is transformed into derived RampIDs in the form of a JSON string in a “RampID” column.
For identity envelope output, all consumer identifier data is transformed into derived RampIDs. A selection logic is applied, then the RampID is additionally obfuscated and encrypted into an identity envelope. Only one identity envelope is returned per row of data. A timestamp column is appended to the end of each row. This column gives the expiration date and time for the identity envelope in Unix format (timezone UTC).

For more information on output options and the format of the output files, see "Output Options and Examples".

Configuration Parameters

See the sections below for information on the required and optional parameters to use, depending on the deployment method being used.

Required Configuration Parameters

Parameter Name	Parameter for Local Configuration	Parameter for Command Line or Configmaps.yaml	Example Value(s)	Notes
AWS user ID	account_id	LR_VAULT_ACCOUNT_ID	AID.….	Provided by LiveRamp
Account type	account_type	LR_VAULT_ACCOUNT_TYPE	awsiam
LiveRamp AWS account ID		AWS_LR_ACCOUNT_ID	461694764112	Provided by LiveRamp
AWS IAM access key ID	lr_access_key_id	LR_VAULT_LR_AWS_ACCESS_KEY_ID	AKI......	Provided by LiveRamp
AWS IAM secret access key	lr_secret_access_key	LR_VAULT_LR_AWS_SECRET_ACCESS_KEY		LiveRamp provides the secret encrypted with customer key
Input File Location	input	LR_VAULT_INPUT	s3://input-bucket gs://input-bucket /tmp/input-folder	For AWS S3 bucket, prefix is “s3://”. For GCS buckets, prefix is “gs://”.
Output file Location	output	LR_VAULT_OUTPUT	s3lr://bucket-name gs://bucket-name s3://bucket-name /tmp/output-folder	For an S3 bucket that belongs to a LiveRamp account, prefix is “‘s3lr://”’. For AWS S3 bucket, prefix is “s3://’. For GCS buckets, prefix is “gs://”.
AWS region for LR resources	lr_region	LR_VAULT_LR_AWS_REGION	eu-central-1	LiveRamp’s AWS Region
Origin of the data being processed	locale	LR_VAULT_LOCALE	us	Two letter country code representing the origin of the data being processed (for example, Australia = “au”, Great Britain = “GB”). Not case sensitive.

Optional Configuration Parameters

Parameter Name	Parameter for Local Configuration	Parameter for Command Line or Configmaps.yaml	Example Value(s)	Notes
Customer Profile	profile	LR_VAULT_PROFILE	prod dev	Default is "prod"
Filename	filename_pattern	LR_VAULT_FILENAME_PATTERN	^test.*	The regex to use to determine which files in the input folder or bucket should be processed (for example, entering “^test.*” would include a file named “test.csv”). The app will process files from the folder/bucket with filenames that match the regex.
Country Header	country_code_column	LR_VAULT_COUNTRY_CODE_COLUMN	COUNTRY_CODE (default value) {{Customer provided}} (optional)	The header name for the column containing country codes for each row. The values in the country code column determine the hygiene applied to phone numbers in the file. Must match the value in input file.
Public Key Encryption	public_key_encryption	LR_VAULT_PUBLIC_KEY_ENCRYPTION	true	Include this parameter and set to “true” to encrypt each row of data in the files before they’re sent to LiveRamp for processing.
Header Mapping	header_mapping	LR_VAULT_HEADER_MAPPING	{{customer determined}} newvalue=defaultvalue,	A list of key=value pairs which can be used to replace the default headers for the identifier columns in the file. For example, if the email columns have the headers “primary_email” and “alt_email”, then the header mapping should be set to “primary_email=email1,alt_email=email2”.
Error Log	customer_logging_enabled	LR_VAULT_CUSTOMER_LOGGING_ENABLED		Include this parameter and set to “true” to enable the generation of an error log file showing any processing errors (such as illegal characters or incorrect header values) and the row number for where the error occurred.
Error Log Location	-v <localFolder>:/var/logs/vault-app	-v <localFolder>:/var/logs/vault-app		Include this parameter to set the delivery location for the error log file.
Mode	mode	LR_VAULT_MODE	default task	The default value is "default" for long-running file processing, set to "task" to enable single file processing (the application will shut down after processing a single file, not available for Kubernetes setup).
Packet Type	packet_type	LR_VAULT_PACKET_TYPE	unencoded	Include this parameter and set to "unencoded" to receive RampIDs rather than RampID packets. Include this parameter only when you are receiving RampIDs.
Envelope Output	envelopes_flow	LR_VAULT_ENVELOPES_FLOW	true	Include this parameter and set to "true" to have the output RampIDs packaged into identity envelopes. Include this parameter only when you are receiving identity envelopes.
Test Mode	dry_run	LR_VAULT_DRY_RUN	true false	The default value is “'false”'. Set to “true” to run the app in dry run mode. Only outputs encrypted packets in dry run mode.
Customer AWS Access Key ID	N/A	AWS_ACCESS_KEY_ID		Only for customers using S3 bucket as input source. Access key ID for client’s AWS.
Customer AWS Secret Access Key	N/A	AWS_SECRET_ACCESS_KEY		Only for customers using S3 bucket as input source. Secret access key for client’s AWS.
Customer AWS Region	N/A	AWS_REGION		Only for customers using S3 bucket as input source. AWS region in which the bucket is residing.
Customer AWS Region	N/A	AWS_DEFAULT_REGION		Only for customers using S3 bucket as input source. AWS region in which the bucket is residing.
Customer GCS Bucket Credentials	N/A	GOOGLE_APPLICATION_CREDENTIALS		Only for customers using GCS bucket as input/output source. Path to your Google Credentials JSON file.
Customer GCS Project	gcp_project_name	LR_VAULT_GCP_PROJECT_NAME		Only for customers using GCS bucket as input/output source. The name of your GCP project. Added if the default profile name can't be found.
Java Tool Options	N/A	JAVA_TOOL_OPTIONS	--env JAVA_TOOL_OPTIONS="-XX:+DisableAttachMechanism -Dcom.sun.management.jmxremote -XX:ActiveProcessorCount=3"	Add the variable (including the quotation marks) and apply the `ActiveProcessorCount` parameter to limit the number of processors used by the application. The additional parameter of -`XX:+DisableAttachMechanism -Dcom.sun.management.jmxremote` must be included. If you don’t specify the number of processors to use, all available processors will be used, depending on the number of files being processed.

In this section:

Running LiveRamp’s Local Encoder in an AWS Fargate Environment

Overall Steps

Prerequisites

Note

Decrypt the AWS IAM Secret Access Key

Note

Set Up IAM Policies for ECR Image Pull and S3 Access

Create an ECS Cluster with Fargate

Create a Task Definition for Fargate

Create a New Task Definition

Create a New Task Definition with JSON

Note

Run the Fargate Task

Format the File

Note

Note

Allowed Identifier Touchpoints

Additional Allowed Touchpoints for Activation Use Cases

Example Header

Example Output Files

Upload the File to the Input Bucket

Configuration Parameters

Required Configuration Parameters

Optional Configuration Parameters

Search results