Skip to main content

Running LiveRamp’s Local Encoder In a Google Cloud Run Environment

Local Encoder enables you to generate securely-encoded RampIDs for your consumer data files within your own cloud environment and then utilize that data for Activation or addressability use cases, depending on your needs. In this way your consumer data is never exposed to an external network, while still enabling full use of the LiveRamp solutions. The encoded RampIDs produced by the application cannot be decoded back to the original consumer identifiers.

Local Encoder can be run on any infrastructure that supports running Docker images. The Local Encoder Docker image is currently distributed via the Amazon Elastic Container Registry (ECR).

For more information on Local Encoder, including information on security, use cases, data you can send, and output options, see "LiveRamp Local Encoder".

For information on running Local Encoder in a Google Cloud Run environment, see the sections below.

Overall Steps

Running the Local Encoder in a Google Cloud Run environment involves the following overall steps:

  1. You provide LiveRamp with your PGP public key.

  2. LiveRamp provides you with credentials.

  3. You decrypt the AWS IAM Secret Access Key.

  4. You use the credentials to configure AWS to access LiveRamp’s Docker image in ECR.

  5. You use Docker to access ECR and pull the Docker image for Local Encoder.

  6. You push the Docker image to GCP.

  7. You deploy on Cloud Run.

  8. To receive RampIDs (rather than RampID packets), you configure the output type.

  9. You deploy the Local Encoder.

  10. You format your data files and then upload them to the appropriate input location.

  11. Local Encoder performs the following operations:

    1. The data is normalized and hygiene is performed.

    2. The identifiers in the data are converted into derived RampIDs.

    3. If appropriate, the derived RampIDs for each record are encoded into secure RampID packets or identity envelopes.

    4. The input identifiers are removed and replaced with the appropriate RampID output type (RampIDs, RampID packets, or identity envelopes).

  12. For Activation use cases the following steps are performed:

    1. The output containing RampID packets is delivered to LiveRamp.

    2. LiveRamp decodes the RampID packets into their individual derived RampIDs.

    3. LiveRamp matches those derived RampIDs to their associated maintained RampIDs.

    4. LiveRamp creates the appropriate fields and segments from the segment data in your LiveRamp platform (such as Connect or Safe Haven).

  13. For Addressability use cases the following steps are performed:

    1. The output containing RampIDs or identity envelopes is output to the destination of your choice.

    2. You leverage the output to build a mapping table of your customer IDs to RampIDs or identity envelopes.

For more information on performing these tasks, see the sections below.

Prerequisites

Running the Local Encoder in a Google Cloud Run environment requires that you have the following prerequisites:

Decrypt the AWS IAM Secret Access Key

In addition to providing the Local Encoder account ID and the AWS IAM Access Key ID, LiveRamp will provide the AWS IAM Secret Access Key in an encrypted format. The AWS IAM Secret Access Key must be decrypted for use in subsequent steps.

To decrypt the AWS IAM Secret Access Key:

  1. Export your public and private key to files, name them “public.key” and “private.key”, and save them in the “/tmp” directory.

  2. Save the encrypted AWS IAM Secret Access Key to a file named “secret.txt” and save it in the “/tmp” directory.

    Note

    If the “/tmp” directory has not already been created, you will need to create it.

  3. From Docker, run the following command:

    docker run -it --rm alpine apk add --no-cache wget gnupg && export GPG_TTY=$(tty) && gpg --import /tmp/public.key && gpg --import /tmp/private.key && base64 -d /tmp/secret.txt | gpg --decrypt > /tmp/output.txt

  4. Open the “output.txt” file to access the decrypted AWS IAM Secret Access Key.

Configure AWS to Access LiveRamp’s Docker Image

To configure AWS to be able to access LiveRamp’s Local Encoder Docker image in ECR, set your AWS IAM access key and secret access key in one of the following methods.

Run AWS Configure

From the command line, run the command “aws configure” with the following values:

AWS Access Key Id: [LR_VAULT_LR_AWS_ACCESS_KEY_ID]
AWS Secret Access Key: [LR_VAULT_LR_AWS_SECRET_ACCESS_KEY]
Default region name: eu-central-1
Default output format: json

Edit Files Manually

To set your keys by editing the files manually (MAC OS example):

  1. From Finder, navigate to the Home directory.

    Local_Encoder-Docker_Run-Finder_Home_Directory.png

  2. Press (command + shift + .) to reveal the hidden folders within the Home directory Note: If you don’t see these files, you might not have the latest version of AWS CLI installed. Follow the instructions in this Amazon article to install the latest version.

  3. From the .aws folder, open the “config” and “credentials” files and then enter your aws_access_key_id and aws_secret_access_key (2.2) values as shown below:

    Local_Encoder-Docker_Run-key_values.png

Pull the Docker Image

The Docker image you need to pull from ECR is “461694764112.dkr.ecr.eu-central-1.amazonaws.com/vault-app:latest”.

Once you’ve configured AWS to access LiveRamp’s Local Encoder Docker image in ECR, perform the following steps to pull the Docker image:

  1. Make sure the Docker Engine is running

  2. Execute the following commands in the CLI:

    aws ecr get-login-password --region eu-central-1 | docker login --username AWS --password-stdin 461694764112.dkr.ecr.eu-central-1.amazonaws.com

    Note

    Make sure to run the command exactly as shown.

    The command output should declare “Login Succeeded”.

    Note

    If you get an error code, check that you have configured AWS correctly.

  3. Run the following command to retrieve the image:

    docker pull 461694764112.dkr.ecr.eu-central-1.amazonaws.com/vault-app:latest

    The docker image should be available on your local machine or VM to use.

  4. To confirm access to the image, run the following command and make sure that the image shows up in the list of images:

    docker images

Push the Docker Image to GCP

  1. Tag the image with the following command:

    docker tag 461694764112.dkr.ecr.eu-central-1.amazonaws.com/vault-app gcr.io/<GCP_PROJECT_ID>/<IMAGE_NAME> 

    if you have pulled a specific version of the image then the command should be:

    docker tag 461694764112.dkr.ecr.eu-central-1.amazonaws.com/vault-app:<VERSION> gcr.io/<GCP_PROJECT_ID>/<IMAGE_NAME> 

  2. Push the image to GCP Container registry with the following command:

    docker push gcr.io/<GCP_PROJECT_ID>/<IMAGE_NAME> 

    You should see the image in container registry and the output should look similar to this:

    I-Local_Encoder_Kubernetes-Push_Docker_Image-sample_output.png

Deploy Local Encoder on Cloud Run

  1. From the Cloud Run UI, select Job from the DEPLOY CONTAINER dropdown.

    I-Run_Local_Encoder_Cloud_Run-Job_menu_selection.png

  2. From the Create job screen, perform the steps below:

    I-Run_Local_Encoder_Cloud_Run-Create_job_screen.png

    1. Select the Docker image from GAR/GCR (Google Artifact Repository/Google Container Registry) 

    2. Enter the job name.

    3. Select the region.

    4. Enter “1” for the number of tasks (since it is a job that is going to run only once).

    5. Click DONE.

  3. Perform the steps below to create the volume:

    I-Run_Local_Encoder_Cloud_Run-Volumes_screen.png

    1. For the Volume type, select Cloud Storage Bucket.

    2. Enter the volume name (such as “input”).

    3. Browse and select the bucket where the input files will be stored.

    4. Click DONE.

  4. Perform the steps below to mount the volume:

    I-Run_Local_Encoder_Cloud_Run-Containers_screen.png

    1. Select the name of the volume. 

    2. Enter the mount path (such as “/tmp/input”).

    3. Click DONE.

  5. Enter the required and optional environment variables and their values, and then click DONE.

    Note

    For more information, see the “Configuration Parameters” section below.

    I-Run_Local_Encoder_Cloud_Run-Variables_screen.png

  6. Perform the steps below to set the container arguments:

    I-Run_Local_Encoder_Cloud_Run-Settings_screen.png

    1. Enter any needed container arguments.

    2. Set the memory and CPU as per your requirements.

    3. Click DONE.

  7. Check the Execute job immediately check box and then click CREATE.

    I-Run_Local_Encoder_Cloud_Run-Create_button.png

Format the File

Input files must include identifier fields and (for Activation use cases where you're receiving RampID packets) can also include segment data fields if desired.

Before uploading a file to the input location, make sure to format the data according to these guidelines:

  • Include a header row in the first line of every file consistent with the contents of the file. Files cannot be processed without headers.

  • If you want to maintain the ability to sort the output file, or if you're utilizing one of the deconfliction options, you must include a column containing row IDs (“RID”) as the first column of the file.

    Note

    The row identifier column is only required to maintain sort order and should not contain any customer personally-identifiable data.

  • Make sure that the only identifiers included are the allowed identifier touchpoints listed below.

  • If you’re sending data for consumers in multiple countries or if you’re including phone numbers, you must include the appropriate country code column (depending on the method used) to identify the country of each record. For more information, see the “Optional Configuration Parameters” section below.

  • Include a maximum of 500 segment data fields in a single file (for Activation use cases where you're receiving RampID packets).

  • Segment data field types can be in the form of a string, numeral, enum, etc.

  • The application supports three file formats: CSV, PSV, and TSV.

  • Make sure to name your files in a format that includes info on the file part, such as "filename-part-0001-of-0200".

    Note

    You must make file names unique, as the application will not process a file that has the same name as a previous file (unless you restart the application to clear the memory).

  • Files must be rectangular (have the same number of columns for every row).

  • If any values contain the file’s delimiter character (for example, a comma in a .csv file), make sure that your values are contained within quotes.

  • The recommended maximum file size is 20GB.

Allowed Identifier Touchpoints

You can include any of the following allowed identifier touchpoints for translation to RampIDs in both Activation and Addressability use cases:

  • Plaintext email address (maximum of three per record)

  • SHA-256 hashed email address (maximum of three per record)

  • Plaintext mobile phone number (maximum of two per record)

  • SHA-256 hashed mobile phone number (maximum of two per record)

  • Plaintext landline phone number (maximum of one per record)

  • SHA-256 hashed landline phone number (maximum of one per record)

Additional Allowed Touchpoints for Activation Use Cases

For Activation use cases (where you're receiving RampID packets, the following additional identifier touchpoints are also allowed for translation to RampIDs:

  • Name and postcode, which consists of first name, last name, and postcode (maximum of one per record) (European countries only)

  • AAID (maximum of one per record)

  • IDFA (maximum of one per record)

  • IMEI (maximum of one per record)

Example Header

See the header shown below for an example of what the header might look like when sending data in a pipe-separated file (psv) for an Activation use case, where segment data fields are included:

RID|EMAIL1|EMAIL2|EMAIL3|SHA256EMAIL1|SHA256EMAIL2|SHA256EMAIL3|MOBILE1|MOBILE2|SHA256MOBILE1|SHA256MOBILE2|LANDLINE1|SHA256LANDLINE1|FIRSTNAME|LASTNAME|POSTCODE|AAID|IDFA|IMEI|ATTRIBUTE_1|...|ATTRIBUTE_N

Replace ATTRIBUTE_1 … N in the example header with the name of your CRM attributes.

Example Output Files

For more information on the output options and the format of the output files, see "Output Options and Examples".

Upload the File to the Input Bucket

Uploading a file to your Local Encoder services input bucket kicks off the encoding operation. To upload your file, run a command similar to the example below (this example shows using an AWS S3, but this could be any local directory):

aws s3 cp [your_file].csv s3://com-liveramp-vault-[your-vpc-id]-input

Once the file has been processed, you’ll get a confirmation message that includes the number of records processed.

  • For RampID packet output, all consumer identifier data in a row is transformed into derived RampIDs, packaged into one data structure and encrypted again, yielding a RampID packet.

  • For RampID output, all consumer identifier data in a row is transformed into derived RampIDs in the form of a JSON string in a “RampID” column.

  • For identity envelope output, all consumer identifier data is transformed into derived RampIDs. A selection logic is applied, then the RampID is additionally obfuscated and encrypted into an identity envelope. Only one identity envelope is returned per row of data. A timestamp column is appended to the end of each row. This column gives the expiration date and time for the identity envelope in Unix format (timezone UTC).

For more information on output options and the format of the output files, see "Output Options and Examples".

Configuration Parameters

See the sections below for information on the required and optional parameters to use, depending on the deployment method being used.

Required Configuration Parameters

Parameter Name

Parameter for Local Configuration

Parameter for Command Line or Configmaps.yaml

Example Value(s)

Notes

AWS user ID

account_id

LR_VAULT_ACCOUNT_ID

AID.….

Provided by LiveRamp

Account type

account_type

LR_VAULT_ACCOUNT_TYPE

awsiam

LiveRamp AWS account ID

AWS_LR_ACCOUNT_ID

461694764112

Provided by LiveRamp

AWS IAM access key ID

lr_access_key_id

LR_VAULT_LR_AWS_ACCESS_KEY_ID

AKI......

Provided by LiveRamp

AWS IAM secret access key

lr_secret_access_key

LR_VAULT_LR_AWS_SECRET_ACCESS_KEY

LiveRamp provides the secret encrypted with customer key

Input File Location

input

LR_VAULT_INPUT

  • s3://input-bucket

  • gs://input-bucket

  • /tmp/input-folder

  • For AWS S3 bucket, prefix is “s3://”.

  • For GCS buckets, prefix is “gs://”.

Output file Location

output

LR_VAULT_OUTPUT

  • s3lr://bucket-name

  • gs://bucket-name

  • s3://bucket-name

  • /tmp/output-folder

  • For an S3 bucket that belongs to a LiveRamp account, prefix is “‘s3lr://”’.

  • For AWS S3 bucket, prefix is “s3://’.

  • For GCS buckets, prefix is “gs://”.

AWS region for LR resources

lr_region

LR_VAULT_LR_AWS_REGION

eu-central-1

LiveRamp’s AWS Region

Origin of the data being processed

locale

LR_VAULT_LOCALE

us

Two letter country code representing the origin of the data being processed (for example, Australia = “au”, Great Britain = “GB”). Not case sensitive.

Optional Configuration Parameters

Parameter Name

Parameter for Local Configuration

Parameter for Command Line or Configmaps.yaml

Example Value(s)

Notes

Customer Profile

profile

LR_VAULT_PROFILE

  • prod

  • dev

Default is "prod"

Filename

filename_pattern

LR_VAULT_FILENAME_PATTERN

  • ^test.*

The regex to use to determine which files in the input folder or bucket should be processed (for example, entering “^test.*” would include a file named “test.csv”). The app will process files from the folder/bucket with filenames that match the regex.

Country Header

country_code_column

LR_VAULT_COUNTRY_CODE_COLUMN

  • COUNTRY_CODE (default value)

  • {{Customer provided}} (optional)

  • The header name for the column containing country codes for each row.

  • The values in the country code column determine the hygiene applied to phone numbers in the file.

  • Must match the value in input file.

Public Key Encryption

public_key_encryption

LR_VAULT_PUBLIC_KEY_ENCRYPTION

true

Include this parameter and set to “true” to encrypt each row of data in the files before they’re sent to LiveRamp for processing.

Header Mapping

header_mapping

LR_VAULT_HEADER_MAPPING

{{customer determined}}

newvalue=defaultvalue,

A list of key=value pairs which can be used to replace the default headers for the identifier columns in the file. For example, if the email columns have the headers “primary_email” and “alt_email”, then the header mapping should be set to “primary_email=email1,alt_email=email2”.

Error Log

customer_logging_enabled

LR_VAULT_CUSTOMER_LOGGING_ENABLED

Include this parameter and set to “true” to enable the generation of an error log file showing any processing errors (such as illegal characters or incorrect header values) and the row number for where the error occurred.

Error Log Location

-v <localFolder>:/var/logs/vault-app

-v <localFolder>:/var/logs/vault-app

Include this parameter to set the delivery location for the error log file.

Mode

mode

LR_VAULT_MODE

  • default

  • task

The default value is "default" for long-running file processing, set to "task" to enable single file processing (the application will shut down after processing a single file, not available for Kubernetes setup).

Packet Type

packet_type

LR_VAULT_PACKET_TYPE

unencoded

Include this parameter and set to "unencoded" to receive RampIDs rather than RampID packets. Include this parameter only when you are receiving RampIDs.

Envelope Output

envelopes_flow

LR_VAULT_ENVELOPES_FLOW

true

Include this parameter and set to "true" to have the output RampIDs packaged into identity envelopes. Include this parameter only when you are receiving identity envelopes.

Test Mode

dry_run

LR_VAULT_DRY_RUN

  • true

  • false

The default value is “'false”'. Set to “true” to run the app in dry run mode. Only outputs encrypted packets in dry run mode.

Customer AWS Access Key

N/A

AWS_ACCESS_KEY

Only for customers using S3 bucket as input source. Access key for client’s AWS.

Customer AWS Secret Access Key

N/A

AWS_SECRET_ACCESS_KEY

Only for customers using S3 bucket as input source. Secret access key for client’s AWS.

Customer AWS Region

N/A

AWS_REGION

Only for customers using S3 bucket as input source. AWS region in which the bucket is residing.

Customer AWS Region

N/A

AWS_DEFAULT_REGION

Only for customers using S3 bucket as input source. AWS region in which the bucket is residing.

Customer GCS Bucket Credentials

N/A

GOOGLE_APPLICATION_CREDENTIALS

Only for customers using GCS bucket as input/output source. Path to your Google Credentials JSON file.

Customer GCS Project

gcp_project_name

LR_VAULT_GCP_PROJECT_NAME

Only for customers using GCS bucket as input/output source. The name of your GCP project. Added if the default profile name can't be found.

Java Tool Options

N/A

JAVA_TOOL_OPTIONS

--env JAVA_TOOL_OPTIONS="-XX:+DisableAttachMechanism -Dcom.sun.management.jmxremote -XX:ActiveProcessorCount=3"

  • Add the variable (including the quotation marks) and apply the ActiveProcessorCount parameter to limit the number of processors used by the application. 

  • The additional parameter of  -XX:+DisableAttachMechanism -Dcom.sun.management.jmxremote must be included.

  • If you don’t specify the number of processors to use, all available processors will be used, depending on the number of files being processed.

In this section:
RampIDs