Local Encoder Release Notes

The processing rate is 1 file per available CPU

The optional feature allows any record errors generated during file processing to be logged into a separate file. This alleviates the need to search row by row within a file to find the errors. This feature is only available when running locally.

A default row identifier [RID] has been instituted for customers looking to build mapping files. This non-PII touchpoint can be customized using header mappings.

The RID will now be returned in the first column of the dataset.

When leveraging Local Encoder to generate LiveRamp Identity Envelopes, the envelope expiration timestamp is now provided as an additional column in the output. The Unix timestamp is based on UTC.

A new Alert notice has been added to application logs when the application fails to start when the input directory exceeds the limit of unprocessed files.

Envelope generation is account specific

To apply, after account is enabled

Header Values for use

Added support for MobileID identifiers, supported headers are IDFA, AAID and IMEI

Configuration parameters can also be set via YAML configuration file. Configuration file properties are overwritten by properties passed trough the command line. In order to use configuration file we should mount a directory where the file is  located to /config directory. Example: 

-v ~/Documents/localencoder:/config - our yaml config file should be located in localencoder folder. Then pass the file name as a command line argument, full command example: 

docker run --rm --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
--env AWS_LR_ACCOUNT_ID=123456789 \
--env AWS_REGION=eu-west-1 \
--env LR_VAULT_ACCOUNT_ID=accountId \
--env LR_VAULT_ACCOUNT_TYPE=awsiam \
--env LR_VAULT_INPUT=/data/vault/input \
--env LR_VAULT_OUTPUT=/data/vault/output \
--env LR_VAULT_LOCALE=au \
--env LR_VAULT_COUNTRY_CODE_COLUMN=COUNTRY_CODE \
-v ~/Documents/tmp:/config \
-v ~/path-to-input-folder:/data/vault/input \
-v ~/path-to-output-folder:/data/vault/output \
vault-app \
-config-file test-configuration.yml 

Configuration file example: 

liveramp:
local_encoder:
         account_type: awsiam
         account_id: 123456789
         profile: dev
         locale: AU
         output: s3://test-in
         input: s3://test-out
         filename_pattern: pattern
         header_mapping: email=mail
         country_code_column: COUNTRY_CODE
         mode: default
         dry_run: false
         gcp_project_name: liveramp-eng-chp-staging
         lr_aws:
             lr_id: lrid123
             lr_access_key_id: lracesss
             lr_secret_access_key: lrsecret
            lr_region: lrregion
         metastore:
             db:
                         url: h2:~/vault-app-metadatastore
                         driver: org.h2.Driver
                         username: sa
                         password: password
                         platform: org.hibernate.dialect.H2Dialect

To list/create/update GCP resources from the command line,

install Cloud SDK

brew install --cask google-cloud-sdk

Log in and configure the project with:

gcloud init

For the application to use your credentials locally, run the following commands:

gcloud auth application-default login export GOOGLE_APPLICATION_CREDENTIALS=       /Users/$USER/.config/gcloud/application_default_credentials.json

When using GCS bucket as input or output source we have to provide GOOGLE_APPLICATION_CREDENTIALS and LR_VAULT_GCP_PROJECT_NAME. Also must mount the gcloud config folder to our docker image

docker run --rm --cap-add IPC_LOCK --ulimit memlock=-1:-1 \ --env GOOGLE_APPLICATION_CREDENTIALS=./tmp/config/application_default_credentials.json \ --env LR_VAULT_GCP_PROJECT_NAME=liveramp-eng-chp-staging \ -v $HOME/.config/gcloud:/tmp/config \ [rest of environment variables]    Vault-app

For GCS buckets, prefix is gs://

Added new parameter LR_VAULT_COUNTRY_CODE_COLUMN or country_code_column(in configuration file). If LR_VAULT_COUNTRY_CODE_COLUMN parameter exists and is not empty, validation and normalization of phone numbers will be done using country code from that column, otherwise it will use locale. 

If the parameter value is passed but not found in the header an error is thrown and processing is stoped.

Added Support for: BR, MX, NZ, CA, US