Local Encoder Release Notes
The sections below outline the released versions of Local Encoder.
Version 1.8.2
Release Date: 6/13/24
Tags: ‘latest’, ‘v1.8.2', ‘v1.8.2-ubuntu’
Changed
The 10-file limit for the input bucket has been removed
The application now uses parallel processing
The processing rate is 1 file per available CPU
Fixed
AWS and GCP client libs loading files in memory.
Fix the “connection reset” error for input buckets.
Resolved timeout issue for AWS client with large file uploads.
Version 1.8.1
Release Date: 4/30/24
Tags: ‘v1.8.1', ‘v1.8.1-ubuntu’
Changed
Add new ENV variable -> LR_VAULT_S3_TARGET_THROUGHPUT this variable has default settings, it doesn’t need to be changed unless there are issues with S3 upload.
Fixed
Files are not loaded in memory during AWS & GCP uploads
Fix the “connection reset” error for input buckets
Version 1.8
Release Date: 3/13/24
Tags: ‘v1.8', ‘v1.8-ubuntu’
Changed
The limit of 10 files for the input folder has changed. There is no longer a limit to the number of files in the input bucket. Restarting the application will reprocess all files found in the input bucket.
The application now uses parallel file processing to process the files more efficiently
The application logging has changed to improve performance. There is no lost information but the logs will be rendered in a new manner.
Fixed
Recognition of Brazilian Mobile Phone Numbers
Broader Envelope support for use with ATS Direct
Version 1.7
Release Date: 12/12/23
Tags: ‘v1.7', ‘v1.7-ubuntu’
Added
File Processing Error Logging
The optional feature allows any record errors generated during file processing to be logged into a separate file. This alleviates the need to search row by row within a file to find the errors. This feature is only available when running locally.
Defined Row Identifier
A default row identifier [RID] has been instituted for customers looking to build mapping files. This non-PII touchpoint can be customized using header mappings.
The RID will now be returned in the first column of the dataset.
LiveRamp Identity Envelope Expiration Date
When leveraging Local Encoder to generate LiveRamp Identity Envelopes, the envelope expiration timestamp is now provided as an additional column in the output. The Unix timestamp is based on UTC.
Application Logging
A new Alert notice has been added to application logs when the application fails to start when the input directory exceeds the limit of unprocessed files.
Fixed
Resolved an issue preventing the use of header mapping while files of a TSV or PSV format.
Version 1.6.3
Release Date: 10/26/23
Tags: ‘v1.6.3, ‘v1.6.3-ubuntu’
Added
Envelope output contains row identifier value RID. Header Value: A defined row identifier must be included in the input file to maintain the sort order of records. The default row identifier value is 'rid'. This column value is only required if you need to keep a sort order.
Fixed
Custom Header Mappings: A list of key=value pairs that can be used to replace the default headers for the identifier columns in the input file. Header Mappings can be utilized on all three supported file formats: CSV, PSV, and TSV
Version 1.6.2
Release Date: 06/29/23
Tags: ‘v1.6.2, ‘v1.6.2-ubuntu’
Changed
Rotated LiveRamp PGP public key
Version 1.6.1
Release Date: 05/23/23
Tags: ‘v1.6.1', ‘v1.6.1-ubuntu'
Fixed
Gradle vulnerabilities
Version 1.6.0
Release Date: 04/11/23
Tags: ‘v1.6.0', ‘v1.6.0-ubuntu'
Added
An alternative application version using the tag v#.#.#-ubuntu. The standard Local Encoder uses an Alpine Linux base image
Support for envelope generation with checks for opt-outs
Envelope generation is account specific
To apply, after account is enabled
the param is envelopes_flow: true or --env LR_VAULT_ENVELOPES_FLOW=true
Support for SHA256 phones (landline and mobile)
Header Values for use
SHA256MOBILE1, SHA256MOBILE2
SHA256LANDLINE1
Version 1.5.2
Release Date: 12/28/22
Tags: ‘v1.5.2'
Fixed
Gradle vulnerabilities
Version 1.5.1
Release Date: 08/12/22
Tags: ‘v1.5.1'
Changed
LiveRamp PGP keys are used for end to end encryption
Derived RampIDs output is structured in simpler json format
Version 1.5.0
Release Date: 07/20/22
Tags: ‘v1.5.0'
Changed
Fixed IV vulnerability in IDLPacket AES encryption
Version 1.4.8
Release Date: 07/07/22
Tags: ‘v1.4.8'
Added
Included encrypted prefix in the output file name when using encryption flow
Version 1.4.7
Release Date: 06/08/22
Tags: ‘v1.4.7'
Added
Added possibility of additional encryption of records processed by local encoder
Version 1.4.6
Release Date: 05/12/22
Tags: ‘v1.4.6'
Changed
Upgraded H2 database connector to v2.1.210 to fix RCE vulnerability
Version 1.4.5
Release Date: 04/08/22
Tags: ‘v1.4.5'
Changed
Spring Boot update to 2.6.6
Version 1.4.4
Release Date: 01/26/22
Tags: ‘v1.4.4'
Changed
Log4j update to 2.17.1
Version 1.4.3
Release Date: 12/29/21
Tags: ‘v1.4.3'
Fixed
Issue when all identifier headers are mapped
Version 1.4.2
Release Date: 12/16/21
Tags: ‘v1.4.2'
Changed
Log4j update to 2.16.0
Version 1.4.0
Release Date: 10/29/21
Tags: ‘v1.4.0'
Added
Supported pseudonymisation of Mobile Device IDs
Added support for MobileID identifiers, supported headers are IDFA, AAID and IMEI
Upgrade configuration from flags to configuration file
Configuration parameters can also be set via YAML configuration file. Configuration file properties are overwritten by properties passed trough the command line. In order to use configuration file we should mount a directory where the file is located to /config directory. Example:
-v ~/Documents/localencoder:/config
- our yaml config file should be located in localencoder folder. Then pass the file name as a command line argument, full command example:docker run --rm --cap-add IPC_LOCK --ulimit memlock=-1:-1 \ --env AWS_LR_ACCOUNT_ID=123456789 \ --env AWS_REGION=eu-west-1 \ --env LR_VAULT_ACCOUNT_ID=accountId \ --env LR_VAULT_ACCOUNT_TYPE=awsiam \ --env LR_VAULT_INPUT=/data/vault/input \ --env LR_VAULT_OUTPUT=/data/vault/output \ --env LR_VAULT_LOCALE=au \ --env LR_VAULT_COUNTRY_CODE_COLUMN=COUNTRY_CODE \ -v ~/Documents/tmp:/config \ -v ~/path-to-input-folder:/data/vault/input \ -v ~/path-to-output-folder:/data/vault/output \ vault-app \ -config-file test-configuration.yml
Configuration file example:
liveramp: local_encoder: account_type: awsiam account_id: 123456789 profile: dev locale: AU output: s3://test-in input: s3://test-out filename_pattern: pattern header_mapping: email=mail country_code_column: COUNTRY_CODE mode: default dry_run: false gcp_project_name: liveramp-eng-chp-staging lr_aws: lr_id: lrid123 lr_access_key_id: lracesss lr_secret_access_key: lrsecret lr_region: lrregion metastore: db: url: h2:~/vault-app-metadatastore driver: org.h2.Driver username: sa password: password platform: org.hibernate.dialect.H2Dialect
GCS buckets as input/output source
To list/create/update GCP resources from the command line,
install Cloud SDK
brew install --cask google-cloud-sdk
Log in and configure the project with:
gcloud init
For the application to use your credentials locally, run the following commands:
gcloud auth application-default login export GOOGLE_APPLICATION_CREDENTIALS= /Users/$USER/.config/gcloud/application_default_credentials.json
When using GCS bucket as input or output source we have to provide GOOGLE_APPLICATION_CREDENTIALS and LR_VAULT_GCP_PROJECT_NAME. Also must mount the gcloud config folder to our docker image
docker run --rm --cap-add IPC_LOCK --ulimit memlock=-1:-1 \ --env GOOGLE_APPLICATION_CREDENTIALS=./tmp/config/application_default_credentials.json \ --env LR_VAULT_GCP_PROJECT_NAME=liveramp-eng-chp-staging \ -v $HOME/.config/gcloud:/tmp/config \ [rest of environment variables] Vault-app
For GCS buckets, prefix is gs://
Number of columns validation for every row
Support multi-country in Local Encoder
Added new parameter
LR_VAULT_COUNTRY_CODE_COLUMN
orcountry_code_column
(in configuration file). IfLR_VAULT_COUNTRY_CODE_COLUMN
parameter exists and is not empty, validation and normalization of phone numbers will be done using country code from that column, otherwise it will use locale.If the parameter value is passed but not found in the header an error is thrown and processing is stoped.
Support additional locales in Local Encoder (BR, MX, NZ, CA, US)
Added Support for: BR, MX, NZ, CA, US
Changed
Improvement for application.properties files
Upgrade of spring boot to version 2.5.4, spring integration to 5.5.3 and spring integration aws to 2.5.1
Upgrade gradle version from 6.4.1 to 7.2
Fixed
Use delimiter method when joining output header
Fill-rate metrics shows wrong percentage
Version 1.3.0
Release Date: 07/07/21
Tags: ‘v1.3.0'
Added
Support for .csv escape sequence
Support for NAP identifiers for Europe
Version 1.2.0
Release Date: 05/31/21
Tags: ‘v1.2.0'
Added
Dry-Run Mode
S3 object lambda access point support
S3 output to clients' account
Separate AWS regions for client and LR AWS account
Changed
Disable MD5 check strategy for S3 objects
S3 input uses default AWS credentials
Base docker image openjdk15:jre-15.0.2_7-alpine
Fixed
Single file processing (task mode)
Metadata store initialization
S3 input
Docker container shut down in case of exception in task mode
Connection timeout for S3 input
Version 1.1.0
Version 1.1.0
Version 1.0.0
Release Date: 11/26/20
Tags: ‘v1.0.0'
Fixed
Fixed issue with processing sha256email column. Value is Hex decoded first.