Skip to main content

Local Encoder Release Notes

The sections below outline the released versions of Local Encoder.

Version 1.8.3

Release Date: 8/21/24

Tags:  ‘latest’, ‘v1.8.3’, or ‘v1.8.3-ubuntu’ 

Added

  • Multi-arch Docker builds (AMD64 and ARM64): Optimized the application build to target multiple CPU architecture combinations to ensure optimal performance across architectures.

  • Support AWS SDK for Java 1.x and 2.x Environment Variables: The AWS Version 2.x release changed the names of specific variables. Both 1.x and 2.x environment variables are now supported.

  • Multi-Region LiveRamp AWS infrastructure support: In the following quarter, LiveRamp will release regional infrastructure for Local Encoder to limit cross-border data transfer. No changes are required at this time.

Changed

  • CRITICAL: AWS S3 uploads now require a minimum of 2GB of RAM per CPU. This will ensure sufficient memory to transfer the data into the application for processing.

Fixed

  • Disabled the JAVA agent flag: Prevents modifying the behavior of the running application.

  • Disabled overriding AWS HTTP Client: Disabled unnecessary system and environmental settings. 

Version 1.8.2

Release Date: 6/13/24

Tags: ‘latest’, ‘v1.8.2', ‘v1.8.2-ubuntu’

Changed

  • The 10-file limit for the input bucket has been removed  

  • The application now uses parallel processing

    • The processing rate is 1 file per available CPU

Fixed

  • AWS and GCP client libs loading files in memory.

  • Fix the “connection reset” error for input buckets.

  • Resolved timeout issue for AWS client with large file uploads.

Version 1.8.1

Release Date: 4/30/24

Tags: ‘v1.8.1', ‘v1.8.1-ubuntu’

Changed

  • Add new ENV variable -> LR_VAULT_S3_TARGET_THROUGHPUT this variable has default settings, it doesn’t need to be changed unless there are issues with S3 upload.

Fixed

  • Files are not loaded in memory during AWS & GCP uploads

  • Fix the “connection reset” error for input buckets

Version 1.8

Release Date: 3/13/24

Tags: ‘v1.8', ‘v1.8-ubuntu’

Changed

  • The limit of 10 files for the input folder has changed. There is no longer a limit to the number of files in the input bucket. Restarting the application will reprocess all files found in the input bucket. 

  • The application now uses parallel file processing to process the files more efficiently

  • The application logging has changed to improve performance. There is no lost information but the logs will be rendered in a new manner.

Fixed

  • Recognition of Brazilian Mobile Phone Numbers

  • Broader Envelope support for use with ATS Direct

Version 1.7

Release Date: 12/12/23

Tags: ‘v1.7', ‘v1.7-ubuntu’

Added

  • File Processing Error Logging

    • The optional feature allows any record errors generated during file processing to be logged into a separate file. This alleviates the need to search row by row within a file to find the errors. This feature is only available when running locally.

  • Defined Row Identifier

    • A default row identifier [RID] has been instituted for customers looking to build mapping files. This non-PII touchpoint can be customized using header mappings.

    • The RID will now be returned in the first column of the dataset.

  • LiveRamp Identity Envelope Expiration Date

    • When leveraging Local Encoder to generate LiveRamp Identity Envelopes, the envelope expiration timestamp is now provided as an additional column in the output. The Unix timestamp is based on UTC.

  • Application Logging

    • A new Alert notice has been added to application logs when the application fails to start when the input directory exceeds the limit of unprocessed files.

Fixed

  • Resolved an issue preventing the use of header mapping while files of a TSV or PSV format.

Version 1.6.3

Release Date: 10/26/23

Tags: ‘v1.6.3, ‘v1.6.3-ubuntu’

Added

  • Envelope output contains row identifier value RID. Header Value: A defined row identifier must be included in the input file to maintain the sort order of records.   The default row identifier value is 'rid'. This column value is only required if you need to keep a sort order.

Fixed

  • Custom Header Mappings: A list of key=value pairs that can be used to replace the default headers for the identifier columns in the input file.  Header Mappings can be utilized on all three supported file formats: CSV, PSV, and TSV

Version 1.6.2

Release Date: 06/29/23

Tags: ‘v1.6.2, ‘v1.6.2-ubuntu’

Changed

  • Rotated LiveRamp PGP public key

Version 1.6.1

Release Date: 05/23/23

Tags: ‘v1.6.1', ‘v1.6.1-ubuntu'

Fixed

  • Gradle vulnerabilities

Version 1.6.0

Release Date: 04/11/23

Tags: ‘v1.6.0', ‘v1.6.0-ubuntu'

Added

  • An alternative application version using the tag v#.#.#-ubuntu. The standard Local Encoder uses an Alpine Linux base image

  • Support for envelope generation with checks for opt-outs

    • Envelope generation is account specific

    • To apply, after account is enabled

      • the param is envelopes_flow: true or --env LR_VAULT_ENVELOPES_FLOW=true

  • Support for SHA256 phones (landline and mobile)

    • Header Values for use

      • SHA256MOBILE1, SHA256MOBILE2

      • SHA256LANDLINE1

Version 1.5.2

Release Date: 12/28/22

Tags: ‘v1.5.2'

Fixed

  • Gradle vulnerabilities

Version 1.5.1

Release Date: 08/12/22

Tags: ‘v1.5.1'

Changed

  • LiveRamp PGP keys are used for end to end encryption

  • Derived RampIDs output is structured in simpler json format

Version 1.5.0

Release Date: 07/20/22

Tags: ‘v1.5.0'

Changed

  • Fixed IV vulnerability in IDLPacket AES encryption

Version 1.4.8

Release Date: 07/07/22

Tags: ‘v1.4.8'

Added

  • Included encrypted prefix in the output file name when using encryption flow

Version 1.4.7

Release Date: 06/08/22

Tags: ‘v1.4.7'

Added

  • Added possibility of additional encryption of records processed by local encoder

Version 1.4.6

Release Date: 05/12/22

Tags: ‘v1.4.6'

Changed

  • Upgraded H2 database connector to v2.1.210 to fix RCE vulnerability

Version 1.4.5

Release Date: 04/08/22

Tags: ‘v1.4.5'

Changed

  • Spring Boot update to 2.6.6

Version 1.4.4

Release Date: 01/26/22

Tags: ‘v1.4.4'

Changed

  • Log4j update to 2.17.1

Version 1.4.3

Release Date: 12/29/21

Tags: ‘v1.4.3'

Fixed

  • Issue when all identifier headers are mapped

Version 1.4.2

Release Date: 12/16/21

Tags: ‘v1.4.2'

Changed

  • Log4j update to 2.16.0

Version 1.4.0

Release Date: 10/29/21

Tags: ‘v1.4.0'

Added

  • Supported pseudonymisation of Mobile Device IDs

    • Added support for MobileID identifiers, supported headers are IDFA, AAID and IMEI

  • Upgrade configuration from flags to configuration file

    • Configuration parameters can also be set via YAML configuration file. Configuration file properties are overwritten by properties passed trough the command line. In order to use configuration file we should mount a directory where the file is  located to /config directory. Example: 

      -v ~/Documents/localencoder:/config - our yaml config file should be located in localencoder folder. Then pass the file name as a command line argument, full command example: 

      docker run --rm --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
--env AWS_LR_ACCOUNT_ID=123456789 \
--env AWS_REGION=eu-west-1 \
--env LR_VAULT_ACCOUNT_ID=accountId \
--env LR_VAULT_ACCOUNT_TYPE=awsiam \
--env LR_VAULT_INPUT=/data/vault/input \
--env LR_VAULT_OUTPUT=/data/vault/output \
--env LR_VAULT_LOCALE=au \
--env LR_VAULT_COUNTRY_CODE_COLUMN=COUNTRY_CODE \
-v ~/Documents/tmp:/config \
-v ~/path-to-input-folder:/data/vault/input \
-v ~/path-to-output-folder:/data/vault/output \
vault-app \
-config-file test-configuration.yml 

    • Configuration file example: 

      liveramp:
local_encoder:
         account_type: awsiam
         account_id: 123456789
         profile: dev
         locale: AU
         output: s3://test-in
         input: s3://test-out
         filename_pattern: pattern
         header_mapping: email=mail
         country_code_column: COUNTRY_CODE
         mode: default
         dry_run: false
         gcp_project_name: liveramp-eng-chp-staging
         lr_aws:
             lr_id: lrid123
             lr_access_key_id: lracesss
             lr_secret_access_key: lrsecret
            lr_region: lrregion
         metastore:
             db:
                         url: h2:~/vault-app-metadatastore
                         driver: org.h2.Driver
                         username: sa
                         password: password
                         platform: org.hibernate.dialect.H2Dialect

  • GCS buckets as input/output source

    • To list/create/update GCP resources from the command line,

    • install Cloud SDK

      brew install --cask google-cloud-sdk 

    • Log in and configure the project with:

      gcloud init 

    • For the application to use your credentials locally, run the following commands:

      gcloud auth application-default login export GOOGLE_APPLICATION_CREDENTIALS=       /Users/$USER/.config/gcloud/application_default_credentials.json 

    • When using GCS bucket as input or output source we have to provide GOOGLE_APPLICATION_CREDENTIALS and LR_VAULT_GCP_PROJECT_NAME. Also must mount the gcloud config folder to our docker image

      docker run --rm --cap-add IPC_LOCK --ulimit memlock=-1:-1 \ --env GOOGLE_APPLICATION_CREDENTIALS=./tmp/config/application_default_credentials.json \ --env LR_VAULT_GCP_PROJECT_NAME=liveramp-eng-chp-staging \ -v $HOME/.config/gcloud:/tmp/config \ [rest of environment variables]    Vault-app 

    • For GCS buckets, prefix is gs://

  • Number of columns validation for every row

  • Support multi-country in Local Encoder

    • Added new parameter LR_VAULT_COUNTRY_CODE_COLUMN or country_code_column(in configuration file). If LR_VAULT_COUNTRY_CODE_COLUMN parameter exists and is not empty, validation and normalization of phone numbers will be done using country code from that column, otherwise it will use locale. 

      If the parameter value is passed but not found in the header an error is thrown and processing is stoped.

  • Support additional locales in Local Encoder (BR, MX, NZ, CA, US)

    • Added Support for: BR, MX, NZ, CA, US

Changed

  • Improvement for application.properties files

  • Upgrade of spring boot to version 2.5.4, spring integration to 5.5.3 and spring integration aws to 2.5.1

  • Upgrade gradle version from 6.4.1 to 7.2

Fixed

  • Use delimiter method when joining output header

  • Fill-rate metrics shows wrong percentage

Version 1.3.0

Release Date: 07/07/21

Tags: ‘v1.3.0'

Added

  • Support for .csv escape sequence

  • Support for NAP identifiers for Europe

Version 1.2.0

Release Date: 05/31/21

Tags: ‘v1.2.0'

Added

  • Dry-Run Mode

  • S3 object lambda access point support

  • S3 output to clients' account

  • Separate AWS regions for client and LR AWS account

Changed

  • Disable MD5 check strategy for S3 objects

  • S3 input uses default AWS credentials

  • Base docker image openjdk15:jre-15.0.2_7-alpine

Fixed

  • Single file processing (task mode)

  • Metadata store initialization

  • S3 input

  • Docker container shut down in case of exception in task mode

  • Connection timeout for S3 input

Version 1.1.0

Version 1.1.0

Version 1.0.0

Release Date: 11/26/20

Tags: ‘v1.0.0'

Fixed

  • Fixed issue with processing sha256email column. Value is Hex decoded first.

In this section: