Skip to main content

LiveRamp Local Encoder

The LiveRamp Local Encoder allows you to access some of the benefits of LiveRamp Activation or Addressability solutions without having to move your consumer data outside of your control.

Local Encoder enables you to generate securely-encoded RampIDs for your consumer data files within your own cloud environment and then utilize that data for activation or addressability use cases, depending on your needs. In this way your consumer data is never exposed to an external network, while still enabling full use of the LiveRamp solutions. The encoded RampIDs produced by the application cannot be decoded back to the original consumer identifiers.

Note

As of August 21, 2024, a new Local Encoder (Vault App) version is now available through the ‘latest’ or ‘v1.8.3’ tag in our Amazon ECR. This mandatory update is available now and is designed to enhance your security and ensure the continued functionality of your application. Access to earlier versions will end on December 31, 2024. To learn more about this release, see “Local Encoder Release Notes”.

Depending on your needs, the Local Encoder has several RampID output options that determine how Local Encoder generates your output file:

  • RampID packets: Used for Activation by brands and marketers, these RampID packets can be delivered to LiveRamp, where they can be transformed into RampIDs and used to generate fields and segments in your LiveRamp application.

  • Identity envelopes: Typically used by publishers and platforms who want to get RampIDs into the bidstream to safely engage with the programmatic advertising ecosystem.

    Note

    Identity envelope output is only available with Local Encoder Version 1.6 or later.

  • RampIDs: Typically used by publishers and platforms who want to create a RampID mapping.

See the diagrams below for the differences between LiveRamp-hosted ID creation and Local Encoder ID creation (RampID packet example shown).

I-Local_Encoder-LiveRamp_Hosted_ID_creationg.png
I-Local_Encoder-Local_Encoder_ID_creationg.png

Local Encoder is run as a Docker image, distributed via the AWS Elastic Container Registry. Local Encoder can be run on any infrastructure that supports running Docker images. Contact your LiveRamp representative to see if the Local Encoder is suitable for your needs.

For additional details, see the sections below. Once you're ready to deploy the Local Encoder, see the following articles for more information:

Local Encoder Security

No matter which method you use to implement the Local Encoder, the security protocols listed below apply.

  • Local Encoder is distributed as a Docker image. You obtain the Docker image from LiveRamp’s AWS ECR repository.

  • The application will need read-only access to an input location and write access to an output location. The input location is owned and controlled by you, and the output location is either owned by you or by the LiveRamp platform.

  • LiveRamp manages the keys and secrets. The application has read-only access to these resources.

  • The only network calls made are to cloud key management services.

Using Local Encoder for Activation

Brands or marketers can use Local Encoder to get their consumer data into LiveRamp for activation at destination platforms without having that consumer data leave their environment.

For example, you might be a brand with CRM data who wants to send that data to destination platforms for targeting but not allow your consumer PII to leave your control. You can use Local Encoder to generate RampID packets containing derived RampIDs from your data files, which LiveRamp first ingests into your Connect or Safe Haven account and then uses to create segments from that data. You can then use your LiveRamp application to distribute those segments to your desired destinations, where they’ll be available for activation.

Note

For Activation use cases, you can receive RampID packets. You cannot receive RampIDs or identity envelopes.

Using Local Encoder for Addressability

Platforms and publishers can use Local Encoder to build a mapping between their platform users and RampIDs or identity envelopes without having consumer data leave their environment. This mapping can then be used to allow marketers to use LiveRamp to activate on RampID in their platform for improved matching. This mapping can also be used to transact on RampIDs (by means of identity envelopes) in the bidstream.

For example, you might be a platform who wants to transact on RampIDs in the bidstream with SSPs but not allow your consumer PII to leave your environment. You can use Local Encoder to generate identity envelopes (which contain derived RampIDs). These envelopes can then be used to create a mapping between your platform users and their envelopes. When a mapped user visits your website, you can pass their associated envelope into the bidstream, where they are utilized by SSPs in bid requests.

Note

For Addressability use cases, you can receive RampIDs or identity envelopes. You cannot receive RampID packets.

Deconfliction Options

Deconfliction is needed for analytics use cases and works best against universe datasets, meaning the full customer view across CRM-based data, subscriber data, or transaction data. These are typically large datasets, and because of the number of different touchpoints included (address, email, phone) and the sharing of touchpoints among consumers, one record in your dataset might match to some RampIDs in the LiveRamp Identity Graph that are also connected to other consumers (such as family members or relatives). This creates identity conflicts because some of these RampIDs might match to other records in your systems. These conflicts are the inevitable result of the sharing of devices, the use of multiple email addresses (and the sharing of email addresses), and other similar consumer behavior.

For certain use cases, you might want to include as many RampIDs as possible to maximize match and reach, and for these, deconfliction is not recommended. However, for other use cases, such as measurement, you might want to minimize noise across first-party datasets and the deconfliction configuration enables tuning of the outputs to maximize accuracy and fit the LiveRamp graph to your universe. This can be particularly useful for large datasets where overconnected RampIDs can introduce noise into analytics. Some use cases that benefit from this configuration are closed loop measurement, reach and frequency, or mutually exclusive segments as examples.

Using deconfliction on large universe datasets can help match your records to the RampIDs that are determined to be the most relevant for each record by looking at the strength of the connection of each RampID to each touchpoint. It includes references to how active the linkage is in the digital ecosystem.

To utilize deconfliction, you’ll need to provide a RID (Record ID) for each record, and for the configuration to drive the highest value, the full universe is required to deconflict across the table. The output you receive will include the best fidelity RampIDs for each RID.

There are two deconfliction options you can choose from:

  • Maximized first-party fidelity: This configuration returns the RampIDs that are determined to be most relevant but preserves additional RIDs even if there are RampIDs that indicate LiveRamp could consolidate them. This option is ideal for publishers and other data owners.

  • Standard: This configuration returns the RampIDs that are determined to be most relevant and removes RIDs that are determined to be duplicates (based on linking to the same RampID). This is the default option and covers most advertiser use cases.

To learn more about deconfliction or to have deconfliction enabled for your account, contact your LiveRamp representative.

Available Integrations

If you’re using Local Encoder to generate identity envelopes, you can integrate Local Encoder with our ATS Direct and Google PAIR programs. This allows you to use Local Encoder to add the associated ID for these programs into the prebid module.

To find out about enabling these integrations, contact your LiveRamp representative.

Integrating with ATS Direct

ATS Direct enables direct media buys between publishers using ATS and marketers distributing their segments via LiveRamp Connect. ATS Direct uses envelopes from ATS to find matching segments. Matched users will be associated with a Deal ID which will be returned with the ATS envelope back to the user’s browser. Deal IDs for matched users will be used by the Google Publisher Tag (GPT) to communicate with Google Ad Manager (GAM) to present the user with the predefined ad creative.

You can integrate Local Encoder envelope generation with our ATS Direct program by setting up the ATS API. Once enabled, ATS will return an identity envelope and the IDs needed for ATS Direct when refreshing the envelope using the ATS API refresh endpoint.

For more information, see “ATS Direct for Publishers”.

Integrating with Google PAIR

LiveRamp’s Google PAIR program is a first-party data solution that allows Display & Video 360 publishers and advertisers to match their respective first-party data in a privacy-compliant way.

You can integrate Local Encoder envelope generation with our Google PAIR program by setting up an ATS for API configuration. Once enabled, ATS will return an identity envelope and a PAIR ID when refreshing the envelope using the ATS API refresh endpoint.

For more information, see “Google PAIR and ATS”.

Refreshing Identity Envelopes

If you’re using Local Encoder to obtain identity envelopes, you’ll need to refresh those envelopes periodically, An envelope is valid for 15 days from the date of creation. 

A timestamp column is appended to the end of each row. This column gives the expiration date and time for the identity envelope in Unix format (timezone UTC). You can use this timestamp column to determine when you need to refresh the envelope.

You can refresh envelopes by calling the Refreshing Envelopes endpoint of the ATS API (for more information, see “Authenticated Traffic Solution API Documentation”).

Note

Envelopes are also refreshed any time you reprocess the data through Local Encoder.

How Local Encoder Works

Once you have the Local Encoder configured, the typical workflow consists of these overall steps:

  1. You format your consumer data file(s) and then upload them to the input location of your secure VPC (virtual private cloud).

  2. You run the encoding operation within your VPC:

    1. The data is normalized and hygiene is performed.

    2. The identifiers in the data are converted into derived RampIDs.

    3. If appropriate, the derived RampIDs for each record are encoded into secure RampID packets or identity envelopes.

    4. The input identifiers are removed and replaced with the appropriate RampID output type (RampIDs, RampID packets, or identity envelopes).

  3. For Activation use cases the following steps are performed:

    1. The output containing RampID packets is delivered to LiveRamp.

    2. LiveRamp decodes the RampID packets into their individual derived RampIDs.

    3. LiveRamp matches those derived RampIDs to their associated maintained RampIDs.

    4. LiveRamp creates the appropriate fields and segments from the segment data in your LiveRamp platform (such as Connect or Safe Haven).

  4. For Addressability use cases the following steps are performed:

    1. The output containing RampIDs or identity envelopes is output to the destination of your choice.

    2. You leverage the output to build a mapping table of your customer IDs to RampIDs or identity envelopes.

See the diagram below for an overview of the Local Encoder workflow.

I-Local_Encoder-Local_Encoder_workflow.png

Optimizing Local Encoder

Use the information below to optimize the performance of Local Encoder.

Allocate Sufficient Resources

The application operates entirely in memory, so make sure to allocate sufficient resources to the container to handle the data processing. During processing, the file size increases to 10 times its original size (the recommended initial file size is 500MB or smaller).

Set an Appropriate Transfer Speed

The application starts processing the file immediately as it streams from the input bucket. A transfer speed of 10GBPS is recommended for optimal performance.

Allocate the Appropriate Number of CPUs

Starting from version 1.8.0, the application supports parallel processing. It will process one file per available CPU, so if you want the application to be able to process 10 files at a time you should allocate 10 CPUs. This should be considered when configuring your infrastructure resources. 

When you run the application, you can use the ActiveProcessorCount parameter to specify the number of CPUs that will be utilized. See the individual implementation articles for more information.

Adjust File Sizes

If you want to reduce the processing time, you can partition your files. For example, a single file containing 21 million email addresses will typically be processed in 45 minutes. If you divide those records into two files of 10.5 million records each and have at least 2 CPUs allocated, the processing time will typically be 22.5 minutes.

Note

These typical processing times are estimates based on internal testing. Your processing times might be different.

Data You Can Send

The Local Encoder process starts when you upload a column-based file containing your customer records to an input location that you own and control. Each row of the file should contain the identifiers you want to have translated into your desired output. For Activation use cases, you can also include segment data which will be used to create fields and segments in your LiveRamp platform.

Supported identifier fields for translation to RampIDs:

  • Plaintext email addresses

  • SHA256-hashed email addresses

  • Plaintext phone numbers

  • SHA256-hashed phone numbers

  • Plaintext name and postcode (which consists of first name, last name, and postcode or ZIP Code) (European countries only)

  • Mobile device IDs (IDFAs and AAIDs)

Note

Name and postcode, mobile device ID, and IMEI are not supported for Addressability use cases. They are only supported for Activation use cases.

Other fields you can include:

  • Row IDs (RIDs)

  • Segment data (for Activation use cases)

Example Header

See the header shown below for an example of what the header might look like when sending data in a pipe-separated file (psv) for an Activation use case, where segment data fields are included:

RID|EMAIL1|EMAIL2|EMAIL3|SHA256EMAIL1|SHA256EMAIL2|SHA256EMAIL3|MOBILE1|MOBILE2|SHA256MOBILE1|SHA256MOBILE2|LANDLINE1|SHA256LANDLINE1|FIRSTNAME|LASTNAME|POSTCODE|AAID|IDFA|IMEI|ATTRIBUTE_1|...|ATTRIBUTE_N

Replace ATTRIBUTE_1 … N in the example header with the name of your CRM attributes.

Output Options and Examples

See the sections below for more information on the output options for Local Encoder and output file examples.

RampIDs

RampIDs are LiveRamp’s proprietary, pseudonymous identifiers for representing individuals. RampID creation involves hashing, salting, and private key encryption in order to prevent consumer data from being re-identified. RampIDs are unique per each LiveRamp client and have a partner-specific encoding.

RampIDs form the basis for using the LiveRamp product. Once your data has been converted to RampIDs, you are able to use the LiveRamp platform for collaboration, activation, and measurement activities.

For RampID output, all consumer identifier data in a row is transformed into derived RampIDs in the form of a JSON string in a “RampID” column:

  • Each plaintext email address returns a derived RampID for each hash type (MD5, SHA1, and SHA-356)

  • Each SHA-256 hashed email address returns one derived RampID

  • Each phone number returns one derived RampID

See below for an example of an output file for RampID output, based on the input file shown below:

RID|EMAIL1|MOBILE1
123|solomon@mac.com|(629) 555-1536
456|maratb@gmail.com|(365) 555-0786

The output file will follow the format shown below:

RID|RampID

The JSON string might look like the example below:

RID, [
  {
    "header": "EMAIL1",
    "RampID": {
      "MD5": "Xi2247b9-ap-AcsWdsKFquY8vUnNig7fYQC-zkiqN9MKJQLvXaq_dppPrOiQBIy8PUYN2h",
      "SHA1": "Xi2247LbvDme7s-hRJXnG1NXe6CwFpRlNC60vFHOOpQXROEBiSF0LZYmdGYcbMcechi_iL",
      "SHA256": "Xi2247FbP--uJe0dFhLUrVD1MOTO6Iuncg5XbBOrR3MFywWgJMP9gzt2pdnVfCWQiUo-d_"
    }
  },
  {
    "header": "MOBILE1",
    "RampID": {
      "NONE": "Xi2247zPBI2V6OAVfpzBWhKy3ay8WJ66flt_1_hDYr0-y3mcf1HHDt2ZgRTyvhWxqp08Y0"
    }
  }
]

RampID Packets

For Activation use cases, Local Encoder can generate RampID packets by bundling individual RampIDs for a given consumer record and then adding an additional layer of encryption that involves a randomly-generated code. The randomly-generated code provides an added layer of protection, ensuring that the same client record (with the same RampID) will produce a packet which looks different every time the application runs.

The packets produced by the application are sent to LiveRamp, where they are unpacked and the included RampIDs are loaded into the client’s LiveRamp account.

For RampID packet output, all consumer identifier data in a row is transformed into derived RampIDs, packaged into one data structure and encrypted again, yielding a RampID packet.

See below for an example of an output file for RampID packet output, based on the input file shown below:

RID|EMAIL1|MOBILE1|FIRSTNAME|LASTNAME|POSTCODE|DOG_OWNER|NUM_DOGS
123|solomon@mac.com|(629) 555-1536|Solomon|Gumbel|10012|true|3
456|maratb@gmail.com|(365) 555-0786|Marta|Baldwin|94602|false|0

The output file will follow the format shown below:

RID|RampID_PACKET|DOG_OWNER|NUM_DOGS
123|dVd8HezBpkyxiOpiUwh1+iYBBRx+RrBMM02rnvDoZT8i=|true|3
456|Tt16SRP6Svrnc2CQPrnHsLEcDR+WWLscjWdYmY2JR5h=|false|0

Identity Envelopes

Identity envelopes are typically utilized by publishers and platforms who want to get RampIDs into the bidstream to safely engage with the programmatic advertising ecosystem or who want to enable brands and marketers using LiveRamp to distribute RampID-based data for improved targeting.

Note

Identity envelope output is only available with Local Encoder Version 1.6 or later.

An identity envelope is an opaque and encrypted structure containing RampIDs and metadata. Each envelope is created with a client-specific encryption key that makes it unique. An envelope is valid for 15 days from the date of creation (the output file contains the expiration date and time for each envelope).

Note

To refresh expired envelopes, use the Refreshing Envelopes endpoint of the ATS API (for more information, see “Authenticated Traffic Solution API Documentation”).

For identity envelope output, all consumer identifier data is transformed into derived RampIDs. A selection logic is applied, and then the RampID is additionally obfuscated and encrypted into an identity envelope. Only one identity envelope is returned per row of data.

A timestamp column is appended to the end of each row. This column gives the expiration date and time for the identity envelope in Unix format (timezone UTC). You can use this timestamp column to determine when you need to refresh the envelope.

An envelope is represented as a Base64-encoded string.

Identity envelope string example:

AY3SYJlFDI1RqsMv9gZOjOab6hGuwWKmbE4K70Z-Zex27DD6M9U0Jg

The string is different each time an envelope is created for the same underlying data as a result of randomization.

Envelopes can be decrypted using the Envelope Decryption API or by SSPs running the Sidecar application.

See below for an example of an output file for identity envelope output, based on the input file shown below:

RID|EMAIL1|MOBILE1
123|solomon@mac.com|(629) 555-1536
456|maratb@gmail.com|(365) 555-0786

The output file will follow the format shown below:

RID|ENVELOPE|TIMESTAMP
123|AY3SYJlFDI1RqsMv9gZOjOab6hGuwWKmbE4K70Z-Zex27DD6M9U0Jg|1701715163
456|ApKVQQBzOR41WDJQoK_iWEaJNsvVUcOHgOWEwB_SzNSRvHPNegb8-EpxIu3G-1ikxzGUwJZQcHOZLsZLySU|1701715163