Auto-Scale Processing Capacity
Auto-Scale is an experimental processing capacity option for SQL and PySpark questions and flow nodes in Hybrid and Hybrid Confidential Compute clean rooms. Instead of choosing a fixed capacity tier yourself, you let LiveRamp Clean Room analyze the question's code and data volume and tune the Spark run configuration for you.
When a run fails because it ran short of resources, the system moves up a tier and runs the question again. When a run succeeds, the system saves that configuration and reuses it for later runs.
Auto-Scale is useful when you do not yet know which capacity tier a question needs. Without it, finding the right tier takes a test-and-learn cycle of raising the setting manually after each failure.
Note
Auto-Scale is under active development, and its behavior might change based on customer feedback.
How Auto-Scale Differs from a Fixed Tier
The fixed processing capacity tiers are Default, Medium, Large, Extra Large, 2X Large, and 3X Large. Each tier allocates the same resources to every run until you change the setting. If a run exceeds those resources, it fails, and you raise the tier and run the question again yourself. For descriptions of each tier, see "Clean Room FAQs".
Auto-Scale replaces that manual cycle. Beyond raising the capacity tier, it also tunes Spark settings that a fixed tier leaves alone, such as partitioning, network timeouts, and join thresholds.
How Auto-Scale Works
The first run starts at Medium capacity. Auto-Scale needs no run history to begin, so a question can use it immediately.
If the run fails because of a capacity or resource constraint, the system moves up a tier and runs the question again. The UI does not show the failed attempt.
On each retry, the system evaluates the specific cause of the failure, such as an out-of-memory error or a CPU limit, and adjusts the configuration accordingly.
The cycle repeats until the run succeeds.
When a run succeeds, the system saves that configuration as the baseline and reuses it for the next run of that question.
The system does not move back down a tier after later successful runs. The saved configuration stays at the level the system reached.
Set a Question's Processing Capacity to Auto-Scale
To change this setting, you need to be the question's owner. For information, see "Question Permissions and Overrides".
Before you save, be aware of the following:
Auto-Scale discards any manual configuration overrides on the question. See "Considerations".
The setting applies to all of the question's associated clean room questions (CRQs). Clean Room displays a warning about this when you save.
Enter the clean room that contains the question.
From the navigation menu, select Clean Room → Questions to open the Questions page.
From the question's More Options menu (the three dots), select Advanced Question Settings.
Under Processing capacity needed to run the question, select (Experimental) Auto-Scale.
Click . If Clean Room warns that the change will change the processing capacity for all associated CRQs, confirm to continue.
Turn Off Auto-Scale
Auto-Scale is optional. To turn it off, set the question's processing capacity to another value. Select the capacity tier the question used before you switched to Auto-Scale. If you do not know the previous tier, select Default.
Follow steps 1 through 3 of the previous procedure.
Under Processing capacity needed to run the question, select the previous tier, or Default if you do not know it.
Click , and confirm the warning if one appears.
Turning off Auto-Scale does not restore the manual configuration overrides that Auto-Scale discarded.
Considerations
Auto-Scale discards manual configuration overrides: If you have hand-tuned the Spark configuration for a question, switching to Auto-Scale removes those settings and replaces them with its own. Record your settings before you switch, in case you want to restore them.
The first run takes longer: Expect a longer-than-usual runtime for the first run while the system calibrates. After the system settles on a configuration, you should see fewer failures and fewer manual adjustments for that question.
Editing a question can restart calibration: Edits that change a question's capacity needs might cause the tuning cycle to start over.
Auto-Scale does not scale back down: After a run succeeds, the system keeps the configuration it reached, even if later runs need fewer resources.
Flow nodes have their own setting: A flow node does not inherit the processing capacity of its question. Set the capacity on each node separately. For information, see "Create a Flow" and "The Flow Builder Page".
Cloning resets what the system learned: When you clone a clean room or a question, the saved configuration does not carry over. The cloned question starts calibration from the beginning.