使用 Ax 進行多目標 NAS

建立於：2022 年 8 月 19 日 | 最後更新：2024 年 7 月 31 日 | 最後驗證：2024 年 11 月 05 日

作者： David Eriksson、Max Balandat 和 Meta 的 Adaptive Experimentation 團隊。

在本教學中，我們將展示如何使用 Ax 對熱門的 MNIST 資料集上的簡單神經網路模型執行多目標神經架構搜尋 (NAS)。 雖然底層方法通常會用於更複雜的模型和更大的資料集，但我們選擇了一個可以在不到 20 分鐘內在筆記型電腦上輕鬆端到端運行的教學。

在許多 NAS 應用程式中，存在多個感興趣目標之間的自然權衡。 例如，當在裝置上部署模型時，我們可能希望最大化模型效能（例如，準確度），同時最小化競爭指標，例如功耗、推理延遲或模型大小，以滿足部署約束。 通常，我們可以透過接受最低限度的模型效能來大幅降低計算需求或預測延遲。 有效探索此類權衡的原則性方法是可擴展和永續 AI 的關鍵推動者，並且在 Meta 中有許多成功的應用 - 例如，請參閱我們關於自然語言理解模型的 案例研究。

在我們的範例中，我們將調整兩個隱藏層的寬度、學習率、dropout 概率、批次大小和訓練週期數。 目標是在效能（驗證集上的準確度）和模型大小（模型參數的數量）之間進行權衡。

本教學使用以下 PyTorch 函式庫

• PyTorch Lightning（指定模型和訓練迴圈）

• TorchX（用於遠端/非同步執行訓練任務）

• BoTorch（為 Ax 演算法提供支援的貝氏最佳化函式庫）

定義 TorchX App

我們的目標是最佳化在 mnist_train_nas.py 中定義的 PyTorch Lightning 訓練任務。 若要使用 TorchX 執行此操作，我們編寫一個輔助函式，該函式接收訓練任務的架構和超參數的值，並使用適當的設定建立一個 TorchX AppDef。

from pathlib import Path

import torchx

from torchx import specs
from torchx.components import utils


def trainer(
    log_path: str,
    hidden_size_1: int,
    hidden_size_2: int,
    learning_rate: float,
    epochs: int,
    dropout: float,
    batch_size: int,
    trial_idx: int = -1,
) -> specs.AppDef:

    # define the log path so we can pass it to the TorchX ``AppDef``
    if trial_idx >= 0:
        log_path = Path(log_path).joinpath(str(trial_idx)).absolute().as_posix()

    return utils.python(
        # command line arguments to the training script
        "--log_path",
        log_path,
        "--hidden_size_1",
        str(hidden_size_1),
        "--hidden_size_2",
        str(hidden_size_2),
        "--learning_rate",
        str(learning_rate),
        "--epochs",
        str(epochs),
        "--dropout",
        str(dropout),
        "--batch_size",
        str(batch_size),
        # other config options
        name="trainer",
        script="mnist_train_nas.py",
        image=torchx.version.TORCHX_IMAGE,
    )

設定 Runner

Ax 的 Runner 抽象允許編寫與各種後端的介面。 Ax 已經附帶了 TorchX 的 Runner，因此我們只需要設定它。 為了本教學的目的，我們以完全非同步的方式在本地運行任務。

為了在叢集上啟動它們，您可以改為指定不同的 TorchX 排程器並適當地調整設定。 例如，如果您有一個 Kubernetes 叢集，您只需要將排程器從 local_cwd 變更為 kubernetes)。

import tempfile
from ax.runners.torchx import TorchXRunner

# Make a temporary dir to log our results into
log_dir = tempfile.mkdtemp()

ax_runner = TorchXRunner(
    tracker_base="/tmp/",
    component=trainer,
    # NOTE: To launch this job on a cluster instead of locally you can
    # specify a different scheduler and adjust arguments appropriately.
    scheduler="local_cwd",
    component_const_params={"log_path": log_dir},
    cfg={},
)

設定 SearchSpace

首先，我們定義我們的搜尋空間。 Ax 支援整數和浮點數類型的範圍參數，以及可以具有非數值類型（例如字串）的選擇參數。 我們將調整隱藏大小、學習率、dropout 和週期數作為範圍參數，並將批次大小調整為有序的選擇參數，以強制它為 2 的冪。

from ax.core import (
    ChoiceParameter,
    ParameterType,
    RangeParameter,
    SearchSpace,
)

parameters = [
    # NOTE: In a real-world setting, hidden_size_1 and hidden_size_2
    # should probably be powers of 2, but in our simple example this
    # would mean that ``num_params`` can't take on that many values, which
    # in turn makes the Pareto frontier look pretty weird.
    RangeParameter(
        name="hidden_size_1",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="hidden_size_2",
        lower=16,
        upper=128,
        parameter_type=ParameterType.INT,
        log_scale=True,
    ),
    RangeParameter(
        name="learning_rate",
        lower=1e-4,
        upper=1e-2,
        parameter_type=ParameterType.FLOAT,
        log_scale=True,
    ),
    RangeParameter(
        name="epochs",
        lower=1,
        upper=4,
        parameter_type=ParameterType.INT,
    ),
    RangeParameter(
        name="dropout",
        lower=0.0,
        upper=0.5,
        parameter_type=ParameterType.FLOAT,
    ),
    ChoiceParameter(  # NOTE: ``ChoiceParameters`` don't require log-scale
        name="batch_size",
        values=[32, 64, 128, 256],
        parameter_type=ParameterType.INT,
        is_ordered=True,
        sort_values=True,
    ),
]

search_space = SearchSpace(
    parameters=parameters,
    # NOTE: In practice, it may make sense to add a constraint
    # hidden_size_2 <= hidden_size_1
    parameter_constraints=[],
)

設定指標

Ax 具有 指標 的概念，該指標定義結果的屬性以及如何獲得這些結果的觀察值。 這允許例如編碼如何從某些分散式執行後端提取資料並在作為輸入傳遞給 Ax 之前進行後處理。

在本教學中，我們將使用 多目標最佳化，目標是最大化驗證準確度並最小化模型參數的數量。 後者代表模型延遲的簡單代理，對於小型 ML 模型來說，很難準確估計（在實際應用程式中，我們會在裝置上運行模型時對延遲進行基準測試）。

在我們的範例中，TorchX 將在本機以完全非同步的方式執行訓練任務，並根據試驗索引將結果寫入 log_dir（請參閱上面的 trainer() 函式）。我們將定義一個知道此記錄目錄的度量類別。透過繼承 TensorboardCurveMetric，我們可以免費獲得讀取和解析 TensorBoard 記錄的邏輯。

from ax.metrics.tensorboard import TensorboardMetric
from tensorboard.backend.event_processing import plugin_event_multiplexer as event_multiplexer

class MyTensorboardMetric(TensorboardMetric):

    # NOTE: We need to tell the new TensorBoard metric how to get the id /
    # file handle for the TensorBoard logs from a trial. In this case
    # our convention is to just save a separate file per trial in
    # the prespecified log dir.
    def _get_event_multiplexer_for_trial(self, trial):
        mul = event_multiplexer.EventMultiplexer(max_reload_threads=20)
        mul.AddRunsFromDirectory(Path(log_dir).joinpath(str(trial.index)).as_posix(), None)
        mul.Reload()

        return mul

    # This indicates whether the metric is queryable while the trial is
    # still running. We don't use this in the current tutorial, but Ax
    # utilizes this to implement trial-level early-stopping functionality.
    @classmethod
    def is_available_while_running(cls):
        return False

現在，我們可以實例化準確度和模型參數數量的度量。在此，curve_name 是 TensorBoard 記錄中度量的名稱，而 name 是 Ax 內部使用的度量名稱。我們也指定 lower_is_better 來表示這兩個度量的有利方向。

val_acc = MyTensorboardMetric(
    name="val_acc",
    tag="val_acc",
    lower_is_better=False,
)
model_num_params = MyTensorboardMetric(
    name="num_params",
    tag="num_params",
    lower_is_better=True,
)

設定 OptimizationConfig

告訴 Ax 應該最佳化的方式是透過 OptimizationConfig。在這裡，我們使用 MultiObjectiveOptimizationConfig，因為我們將執行多目標最佳化。

此外，Ax 支援透過指定目標閾值來限制不同度量，這些閾值限定了我們想要探索的結果空間中的感興趣區域。在此範例中，我們將限制驗證準確度至少為 0.94 (94%)，模型參數數量最多為 80,000。

from ax.core import MultiObjective, Objective, ObjectiveThreshold
from ax.core.optimization_config import MultiObjectiveOptimizationConfig


opt_config = MultiObjectiveOptimizationConfig(
    objective=MultiObjective(
        objectives=[
            Objective(metric=val_acc, minimize=False),
            Objective(metric=model_num_params, minimize=True),
        ],
    ),
    objective_thresholds=[
        ObjectiveThreshold(metric=val_acc, bound=0.94, relative=False),
        ObjectiveThreshold(metric=model_num_params, bound=80_000, relative=False),
    ],
)

建立 Ax 實驗

在 Ax 中，Experiment 物件是儲存所有問題設定資訊的物件。

from ax.core import Experiment

experiment = Experiment(
    name="torchx_mnist",
    search_space=search_space,
    optimization_config=opt_config,
    runner=ax_runner,
)

選擇產生策略

GenerationStrategy 是我們想要執行最佳化方式的抽象表示。雖然可以自訂（如果您想這麼做，請參閱本教學課程），但在大多數情況下，Ax 可以根據搜尋空間、最佳化配置和我們想要執行的試驗總數自動確定適當的策略。

通常，Ax 選擇在啟動基於模型的貝氏最佳化策略之前，評估一些隨機配置。

total_trials = 48  # total evaluation budget

from ax.modelbridge.dispatch_utils import choose_generation_strategy

gs = choose_generation_strategy(
    search_space=experiment.search_space,
    optimization_config=experiment.optimization_config,
    num_trials=total_trials,
  )

[INFO 02-03 16:47:51] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there is at least one ordered parameter and there are no unordered categorical parameters.
[INFO 02-03 16:47:51] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trials=None max_initialization_trials=None num_tunable_parameters=6 num_trials=48 use_batch_trials=False
[INFO 02-03 16:47:51] ax.modelbridge.dispatch_utils: calculated num_initialization_trials=9
[INFO 02-03 16:47:51] ax.modelbridge.dispatch_utils: num_completed_initialization_trials=0 num_remaining_initialization_trials=9
[INFO 02-03 16:47:51] ax.modelbridge.dispatch_utils: `verbose`, `disable_progbar`, and `jit_compile` are not yet supported when using `choose_generation_strategy` with ModularBoTorchModel, dropping these arguments.
[INFO 02-03 16:47:51] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+BoTorch', steps=[Sobol for 9 trials, BoTorch for subsequent trials]). Iterations after 9 will take longer to generate due to model-fitting.

配置排程器

Scheduler 充當最佳化的迴圈控制。它與後端通信以啟動試驗、檢查其狀態並檢索結果。在本教學課程中，它只是讀取和解析本機儲存的日誌。在遠端執行設定中，它將呼叫 API。以下來自 Ax 排程器教學課程的說明總結了排程器如何與用於執行試驗評估的外部系統交互

Scheduler 需要 Experiment 和 GenerationStrategy。可以透過 SchedulerOptions 傳入一組選項。在這裡，我們配置了總評估次數以及 max_pending_trials，即應該同時運行的最大試驗數。在我們的本機設定中，這是作為單獨進程運行的訓練任務數量，而在遠端執行設定中，這將是您想要平行使用的機器數量。

from ax.service.scheduler import Scheduler, SchedulerOptions

scheduler = Scheduler(
    experiment=experiment,
    generation_strategy=gs,
    options=SchedulerOptions(
        total_trials=total_trials, max_pending_trials=4
    ),
)

[WARNING 02-03 16:47:51] ax.service.utils.with_db_settings_base: Ax currently requires a sqlalchemy version below 2.0. This will be addressed in a future release. Disabling SQL storage in Ax for now, if you would like to use SQL storage please install Ax with mysql extras via `pip install ax-platform[mysql]`.
[INFO 02-03 16:47:51] Scheduler: `Scheduler` requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to `True` on experiment.

執行最佳化

現在一切都已配置完成，我們可以讓 Ax 以完全自動化的方式運行最佳化。排程器將定期檢查日誌中所有目前運行試驗的狀態，如果試驗完成，排程器將更新其在實驗中的狀態，並獲取貝氏最佳化演算法所需的觀察值。

scheduler.run_all_trials()

[INFO 02-03 16:47:51] Scheduler: Fetching data for newly completed trials: [].
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:47:51] Scheduler: Running trials [0]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:47:52] Scheduler: Running trials [1]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:47:53] Scheduler: Running trials [2]...
/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:47:54] Scheduler: Running trials [3]...
[INFO 02-03 16:47:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:47:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 16:47:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:47:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 16:47:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:47:57] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 16:47:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:47:59] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 16:48:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:03] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 02-03 16:48:08] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:08] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 02-03 16:48:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:16] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 02-03 16:48:27] Scheduler: Fetching data for newly completed trials: [1, 3].
[INFO 02-03 16:48:27] Scheduler: Retrieved COMPLETED trials: [1, 3].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:48:27] Scheduler: Running trials [4]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:48:28] Scheduler: Running trials [5]...
[INFO 02-03 16:48:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:29] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 16:48:30] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:30] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 16:48:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:32] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 16:48:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:34] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 16:48:37] Scheduler: Fetching data for newly completed trials: [2].
[INFO 02-03 16:48:37] Scheduler: Retrieved COMPLETED trials: [2].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:48:37] Scheduler: Running trials [6]...
[INFO 02-03 16:48:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 16:48:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 16:48:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:41] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 16:48:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 16:48:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:46] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 02-03 16:48:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:48:52] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 02-03 16:48:59] Scheduler: Fetching data for newly completed trials: [4].
[INFO 02-03 16:48:59] Scheduler: Retrieved COMPLETED trials: [4].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:48:59] Scheduler: Running trials [7]...
[INFO 02-03 16:49:00] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:00] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 16:49:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:01] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 16:49:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:03] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 16:49:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:05] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 16:49:08] Scheduler: Fetching data for newly completed trials: [6].
[INFO 02-03 16:49:08] Scheduler: Retrieved COMPLETED trials: [6].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/modelbridge/cross_validation.py:463: UserWarning:

Encountered exception in computing model fit quality: RandomModelBridge does not support prediction.

[INFO 02-03 16:49:08] Scheduler: Running trials [8]...
[INFO 02-03 16:49:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:09] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 16:49:10] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:10] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 16:49:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:12] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 16:49:14] Scheduler: Fetching data for newly completed trials: [5].
[INFO 02-03 16:49:14] Scheduler: Retrieved COMPLETED trials: [5].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:49:17] Scheduler: Running trials [9]...
[INFO 02-03 16:49:18] Scheduler: Fetching data for newly completed trials: [0].
[INFO 02-03 16:49:18] Scheduler: Retrieved COMPLETED trials: [0].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:49:23] Scheduler: Running trials [10]...
[INFO 02-03 16:49:24] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:24] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 16:49:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:25] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 16:49:27] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:27] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 16:49:29] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:29] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 16:49:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:32] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 02-03 16:49:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:38] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 02-03 16:49:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:49:45] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 02-03 16:49:57] Scheduler: Fetching data for newly completed trials: [9].
[INFO 02-03 16:49:57] Scheduler: Retrieved COMPLETED trials: [9].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:50:01] Scheduler: Running trials [11]...
[INFO 02-03 16:50:02] Scheduler: Fetching data for newly completed trials: [8].
[INFO 02-03 16:50:02] Scheduler: Retrieved COMPLETED trials: [8].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:50:06] Scheduler: Running trials [12]...
[INFO 02-03 16:50:08] Scheduler: Fetching data for newly completed trials: [7, 10].
[INFO 02-03 16:50:08] Scheduler: Retrieved COMPLETED trials: [7, 10].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:50:14] Scheduler: Running trials [13]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:50:17] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:50:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:17] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:50:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:18] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:50:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:20] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:50:22] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:22] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:50:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:25] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:50:30] Scheduler: Fetching data for newly completed trials: [11].
[INFO 02-03 16:50:30] Scheduler: Retrieved COMPLETED trials: [11].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:50:36] Scheduler: Running trials [14]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:50:40] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:50:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:40] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:50:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:41] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:50:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:43] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:50:45] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:45] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:50:48] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:50:48] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:50:54] Scheduler: Fetching data for newly completed trials: [12].
[INFO 02-03 16:50:54] Scheduler: Retrieved COMPLETED trials: [12].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:51:01] Scheduler: Running trials [15]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:51:05] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:51:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:05] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:51:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:06] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:51:07] Scheduler: Fetching data for newly completed trials: [13].
[INFO 02-03 16:51:07] Scheduler: Retrieved COMPLETED trials: [13].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:51:12] Scheduler: Running trials [16]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:51:17] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:51:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:17] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:51:18] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:18] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:51:19] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:19] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:51:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:21] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:51:25] Scheduler: Fetching data for newly completed trials: [15].
[INFO 02-03 16:51:25] Scheduler: Retrieved COMPLETED trials: [15].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:51:31] Scheduler: Running trials [17]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:51:36] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:51:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:51:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:51:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:38] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:51:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:41] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:51:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:44] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:51:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:49] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 16:51:57] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:51:57] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 16:52:08] Scheduler: Fetching data for newly completed trials: [14].
[INFO 02-03 16:52:08] Scheduler: Retrieved COMPLETED trials: [14].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:52:15] Scheduler: Running trials [18]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:52:21] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:52:21] Scheduler: Fetching data for newly completed trials: [16].
[INFO 02-03 16:52:21] Scheduler: Retrieved COMPLETED trials: [16].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:52:29] Scheduler: Running trials [19]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:52:35] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:52:35] Scheduler: Fetching data for newly completed trials: [17].
[INFO 02-03 16:52:35] Scheduler: Retrieved COMPLETED trials: [17].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:52:44] Scheduler: Running trials [20]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:52:50] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:52:50] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:52:50] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:52:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:52:51] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:52:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:52:53] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:52:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:52:55] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:52:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:52:58] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:53:03] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:53:03] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 16:53:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:53:11] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 16:53:22] Scheduler: Fetching data for newly completed trials: [18].
[INFO 02-03 16:53:22] Scheduler: Retrieved COMPLETED trials: [18].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:53:29] Scheduler: Running trials [21]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:53:34] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:53:34] Scheduler: Fetching data for newly completed trials: [19].
[INFO 02-03 16:53:34] Scheduler: Retrieved COMPLETED trials: [19].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:53:43] Scheduler: Running trials [22]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:53:49] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:53:49] Scheduler: Fetching data for newly completed trials: [20].
[INFO 02-03 16:53:49] Scheduler: Retrieved COMPLETED trials: [20].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:53:59] Scheduler: Running trials [23]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:54:04] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:54:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:04] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:54:05] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:05] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:54:06] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:06] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:54:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:09] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:54:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:12] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:54:17] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:17] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 16:54:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:25] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 16:54:36] Scheduler: Fetching data for newly completed trials: [21].
[INFO 02-03 16:54:36] Scheduler: Retrieved COMPLETED trials: [21].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:54:45] Scheduler: Running trials [24]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:54:51] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:54:51] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:51] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:54:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:52] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:54:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:54:53] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:54:55] Scheduler: Fetching data for newly completed trials: [23].
[INFO 02-03 16:54:55] Scheduler: Retrieved COMPLETED trials: [23].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:55:05] Scheduler: Running trials [25]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:55:10] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:55:11] Scheduler: Fetching data for newly completed trials: [22].
[INFO 02-03 16:55:11] Scheduler: Retrieved COMPLETED trials: [22].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:55:22] Scheduler: Running trials [26]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:55:32] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:55:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:55:32] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:55:33] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:55:33] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:55:34] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:55:34] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:55:37] Scheduler: Fetching data for newly completed trials: [25].
[INFO 02-03 16:55:37] Scheduler: Retrieved COMPLETED trials: [25].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:55:46] Scheduler: Running trials [27]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:55:53] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:55:53] Scheduler: Fetching data for newly completed trials: [24].
[INFO 02-03 16:55:53] Scheduler: Retrieved COMPLETED trials: [24].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:56:01] Scheduler: Running trials [28]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:56:07] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:56:07] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:07] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:56:08] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:08] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:56:09] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:09] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:56:11] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:11] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:56:15] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:15] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:56:20] Scheduler: Fetching data for newly completed trials: [26].
[INFO 02-03 16:56:20] Scheduler: Retrieved COMPLETED trials: [26].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:56:29] Scheduler: Running trials [29]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:56:35] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:56:35] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:35] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:56:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:36] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:56:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:38] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:56:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:40] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:56:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:43] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:56:48] Scheduler: Fetching data for newly completed trials: [27].
[INFO 02-03 16:56:48] Scheduler: Retrieved COMPLETED trials: [27].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:56:55] Scheduler: Running trials [30]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:56:59] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:56:59] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:56:59] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:57:00] Scheduler: Fetching data for newly completed trials: [28].
[INFO 02-03 16:57:00] Scheduler: Retrieved COMPLETED trials: [28].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:57:12] Scheduler: Running trials [31]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:57:21] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:57:21] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:21] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:57:22] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:22] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:57:23] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:23] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:57:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:26] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:57:29] Scheduler: Fetching data for newly completed trials: [29].
[INFO 02-03 16:57:29] Scheduler: Retrieved COMPLETED trials: [29].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:57:44] Scheduler: Running trials [32]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:57:54] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:57:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:54] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:57:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:55] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:57:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:57:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:57:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:58:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:58:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:58:07] Scheduler: Fetching data for newly completed trials: [31].
[INFO 02-03 16:58:07] Scheduler: Retrieved COMPLETED trials: [31].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:58:19] Scheduler: Running trials [33]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:58:25] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:58:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:58:25] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:58:26] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:58:26] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:58:27] Scheduler: Fetching data for newly completed trials: [30].
[INFO 02-03 16:58:27] Scheduler: Retrieved COMPLETED trials: [30].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:58:38] Scheduler: Running trials [34]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:58:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:58:45] Scheduler: Fetching data for newly completed trials: 32 - 33.
[INFO 02-03 16:58:45] Scheduler: Retrieved COMPLETED trials: 32 - 33.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:58:55] Scheduler: Running trials [35]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:59:05] Scheduler: Running trials [36]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:59:12] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 16:59:12] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:59:12] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 16:59:13] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:59:13] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 16:59:14] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:59:14] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 16:59:16] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:59:16] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 16:59:20] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:59:20] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 16:59:25] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:59:25] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 16:59:32] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 16:59:32] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 16:59:44] Scheduler: Fetching data for newly completed trials: 34 - 35.
[INFO 02-03 16:59:44] Scheduler: Retrieved COMPLETED trials: 34 - 35.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 16:59:55] Scheduler: Running trials [37]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:00:07] Scheduler: Running trials [38]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:00:14] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:00:14] Scheduler: Fetching data for newly completed trials: [36].
[INFO 02-03 17:00:14] Scheduler: Retrieved COMPLETED trials: [36].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:00:26] Scheduler: Running trials [39]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:00:36] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:00:36] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:00:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 17:00:37] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:00:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 17:00:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:00:38] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 17:00:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:00:40] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 17:00:44] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:00:44] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 17:00:49] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:00:49] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 17:00:57] Scheduler: Fetching data for newly completed trials: 37 - 38.
[INFO 02-03 17:00:57] Scheduler: Retrieved COMPLETED trials: 37 - 38.
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:01:08] Scheduler: Running trials [40]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:01:21] Scheduler: Running trials [41]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:01:30] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:01:30] Scheduler: Fetching data for newly completed trials: [39].
[INFO 02-03 17:01:30] Scheduler: Retrieved COMPLETED trials: [39].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:01:43] Scheduler: Running trials [42]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:01:54] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:01:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:01:54] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 17:01:55] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:01:55] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 17:01:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:01:56] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 17:01:58] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:01:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 17:02:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:02:02] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 17:02:07] Scheduler: Fetching data for newly completed trials: [40].
[INFO 02-03 17:02:07] Scheduler: Retrieved COMPLETED trials: [40].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:02:15] Scheduler: Running trials [43]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:02:20] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:02:20] Scheduler: Fetching data for newly completed trials: [41].
[INFO 02-03 17:02:20] Scheduler: Retrieved COMPLETED trials: [41].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:02:31] Scheduler: Running trials [44]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:02:38] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:02:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:02:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 17:02:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:02:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 17:02:40] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:02:40] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 17:02:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:02:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 17:02:46] Scheduler: Fetching data for newly completed trials: [42].
[INFO 02-03 17:02:46] Scheduler: Retrieved COMPLETED trials: [42].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:02:58] Scheduler: Running trials [45]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:03:05] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:03:05] Scheduler: Fetching data for newly completed trials: [44].
[INFO 02-03 17:03:05] Scheduler: Retrieved COMPLETED trials: [44].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:03:18] Scheduler: Running trials [46]...
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:03:26] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 17:03:26] Scheduler: Fetching data for newly completed trials: [43].
[INFO 02-03 17:03:26] Scheduler: Retrieved COMPLETED trials: [43].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:03:37] Scheduler: Running trials [47]...
[INFO 02-03 17:03:38] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 17:03:39] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 17:03:41] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:41] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 17:03:43] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 17:03:46] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:46] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 17:03:51] Scheduler: Fetching data for newly completed trials: [45].
[INFO 02-03 17:03:51] Scheduler: Retrieved COMPLETED trials: [45].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:03:52] Scheduler: Done submitting trials, waiting for remaining 2 running trials...
[INFO 02-03 17:03:52] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:52] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 2).
[INFO 02-03 17:03:53] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:53] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 2).
[INFO 02-03 17:03:54] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:54] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 2).
[INFO 02-03 17:03:56] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:03:56] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 2).
[INFO 02-03 17:04:00] Scheduler: Fetching data for newly completed trials: [47].
[INFO 02-03 17:04:00] Scheduler: Retrieved COMPLETED trials: [47].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[INFO 02-03 17:04:00] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:04:00] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 1).
[INFO 02-03 17:04:01] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:04:01] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 1).
[INFO 02-03 17:04:02] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:04:02] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 1).
[INFO 02-03 17:04:04] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:04:04] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 1).
[INFO 02-03 17:04:08] Scheduler: Fetching data for newly completed trials: [].
[INFO 02-03 17:04:08] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 1).
[INFO 02-03 17:04:13] Scheduler: Fetching data for newly completed trials: [46].
[INFO 02-03 17:04:13] Scheduler: Retrieved COMPLETED trials: [46].
/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.


OptimizationResult()

評估結果

我們現在可以使用 Ax 附帶的輔助函式和視覺化來檢查最佳化的結果。

首先，我們產生一個包含實驗結果摘要的 dataframe。此 dataframe 中的每一列都對應於一個試驗（即已運行的訓練任務），並包含有關試驗狀態、已評估的參數配置和已觀察到的度量值的信息。這提供了一種簡單的方法來健全檢查最佳化。

from ax.service.utils.report_utils import exp_to_df

df = exp_to_df(experiment)
df.head(10)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 02-03 17:04:13] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

	trial_index	arm_name	trial_status	generation_method	is_feasible	num_params	val_acc	hidden_size_1	hidden_size_2	learning_rate	epochs	dropout	batch_size
0	0	0_0	COMPLETED	Sobol	False	16810.0	0.883973	19	66	0.003182	4	0.190970	32
1	1	1_0	COMPLETED	Sobol	False	40280.0	0.918498	50	18	0.000614	2	0.425028	128
2	2	2_0	COMPLETED	Sobol	False	112720.0	0.963316	127	96	0.001233	3	0.334354	256
3	3	3_0	COMPLETED	Sobol	False	30653.0	0.907889	37	35	0.000234	1	0.037933	64
4	4	4_0	COMPLETED	Sobol	False	26056.0	0.923292	28	108	0.000339	1	0.062996	32
5	5	5_0	COMPLETED	Sobol	True	71312.0	0.953213	87	32	0.006412	3	0.297078	128
6	6	6_0	COMPLETED	Sobol	False	50051.0	0.866982	59	55	0.000116	2	0.458433	256
7	7	7_0	COMPLETED	Sobol	False	19530.0	0.937533	24	21	0.002236	4	0.161959	64
8	8	8_0	COMPLETED	Sobol	False	18080.0	0.906884	20	80	0.000163	3	0.388082	64
9	9	9_0	COMPLETED	BoTorch	True	38672.0	0.954845	41	128	0.002599	2	0.170741	128

我們還可以視覺化驗證準確度和模型參數數量之間權衡的帕雷托前沿。

提示

Ax 使用 Plotly 產生互動式繪圖，這讓您可以執行諸如縮放、裁剪或懸停等操作，以便查看繪圖組件的詳細資訊。試用一下，如果您想了解更多資訊，請查看視覺化教學課程）。

最終的最佳化結果顯示在下圖中，其中顏色對應於每個試驗的迭代次數。我們看到我們的方法能夠成功地探索權衡，並且找到了具有高驗證準確度的大型模型以及具有相對較低驗證準確度的小型模型。

from ax.service.utils.report_utils import _pareto_frontier_scatter_2d_plotly

_pareto_frontier_scatter_2d_plotly(experiment)

/usr/local/lib/python3.10/dist-packages/ax/core/map_data.py:195: FutureWarning:

The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

[WARNING 02-03 17:04:13] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column.

為了更好地了解我們的代理模型已經學到了關於黑盒目標的什麼，我們可以看一下留一法交叉驗證結果。由於我們的模型是高斯過程，它們不僅提供點預測，還提供關於這些預測的不確定性估計。一個好的模型意味著預測的均值（圖中的點）接近 45 度線，並且置信區間以預期頻率覆蓋 45 度線（這裡我們使用 95% 置信區間，所以我們期望它們包含真實觀察值的時間為 95%）。

如下圖所示，模型大小 (num_params) 度量比驗證準確度 (val_acc) 度量更容易建模。

from ax.modelbridge.cross_validation import compute_diagnostics, cross_validate
from ax.plot.diagnostic import interact_cross_validation_plotly
from ax.utils.notebook.plotting import init_notebook_plotting, render

cv = cross_validate(model=gs.model)  # The surrogate model is stored on the ``GenerationStrategy``
compute_diagnostics(cv)

interact_cross_validation_plotly(cv)

我們還可以製作等高線圖，以更好地了解不同的目標如何取決於兩個輸入參數。在下圖中，我們顯示了模型預測的驗證準確度作為兩個隱藏大小的函數。驗證準確度隨著隱藏大小的增加而明顯增加。

from ax.plot.contour import interact_contour_plotly

interact_contour_plotly(model=gs.model, metric_name="val_acc")

類似地，我們在下圖中展示了模型參數的數量與隱藏層大小的關係，可以看到它也隨著隱藏層大小的增加而增加（對 hidden_size_1 的依賴性更大）。

interact_contour_plotly(model=gs.model, metric_name="num_params")

致謝

我們感謝 TorchX 團隊（特別是 Kiuk Chung 和 Tristan Rice）在將 TorchX 與 Ax 整合方面的幫助。