Importing plotly failed. Interactive plots will not work. 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter is_subgraph_orchestration 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter is_automode 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter IsImageTask 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter IsTextDNNTask 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter max_nodes 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter distributed_dnn_max_node_check 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter enable_distributed_featurization 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter enable_distributed_dnn_training 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter enable_distributed_dnn_training_ort_ds 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter enable_cache 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter use_incremental_learning_override 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter is_gpu_tmp 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter vm_priority 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter miro_flight 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter many_models_process_count_per_node 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter automl_many_models_scenario 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter enable_mltable_quick_profile 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter _enable_future_regressors 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter ensemble_download_models_timeout_sec 2023-07-09:17:36:16,74 WARNING [automl_base_settings.py:752] Received unrecognized parameter stack_meta_learner_train_percentage 2023-07-09:17:36:16,74 INFO [_distributed_helper.py:48] Horovod import succeeded 2023-07-09:17:36:16,74 INFO [_distributed_helper.py:50] Initializing horovod. 2023-07-09:17:36:33,21 INFO [logging_utilities.py:399] [RunId:zen_sheep_883mz7gy7x_HD_1]CPU logical cores: 4, CPU cores: 2, virtual memory: 33667153920, swap memory: 0. 2023-07-09:17:36:33,22 INFO [logging_utilities.py:410] [RunId:zen_sheep_883mz7gy7x_HD_1]Platform information: Linux. 2023-07-09:17:36:33,76 INFO [logging_utilities.py:399] [RunId:zen_sheep_883mz7gy7x_HD_1]CPU logical cores: 4, CPU cores: 2, virtual memory: 33667153920, swap memory: 0. 2023-07-09:17:36:33,76 INFO [logging_utilities.py:410] [RunId:zen_sheep_883mz7gy7x_HD_1]Platform information: Linux. Building model 2023-07-09:17:36:34,145 INFO [forecast_tcn_wrapper.py:500] Building model 2023-07-09:17:36:34,148 INFO [tcn_model_utl.py:234] Model used the following hyperparameters: num_cells=3, multilevel=CELL, depth=1, num_channels=64, dropout_rate=0.5, dilation=2 Start time: 2023-07-09T17:27:42.03935Z, latest permissible end time: 2023-07-09 18:21:42.039350+00:00 2023-07-09:17:36:35,283 INFO [forecast_tcn_wrapper.py:589] Start time: 2023-07-09T17:27:42.03935Z, latest permissible end time: 2023-07-09 18:21:42.039350+00:00 the name of the metric used EarlyStoppingCallback normalized_root_mean_squared_error 2023-07-09:17:36:35,283 INFO [forecast_tcn_wrapper.py:592] the name of the metric used EarlyStoppingCallback normalized_root_mean_squared_error The patience used in used EarlyStoppingCallback 20 2023-07-09:17:36:35,283 INFO [forecast_tcn_wrapper.py:593] The patience used in used EarlyStoppingCallback 20 the name of the improvement passed to EarlyStoppingCallback 0.001 2023-07-09:17:36:35,283 INFO [forecast_tcn_wrapper.py:594] the name of the improvement passed to EarlyStoppingCallback 0.001 LR Factor 0.5 2023-07-09:17:36:35,283 INFO [forecast_tcn_wrapper.py:595] LR Factor 0.5 Apply log transform to label during training: True 2023-07-09:17:36:35,283 INFO [forecast_tcn_wrapper.py:619] Apply log transform to label during training: True 2023-07-09:17:36:35,284 INFO [forecaster.py:761] No GPU of compute capability >= 7.0 detected; AMP is disabled. Trying with batch_size: 16 2023-07-09:17:36:36,196 INFO [forecast_tcn_wrapper.py:232] Trying with batch_size: 16 2023-07-09:17:36:37,372 WARNING [metrics.py:198] azureml.automl.runtime.shared.metrics.compute_metrics_regression is deprecated. Please use azureml.automl.runtime.shared.score.scoring.score_regression 2023-07-09:17:36:37,530 WARNING [metrics.py:198] azureml.automl.runtime.shared.metrics.compute_metrics_regression is deprecated. Please use azureml.automl.runtime.shared.score.scoring.score_regression 2023-07-09:17:36:37,630 ERROR [runner.py:61] TCN runner script terminated with an exception of type: 2023-07-09:17:36:37,704 INFO [logging_handler.py:290] Sending 3776 bytes 2023-07-09:17:36:37,704 INFO [logging_handler.py:304] Finish uploading in 0.066171 seconds. 2023-07-09:17:36:37,706 INFO [run.py:2347] fail is not setting status for submitted runs. Cleaning up all outstanding Run operations, waiting 300.0 seconds 2 items cleaning up... Cleanup took 0.12673616409301758 seconds Traceback (most recent call last): File "hd_forecasting_dnn_driver.py", line 14, in runner.run(mltable_data_json=mltable_data_json) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/contrib/automl/dnn/forecasting/wrapper/dispatched/invoker/runner.py", line 58, in run _run(mltable_data_json, **kwargs) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/contrib/automl/dnn/forecasting/wrapper/dispatched/invoker/runner.py", line 144, in _run model.train( File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/contrib/automl/dnn/forecasting/wrapper/forecast_tcn_wrapper.py", line 238, in train self.forecaster.fit( File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/forecast/forecast/forecaster.py", line 198, in fit self._callbacks.on_val_end(epoch, val_loss, metrics) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/forecast/forecast/callbacks/callback.py", line 238, in on_val_end cb.on_val_end(epoch, loss, metrics) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/contrib/automl/dnn/forecasting/callbacks/run_update.py", line 89, in on_val_end scores = self._score_metrics(loss=loss, metrics=FORECASTING_SCALAR_SET | REGRESSION_SCALAR_SET) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/contrib/automl/dnn/forecasting/callbacks/run_update.py", line 160, in _score_metrics y_pred = self.model_wrapper._predict(data_loader=data_loader).reshape(-1) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/contrib/automl/dnn/forecasting/wrapper/forecast_tcn_wrapper.py", line 723, in _predict predictions = np.asarray(self.forecaster.predict(data_loader)) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/forecast/forecast/forecaster.py", line 237, in predict batch = self._batch_transform.do(batch) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/forecast/forecast/data/batch_transforms/batch_transforms.py", line 139, in do series_indices = self._get_series_ids(batch[PAST_IND_KEY]) File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/forecast/forecast/data/batch_transforms/batch_transforms.py", line 224, in _get_series_ids return self._series_indexer(past_regressor).long() if self._series_indexer is not None else None File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/contrib/automl/dnn/forecasting/wrapper/forecast_tcn_wrapper.py", line 894, in indexer Contract.assert_true(grain_num in ForecastTCNWrapper._s_grain_map, File "/azureml-envs/azureml-automl-dnn-forecasting-gpu/lib/python3.8/site-packages/azureml/automl/core/shared/_diagnostics/contract.py", line 49, in assert_true raise InvalidOperationException( azureml.automl.core.shared.exceptions.InvalidOperationException: InvalidOperationException: Message: Assertion Failed. Invalid Operation. Target: batch. Reference Code: 54a98e68-7b31-11ec-b7df-8c1645fec84b. Details: One of the time series was not found in the data set. InnerException: None ErrorResponse { "error": { "code": "SystemError", "message": "Encountered an internal AutoML error. Error Message/Code: InvalidOperationException. Additional Info: InvalidOperationException:\n\tMessage: Assertion Failed. Invalid Operation. Target: batch. Reference Code: 54a98e68-7b31-11ec-b7df-8c1645fec84b. Details: One of the time series was not found in the data set.\n\tInnerException: None\n\tErrorResponse \n{\n \"error\": {\n \"message\": \"Assertion Failed. Invalid Operation. Target: batch. Reference Code: 54a98e68-7b31-11ec-b7df-8c1645fec84b. Details: One of the time series was not found in the data set.\",\n \"target\": \"batch\",\n \"reference_code\": \"54a98e68-7b31-11ec-b7df-8c1645fec84b\"\n }\n}", "details_uri": "https://aka.ms/automltroubleshoot", "target": "batch", "inner_error": { "code": "ClientError", "inner_error": { "code": "AutoMLInternal" } }, "reference_code": "54a98e68-7b31-11ec-b7df-8c1645fec84b" } }