update_acs

embeddings=azureml://subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourcegroups/enterpriseit-ai/workspaces/confluence-fdt/datastores/workspaceblobstore/paths/azureml/b6caa2e2-a343-46ee-89ba-0935d6862b2a/embeddings/
acs_config={"index_name":"calm-cabbage-58b6jhctzf"}
connection_id=None
output=/mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/cap/data-capability/wd/index
verbosity=1
[2025-03-03 03:43:38] INFO     azureml.rag.update_acs - Reading connection id from environment variable (update_acs.py:495)
[2025-03-03 03:43:38] INFO     azureml.rag.update_acs.update_acs - ActivityStarted, update_acs (activity.py:108)
[2025-03-03 03:43:38] INFO     azureml.rag.connections - Getting workspace connection: enterpriseitaisearch, with input credential: <class 'NoneType'>. (connections.py:332)
[2025-03-03 03:43:38] INFO     azureml.rag.connections - Getting credential from AzureMLTokenAuthentication._initialize_aml_token_auth (connections.py:341)
[2025-03-03 03:43:39] INFO     azureml.rag.connections - Getting workspace connection via MLClient with auth: <class 'azureml.dataprep.api._aml_auth._azureml_token_authentication.AzureMLTokenAuthentication'>, subscription_id: bf4f4af4-8cb7-420c-b682-722abbdff681, resource_group_name: enterpriseit-ai, workspace_name: confluence-fdt. (connections.py:348)
Method connections: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
[2025-03-03 03:43:39] INFO     azureml.rag.connections - Using ml_client base_url: https://eastus.api.azureml.ms/rp/workspaces, original_base_url: https://management.azure.com. (connections.py:367)
[2025-03-03 03:43:40] INFO     azureml.rag.connections - Parsed Connection: /subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourceGroups/enterpriseit-ai/providers/Microsoft.MachineLearningServices/workspaces/confluence-fdt/connections/enterpriseitaisearch (connections.py:386)
[2025-03-03 03:43:40] INFO     azureml.rag.connections - Got connection: /subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourceGroups/enterpriseit-ai/providers/Microsoft.MachineLearningServices/workspaces/confluence-fdt/connections/enterpriseitaisearch as <class 'azure.ai.ml.entities._workspace.connections.connection_subtypes.AzureAISearchConnection'>. (connections.py:443)
[2025-03-03 03:43:40] INFO     azureml.rag.update_acs - got embeddings uri as input: azureml://subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourcegroups/enterpriseit-ai/workspaces/confluence-fdt/datastores/workspaceblobstore/paths/azureml/b6caa2e2-a343-46ee-89ba-0935d6862b2a/embeddings/ (update_acs.py:407)
[2025-03-03 03:43:40] INFO     azureml.rag.update_acs - extracted embeddings directory name: embeddings (update_acs.py:410)
[2025-03-03 03:43:40] INFO     azureml.rag.update_acs - extracted embeddings container path: azureml://subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourcegroups/enterpriseit-ai/workspaces/confluence-fdt/datastores/workspaceblobstore/paths/azureml/b6caa2e2-a343-46ee-89ba-0935d6862b2a/ (update_acs.py:412)
[2025-03-03 03:43:40] INFO     azureml.rag.update_acs - mounting embeddings container from: 
azureml://subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourcegroups/enterpriseit-ai/workspaces/confluence-fdt/datastores/workspaceblobstore/paths/azureml/b6caa2e2-a343-46ee-89ba-0935d6862b2a/ 
   to: 
/mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount (update_acs.py:417)
[2025-03-03 03:43:42] INFO     azureml.rag.azureml.rag.embeddings - loading embeddings from: /mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount/embeddings (__init__.py:573)
[2025-03-03 03:43:43] INFO     azureml.rag.connections - Getting workspace connection: kshe-m7llzpu3-eastus2_aoai, with input credential: <class 'NoneType'>. (connections.py:332)
[2025-03-03 03:43:43] INFO     azureml.rag.connections - Getting credential from AzureMLTokenAuthentication._initialize_aml_token_auth (connections.py:341)
[2025-03-03 03:43:43] INFO     azureml.rag.connections - Getting workspace connection via MLClient with auth: <class 'azureml.dataprep.api._aml_auth._azureml_token_authentication.AzureMLTokenAuthentication'>, subscription_id: bf4f4af4-8cb7-420c-b682-722abbdff681, resource_group_name: enterpriseit-ai, workspace_name: confluence-fdt. (connections.py:348)
[2025-03-03 03:43:43] INFO     azureml.rag.connections - Using ml_client base_url: https://eastus.api.azureml.ms/rp/workspaces, original_base_url: https://management.azure.com. (connections.py:367)
[2025-03-03 03:43:43] INFO     azureml.rag.connections - Parsed Connection: /subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourceGroups/enterpriseit-ai/providers/Microsoft.MachineLearningServices/workspaces/confluence-fdt/connections/kshe-m7llzpu3-eastus2_aoai (connections.py:386)
[2025-03-03 03:43:43] INFO     azureml.rag.connections - Got connection: /subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourceGroups/enterpriseit-ai/providers/Microsoft.MachineLearningServices/workspaces/confluence-fdt/connections/kshe-m7llzpu3-eastus2_aoai as <class 'azure.ai.ml.entities._workspace.connections.connection_subtypes.AzureOpenAIConnection'>. (connections.py:443)
[2025-03-03 03:43:43] INFO     azureml.rag.connections - The connection 'kshe-m7llzpu3-eastus2_aoai' is a <class 'azure.ai.ml.entities._workspace.connections.connection_subtypes.AzureOpenAIConnection'> with api_key auth type. (connections.py:184)
[2025-03-03 03:43:43] INFO     azureml.rag.azureml.rag.embeddings - loading embeddings from file format version: 2 (__init__.py:590)
[2025-03-03 03:43:44] INFO     azureml.rag.azureml.rag.embeddings - found following sources partitions: [PosixPath('/mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount/embeddings/sources.parquet')] (__init__.py:651)
[2025-03-03 03:43:44] INFO     azureml.rag.azureml.rag.embeddings - processing partition: /mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount/embeddings/sources.parquet (__init__.py:653)
[2025-03-03 03:43:44] INFO     azureml.rag.azureml.rag.embeddings - found following deleted sources partitions: [] (__init__.py:673)
[2025-03-03 03:43:45] INFO     azureml.rag.azureml.rag.embeddings - found following deleted document partitions: [] (__init__.py:695)
[2025-03-03 03:43:45] INFO     azureml.rag.azureml.rag.embeddings - found following embedding partitions: [PosixPath('/mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount/embeddings/embeddings.parquet'), PosixPath('/mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount/embeddings/embeddings_0.parquet')] (__init__.py:721)
[2025-03-03 03:43:45] INFO     azureml.rag.azureml.rag.embeddings - processing partition: /mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount/embeddings/embeddings.parquet (__init__.py:723)
[2025-03-03 03:43:45] INFO     azureml.rag.azureml.rag.embeddings - processing partition: /mnt/azureml/cr/j/77afdcc0cb07485893f7c829bf10342c/exe/wd/embeddings_mount/embeddings/embeddings_0.parquet (__init__.py:723)
[2025-03-03 03:43:46] INFO     azureml.rag.update_acs.update_acs - ActivityStarted, update_acs (activity.py:108)
[2025-03-03 03:43:46] INFO     azureml.rag.update_acs - Updating ACS index (update_acs.py:294)
[2025-03-03 03:43:46] INFO     azureml.rag.connections - Getting workspace connection: enterpriseitaisearch, with input credential: <class 'NoneType'>. (connections.py:332)
[2025-03-03 03:43:46] INFO     azureml.rag.connections - Getting credential from AzureMLTokenAuthentication._initialize_aml_token_auth (connections.py:341)
[2025-03-03 03:43:46] INFO     azureml.rag.connections - Getting workspace connection via MLClient with auth: <class 'azureml.dataprep.api._aml_auth._azureml_token_authentication.AzureMLTokenAuthentication'>, subscription_id: bf4f4af4-8cb7-420c-b682-722abbdff681, resource_group_name: enterpriseit-ai, workspace_name: confluence-fdt. (connections.py:348)
[2025-03-03 03:43:46] INFO     azureml.rag.connections - Using ml_client base_url: https://eastus.api.azureml.ms/rp/workspaces, original_base_url: https://management.azure.com. (connections.py:367)
[2025-03-03 03:43:46] INFO     azureml.rag.connections - Parsed Connection: /subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourceGroups/enterpriseit-ai/providers/Microsoft.MachineLearningServices/workspaces/confluence-fdt/connections/enterpriseitaisearch (connections.py:386)
[2025-03-03 03:43:46] INFO     azureml.rag.connections - Got connection: /subscriptions/bf4f4af4-8cb7-420c-b682-722abbdff681/resourceGroups/enterpriseit-ai/providers/Microsoft.MachineLearningServices/workspaces/confluence-fdt/connections/enterpriseitaisearch as <class 'azure.ai.ml.entities._workspace.connections.connection_subtypes.AzureAISearchConnection'>. (connections.py:443)
[2025-03-03 03:43:46] INFO     azureml.rag.connections - The connection 'enterpriseitaisearch' is a <class 'azure.ai.ml.entities._workspace.connections.connection_subtypes.AzureAISearchConnection'> with api_key auth type. (connections.py:184)
[2025-03-03 03:43:46] INFO     azureml.rag.update_acs - Using Index fields: {
  "content": "content",
  "url": "url",
  "filename": "filepath",
  "title": "title",
  "metadata": "meta_json_string",
  "embedding": "contentVector"
} (update_acs.py:309)
[2025-03-03 03:43:46] INFO     azureml.rag.update_acs - Ensuring search index calm-cabbage-58b6jhctzf exists (update_acs.py:127)
[2025-03-03 03:43:47] ERROR    azureml.rag.update_acs.update_acs - ActivityCompleted: Activity=update_acs, HowEnded=Failure, Duration=1006.24 [ms], Exception=HttpResponseError (activity.py:127)
[2025-03-03 03:43:48] ERROR    azureml.rag.update_acs - Failed to update ACS index (update_acs.py:429)
[2025-03-03 03:43:48] ERROR    azureml.rag.update_acs.update_acs - ServiceError: intepreted error = Rag system error, original error = Operation returned an invalid status 'Forbidden' (exceptions.py:124)
[2025-03-03 03:43:53] ERROR    azureml.rag.update_acs.update_acs - update_acs failed with exception: Traceback (most recent call last):
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 465, in main_wrapper
    map_exceptions(main, activity_logger, args, logger, activity_logger)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/utils/exceptions.py", line 126, in map_exceptions
    raise e
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/utils/exceptions.py", line 118, in map_exceptions
    return func(*func_args, **kwargs)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 456, in main
    raise e
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 420, in main
    create_index_from_raw_embeddings(
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 311, in create_index_from_raw_embeddings
    create_search_index_sdk(acs_config, connection_credential, emb)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 131, in create_search_index_sdk
    if acs_config["index_name"] not in index_client.list_index_names():
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azure/core/paging.py", line 123, in __next__
    return next(self._page_iterator)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azure/core/paging.py", line 75, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azure/search/documents/indexes/_generated/operations/_indexes_operations.py", line 502, in get_next
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Forbidden'
 (update_acs.py:467)
[2025-03-03 03:43:53] ERROR    azureml.rag.update_acs.update_acs - ActivityCompleted: Activity=update_acs, HowEnded=Failure, Duration=14861.16 [ms], Exception=HttpResponseError (activity.py:127)
Traceback (most recent call last):
  File "/azureml-envs/rag-embeddings/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/azureml-envs/rag-embeddings/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 499, in <module>
    main_wrapper(args, logger)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 465, in main_wrapper
    map_exceptions(main, activity_logger, args, logger, activity_logger)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/utils/exceptions.py", line 126, in map_exceptions
    raise e
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/utils/exceptions.py", line 118, in map_exceptions
    return func(*func_args, **kwargs)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 456, in main
    raise e
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 420, in main
    create_index_from_raw_embeddings(
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 311, in create_index_from_raw_embeddings
    create_search_index_sdk(acs_config, connection_credential, emb)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azureml/rag/tasks/update_acs.py", line 131, in create_search_index_sdk
    if acs_config["index_name"] not in index_client.list_index_names():
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azure/core/paging.py", line 123, in __next__
    return next(self._page_iterator)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azure/core/paging.py", line 75, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/azureml-envs/rag-embeddings/lib/python3.9/site-packages/azure/search/documents/indexes/_generated/operations/_indexes_operations.py", line 502, in get_next
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Forbidden'