Name: tkt-classify-ept-11262048-7b659cb57-f9jr4 Namespace: azureml-ticketbot Priority: 0 Service Account: default Node: Labels: azuremlappname=tkt-classify-ept-11262048 isazuremlapp=true mmsserviceid=99bb3135-2fd4-48a5-a3bf-25cfbf8d417e pod-template-hash=7b659cb57 workspace.id=a7026364-1ab9-4fc1-a1f6-3d100190b9e6 Annotations: Status: Pending IP: IPs: Controlled By: ReplicaSet/tkt-classify-ept-11262048-7b659cb57 Init Containers: amlappinit: Image: mcr.microsoft.com/azureml/dependency-unpacker:20220830 Port: Host Port: Limits: nvidia.com/gpu: 1 Requests: cpu: 8 memory: 28G nvidia.com/gpu: 1 Environment: AML_APP_INSIGHTS_KEY: Optional: false AML_APP_INSIGHTS_ENABLED: Optional: false AML_APP_INSIGHTS_ENDPOINT: Optional: false AML_MODEL_DC_STORAGE_ENABLED: Optional: false AML_MODEL_DC_EVENT_HUB_ENABLED: Optional: false AML_MODEL_DC_STORAGE: Optional: false AML_MODEL_DC_EVENT_HUB: Optional: false AML_MODEL_DEBUG_ENABLED: Optional: false AML_MODEL_DEBUG_KEY: Optional: false AML_INIT_CONFIG_LOCATION: Optional: false AZUREML_ENTRY_SCRIPT: Optional: false AML_APP_ROOT: Optional: false AZUREML_MODEL_DIR: Optional: false POD_NAME: tkt-classify-ept-11262048-7b659cb57-f9jr4 (v1:metadata.name) WORKSPACE_NAME: TicketBot SERVICE_NAME: tkt-classify-ept-11262048 SERVICE_ID: 99bb3135-2fd4-48a5-a3bf-25cfbf8d417e COMPUTE_NAME: scoring-cluster COMPUTE_LOCATION: westeurope COMPUTE_TYPE: AKS SERVICE_VERSION: 1 SERVICE_PATH_PREFIX: api/v1/service/tkt-classify-ept-11262048 AZURE_TENANT_ID: ... AZURE_CLIENT_ID: ... AZURE_CLIENT_SECRET: ... SUBSCRIPTION_ID: f2b7d77f-ccc3-476f-bcc0-c969052eef48 RESOURCE_GP: InternalTicketBot STORAGE_ACCOUNT: ticketbot2478182304 STORAGE_KEY: FLkrrJU2vXgAVAx8XlLYGlDwhgV7C8vlsBVlpZxq8UPBSKEwdYSPdMXq6jEOw++DYCORXUx0m1Ua+AStRixV1w== ENV_NAME: tkt-scoring-env DOWNLOAD_TO: ./downloaded_models MODEL_NAME: bert-base-uncased_ZZ_CODEGRUPPE__ID2 AZUREML_SOURCE_DIRECTORY: src Mounts: /staging from staging (rw) /structure from structure (rw) /var/azureml-app from emptymount (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cst6q (ro) Containers: tkt-classify-ept-11262048: Image: a70263641ab94fc1a1f63d100190b9e6.azurecr.io/azureml/azureml_130ec9546d350153a38fdabeb1108901 Ports: 5001/TCP, 5001/TCP Host Ports: 0/TCP, 0/TCP Command: runsvdir /var/runit Limits: nvidia.com/gpu: 1 Requests: cpu: 8 memory: 28G nvidia.com/gpu: 1 Liveness: http-get http://:health-port/ delay=310s timeout=120s period=5s #success=1 #failure=3 Readiness: http-get http://:health-port/ delay=0s timeout=120s period=5s #success=1 #failure=3 Environment: AML_APP_INSIGHTS_KEY: Optional: false AML_APP_INSIGHTS_ENABLED: Optional: false AML_APP_INSIGHTS_ENDPOINT: Optional: false AML_MODEL_DC_STORAGE_ENABLED: Optional: false AML_MODEL_DC_EVENT_HUB_ENABLED: Optional: false AML_MODEL_DC_STORAGE: Optional: false AML_MODEL_DC_EVENT_HUB: Optional: false AML_MODEL_DEBUG_ENABLED: Optional: false AML_MODEL_DEBUG_KEY: Optional: false AML_INIT_CONFIG_LOCATION: Optional: false AZUREML_ENTRY_SCRIPT: Optional: false AML_APP_ROOT: Optional: false AZUREML_MODEL_DIR: Optional: false POD_NAME: tkt-classify-ept-11262048-7b659cb57-f9jr4 (v1:metadata.name) WORKSPACE_NAME: TicketBot SERVICE_NAME: tkt-classify-ept-11262048 SERVICE_ID: 99bb3135-2fd4-48a5-a3bf-25cfbf8d417e COMPUTE_NAME: scoring-cluster COMPUTE_LOCATION: westeurope COMPUTE_TYPE: AKS SERVICE_VERSION: 1 SERVICE_PATH_PREFIX: api/v1/service/tkt-classify-ept-11262048 AZURE_TENANT_ID: ... AZURE_CLIENT_ID: ... AZURE_CLIENT_SECRET: ... SUBSCRIPTION_ID: f2b7d77f-ccc3-476f-bcc0-c969052eef48 RESOURCE_GP: InternalTicketBot STORAGE_ACCOUNT: ticketbot2478182304 STORAGE_KEY: FLkrrJU2vXgAVAx8XlLYGlDwhgV7C8vlsBVlpZxq8UPBSKEwdYSPdMXq6jEOw++DYCORXUx0m1Ua+AStRixV1w== ENV_NAME: tkt-scoring-env DOWNLOAD_TO: ./downloaded_models MODEL_NAME: bert-base-uncased_ZZ_CODEGRUPPE__ID2 AZUREML_SOURCE_DIRECTORY: src Mounts: /structure from structure (rw) /usr/local/nvidia from nvidia (rw) /var/azureml-app from emptymount (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cst6q (ro) Conditions: Type Status PodScheduled False Volumes: staging: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: tkt-classify-ept-1126204870689d11c81d4438b92f2073a07f708c-pvc ReadOnly: true structure: Type: HostPath (bare host directory volume) Path: /mnt/amlMount/89e9641cb4e545a6bfbd2a0d125cf4e5 HostPathType: DirectoryOrCreate emptymount: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: nvidia: Type: HostPath (bare host directory volume) Path: /usr/local/nvidia HostPathType: kube-api-access-cst6q: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s nvidia.com/gpu:NoSchedule op=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 12h default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Warning FailedScheduling 12h (x3 over 12h) default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Warning FailedScheduling 12h (x2 over 12h) default-scheduler 0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod. Normal NotTriggerScaleUp 12h cluster-autoscaler pod didn't trigger scale-up: 1 pod has unbound immediate PersistentVolumeClaims Normal NotTriggerScaleUp 12h (x89 over 12h) cluster-autoscaler pod didn't trigger scale-up: 1 Insufficient cpu Normal NotTriggerScaleUp 12h cluster-autoscaler pod didn't trigger scale-up: Normal NotTriggerScaleUp 29m cluster-autoscaler pod didn't trigger scale-up: Warning FailedScheduling 26m (x23 over 29m) default-scheduler no nodes available to schedule pods Warning FailedScheduling 25m default-scheduler 0/1 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. Normal NotTriggerScaleUp 2m47s (x151 over 27m) cluster-autoscaler pod didn't trigger scale-up: 1 Insufficient cpu Warning FailedScheduling 20s (x6 over 20m) default-scheduler 0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.