Amls Experiment Run Stuck In Status "running"

I made an Azure Machine Learning Service Experiment run and logged neural network losses with Jupyter Notebook. Logging worked fine and NN training completed as it should. However,

Solution 1:

this totally happens from time to time. it is certainly frustrating especially because the "Cancel" button it grayed out. You can use either the CLI or Python SDK to cancel the run.

SDK

>= 1.16.0

As of version 1.16.0 you no longer an Experiment object is no longer needed. Instead you can access using the Run or Workspace objects directly

from azureml.core import Workspace, Experiment, Run, VERSION
print("SDK version:", VERSION)

ws = Workspace.from_config()

run = ws.get_run('YOUR_RUN_ID')
run = Run().get(ws, 'YOUR_RUN_ID') # also works
run.cancel()

< 1.16.0

from azureml.core import Workspace, Experiment, Run, VERSION
print("SDK version:", VERSION)

ws = Workspace.from_config()
exp = Experiment(workspace = ws, name = 'YOUR_EXP_NAME')

run = Run(exp, run_id='YOUR STEP RUN ID')

run.cancel() # or run.fail()

CLI

More CLI details here

az login
az ml run cancel --run YOUR_RUN_ID