Azure Data Factory - jobs not even running?

Question

I have a persistent issue with ADF since Friday. Occasionally our pipelines will show "in progress" however the activities just wont move. Not even in a queued state, they just won't start at all.

Right now we have a pipeline that has been running for 10 minutes with no movement in the activities. I have tried both self hosted IR and azure auto resolve IR - both the same. I have submitted screenshots, pipeline run IDs and support tickets to Microsoft who have been zero help in the slightest.

My issue isn't how long the activities are taking to run, it's the movement between the activities even running that's the issue. Its as though ADF is struggling to even start them!

Has anyone had a similar issue? My ADF is in the west europe region

Answer

Hi @Jordan Dwyer

Thank you for your patience while we worked to resolve the issue with your pipelines. I'm glad to inform you that the issue was mitigated last night, and you should no longer see the error you were experiencing. If you are still experiencing the issue, please let me know and I will investigate further. Thank you for bringing this to our attention, and please don't hesitate to reach out if you have any further questions or concerns.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer

Hello @Jordan Dwyer

It sounds like your pipeline is stuck and the activities are not moving. There could be several reasons why this is happening. First, have you checked if your pipeline has a concurrency policy assigned to it? If it does, verify that there are no old pipeline runs in progress.

You can do this by going to the Monitoring view and checking if there's anything in the past 45 days that's in progress.

If there is something in progress, you can cancel it and the new pipeline run should start.

Second, it is possible that your run was impacted by a transient network issue, credential failures, or service outages. If this happens, Azure Data Factory has an internal recovery process that monitors all the runs and starts them when it notices something went wrong.

You can rerun pipelines and activities as described here: Visually monitor Azure Data Factory

You could rerun activities if you had canceled activity or had a failure as per Rerun from activity failures.

This process happens every one hour, so if your run is stuck for more than an hour, create a support case.

Lastly, it is possible that you are hitting capacity issues in your Self-Hosted Integration Runtime (SHIR). This can happen if you have not scaled up SHIR as per your workload. If you encounter a capacity issue from SHIR, upgrade the VM to increase the node to balance the activities.

If you receive an error message about a self-hosted IR general failure or error, a self-hosted IR upgrade, or self-hosted IR connectivity issues, which can generate a long queue, go to Troubleshoot self-hosted integration runtime.

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.

Share via

Azure Data Factory - jobs not even running?

2 answers