We have released the latest set of changes to our production servers with regard to this outage, and are confident that this issue has now been resolved.
Posted Mar 05, 2019 - 18:15 UTC
Update
Our monitoring is showing all systems to be performing normally.
Posted Mar 05, 2019 - 15:40 UTC
Monitoring
Our latest fix has resolved our scaling problems. Our logs and metrics are showing normal expected performance. We will monitor the situation closely for the rest of the day.
Posted Mar 05, 2019 - 13:14 UTC
Identified
Our application logic for scaling our services up was incorrect and stopped processing some inbound events from Atlassian. We are deploying a fix for this scaling-up logic and should start processing events correctly again in the next hour or so.
Posted Mar 05, 2019 - 12:53 UTC
Update
We are continuing to investigate this issue.
Posted Mar 05, 2019 - 11:50 UTC
Investigating
We are currently experiencing an issue where the scaling we rolled out to fix to mitigate the script execution delays has failed and is causing no scripts to be executed currently.
We are currently investigating this issue and will post a further update once we have more information.
Posted Mar 05, 2019 - 11:47 UTC
Monitoring
We have identified the cause of the issue and have introduced a fix to scale up the resources for the script runner Jira Cloud service which appears to have mitigated the issue.
We are currently monitoring and will post a futher update later on today.
Posted Mar 05, 2019 - 11:32 UTC
Update
We are currently scaling up our services to allow us to better handle the load from scripts in order to help mitigate the delays.
Posted Mar 05, 2019 - 10:15 UTC
Investigating
Since yesterday morning some customers have experienced delays when waiting for scripts to run.
We rolled out a prevention mechanism last night but it hasn’t worked as expected, and are contiuning to work to diagnose the root cause so that we can release a fix to mitigate this issue.
We will post a further update once we have identified the cause of the issue.
Posted Mar 05, 2019 - 09:10 UTC
This incident affected: ScriptRunner (ScriptRunner for Jira Cloud).