Stopping cluster overnight prevents scheduled jobs from running after cluster startup.

[lpreson]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
CronJob
ScheduledJob
"Too many missed start times to list"

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Feature Request

Kubernetes version (use kubectl version):

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): CentOS Linux release 7.3.1611 (Core)
• Kernel (e.g. uname -a):Linux 4.9.11-1.el7.elrepo.x86_64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Sat Feb 18 18:16:50 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Custom
• Others:

What happened:
Cluster shutdown nightly to reduce costs. On startup the cluster refuses to run jobs.
The controller manager errors with:
E0307 15:40:23.754617 1 controller.go:163] Cannot determine if default/ needs to be started: Too many missed start times to list

What you expected to happen:
The cluster will run scheduled jobs after being restored.

How to reproduce it (as minimally and precisely as possible):
Set a regular schedule such as */1 * * * *
Shutdown for 100 scheduled executions of the job (100 minutes).

Anything else we need to know:
This seems to be related to:
Function: getRecentUnmetScheduleTimes
v1.4: https://github.com/kubernetes/kubernetes/blob/release-1.4/pkg/controller/scheduledjob/utils.go#L169

[0xmichalis]

@kubernetes/sig-apps-bugs

[soltysh]

There was similar problem recently fixed #36311, although in this particular use case it's hard to distinguish whether it was down or something else broke, from a controller point of view. I could only suggest tweaking .spec.startingDeadlineSeconds for the time.

[lpreson]

@soltysh if I set .spec.startingDeadlineSeconds to a high number will it trigger once on resume or trigger all missed jobs?

[soltysh]

@lpreson up to .spec.startingDeadlineSeconds but then again you could try setting .spec.restartPolicy to Forbid, this way only one will be run at the time. I must admit your use case is far from normal, due to shutting down the cluster, but reasonable for the controller to work stably.

[lpreson]

@soltysh This could be a common approach for money saving in the cloud when running non production clusters.

Thank you for the interim suggestion, I will try this out.

[0xmichalis]

Also reported in #45825 - @soltysh @erictune is it something we are targetting to fix in 1.7?

[soltysh]

With the current time-frame I'm not sure. I'll be sweeping through my bugs over the next few weeks. I'll see what I can go with.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[skynardo]

@lpreson - I was wondering if setting .spec.startingDeadlineSecond and or setting .spec.restartPolicy to Forbid resolved this issue. We have a similar issue with 2 of our non-prod clusters that we shut down each night.

[mattfarina]

/remove-lifecycle stale

@skynardo Can you share the version of Kubernetes you are still experiencing this on?

@kow3ns did you know about this one?

[ApsOps]

I'm still experiencing this on v1.8.6.

[mludvig]

I've got the same problem with Minikube 0.25.0 running Kubernetes server 1.9.0 in a VirtualBox on my laptop. When I close the lid and laptop suspends the minikube VM obviously stops and upon resume my scheduled job keeps failing with:

Cannot determine if job needs to be started: Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.

[soltysh]

This is due to the hard-coded value of how many missed start times the cronjob controller can handle. It's reasonable to have it configurable, I guess.

[soltysh]

Actually I was thinking about it more this morning. Currently if we exceed the artificial, hardcoded limit the cronjob will always error out without any chance to recover the cronjob. Maybe we should error out as we do today and pause the cronjob to prevent it from further erroring out, but at the same time giving the user the ability to restart it. I'd like to hear what others think about such approach.

[mludvig]

Some restart button was exactly what I was looking for in the dashboard! Couldn't find a way to do it and had to delete and recreate the job :(
Definitely +1 from me ;)

[skynardo]

I would prefer it be configurable. I could then set it > the number of scheduled starts it would have had overnight while our EC2 instances are shut down. I need something automated. I was thinking about adding a check to my monitoring pod to see if the scheduled job was dead then recreate it.

[boosty]

I would prefer it be configurable

+1

[gtaylor]

This just bit us today, too. We had to work around the issue by deleting and re-adding part of our Helm chart on the cluster :(

Would be great to be able to disable/re-enable a CronJob.

[dzoeteman]

Not only being able to restart the CronJob would be nice, but also exclude actually suspending a CronJob - in this case, the user has specified he doesn't want the CronJob to be run, so it should be okay to start again on schedule as soon as the user has resumed the CronJob.

If the above is not a major change - I'd like to pick it up if someone else hasn't yet.

[soltysh]

Not only being able to restart the CronJob would be nice, but also exclude actually suspending a CronJob - in this case, the user has specified he doesn't want the CronJob to be run, so it should be okay to start again on schedule as soon as the user has resumed the CronJob.

@dzoeteman suspension is already supported, see .spec.suspend which is false by default.

[gtaylor]

If I flip a "stuck" CronJob to spec.suspend = true then back to false, do I get un-stuck? Or is the answer for sure "Delete and re-create the CronJob?"

[Bo0km4n]

Same issue happen on Kubernetes 1.18.6

[zentale]

Another fix for this same thing? #89397

[soltysh]

This is being fixed in the new controller #93370

[Skybladev2]

Still experience it with minikube v1.16.0 (Kubernetes v1.20.0)

[alculquicondor]

@Skybladev2 did you enable the CronJobControllerV2 feature gate?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[unixfox]

/remove-lifecycle stale

[soltysh]

This should now be resolved with the new controller implementation.
/close

[k8s-ci-robot]

@soltysh: Closing this issue.

In response to this:

This should now be resolved with the new controller implementation.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hughesadam87]

FWIW - another use case besides just shutting the cluster down is pausing the job. In GCP it's easy to pause a cron job, and we often do this. If the job is having an issue, I'll pause the prod container for hours while its being debugged and then run into this issue when trying to resume. We are using old GKE/K8s version so am hoping it goes away when upgraded.

The "Run Now" still works for one-off runs, but then the cron does not resume after. I'm going to try the restartPolicy=forbid as mentioned above