Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Remove the "100 missed start times" CronJob limit #89397

Closed
wants to merge 1 commit into from
Closed

WIP: Remove the "100 missed start times" CronJob limit #89397

wants to merge 1 commit into from

Conversation

vllry
Copy link
Contributor

@vllry vllry commented Mar 24, 2020

(This code was written as a patch, and needs some cleanup to directly solve the problem in k/k).

Removes the 100 missed schedule times limit from CronJobs.

Introduces a binary search approach with the schedule.Next() function in the existing 3rd party cron module. The binary search provides a much sharper upper bound on the timespan checked, and number of Next() calls. EG, a minutely CronJob that has been failing for a year requires checking only ~20 Next() calls (log2(60*24*365) + 1) in order to get the most recent missed schedule.

Co-Authored-By: Tom Wanielista tomwans@users.noreply.github.com
Co-Authored-By: Kevin Yang 16491459+yangkev@users.noreply.github.com

What type of PR is this?
/kind bug

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Removes a well-known limitation, where CronJobs could only miss 100 consecutive start times before needing to be re-created.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
Needs doc updates, TBD.

@yangkev
Check next cron schedules in a binary-search fashion
d4dcc26 
Co-Authored-By: Tom Wanielista <tomwans@users.noreply.github.com>

Co-Authored-By: Kevin Yang <16491459+yangkev@users.noreply.github.com>
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Mar 24, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vllry
To complete the pull request process, please assign erictune
You can assign the PR to them by writing /assign @erictune in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 24, 2020
@vllry
Copy link
Contributor Author

vllry commented Mar 24, 2020

/uncc @soltysh
/uncc @smarterclayton

/test pull-kubernetes-conformance-kind-ga-only-parallel

@vllry
Copy link
Contributor Author

vllry commented Mar 24, 2020

/test pull-kubernetes-e2e-kind

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Mar 24, 2020 •
edited

@vllry: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-verify d4dcc26 link /test pull-kubernetes-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 22, 2020
@k8s-ci-robot
Copy link
Contributor

@vllry: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 22, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 22, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@alexey-gavrilov-flant
Copy link

/reopen

@k8s-ci-robot
Copy link
Contributor

@alexey-gavrilov-flant: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@zentale zentale mentioned this pull request Oct 13, 2020
Closed
@ltagliamonte-dd
Copy link
Contributor

@vllry is there any reason why this PR didn't get attention?
Few people are "suffering" from:

Cannot determine if job needs to be started: too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew

@soltysh
Copy link
Contributor

soltysh commented Nov 26, 2020

There's a new cronjob controller being written see #93370

@2rs2ts
Copy link
Contributor

2rs2ts commented Dec 2, 2020

@soltysh The new cronjob controller has the same problem, according to this comment #81557 (comment)

I think this PR might be redundant with that one, #81557, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/apps Categorizes an issue or PR as relevant to SIG Apps. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@vllry @k8s-ci-robot @fejta-bot @alexey-gavrilov-flant @ltagliamonte-dd @soltysh @2rs2ts