Kuruko2 schedule job commands sometime runs twice

[algiecaballesbbo]

When running testing executing a simple command execute: node -v and scheduled it to run every minute, there are instance that the job runs twice. please refer to the attached images for more details

Info:
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
Rails 5.2.1
rvm 1.29.4 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
node v8.4.0
RAILS_ENV=production
NUM_OF_WORKERS=1

[eagletmt]

I never see such situation...
Could you paste log/job-scheduler.log and log/production.log?
Which RDB are you using for Kuroko2?

Kuroko2 currently does not support Rails 5.2 series, so you cannot use Kuroko2 with Rails 5.2.1.
https://github.com/cookpad/kuroko2/blob/v0.5.0/kuroko2.gemspec#L20
You can check installed gems by running bundle show command.

[algiecaballesbbo]

For you question Kuroko2 currently does not support Rails 5.2 series, so you cannot use Kuroko2 with Rails 5.2.1.? , its currently working in our production using this version on rails. the only problem we have is it has duplicate process each commands when running scheduled jobs where we only use only 1 worker.
Second question Which RDB are you using for Kuroko2?, we are using MariaDB v5.5.56
Our installed gems in the production:

Gems included by the bundle:
  * actioncable (5.2.1)
  * actionmailer (5.2.1)
  * actionpack (5.2.1)
  * actionview (5.2.1)
  * activejob (5.2.1)
  * activemodel (5.2.1)
  * activerecord (5.2.1)
  * activestorage (5.2.1)
  * activesupport (5.2.1)
  * addressable (2.5.2)
  * archive-zip (0.11.0)
  * arel (9.0.0)
  * aws-sdk (2.11.96)
  * aws-sdk-core (2.11.96)
  * aws-sdk-resources (2.11.96)
  * aws-sigv4 (1.0.3)
  * bindex (0.5.0)
  * bootsnap (1.3.1)
  * builder (3.2.3)
  * bundler (1.16.3)
  * byebug (10.0.2)
  * capybara (3.4.2)
  * childprocess (0.9.0)
  * chromedriver-helper (1.2.0)
  * chrono (0.4.0)
  * coffee-rails (4.2.2)
  * coffee-script (2.3.0)
  * coffee-script-source (1.12.2)
  * commonmarker (0.16.8)
  * concurrent-ruby (1.0.5)
  * crass (1.0.4)
  * dalli (2.7.8)
  * dotenv (0.11.1)
  * dotenv-deployment (0.0.2)
  * dotenv-rails (0.11.1)
  * erubi (1.7.1)
  * execjs (2.7.0)
  * faraday (0.12.2)
  * ffi (1.9.25)
  * font-awesome-rails (4.7.0.4)
  * foreman (0.85.0)
  * globalid (0.4.1)
  * haml (5.0.4)
  * hashie (3.5.7)
  * hipchat (1.3.0)
  * html-pipeline (2.8.4)
  * http_accept_language (2.1.1)
  * httparty (0.16.2)
  * i18n (1.1.0)
  * io-like (0.3.0)
  * jbuilder (2.7.0)
  * jmespath (1.4.0)
  * jquery-rails (4.3.3)
  * json (2.1.0)
  * jwt (1.5.6)
  * kaminari (1.1.1)
  * kaminari-actionview (1.1.1)
  * kaminari-activerecord (1.1.1)
  * kaminari-core (1.1.1)
  * kuroko2 (0.4.6)
  * listen (3.1.5)
  * loofah (2.2.2)
  * mail (2.7.0)
  * marcel (0.3.2)
  * method_source (0.9.0)
  * mimemagic (0.3.2)
  * mini_mime (1.0.1)
  * mini_portile2 (2.3.0)
  * minitest (5.11.3)
  * momentjs-rails (2.20.1)
  * msgpack (1.2.4)
  * multi_json (1.13.1)
  * multi_xml (0.6.0)
  * multipart-post (2.0.0)
  * mysql2 (0.5.2)
  * nio4r (2.3.1)
  * nokogiri (1.8.4)
  * oauth2 (1.4.0)
  * oj (3.6.5)
  * omniauth (1.8.1)
  * omniauth-google-oauth2 (0.2.10)
  * omniauth-oauth2 (1.3.1)
  * passenger (5.3.3)
  * public_suffix (3.0.2)
  * rack (2.0.5)
  * rack-accept-default (0.0.2)
  * rack-store (0.0.4)
  * rack-test (1.1.0)
  * rails (5.2.1)
  * rails-dom-testing (2.0.3)
  * rails-html-sanitizer (1.0.4)
  * rails_bootstrap_sortable (2.0.3)
  * railties (5.2.1)
  * rake (12.3.1)
  * rb-fsevent (0.10.3)
  * rb-inotify (0.9.10)
  * redcarpet (3.4.0)
  * redis (4.0.1)
  * responders (2.4.0)
  * retryable (3.0.1)
  * rinku (2.0.4)
  * ruby-enum (0.7.2)
  * ruby_dep (1.5.0)
  * rubyzip (1.2.1)
  * sass (3.4.25)
  * sass-rails (5.0.7)
  * select2-rails (4.0.3)
  * selenium-webdriver (3.13.1)
  * serverengine (1.5.11)
  * sigdump (0.2.4)
  * slim (3.0.9)
  * slim-rails (3.1.3)
  * spring (2.0.2)
  * spring-watcher-listen (2.0.1)
  * sprockets (3.7.2)
  * sprockets-rails (3.2.1)
  * temple (0.8.0)
  * the_garage (2.4.3)
  * thor (0.19.4)
  * thread_safe (0.3.6)
  * tilt (2.0.8)
  * tzinfo (1.2.5)
  * uglifier (2.7.2)
  * visjs-rails (4.21.0.0)
  * weak_parameters (0.5.0)
  * web-console (3.6.2)
  * websocket-driver (0.7.0)
  * websocket-extensions (0.1.3)
  * xpath (3.1.0)

[riseshia]

Could you paste log/job-scheduler.log and log/production.log?

Please check this one, too.

[joefrey25]

Hi @riseshia san and @eagletmt san

This is lately the incident we have encountered in our production server.
log.txt

[joefrey25]

hi @eagletmt san @riseshia san
This is just a follow up question. How do you properly restart kuroko2 services daemon in the server?
Do you have any proper commands that we can restart these services?

TIA

[riseshia]

I read pasted logs, It seems that more than one job-scheduler is running

D, [2018-08-31T04:00:26.906791 #3825] DEBUG -- :    (0.2ms)  BEGIN
D, [2018-08-31T04:00:26.907739 #3825] DEBUG -- :   Kuroko2::Tick Load (0.4ms)  SELECT  `kuroko2_ticks`.* FROM `kuroko2_ticks` ORDER BY `kuroko2_ticks`.`id` ASC LIMIT 1
D, [2018-08-31T04:00:26.908511 #3825] DEBUG -- :   Kuroko2::Tick Update (0.2ms)  UPDATE `kuroko2_ticks` SET `at` = '2018-08-30 19:00:26' WHERE `kuroko2_ticks`.`id` = 1
D, [2018-08-31T04:00:26.909129 #3825] DEBUG -- :   Kuroko2::JobSchedule Load (0.2ms)  SELECT  `kuroko2_job_schedules`.* FROM `kuroko2_job_schedules` ORDER BY `kuroko2_job_schedules`.`id` ASC LIMIT 1000
D, [2018-08-31T04:00:26.909960 #3825] DEBUG -- :   Kuroko2::JobDefinition Load (0.2ms)  SELECT `kuroko2_job_definitions`.* FROM `kuroko2_job_definitions` WHERE `kuroko2_job_definitions`.`id` IN (2, 5)
D, [2018-08-31T04:00:26.910801 #3825] DEBUG -- :   Kuroko2::JobSuspendSchedule Load (0.2ms)  SELECT `kuroko2_job_suspend_schedules`.* FROM `kuroko2_job_suspend_schedules` WHERE `kuroko2_job_suspend_schedules`.`job_definition_id` IN (2, 5)
D, [2018-08-31T04:00:26.922660 #3825] DEBUG -- :    (0.9ms)  COMMIT
D, [2018-08-31T04:00:26.934955 #3855] DEBUG -- :    (0.1ms)  BEGIN
D, [2018-08-31T04:00:26.935703 #3855] DEBUG -- :   Kuroko2::Tick Load (0.3ms)  SELECT  `kuroko2_ticks`.* FROM `kuroko2_ticks` ORDER BY `kuroko2_ticks`.`id` ASC LIMIT 1
D, [2018-08-31T04:00:26.936373 #3855] DEBUG -- :   Kuroko2::Tick Update (0.2ms)  UPDATE `kuroko2_ticks` SET `at` = '2018-08-30 19:00:26' WHERE `kuroko2_ticks`.`id` = 1
D, [2018-08-31T04:00:26.936966 #3855] DEBUG -- :   Kuroko2::JobSchedule Load (0.2ms)  SELECT  `kuroko2_job_schedules`.* FROM `kuroko2_job_schedules` ORDER BY `kuroko2_job_schedules`.`id` ASC LIMIT 1000
D, [2018-08-31T04:00:26.937774 #3855] DEBUG -- :   Kuroko2::JobDefinition Load (0.2ms)  SELECT `kuroko2_job_definitions`.* FROM `kuroko2_job_definitions` WHERE `kuroko2_job_definitions`.`id` IN (2, 5)
D, [2018-08-31T04:00:26.938588 #3855] DEBUG -- :   Kuroko2::JobSuspendSchedule Load (0.2ms)  SELECT `kuroko2_job_suspend_schedules`.* FROM `kuroko2_job_suspend_schedules` WHERE `kuroko2_job_suspend_schedules`.`job_definition_id` IN (2, 5)
D, [2018-08-31T04:00:26.947863 #3855] DEBUG -- :    (0.7ms)  COMMIT

Base on your comment (#122 (comment)), It may fail to restart job scheduler, but just launch one more job scheduler.

we using systemd to manage these processes, except executor which need to be restarted when it's not working.
Please check example service unit files: https://github.com/cookpad/kuroko2/blob/master/docs/admin_guide.md#daemons

[joefrey25]

@riseshia san

Thanks for the response,

Base on your comment (#122 (comment)), It may fail to restart job scheduler, but just launch one more job scheduler.

Yes we just launch 1 job in scheduler and it will only run everyday at 3am

we using systemd to manage these processes, except executor which need to be restarted when it's not working.
Please check example service unit files: https://github.com/cookpad/kuroko2/blob/master/docs/admin_guide.md#daemons

Yes we used this systemd file in our server and we also do have a cronjob to restart the services

BTW. may we know the versions of rails, ruby and web server(what you used) you guys running for this?
Thanks!

[joefrey25]

Hi @riseshia san

I just want to confirm this if this is another issue in kuroko2. I'm currently downgraded my rails version.
current version of rails 5.1.6
current version of ruby 2.5.1

As I can see there's a new table kuroko2_script_revisions and there's no documentation on how to do this script revisions.

Sorry for too much question.

TIA