summaryrefslogtreecommitdiff
path: root/scheduler
AgeCommit message (Collapse)AuthorFilesLines
2011-02-21Trying to shutdown the scheduler in a nicer waylmr2-1/+6
Signed-off-by: Jean-Marc Eurin <jmeurin@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@5256 592f7852-d20e-0410-864c-8624ca9c26a4
2011-01-29scheduler: Introduce site extensibility to HostSchedulerlmr1-1/+10
Signed-off-by: Dale Curtis <dalecurtis@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@5214 592f7852-d20e-0410-864c-8624ca9c26a4
2010-11-08Testing that we can access the drone before adding itlmr2-5/+16
Signed-off-by: Jean-Marc Eurin <jmeurin@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4909 592f7852-d20e-0410-864c-8624ca9c26a4
2010-11-08Massive permission and indentation fix on the autotest treelmr1-8/+8
Once again I went through the autotest tree with my automated script to check all autotest source, and discovered several places that needed permission and indentation (including trailing whitespace) fixes. I have checked the generated patch and indeed it looks OK. Will commit this shortly, as preparation for autotest 0.13. Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4907 592f7852-d20e-0410-864c-8624ca9c26a4
2010-10-28Adding some logging in the drones.lmr1-0/+5
Adding some logging in the drones. Signed-off-by: Jean-Marc Eurin <jmeurin@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4900 592f7852-d20e-0410-864c-8624ca9c26a4
2010-10-28Bugfix for my previous change r4897lmr2-6/+9
Bugfix for my previous change. The unittest unfortunately hid the problem because our self.god.stub_with() stubs non-existant things out with a new value without checking to see if the thing being stubbed out existed in the first place. Signed-off-by: Gregory Smith <gps@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4898 592f7852-d20e-0410-864c-8624ca9c26a4
2010-10-28Allow limiting of the number of hosts that go through the scheduler'slmr3-4/+45
Allow limiting of the number of hosts that go through the scheduler's periodic Repair Failed -> Verifying cleanup cycle at once. When you've got an infrastructure issue that kicks all your hosts into repair failed, launching reverifies on all of them at once can overwhelm the server. Signed-off-by: Gregory Smith <gps@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4897 592f7852-d20e-0410-864c-8624ca9c26a4
2010-10-19Add some site-specific parsing utilitieslmr2-4/+16
Add some site-specific parsing utilities. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4871 592f7852-d20e-0410-864c-8624ca9c26a4
2010-10-18Scheduler notification e-mail: Handle job begin and/or end types not setlmr1-5/+9
In some occasions, the job end time might not be set (job stuck that had to be aborted, for example). If that happens, the operation t_end - t_begin used to get the job execution time will fail with an IndexError, since t_end is None. Handle that case properly. Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4859 592f7852-d20e-0410-864c-8624ca9c26a4
2010-10-18Scheduler notification e-mail: Order failures by reasonlmr1-0/+1
Ordering by reason makes it easy to group failures and easily identifying them. Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4858 592f7852-d20e-0410-864c-8624ca9c26a4
2010-09-23Scheduler e-mail notification: Code cleanupslmr1-25/+30
Make changes on the scheduler notification code to allow for shorter code. Also, introduce a new common_lib.utils function that generates a pretty print representation of any matrix, that will adjust the length of all columns automagically, making the resulting e-mail report much prettier to see, even with very large test names. Example: header = ("Word1", "Word2", "Word3") rows = [["Heeeeeeeeeeeeeey", "Hey", "Eh"], ["Orange", "Nonononono", "Pineapple"], ["Egg", "Spam", "Bacon"]] result = matrix_to_string(rows, header) Gives: Word1 Word2 Word3 Heeeeeeeeeeeeeey Hey Eh Orange Nonononono Pineapple Egg Spam Bacon Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4806 592f7852-d20e-0410-864c-8624ca9c26a4
2010-09-16[PATCH] Read all labels from host_keyvals file and handle the commaericli1-1/+2
(,) within labels properly. Chromium OS autotest project had assigned labels with comma to test hosts and this situation makes the current host_keyvals file impossible to parse out correct label values. This change will fix/enhance that. git-svn-id: svn://test.kernel.org/autotest/trunk@4793 592f7852-d20e-0410-864c-8624ca9c26a4
2010-09-12Fix scheduler email notifications when the job is not run at alllmr1-5/+9
Fix scheduler email notifications when the job is not run at all Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4779 592f7852-d20e-0410-864c-8624ca9c26a4
2010-08-27Scheduler: Make the e-mail notification code clearerlmr1-14/+24
Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4768 592f7852-d20e-0410-864c-8624ca9c26a4
2010-08-26Bugfix for r4766lmr1-19/+28
Changes from v2: * Make sure results from SERVER_JOB and CLIENT_JOB.* will not disturb the user test stats in case of a job failure Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4767 592f7852-d20e-0410-864c-8624ca9c26a4
2010-08-26Sometimes, we just want to have important data about the jobjamesren1-14/+103
readily available, without having to open the web interface. This patch adds important info to the regular status e-mails: * Number of tests executed * Success rate * Details about test failures So people can decide whether they actually want to inspect the results more carefully. Changes from v1: * Refactored the original code, that repeated a lot of logic to generate the subject and the body of the status e-mail. By refactoring it becomes a lot easier to make changes on the e-mail body if needed. * Fixed a bug, where job failures were being treated incorrectly. * Made floats to be displayed only with 2 decimals. Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4766 592f7852-d20e-0410-864c-8624ca9c26a4
2010-08-10Adds a diagnostic message.jamesren1-0/+3
Signed-off-by: Gregory Smith <gps@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4747 592f7852-d20e-0410-864c-8624ca9c26a4
2010-07-16Begin adding support for parameterized jobs.jamesren1-1/+2
This will allow test developers to specify certain parameters that a test control file may take, so that users can then easily set those parameters on job create. Enabling this feature removes the ability to edit the control file directly on job creation. Feature is currently INCOMPLETE. Do not attempt to use. This feature will be committed in small pieces for the sake of having smaller code reviews. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4720 592f7852-d20e-0410-864c-8624ca9c26a4
2010-06-16Fix a logging messagejamesren1-3/+5
Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4623 592f7852-d20e-0410-864c-8624ca9c26a4
2010-06-10Set host status to RUNNING on QueueTask abort, since queue entry will be injamesren2-4/+13
GATHERING state. Also modify a logging string to be more precise. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4591 592f7852-d20e-0410-864c-8624ca9c26a4
2010-06-08Don't try stopping the job on HQE abort, and have the dispatcher stop alljamesren3-3/+22
necessary jobs in bulk. This avoids a scheduler crash on an assertion. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4585 592f7852-d20e-0410-864c-8624ca9c26a4
2010-05-12Add a run of build_externals to the initialization of a drone when the ↵jamesren1-0/+3
scheduler is first started. Signed-off-by: Scott Zawalski <scottz@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4488 592f7852-d20e-0410-864c-8624ca9c26a4
2010-05-10Scheduler VerifyTask should only delete queued manual reverify tasks,jamesren1-2/+2
not all queued verify tasks. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4487 592f7852-d20e-0410-864c-8624ca9c26a4
2010-04-28Fix an error in drone sets in monitor_db. Also added more unit tests.jamesren2-9/+102
Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4449 592f7852-d20e-0410-864c-8624ca9c26a4
2010-04-27Fix process counting for SelfThrottledPostJobTask. Would previouslyjamesren1-1/+2
lose a slot for a process permanently if the paired results were lost before the process started. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4447 592f7852-d20e-0410-864c-8624ca9c26a4
2010-04-21Add ability to associate drone sets with jobs. This restricts a job tojamesren6-26/+231
running on a specified set of drones. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4439 592f7852-d20e-0410-864c-8624ca9c26a4
2010-04-16monitor_db.py: Fix SyntaxWarninglmr1-3/+2
I've noticed that monitor_db.py issues a SyntaxWarning as soon as it is started: 19:56:55 INFO | Killing monitor_db 19:56:55 INFO | STARTING monitor_db with log file /usr/local/autotest/logs/scheduler.log.2010-04-15-19.56.55 /usr/local/autotest/scheduler/monitor_db.py:1779: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert (self.TASK_TYPE is not None, I noticed that the whole statement fits under 80 chars, so the parenthesis can be removed safely, getting rid of the warning. Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4421 592f7852-d20e-0410-864c-8624ca9c26a4
2010-04-12Reset host status to READY on aborting a WAITING entryjamesren1-1/+2
Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4399 592f7852-d20e-0410-864c-8624ca9c26a4
2010-03-25Fixes to drone_manager behavior.jamesren3-8/+45
Fixed bug where a drone that a user is not allowed to access may get a job scheduled on it anyways if all drones are over capacity. Modified capacity computation to handle drones with the same ratio of running-to-max processes, but different total max processes. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4345 592f7852-d20e-0410-864c-8624ca9c26a4
2010-03-13Set hostless queue entries to STARTING upon scheduling the agent. Thisjamesren2-0/+24
fixes an issue where the scheduler created multiple HostlessQueueTask objects for a single hostless queue entry, causing several autoserv processes to be launched when the agents are run. Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4304 592f7852-d20e-0410-864c-8624ca9c26a4
2010-03-11Add support to autoserv for a --control-filename parameter, to allow users tombligh1-1/+1
control where in the results directory autoserv will store the server control file. This also changes the archving stage in the scheduler to make use of this argument, so that the control file from the archving stage is written to control.archive and does not overwrite the control.srv from the job itself. Signed-off-by: John Admanski <jadmanski@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4294 592f7852-d20e-0410-864c-8624ca9c26a4
2010-03-02Abstract out common models used in the frontend's models.py so that django ↵jamesren3-12/+14
is not required to interact with non Django portions of the code. This includes the enums RebootBefore, RebootAfter and Test.Type git-svn-id: svn://test.kernel.org/autotest/trunk@4280 592f7852-d20e-0410-864c-8624ca9c26a4
2010-03-02Adding "executable" property to scheduler unittestsjamesren3-0/+0
Signed-off-by: James Ren <jamesren@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4279 592f7852-d20e-0410-864c-8624ca9c26a4
2010-03-02Part of http://patchwork.test.kernel.org/patch/1850/ didn't make it into thejamesren2-2/+1201
repository. Trying again. git-svn-id: svn://test.kernel.org/autotest/trunk@4278 592f7852-d20e-0410-864c-8624ca9c26a4
2010-02-26Minor fix to new metahost handlers code in scheduler to ensure handlers get ↵jamesren1-0/+3
a tick every cycle, even if there are no queued metahost jobs. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4274 592f7852-d20e-0410-864c-8624ca9c26a4
2010-02-19Change email of hosts without platforms to a warning in the logs. We don't ↵jamesren1-3/+2
do anything about this and it is just extra spam for us at this point in time. This will only affect admins who use the SCHEDULER.notify_email variable Signed-off-by: Scott Zawalski <scottz@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4258 592f7852-d20e-0410-864c-8624ca9c26a4
2010-02-19Add a site_monitor_db_babysitter module. If found, it will use itsjamesren1-2/+16
SiteMonitorProc as a base class for MonitorProc so that site code can override the monitor_db arguments and/or setup environment variables needed for the scheduler to run properly at the given site. Signed-off-by: Gregory Smith <gps@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4253 592f7852-d20e-0410-864c-8624ca9c26a4
2010-02-19Refactor scheduler models into a separate module, scheduler_models. This ↵jamesren6-1657/+593
module doesn't depend on monitor_db, only the other way around. The separation and isolation of dependencies should help us organize the scheduler code a bit better. This was made possible largely by the many changes we made late last year to improve statelessness of the scheduler. It was motivated here by my work on pluggable metahost handlers, which will need to depend on scheduler models. Without this separation, we'd end up with circular dependencies. Also includes some fixes for metahost schedulers. Signed-off-by: Steve Howard <showard@google.com> Property changes on: scheduler/scheduler_models.py git-svn-id: svn://test.kernel.org/autotest/trunk@4252 592f7852-d20e-0410-864c-8624ca9c26a4
2010-02-12First iteration of pluggable metahost handlers. This change adds the basic ↵jamesren5-102/+309
framework and moves the default, label-based metahost assignment code into a handler. It includes some refactorings to the basic scheduling code to make things a bit cleaner. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4232 592f7852-d20e-0410-864c-8624ca9c26a4
2010-02-03When archiving results, we need to append a slash to the path to ensure it ↵showard1-0/+3
gets correctly handled as a directory. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4196 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-25Ensure we reset pidfile age when the pidfile is read. I had dropped the ↵showard1-1/+7
call to register_pidfile() from get_pidfile_info() in my previous change, but now I realize the purpose of it was to reset the pidfile age. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4164 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-25Fix scheduler functional test for recent change to parse hostless jobs.showard1-0/+6
Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4163 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-25Don't implicitly register pidfiles when get_pidfile_contents() is called. ↵showard1-1/+0
The scheduler is now registering and unregistering pidfiles correctly on its own, and this was causing files to get accidentally re-registered after being unregistered, causing pidfile leaks. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4162 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-25Basic support for "summary results" -- articifial test results that are ↵showard1-9/+1
explicitly recorded by a server-side control file or code that it calls. This CL just adds the record_summary() method to the server_job object. It lacks any special parser support or TKO DB changes, those will come later. This also includes a couple of minor changes to support conitnuous parsing and final reparsing for hostless jobs. Since hostless jobs are a common intended use case for summary results, they'll need full parsing support to be useful. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4161 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-20don't set the current user to my_user in frontend_test_utils. let it ↵showard1-5/+4
default to the new autotest_system user. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4156 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-20New code for performing explicit joins with custom join conditions.showard1-2/+2
* added ExtendedManager.join_custom_field(), which uses the introspection magic from populate_relationships (now factored out) to infer the type of relationship between two models and construct the correct join. join_custom_field() presents a much simpler, more Django-y interface for doing this sort of thing -- compare with add_join() above it. * changed TKO custom fields code to use join_custom_field() * added some cases to AFE rpc_interface_unittest to ensure populate_relationships() usage didn't break * simplified _CustomQuery and got rid of _CustomSqlQ. _CustomQuery can do the work itself and its cleaner this way. * added add_where(), an alternative to extra(where=...) that fits more into Django's normal representation of WHERE clauses, and therefore supports & and | operators later Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4155 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-15In scheduler check for existence of results before trying to write the ↵showard1-1/+1
.archiver_failed file. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4130 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-15Support for job keyvalsshoward2-4/+21
* can be passed as an argument to create_job, stored in AFE DB * scheduler reads them from the AFE DB and writes them to the job-level keyval file before the job starts * parser reads them from the keyval file and writes them to the TKO DB in a new table Since the field name "key" happens to be a MySQL keyword, I went ahead and made db.py support proper quoting of field names. Evetually it'd be really nice to deprecate db.py and use Django models exclusively, but that is a far-off dream. Still lacking support in the AFE and TKO web clients and CLIs, at least the TKO part will be coming soon Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4123 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-15* fix a bug with restricted drone users config parsingshoward2-7/+14
* display allowed users in the status server * ensure drone users reparse correctly * fix a bug in max_runnable_processes introduced by user restrictions (could pass empty set to max()) Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4121 592f7852-d20e-0410-864c-8624ca9c26a4
2010-01-15In periodic reverificaiton, use schedule_special_task() instead of straight ↵showard1-1/+1
object creation. This is the right path to use for creating tasks -- it include duplication avoidance and automatic owner tagging. Signed-off-by: Steve Howard <showard@google.com> git-svn-id: svn://test.kernel.org/autotest/trunk@4120 592f7852-d20e-0410-864c-8624ca9c26a4
generated by cgit v1.2.3 (git 2.39.1) at 2024-11-22 04:40:47 +0000