14:00:27 #startmeeting 14:00:27 Meeting started Tue Aug 14 14:00:27 2012 UTC. The chair is quaid. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:27 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:00:40 #meetingname Infra Team weekly sync 14:00:40 The meeting name has been set to 'infra_team_weekly_sync' 14:00:52 #topic roll call & agenda 14:00:58 * quaid is here :) 14:01:07 * mburns lurking, but distracted 14:01:56 * RobertM here 14:02:19 quiet today 14:02:42 ''Agenda'' 14:02:42 * Post-release check-in 14:02:42 * Puppet 14:02:42 * Hosting search 14:02:42 * Jenkins 14:02:44 * All other business 14:03:21 * eedri here 14:03:30 ok, let's go ahead with topics 14:03:37 #topic Post-release check-in 14:04:00 Anything noteworthy from last week's release for this team? 14:04:38 quaid, Other then the fact that the new structure was in place and worked fine. 14:05:03 web traffic to ovirt.org was less than for the 3.0 release, so no risk of overwhelming the services 14:05:45 tracking and reporting on those sort of trends (downloads, sit visits, etc.) is part of my $dayjob, btw 14:06:18 and I'll be doing all the tooling & how-to within oVirt directly, so we'll all be able to watch the same trends 14:06:52 anything we noticed to consider for next release? 14:07:20 quaid, What tracker are you using? 14:07:33 I realized that we were doing changes on infra right up to the release; Fedora Infra for example has change freeze for some weeks leading up to the release 14:07:44 we may want to consider something similar as the project grows 14:07:51 quaid, No I think we implemented the changes that were needed to make 3.2 go smoothly. 14:08:04 quaid, +1 on that 14:08:27 quaid, probably focus on monitor and fix and not push new features around releases 14:08:46 RobertM: combo is: google analytics, mlstats + small custom subscriber count script, and still need to find something good for Apache logs, that's about it 14:08:57 quaid, +1 but part of the reason is the group really didn't form until right before release. 14:09:06 yep 14:09:39 I don't think we were ever at risk, it's just a better practice, where we can do it 14:09:49 We addressed the issue with the repo after beta2. 14:10:06 right 14:10:17 We should be in much better shape come 3.2 14:10:44 RobertM: do you agree that we'd normally want to have a change freeze for some $time_period leading up to the release 14:11:32 quaid, Yes 14:11:34 change yes, but net-new services should be ok as long as they don't interact with the existing stuff 14:11:53 * ewoud should really set up a notification for this meeting 14:13:00 quaid, But we have to be flexable. Since releases themselves can require a change. AKA the repo structure had to change to make room for 3.1 14:13:22 #agreed We'll set some light-but-flexible policy about having a change-to-existing-infra freeze for a week (or similar time period) before a releases 14:13:35 It should have been caught and fixed before had but sometimes stuff like that will come out of no where. 14:13:48 right 14:14:09 we can also just use a change process, as a group we can approve doing changes in the freeze window 14:14:43 the idea is to just have a time where we are more careful than normal just-in-case, and the care is more about checking with others to be sure we aren't risking breaking something on release 14:15:57 logical 14:16:59 anything more? 14:17:12 No 14:17:37 Should add that to the release process in the wiki 14:19:17 #action add change notice procedure process to release process in the wiki 14:19:26 #topic Puppet 14:19:40 I think that's my cue 14:19:49 you've seen my proposal on the list? 14:20:32 In short: I wrote some puppet classes to manage a slave and it's available my github 14:20:35 #link https://github.com/ekohl/ovirt-infra-puppet 14:21:08 opinions? 14:21:43 so far on the ML I saw eedri and quaid respond with some suggestions, anyone else? 14:21:46 ewoud, Are there diff classes for Fedora and EL? 14:21:48 looks like we have some discussion going there, which is good 14:22:15 RobertM: not yet, but we can build those when needed 14:23:21 ewoud, not sure we want to have everything under one module 14:23:27 ewoud, ovirt-infra-puppet 14:23:49 ewoud, might be better to have small and atomic class for each purpose 14:23:53 eedri: the current repo is meant as a parent git repo which you use as an environment 14:24:22 ewoud, the repo is fine, i'm talking abot the moudle 'ovirt_infra' 14:24:29 * quaid puts some comments directly in to the thread 14:25:02 ewoud, or at least as subclasses 14:25:18 ewoud, for example instead of having package {['vim-enhanced', 'git']: in the main ovirt_infra 14:25:36 ewoud, put it in ovirt_infa::required_pkg.. 14:25:48 ewoud, but i think the best approach is to create a class per resource 14:26:00 ewoud, so you would create a class for 'git' seperatly 14:26:22 eedri: I agree, that would be better 14:26:26 ewoud, i know it's more work, but i found it to be a much more flexible way of building complex hostgroups in the future 14:26:47 eedri: I certainly want to split the users into their own classes so you can give a user permissions to a server etc 14:28:16 but to continue working on this I think we need to set some goals 14:28:51 I think first should be to set up some infra to collaborate on it, so that should be a puppet master and a gerrit repo 14:29:28 and the puppet master can be either pure puppet or foreman, where the latter could certainly provide some cool features 14:30:24 would the current kitchen sink have sufficient memory to run foreman? 14:30:54 don't know 14:31:09 can Foreman run on e.g. OpenShift? 14:31:31 ewoud, i agree that we should do a basic design or requirment doc before writing puppet classes 14:31:42 ewoud, maybe start a wiki page on it.. 14:31:53 eedri: I will do that 14:31:57 ewoud, great 14:32:16 +1 14:32:25 ewoud, once we'll get an agreement from the team on the basic classes we can start writing them 14:32:27 quaid: I think you're the only chair so can you fix a #action? 14:32:49 ewoud, maybe you should open a new project in gerrit for puppet classes? 14:32:55 ewoud, instead of git hum 14:33:12 ewoud: anyone can do action, fwiw 14:33:23 #chair ewoud eedri mburns RobertM 14:33:23 Current chairs: RobertM eedri ewoud mburns quaid 14:33:29 #action ewoud write a wiki page on puppet 14:33:58 (I think only #agreed and something else I forget are limited to chair, plus some actions such as giving out chair, opening or closing the meeting, etc.) 14:34:14 eedri: a gerrit project is git and I certainly do want to continue there 14:35:07 I think github is fine for the working group when we go to implentant we will want to move to gerrit. 14:35:13 I posed some more questions/comments to the thread, maybe we can continue from there? 14:35:28 RobertM: yes 14:35:30 RobertM: yeah, do we have an Infra git repo yet? 14:35:35 RobertM, yes, i agree that github is usefull for the devel phase.. 14:35:41 +1 for continue on thread 14:35:44 would this be a stand-alone repo, or part of a big Infra repo? 14:35:51 RobertM, once we have all the classes we want we can push it to gerrit infra project 14:35:57 * quaid actually asked this in the thread, so will take his answers there 14:36:10 ok, good stuff! 14:36:22 going to be very exciting to have Puppet (and Foreman) running) 14:36:38 anything more on this topic here? 14:36:40 quaid, as long as we'll have the power (vms) to run them properly :) 14:37:14 eedri, That is a very good point. 14:37:57 yes 14:38:10 well, that's an upcoming topic, so let's move there 14:38:31 #topic Hosting search 14:39:18 anyone looked for hosting? 14:39:22 last week I got some basic budget approval, I think we can work with US$150/mon for hosting 14:39:27 but I haven't done any searching beyond that 14:39:53 also, not sure what we want - 5 VMs with that? One bare metal server with massive specs so we can run on it and put up our own VMs on it? 14:40:18 anyone interested in leading the specs and research here? 14:40:35 we could be running within a few days, but that research needs to be done to get there 14:40:51 * quaid can help but hasn't been good at leading the research so far 14:40:54 I would like to be part of the group since I have been having so much fun with Jenkins I have a feel for what it requires. 14:41:22 can we just do this from a new mailing list thread? 14:41:43 ok, RobertM will you propose something on the ML? 14:41:55 ewoud, ok 14:42:51 #action RobertM will kick off the hosting proposal thread, where we can do resarch 14:42:53 RobertM: would be interested in $ vs euro and also US-based hosting or EU-based, can you include those? 14:43:02 good point 14:43:20 I assume are budget is in US $ 14:43:24 I'll be paying either with credit card or (preferably) a purchase order 14:43:29 RobertM: yes 14:43:30 are=our 14:44:06 So the week dollar will make euro hasting more expensive. 14:44:12 #info Looking at October++ for Red Hat IT to have possible VMs for us to use permanently, which would replace this paid-for hosting 14:44:17 *weak 14:44:32 RobertM: maybe we can find a US-based company with EU data centers :) 14:45:23 so next agenda item? 14:45:46 #topic Jenkins 14:46:14 Were do we start with Jenkins :) 14:46:27 * ewoud is just following the agenda 14:47:10 eedri, Have you had a change to look into the nightly and package generation? 14:47:29 RobertM, sorry, didn't have time this week 14:47:46 RobertM, but i think we might need to ask on ovirt weekly 14:48:13 RobertM, about if a code change is needed in the various projects for supporting rpm names 14:48:23 RobertM, like it is supported in ovirt-node and vdsm 14:48:58 eedri, Do you want to take care of bring that up on the Wednesday meeting? 14:49:05 RobertM, ok 14:49:55 but do I understand it correct that we currently have 2 jenkins masters? 14:50:14 ewoud, Yes. 14:50:36 #action eedri to bring up makefile changes in MAKEFILE for ovirt projects to support per-commit rpm names 14:51:03 ovirt.org had the master branch dealing with building nightly. 14:51:19 ovirt.info is running the per patch stuff. 14:52:21 jenkins.ovirt.org and jenkins.ovirt.info 14:54:27 wow, is that really all we have for Jenkins today? 14:54:33 * quaid *shocked* 14:54:39 Per Patch jobs are running for both VDSM and Engine on jenkins.ovirt.info 14:54:59 #info Per Patch jobs are running for both VDSM and Engine on jenkins.ovirt.info 14:55:17 are we fine with having two masters for the moment? 14:55:37 RobertM, we need to verify all jobs are ineed runs on the relevant refspec 14:55:38 quaid, We really don't have much of a choice. 14:55:47 RobertM, some users told me it wasn't the case for some patches.. 14:55:49 eedri, Yes we do 14:56:15 quaid: I think there are some slaves under each master 14:56:32 and I'd prefer to move to a single master or at least a single infra 14:56:33 RobertM, i think its due to wrong 'git strategy' for some jobs 14:56:50 ewoud, the secondary master was build primarly as a backup server 14:56:54 ewoud, and testing server 14:56:58 eedri, I went though them all last night. But I would love a 2nd set of eyes 14:57:28 ewoud, the only reason that per patch jobs are running on it is that the master can't handle it (the ec2) 14:57:29 eedri: I know, but I mean a single primary for production which makes it easier to replicate a staging env 14:57:47 ewoud, true, would be possible once we'll have a proper master 14:58:21 ewoud, i don't think that the master will be able to handle all the patch jobs. even if they run on a slave 14:59:14 eedri: even on a decent server? 14:59:26 ewoud, no, on a decent server it will manage 14:59:35 first thing with new hosting will be to move Jenkins master(s) over, yes? 14:59:46 ewoud, i have a jenkins master with more than 50 slaves 14:59:47 ewoud, jenkins.ovirt.info was doing the same jobes as jenkins.ovirt.org in half the time but it is way backed up. With the right hardware we can merge the 2 into 1 master. 15:00:17 ok, so jenkins improvements are pending new hosting now 15:00:18 ewoud, a proper master can defiently handle the load i foresee for ovirt project for the coming future 15:00:34 #agreed First use of new hosting is to be combined Jenkins master 15:00:53 +1 15:01:03 ewoud, it would just need slaves to running jobs 15:01:19 anything else about jenkins or can we move on? 15:01:25 if we want to setup puppet.ovirt.org on the Linode box ... we could possibly use it to build the new Jenkins master, yes? 15:01:35 ewoud: +1 15:01:37 i think the whitelist has been set up quite well also 15:01:49 quaid: yes, could certainly help 15:01:50 need to pay attention for users that are missing from it 15:02:31 ewoud, there is a puppet moudle for jenkins http://puppetlabs.com/blog/module-of-the-week-rtylerjenkins-continuous-integration-server/ 15:02:39 eedri: I know, and I've used it 15:02:44 ewoud, :) 15:02:56 eedri: but then you informed me that a slave doesn't need to run it, so I removed it ;) 15:03:01 anything more? 15:03:06 Building the new master based on puppet will be a good way to test the classes and find the large number of missing packages that are needed that aren't in there already 15:03:10 ewoud, yea.. only the master ;) 15:03:20 I'd like to discus some sort of task tracker 15:03:27 +1 on traker 15:03:37 doesn't have to be fixed in the short term, but things are getting harder to track 15:03:49 RobertM, we'll probably have 2 profiles - 1 for master and 1 for slave 15:03:54 RobertM, for starts 15:04:27 Depends on if we are running jobes on the master or not. 15:04:54 quaid: do you have any experience or know someone who has experience with a task/issue tracker for us? 15:05:34 ewoud, quaid Do we have a preference on what task tracker to use. Those things tend to be personal prefer things 15:05:46 ewoud: I was going to recommend we use Teambox 15:06:01 #topic Task trackers 15:06:28 Teambox is hosted and open source 15:06:35 (the latest version isn't yet opened, but will be) 15:06:53 folks I know who use it really like it 15:06:56 quaid: how open can we make it? 15:07:31 i.e. read only access to anonymous users? 15:07:34 otherwise, we pick another hosting or *shudder* host it ourselves 15:08:03 ewoud: I think so, but we want to be sure it doesn't require an account/login to view, worth double-checking 15:08:35 quaid: can you work out a proposal for the mailing list? 15:09:39 #action quaid to research Teambox, potential propose it as a new thread on task tracking 15:10:58 Anything else? 15:11:05 nothing more on the agenda 15:11:30 quaid: RobertM eedri? 15:11:44 closing in 30 seconds 15:12:32 nothing? 15:12:35 going once 15:12:38 going twice 15:12:41 #endmeeting