14:00:56 #startmeeting 14:00:56 Meeting started Tue Jul 10 14:00:56 2012 UTC. The chair is quaid. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:56 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:01:07 #meetingname oVirt Infra Team weekly 14:01:07 The meeting name has been set to 'ovirt_infra_team_weekly' 14:01:22 #topic roll call aka "quick howdies" 14:01:24 howdy! 14:01:29 * mburns in 14:01:30 * RobertM here 14:01:31 !ydwoh 14:01:35 * eedri here 14:01:40 * sgordon_ here 14:02:04 * ewoud here 14:02:30 I realized I didn't write down an agenda, I was just thinking of running from last week, but could have written it down 14:02:54 * ewoud opens the minutes from last week 14:02:54 * eedri got some stuff to talk about 14:03:09 http://ovirt.org/meetings/ovirt/2012/ovirt.2012-07-03-14.00.html 14:03:23 quaid: ty 14:03:40 ok, I'm going to take from the middle where we left off, eedri et al can add 14:03:43 #topic agenda 14:03:54 * sub-project idea & status 14:03:58 * team mission 14:04:05 eedri: go ahead please 14:04:23 ok, we finally got this week a new physical server as jenkins slave 14:04:42 eedri: if you mean the one I made available, it's not physical but a VM in our RHEV3 cluster 14:04:48 Hi all 14:04:59 sgordon_, Hi there 14:05:02 ok, ok sorry it was that fast, i was sure it was physical :) 14:05:08 heh 14:05:11 change that last mark 14:05:22 no, it's a dell R710 now 14:05:22 anyway, we should decide how to best use it 14:05:35 think the SAN is on SATA even 14:05:35 glad you brought that up, I was trying to work on the "migrate main Jenkins from EC2 plan" and would like to discuss that, too 14:05:51 1st priority imo is trying to run at least one gerrit task on it, since they can't run on ec2 vms 14:06:07 ok 14:06:11 any other agenda items? 14:06:33 eedri: we'll move through the first items quickly, then spend the rest of the time talking Jenkins - it's our most important current topic, I think 14:06:34 i think we can push forward the ovirt-engine rpms sync to ovirt.org 14:06:55 Are we going to be having an open forum at the end for new business? 14:07:07 quaid, agreed 14:07:23 RobertM: yes, but we can put things on the agenda now to increase chance of talking about if you have something in mind right now? 14:07:32 * quaid about to save the agenda page 14:07:48 quaid, That is fine 14:07:51 quaid: puppet as sub from sub-project? 14:08:10 ewoud: can you explain what you mean? 14:08:25 quaid: as agenda item 14:08:48 ok, I'll put it up, you can explain it further when we get there 14:08:50 14[[07Infrastructure team task list14]]4 !10 02http://www.ovirt.org/w/index.php?diff=3836&oldid=3746&rcid=3930 5* 03Rmiddle 5* (+24) 10/* One time tasks */  14:09:11 14[[07Infrastructure team meetings14]]4 !10 02http://ovirt.org/w/index.php?diff=3837&oldid=3734&rcid=3931 5* 03Quaid 5* (+530) 10agenda for today 14:09:37 if anyone has more, we can add it to that page, and work fast to get there today :) 14:09:48 #topic Sub-project idea & status 14:09:52 trust/seed? 14:10:01 as a topic to the agenda? 14:10:08 eedri: sure 14:10:33 yesterday I sent email to board@ 14:10:35 http://lists.ovirt.org/pipermail/board/2012-July/000570.html 14:10:57 opening the topic of what to do to i) recognize Infra formally, and ii) resolve this for all non-coding focused sub-projects 14:12:21 so that will play out on board@ but I don't think we'll get any objections to the idea, just clarity 14:12:32 if anyone wants to participate, that is an open list 14:12:39 http://lists.ovirt.org/mailman/listinfo/board 14:12:49 other thoughts on this topic? 14:12:50 I think we first need to decide on a mission 14:12:51 The charter takes into account none programming maintainer so the sub projects should as well. 14:13:07 RobertM: good point 14:13:26 14[[07Infrastructure team meetings14]]4 !10 02http://ovirt.org/w/index.php?diff=3838&oldid=3837&rcid=3932 5* 03Quaid 5* (+13) 10/* 2012-07-10 */ adding trust seed agenda item 14:13:46 then let's move on to mission :) 14:14:14 #info The charter takes into account none programming maintainer so the sub projects should as well. 14:14:56 +1 14:15:19 ok, let's move on 14:15:23 #topic Team mission 14:15:45 This was my first pass, what is missing? 14:15:48 Strawdog: "To provide a free/open source infrastructure of participation that supports the development of the oVirt project, and to do so by providing full free/open access to the entire infrastructure as a collaboration." 14:16:31 provide full/free access sounds like anyone can manage the infra, no? 14:16:36 I think the goal should be to optimally support the development process, so we agree on that 14:17:05 This is what is in the wiki and I like it lang better. 14:17:06 eedri: I think the open access to the infra means everyone can access the wiki, gerrit, jenkins 14:17:08 This is a community services infrastructure team. That means the project infrastructure is maintained to a professional level by a group of system administrators who are contributing their time. (That time may be contributed as part of their job role, it might be part of a class or workshop, it might be purely voluntary, and so forth.) 14:17:19 ewoud, ok 14:18:07 * eedri thinks on the way we should ask community to ask infra for help 14:18:24 should it be via email/jira/redmine? 14:18:30 sorry if this off topic 14:18:52 eedri: it's a good topic, but yes I think a bit off-topic for now 14:19:04 i think eventually, yes, but sufficient to request on infra@ for now 14:19:22 eedri: how about, anyone should be able to manage the infra, but not everyone is permitted to :) 14:19:26 we'll want something better long term as we grow 14:19:38 quaid, sounds good :) 14:19:45 RobertM: can we turn that statement in to a mission? is it mission-y enough? 14:20:05 this is a great topic for the mailing list, but it's good to quickly see where we stand now 14:21:00 Mission statements are ment to be vague since missions change over time. I this we could rework what is in the wiki pretty easily. 14:21:14 should we include in the mission a statement that implies the way someone can join infra team? 14:21:14 this=think 14:21:16 cool, good point 14:21:26 eedri does touch another topic: how far do we push new technology vs on request 14:21:42 eedri: or at least make it clear there is a reasonable barrier and link to details on how to join? 14:22:14 ewoud: agreed; I get concerned about stretching ourselves too far 14:22:24 quaid, it should be leveled, on one hand we want to state we're proffesional and won't let anyone on, on the other we do want more to get it 14:22:41 s/it/in 14:22:46 Personal opinion is that isn't in the scope of a mission statement. Mission statements are basicly your reason for being. 14:22:47 in the end most of us will have a pet project anyway, that's a fairly natural thing 14:23:22 however, it is important to als keep redundancy in people who can maintain systems 14:24:14 The best option is to automate as much as you can to limit the need to people to need access to the machine. 14:25:03 RobertM, agreed, some initial effort needs to be done in order to achive that goal (i.e puppet, better jenkins infra,etc..) 14:25:15 yep, yep 14:25:58 let's start by reworking from the existing wiki content 14:26:00 RobertM, and still even if all is automated, you'll still get random errors in those systems 14:26:02 14[[07Documentation14]]4 !10 02http://www.ovirt.org/w/index.php?diff=3839&oldid=3828&rcid=3933 5* 03Sgordon 5* (+255) 10/* Source Control */  14:26:15 I think we should organize as follows: a container project named infra with internal projects like puppet, jenkins 14:26:50 eedri: it's a fair point about professional as a barrier, we'll try to make it a welcoming mission 14:27:09 for example, acting professional doesn't mean one is in the profession - we want intern-level people, too, etc. 14:27:10 quaid, +1 14:27:19 eedri, Yes. You do but you might be surprised at how few those are once you get things locked down and more signal task oriented. 14:27:32 RobertM: can't wait! 14:27:41 * quaid excited by the idea of our new Puppet Overlord 14:28:04 RobertM, we'll need to decide on update policies (i.e. when to update jenkins/gerrit/plugins versions..) 14:28:15 Some still has to write those pupper recipes :) 14:28:16 RobertM, which will greatly affect stableness of the infra 14:28:20 #info need to decide on update policies 14:28:34 #action need to find one or more Puppet chiefs to lead us from the darkness 14:28:55 at $employer I'm also working on a project with puppet and foreman 14:28:58 ewoud: ah, I see! yes, that's what I was thinking 14:28:59 so I have some experience 14:29:10 ewoud: about containers, I mean 14:29:14 eedri, I agree but that brings use to the ugly topic of budgets. 14:29:14 * eedri just gave a lecture about integrating jenkins + foreman + puppet 14:29:31 ewoud: we can then split up permissions a bit by Infra project team, etc. 14:29:35 14[[07Documentation14]]4 !10 02http://www.ovirt.org/w/index.php?diff=3840&oldid=3839&rcid=3934 5* 03Sgordon 5* (+61) 10/* Brand */  14:30:09 quaid: yes 14:30:12 14[[07Documentation14]]4 !10 02http://www.ovirt.org/w/index.php?diff=3841&oldid=3840&rcid=3935 5* 03Sgordon 5* (-3) 10/* Installing the oVirt Brand */  14:30:14 ok, does anyone feel interested enough to take the "What is our mission? What our our goals? What our our methods?" to the infra list? 14:30:27 i.e., volunteer to lead that thread? 14:31:21 * RobertM volunteers 14:31:59 #action RobertM to open & run thread about mission/goals/methods 14:32:02 thanks RobertM 14:32:24 ok, ready to talk Jenkins? 14:32:38 you bet 14:32:45 #topic New Jenkins server 14:33:02 eedri: can you start us, please? 14:33:16 sure 14:33:38 few issues on the table 14:34:07 1. jenkins infra (i.e. current ec2 slaves and what our plan to move from it ) 14:34:42 current vms are very slow and can barely handle load of jobs 14:34:42 I can add that my employer is sponsoring a VM in our RHEV3 cluster with 8 GB RAM, 50GB SATA storage and 4 cores and eedri has added it to jenkins.ovirt.org 14:35:15 ewoud, that was the 1st good improvment to the infra, if we have more of those we'll be in a much better state 14:35:28 now that we have a strong vm, we need to decide how to best use it 14:35:51 any ideas? 14:36:18 * eedri thinks trying to run basic tests (unit test/findbugs) per gerrit patch 14:36:29 ewoud, How much control do we have over the VM? 14:36:39 eedri: ok, I've got the "migrate Jenkins" as the next topic, just so we don't miss anything with each topic 14:37:14 RobertM: atm only I have full root, eedri has SSH access to the jenkins user 14:37:22 Hello, I am getting this error.. anybody have any idea whats wrong? Exit message: unable to open disk path /rhev/data-center/f666ff44-9e5a-442c-b2d2-747cdbcd745d/c62c9255-83f4-4c1 Permission Denied 14:37:31 we'll need sudo access for some of the jobs 14:37:33 I have no objections on sharing root, but I wanted to wait till we have thie discussion 14:37:51 ewoud, What OS is installed in the VM? 14:37:58 RobertM: atm fedora 17 14:38:24 How hard is a rebuild? 14:38:32 mostly because right now that's the main ovirt target platform 14:38:39 foגשאק 14:39:10 joncox: right now we're in a meeting here, so might not be the best time for support 14:39:24 ok thx 14:39:33 RobertM: not that hard, I tried to install it with a local puppet profile 14:39:41 joncox: we'll be done in 20 minutes 14:39:50 * eedri tested running 5 parallel gerrit jobs for unit tests on new vm, took ~ 30 min each 14:39:51 it will ensure the proper user exists, packages installed, SSH keys installed 14:39:59 so not optimal 14:40:41 * eedri thinks that one VM won't be able to provide full support to gerrit patch, maybe best to wait for addional servers? 14:41:03 and utilize the current one to speed us current jobs 14:41:12 eedri: if I look at http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/index.html it seems to be mostly IO bound, which was somewhat to be expected with SATA storage 14:41:17 eedri: we can enable gerrit patch for some projects with smaller patch rate? 14:41:32 mburns, good idea, instead of ovirt-engine 14:41:56 mburns, we can start with ovirt-node, how long does an ovirt-node job takes usually? 14:41:57 eedri: or break up the engine repo into multiple sub-components... 14:42:39 eedri: on my internal systems, ~21 min 14:42:54 mburns, you mean run unit test per sub-module in maven? 14:42:55 on bare metal, ~10 min 14:43:10 mburns, not full unit test on main pom? 14:43:26 ~25 min for jenkins.ovirt 14:43:54 eedri: yes, something like that, then run full find-bugs, etc on merge 14:44:42 #action eedri to try to split up current jobs on jenkins.ovirt.org to try to speed us run time 14:44:54 eedri: or you could go one step further 14:45:02 and break it all out into multiple git repos 14:45:09 i.e. have something for the gui 14:45:14 something separate for core 14:45:24 something else for api... 14:45:25 mburns, that won't be so trivial.. 14:45:25 etc... 14:45:32 i realize that 14:45:36 mburns, i don't think it will be accepped 14:45:45 but might be worthwhile effort 14:46:01 mburns, let's raise this in ovirt weekly? 14:46:14 eedri: probably more a topic for engine weekly call 14:46:28 mburns, also, there might be depenencies between some of them 14:46:37 eedri: ack 14:46:57 i know it's not trivial, but might be worth it to bring it up with lpeer and doron_ 14:47:08 but that's a tangent that we don't need to discuss now 14:47:14 mburns, ok, let's bring it up in the ovirt weekly 14:47:46 to summarize this topic 14:47:50 * quaid trying to tell if we've settled the question of what to do with new Jenkins server 14:48:04 ewoud, would you be willing to allow sudo access to jenkins user on that server? 14:48:18 ewoud, and disable require tty in sudoers file 14:48:48 quaid: step 1 is to enable per patch builds on ovirt-node, then watch load 14:48:57 step 2, add per patch builds of vdsm 14:49:07 assuming it can handle that 14:49:18 ewoud, this is the steps we usually do for ec2 jenkins slaves: http://www.ovirt.org/wiki/Jenkins 14:49:51 ewoud: i can generate a list of what i need sudo access for if you would prefer it to be restricted... 14:50:07 vdsm should also provide a list of what it needs 14:50:13 danken, can you provide this? 14:50:30 eedri: sudo yum access should be sufficient for most builds 14:50:40 node is different because it builds an iso which needs root access 14:51:17 how does this sound: 14:51:23 ewoud, also there is a list of rpms needed to be installed in order to run vdsm tests 14:51:38 #agreed new Jenkins server" enable per patch builds on ovirt-node, watch load, then add per patch builds of vdsm 14:52:08 eedri: we should enforce a section in the build where it yum installs it's dependencies... 14:52:21 hiho 14:52:29 I cant add a host to my cluster 14:52:34 wb gestahlt 14:52:45 mburns, we can create a pre-step to parse the spec file and install all required rpms 14:52:47 The status always switches from installing to failed install 14:52:50 gestahlt: just finishing a meeting, so people might be distracted ... 14:53:07 eedri: exactly 14:53:10 quaid: i ask anyway ;) Maybe i will attract an answer 14:53:13 i did something like this for node 14:53:16 mburns, assuming we have sudo access to run yum 14:53:21 eedri: mburns ewoud How are we on that above #agreed? 14:53:27 gestahlt: no problem, just letting you know :) 14:53:30 sorry, back 14:53:31 eedri: right, need sudo yum access 14:53:40 collaegue needed me 14:53:41 quaid: ack for me... 14:53:47 quaid, ack 14:55:01 ok, I want to get to the other jenkins topic 14:55:12 #topic Migrating Jenkins from EC2 14:55:13 to respond to the backlog 14:55:14 eedri: I'm willing to allow sudo 14:55:53 Itamar pointed me at an article a while ago that showed how Jenkins is 10x slower for 10x the cost, which convinced me :) if our experience already wasn't convincing enough 14:55:53 quaid, What options do we have to Migrate to? 14:56:20 we discussed getting a dedicated host, maybe a real bare metal server with enough that we can Do Whatever 14:56:46 I think eventually we'll see more servers come available from Red Hat IT, but we have a few months to cover at least 14:56:50 the master server doesn't need to be a monster server 14:57:01 at long as we'll have powerful slaves to run the jobs on 14:57:10 master should run any jobs as best practise 14:57:18 RobertM: so ... something like $75/mon+ to get a hosted server 14:57:26 it should have enough disk size to hold history and configuration 14:57:31 Guys 14:57:36 eedri: we could keep Master on EC2? just create a big slave server? 14:57:37 If you need VMs with power 14:57:40 i gladly provide 14:57:52 well, what I was going to ask was ... 14:57:59 quaid, i prefer not ec2, just a better VM or a stadard physical server 14:58:15 can a few of us convene on infra@ and discuss some options? can anyone help put together a migration plan? 14:58:18 eedri: what do you mean by 'master should run any jobs'? 14:58:29 s/should/shouldn't 14:58:31 gestahlt: are you on infra@ovirt.org mailing list? because that is a great offer :) 14:58:33 oh yes :) 14:58:41 no, im not on the mailing list 14:59:03 eedri, What are the requirements for Jenkins (number of servers/VMs, specs for them, etc) 14:59:04 http://lists.ovirt.org/mailman/listinfo/infra 14:59:18 quaid: I only need help setting it up. For the help and continuing support, i am willing to offer you free VM hosting 14:59:36 gestahlt: running a jenkins slave is suprisingly easy 14:59:42 definitely want to get to dneary's question, but perhaps we should do that on list? 14:59:58 just create a user, add the SSH keys, install the right packages 15:00:28 never used jenkins and dont know what it is, if you need VMs for it, i can provide.. IF i get oVirt running 15:00:30 quaid, +1 on that, need to check it more thourally. for now we'll do with 4-5 stronger servers/vms that existing ec2 15:00:36 quaid, Fair enough 15:00:42 * dneary has another meeting now 15:00:55 Guys, how much CPU / RAM and Storage do you need? 15:00:57 * quaid too ... 15:01:23 I have another meeting I need to get to as well 15:01:24 gestahlt: the more the better, but a lot seems to be IO/RAM bound 15:01:43 storage on each slave can be 100gb 15:01:54 only master server needs higher storage to keep history builds 15:01:56 btw, next week I'll be on vacation 15:02:15 memory i would go on 16GB at least per server 15:02:24 gestahlt: that would be ovirt team eating it's own dog food 15:02:31 i have 192 cores and 3TB Ram avail 15:02:34 if possible, currently we're running on 8GB 15:02:50 I need like 10-20% total 15:02:55 the rest i can spare 15:02:58 not sure if jenkins is a real CPU consumer 15:03:06 dependes on the jobs that runs on it 15:03:20 mostly IO 15:03:41 I'm a bit distracted now, but this is a fine idea to work out the details right now, just need to migrate the conclusion to the mailing list 15:03:48 yes, master is pretty light for the most part 15:03:49 I have to run. quaid If you need me to volunteer for somethings else you can generally assume yes. 15:03:54 IO i can offer iSCSI (4gbit bonded) or 8 gbit FC 15:03:57 RobertM: thank you very much! 15:04:11 #chair eedri ewoud mburns 15:04:11 Current chairs: eedri ewoud mburns quaid 15:04:13 eedri: http://jenkins.ekohl.nl/munin/ekohl.nl/jenkins.ekohl.nl/cpu.html tells me it can use some CPU, but generally isn't 15:04:18 just in case so you all can close, do #agreed, etc. 15:04:54 ok, so priority is to get gestahlt up and running with ovirt 3.1 15:04:55 quaid: thanks for your time :) 15:05:00 ;) 15:05:25 ewoud, good to know you've got monitoring on it, we can use this to get an idea for what is needed 15:05:27 mburns: Sounds like a deal to me. How many VMs and which config you need? 15:05:40 ewoud, i assume that the cpu was high while jobs were running 15:05:50 gestahlt: i'll defer that decision to eedri and other infra people 15:05:56 we got a 100Mbit line to the company 15:06:04 If thats enough for you 15:06:19 gestahlt, anything will be probably better than the current ec2 infra :) 15:06:27 gestahlt: generally should be 15:06:48 Great 15:07:08 gestahlt, we can start with one server/vm and see how it goes 15:07:14 gestahlt: just FYI, we try to make all decisions ultimately on the mailing list, so we'll want to migrate you/ewoud/eedri's discussion there for conclusion, I think 15:07:28 #info gestahlt has offered free vm hosting for jenkins slaves 15:07:32 just so we can work out the details, it also shows others what those details are so it's not as much of a mystery, etc. 15:07:45 quaid: true 15:07:55 #info just need to get ovirt running in gestahlt 's environment 15:08:50 #info may get hardware available from Red Hat IT eventually, but not for a couple months... 15:09:28 somewhat related, are there plans to support nested virtualisation in ovirt? 15:10:04 ewoud: nothing specific, that i'm aware of 15:10:21 if you want to test creating VMs in VDSM that would be a useful feature to have 15:10:43 but i'm not sure how much effort is really needed beyond enabling it in ovirt-node and the various other distros we run vdsm on 15:11:00 ewoud: yes, we do have some ways to do that now 15:11:10 using vdsm-hook-faqemu 15:11:23 they're not *real* vm's but close enough 15:11:51 and, fwiw, this may even work today with AMD hardware which has nested virt on by default 15:12:11 it does? 15:12:35 ewoud: pretty sure it does 15:12:45 intel hardware doesn't 15:12:49 how about intel? I know the implementation is more complex 15:13:13 but i think intel hardware just needs a kernel flag set 15:13:24 yes, it does work but manually 15:13:32 Hello all, any word on the 3.1 release? Didn't see any updates on the ovirt site. 15:13:40 anyway, this seems to go offtopic and I think we can end this meeting 15:13:43 ewoud: my info comes from http://kashyapc.wordpress.com/2012/01/14/nested-virtualization-with-kvm-intel/ 15:14:02 Okay back 15:14:04 mburns: mine from seeing a collaegue do it on his desktop 15:14:07 I will join that mailing list 15:14:14 gestahlt: excellent 15:14:23 we'll take this discussion to infra then 15:14:32 tj: will answer in 2 min after we finish this meeting 15:14:48 ewoud: eedri: anything else to cover? 15:15:00 Where do i sign up? 15:15:16 mburns, there are other topics for jenkins, but i'm afraid we'll have to talk about in our next meeting? 15:15:26 gestahlt: http://lists.ovirt.org/mailman/listinfo/infra 15:15:37 eedri: yes, or take them to infra@ 15:15:59 mburns, ok 15:16:08 ok, wrapping up in 30 sec 15:16:35 okay done 15:16:58 <- Joachim Jabs 15:17:22 ok, /me ends the meeting 15:17:26 #endmeeting