15:01:09 #startmeeting oVirt Infra 15:01:10 Meeting started Mon Nov 4 15:01:09 2013 UTC. The chair is knesenko. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:10 Useful Commands: #action #agreed #help #info #idea #link #topic. 15:01:19 #chair ewoud obasan 15:01:19 Current chairs: ewoud knesenko obasan 15:01:29 dcaro, ping 15:01:47 * dcaro is here 15:01:50 obasan: pong 15:01:51 #chair dcaro 15:01:51 Current chairs: dcaro ewoud knesenko obasan 15:02:28 #chair eedri 15:02:28 Current chairs: dcaro eedri ewoud knesenko obasan 15:03:00 #topic Hosting 15:03:22 dcaro: have you rebooted the rackspace03 server ? 15:04:35 knesenko: I've tried connecting to the webadmin with no luck, and I haven't found a way of doing it from the ui, I'll open a ticket for it 15:04:46 dcaro: thanks 15:05:02 #action dcaro open a ticket to rackspace to reboot ovirt03 15:05:14 so i am still blocked on installation 15:05:43 any other news/issues on hosting ? 15:06:29 I have some thoughts about gluster way we choose 15:06:58 knesenko: please share 15:07:13 because we are going to use a local disks, and those disks will be used by jenkins slaves I think it will create high IO load on those disks 15:07:46 because currently jenkins slaves are in high load 15:08:02 that's what I think ... 15:08:10 more VMs we will add, more IO load 15:08:47 thoughts ? 15:09:02 can't ovirt use local disks and bind a VM to a specific hypervisor inside a cluster? 15:09:56 we could create a /var/lib/jenkins mount point and pass that through 15:09:59 ewoud, I think that it's possible to bind a vm to a specific hypervisor 15:10:23 ewoud: its possible 15:10:38 knesenko: are you sure that the current bottleneck is the disk io? 15:10:59 dcaro: I am not 100% sure ... but that's what Sasha told ... 15:11:30 knesenko: can we check with iotop or something? 15:11:50 dcaro: i think we should 15:12:08 often it's a fair guess 15:12:26 btw, using tuned could also help 15:12:32 let's make sure we are not fixing a problem we don't have though 15:12:43 the default scheduler isn't that well suited for virtualisation which also schedules 15:15:15 dcaro: its just a guess 15:16:31 so what we decide to do here ? 15:16:41 monitor IO a little bit on those servers ? 15:16:53 knesenko: that's ok, guesses are ok as a starter point, now let's continue the idea and make sure, I vote for gathering some info on current usage before taking any paliative action 15:17:06 dcaro: +1 15:17:19 obasan: can we monitor disks IO on incignia ? 15:17:19 sounds like we're back to monitoring and trending 15:17:35 knesenko, sure 15:18:04 yep :/, I've installed a graphite instance on stats.ovirt.org, I'd like to send there some metrics, maybe we can use it also for that 15:18:06 #action ad rackspace* servers to incingia to monitor IO, CPIU, mem etc .ASAP 15:18:17 dcaro: +1 15:18:20 can you do that ? 15:18:33 if u have time for that 15:18:51 we will have to figure out a way of securing the channel (graphite does not handle any authentication/ssl) 15:19:51 dcaro: do we care ? 15:19:57 dcaro: its just monitoring info ... 15:20:06 dcaro: and without anycredentials ... 15:20:21 knesenko: at least to avoid anyone from sending us fake stats 15:20:38 dcaro: i assume rackspace* servers will send info to stats.ovirt.org - right ? 15:20:51 yep 15:21:16 dcaro: you can create iptables rules on stats.ovirt.org to receive info only from rackspace* hosts 15:21:20 can we ? 15:21:25 sure 15:21:38 dcaro: i think it will e enough for now ... 15:21:42 objections ? 15:22:12 not right now 15:22:42 I don't know graphite though 15:24:00 dcaro: obasan ? 15:24:06 +1 15:24:13 obasan, I don't know either :) 15:24:42 #action dcaro setup grafite to collect info from rackspace servers + iptables rules 15:24:45 ok 15:24:54 #topic Foreman and Puppet 15:25:01 dcaro: ewoud news guys 15:25:02 ?= 15:25:20 I did nothing... :/ 15:25:28 no, but I do intend to finish the r10k deployment patch today 15:25:34 nice! 15:25:35 how about merging that http://gerrit.ovirt.org/#/c/20622/ ? 15:26:05 dcaro: I did describe the general workflow a while back so if you can take a look at that 15:27:43 obasan: I gave my +1, but I'd like dcaro to give it at least +1 as well 15:28:01 ewoud, great ! 15:28:05 looks good to me :) 15:29:21 I also think that we need a testing environment. it's a problem verifying puppet patches like this 15:29:29 dcaro: ewoud do we have F20 hostgroup ? 15:30:00 knesenko: no, but I think everything we do for F19 should work on F20 as well 15:30:06 nm we have only slaves hostgroup 15:30:14 yep 15:31:01 ok 15:31:05 anything else on puppet ? 15:31:08 foreman ? 15:31:22 I'd like to upgrade to 1.3 15:31:36 I also think that we need a testing environment. it's a problem verifying puppet patches like this 15:31:44 I think its important 15:31:49 and prevent future breakage 15:31:53 obasan: agreed, but sometimes you sort of need production load to test 15:32:03 such as tmpwatch that only does something after 12 hours 15:32:29 ewoud, correct. but I still think that we need some kind of a simple way to test functionality 15:33:09 ideally we'd write tests with rspec-puppet and rspec-system to do CI builds before pushing 15:33:15 ewoud, in the tmpwatch case I would like to see it installs and adds the cron entry 15:33:37 ewoud, that sounds interesting. I don't know these tools 15:33:59 ewoud, but maybe we should add that to jenkins if it tests puppet manifests functioning 15:34:01 obasan: http://rspec-puppet.com/tutorial/ you can at least let puppet compile the catalog and verify the package entry is there 15:34:31 it looks good. I recommend a jenkins gerrit job 15:34:39 that will use this tool to auto verify puppet patches 15:35:00 actually, that's http://gerrit.ovirt.org/20367 15:35:13 obasan: that should allow at least basic testing 15:35:26 ewoud, great. why is it not merged yet? problems? 15:35:39 obasan: no reviews 15:35:39 #action obasan add job in jenkins to use http://gerrit.ovirt.org/20367 and verify puppet patches 15:36:00 ewoud, I will review it 15:36:15 obasan: if you need help writing a jenkins job for it, do say so 15:36:30 ewoud, thank you . let's get it merged first 15:37:34 ewoud: obasan +1 15:37:52 for foreman 1.3, should I send an announcement at least a few hours in advance? 15:38:06 ewoud, +1 15:38:37 #info foreman upgrade to 1.3 will be announced on the ML 15:39:09 ok ... anything else in this topic ? 15:39:18 not from me 15:39:32 #topic jenkins 15:39:46 obasan: you upgraded jenkins last week right ? 15:40:09 knesenko, nope. we're on the latest LTS 15:40:23 obasan: ok :\ 15:40:31 where is eedri to update with the status ? 15:40:56 status looks ok on jenkins for me 15:40:58 at least 15:42:27 ok ... 15:42:47 #topic other business 15:42:58 any other topics we want to discuss ? 15:45:28 #endmeeting