14:00:28 #startmeeting 14:00:28 Meeting started Tue Jul 3 14:00:28 2012 UTC. The chair is quaid. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:28 Useful Commands: #action #agreed #help #info #idea #link #topic. 14:00:33 * RobertM here 14:00:41 #chair RobertM ewoud 14:00:41 Current chairs: RobertM ewoud quaid 14:00:48 (just in case) 14:00:59 * mburns here 14:00:59 * ewoud here 14:01:19 #chair mburns 14:01:19 Current chairs: RobertM ewoud mburns quaid 14:01:47 #meetingname oVirt infrastructure team meeting 14:01:47 The meeting name has been set to 'ovirt_infrastructure_team_meeting' 14:02:10 #topic Introductions (short) 14:02:44 I'll start :) 14:03:36 I work at Red Hat on a team focused on helping open source projects be wildly successful, and have been involved with oVirt since just before the open sourcing, doing whatever I could to help, including starting-up and maintaining the infrastructure. 14:03:46 * sgordon here 14:04:26 if anyone else is up for a quick intro, we'll spend just another minute or two on that 14:04:38 http://ovirt.org/wiki/Infrastructure_team_meetings#2012-07-03 is the agenda 14:04:57 * mburns has worked on ovirt since the days when we had a ruby-based ovirt server, i'm currently the tech lead on ovirt-node, and help out with infra stuff when needed but have no qualifications to do so 14:05:29 i work for redhat docs, my interest in infra primarily relates to providing infrastructure to support ovirt docs (once we get the docs more complete) 14:06:03 in particular a docs.ovirt.org site deployed with publican would be of interest and/or an instance of the ccms that has been developed internally to generate the next iteration of docs for RHEV 14:06:22 you can see a test of it in action at fedora-reloaded.com, a project josh wulf ran at fudcon malaysia 14:06:23 * ewoud works as a software engineer with a significant dose of infra 14:06:36 also i got my RHCE on friday, yay me ;) 14:06:43 sgordon: congrats! 14:06:47 * quaid is a lapsed RHCE 14:07:05 #idea Publican quickstart for OpenShift to run Publican-based doc sites 14:07:15 I work at red hat with quaid, I worked for a long time at eWEEK Labs, and have followed / written about open source virt quite a bit 14:07:28 I am Robert and I work for a network of answering service I started following ovirt when it relanched in Nov but have been following it closes for the last month. I have been working on Linux Since 1998 and been doing infra work for 8 years now. 14:07:59 ok, that's everybody mostly just as I'm ready to move on too :) 14:08:09 I know php well python, perl, and C some and just can't seem to get Java 14:08:18 #topic Agenda affirmation - anything to add? 14:08:36 I'm going to rudely paste it here so we have it in the log: 14:08:37 #link http://ovirt.org/wiki/Infrastructure_team_meetings#2012-07-03 14:08:54 Short, one-line introductions. 14:08:54 Agenda affirmation - anything to add or change? 14:08:54 Review of open topics - what are the things we need to be talking about, in general? 14:08:57 Some specific and ongoing tasks are on the Infrastructure team task list: 14:09:00 http://www.ovirt.org/wiki/Infrastructure_team_task_list 14:09:02 How are we going to be more open & enable others to do Infra work? 14:09:05 Being transparent 14:09:07 Building trust 14:09:10 What is an oVirt sub-project and do we want to be one? 14:09:12 http://www.ovirt.org/governance/adding-a-subproject/ 14:09:15 What is the mission of the oVirt infra project? 14:09:17 Strawdog: "To provide a free/open source infrastructure of participation that supports the development of the oVirt project, and to do so by providing full free/open access to the entire infrastructure as a collaboration." 14:09:22 Any other business? (AOB) 14:09:24 14:09:27 if that's too annoying, we won't do that again :) 14:09:29 anything to add before we start? 14:09:52 (can always add later) 14:10:13 #topic Review of open infrastructure topics - what do we care about? 14:11:07 what do you exactly mean by open infra topic? 14:11:08 under review of open topics it might be useful to detail what we have now in terms of "machines" and then discuss what scope we have on top of that 14:11:25 http://www.ovirt.org/wiki/Infrastructure_team_task_list 14:11:53 ewoud: trying to capture what we think are things we need to be talking about, in general - such as what sgordon just suggested 14:12:18 sgordon: I made a stub page for that last night: 14:12:19 http://ovirt.org/wiki/Design_of_oVirt_project_infrastructure 14:12:26 was too tired to get anything in to it :) 14:12:31 I think currently most infra is run on amazon, RH provides some jenkins slaves 14:12:47 Linode, too 14:13:18 my employer has allowed a jenkins slave which I'm installing and show deliver any day now 14:13:26 +1 14:13:46 quaid, can correct but my understanding is there is the kitchen sink box on linode and jekins/gerrit and the slaves are mostly on EC2 14:13:47 I'll announce it when it's final, but 8GB RAM, 50GB HDD and it runs on RHEV 14:14:04 itamar & I have had side conversations about jumping Jenkins to a dedicated server from a hosting provider, until we can get better from e.g. Red Hat IT 14:14:14 RobertM: that's correct 14:15:03 #action Fill out infrastructure design page with a list of hosts and what goes on top of them, as a way to feed our scope discussion 14:15:28 #action Plan to potentially move Jenkins from EC2 to a dedicated host for performance and cost reasons 14:15:52 I think it's also important to sync configurations, or at least manage them centrally 14:15:56 #agreed Linode box == kitchen sink, jenkins, gerrit, and slaves are mostly EC2 14:16:03 #idea Puppet or chef FTW! 14:16:17 I have experience with puppet, none with chef 14:16:24 ewoud: is that something that would help Jenkoins 14:16:39 Puppet is the one I'd like to learn, anyway, so I'm in favor of starting there 14:17:07 quaid: I'm trying to install the host with puppet locally, we could move to a central puppet master if needed 14:17:11 The main issue is the Jenkins slaves and keeping them in sync so both F17 boxes are the same. 14:17:32 RobertM: keeping the configurations in sync? or other items? 14:17:33 And when we move to other slave types. 14:18:14 ah, right 14:18:26 Sorry - lost the connection there for a few minutes 14:18:32 #action setup a Puppet server for ovirt infra 14:18:34 Will read the backlog afterwards 14:18:39 I'd like to publish the puppet profiles on gerrit.ovirt.org, but unsure how to handle the secret stuff like SSL certs 14:18:56 sgordon, I had a question about publican 14:19:01 dneary: http://ovirt.org/meetings/ovirt/2012/ovirt.2012-07-03-14.00.log.txt updates continuously 14:19:08 sgordon, What does it bring to the table relative to a Mediawiki? 14:19:31 but the main advantage of puppet on gerrit is that we could accept patches for it even from untrusted people, as long as we review them 14:19:33 ewoud: what about a git repo that is only visible to root but is distributed on most/all boxes? 14:19:42 i.e., a git repo that is not kept in gerrit but in /root/git/ 14:19:46 dneary, formal documentation 14:20:13 quaid: I'm interested in knowing how fedora / mediawiki does that, but certainly an option to place the secret stuff in its own module 14:20:16 as in, we have a team of writers producing documentation for RHEV, much the same as we do for RHEL 14:20:26 we can't easily contribute that to mediawiki 14:20:32 besides which, the wiki is a mess already :p 14:20:41 ewoud: I'll ask 14:20:46 sgordon, Well, I'd like to see it fixed up :) 14:21:09 (as in, I'd like to make that happen) 14:21:42 quaid, Thanks 14:21:49 How much stuff is in the current Git repo that we don't want out? 14:21:59 the wiki is good for short docs and community collaboratins, and it can be a source for documents that get converted to Publican (cf. Fedora release notes) 14:22:11 but it's not so good for actual guides :) 14:22:46 RobertM: I don't think there is anything; I did start a git report in /root on the Linode with /etc but not sure there are any secrets in there 14:23:11 ok, these are good additions to the list 14:23:59 Unless you have someone watching the wiki heavy handly most wiki's end up being full of old info and no longer true info. 14:24:04 i am the first to admit i have a lot more work to do on actual guides, but yeah, longer term we would like to provide something 'static' 14:24:09 that is correct with each release 14:25:14 RobertM: yeah, we can talk with others about how to manage the wiki better; I'm interested in that, too, although it's more of a topic for a Docs meeting that maybe sgordon can call together :) 14:25:27 sgordon, I think we need need to make the admin and install guides an ongoing process not something that gets released when the project gets released. Making it easier for changes to be submitted. 14:25:38 sgordon: we do seem to have a few docs-related topics that could be at least a thread or three on arch@ - the docs.ovirt.org, better mediawiki, publican training, etc. 14:25:39 well it is an ongoing process 14:25:47 and stimulate the use of categories 14:25:48 it's just im the only committer ;) 14:25:54 on the wiki that is 14:25:56 ewoud: +1 to categories 14:26:36 quaid, that's a good point, about a topic for another meeting -- I've wondered how many sorts of topics arch is supposed to cover 14:26:37 sgordon: we can fix that :) ... I know you won't see the same level of contrib to Publican-based than to a wiki, but helping others with how to submit patches e.g. via gerrit? 14:26:44 sgordon, I just noticed I'm still owner in publican-ovirt, do you want to take over? 14:26:45 jbrooks: "everything" 14:26:59 apevec, i can, i think you already gave me comaintain 14:27:03 Is the doc's in gerrit? Is there an ongoing process to regenerate the doc's after updates? Are they versioned? 14:27:05 quaid, OK, cool, so docs would be another arch meeting? 14:27:08 I think we're getting to another point on the agenda: what should arch do (and not do) 14:27:09 14[[07Infrastructure team task list14]]4 !10 02http://www.ovirt.org/w/index.php?diff=3746&oldid=3732&rcid=3839 5* 03Quaid 5* (+218) 10/* One time tasks */ updating with new tasks gleaned from the meeting so far 14:27:25 sgordon, yep, and I think you proved yourself :) 14:27:41 sgordon, I'll release ownership, be ready at https://admin.fedoraproject.org/pkgdb/acls/name/publican-ovirt 14:27:42 I think I got all we suggested so far 14:27:44 the docs, at this stage just an install guide are in gerrit, the regeneration at this stage is just building an uploading pdf to the wiki 14:28:10 obviously with a docs subsite we would look at releasing the other output formats (html, html-single, epub) 14:28:18 bugs/changes are raised in bugzilla 14:28:25 ewoud: answer to question about secrets & puppet in Fedora: 14:28:29 07:24 < skvidal> quaid: in the private git repo 14:28:29 07:25 < skvidal> quaid: that's assembled so puppet can access it 14:28:29 07:25 < skvidal> quaid: sysadmin-main has access to it 14:28:33 14:28:56 sgordon, ready for grabs! 14:28:58 ewoud: I can get us more specific help as we approach the task 14:29:01 apevec, taken 14:29:06 +1 14:29:10 quaid: ok 14:29:20 any other topics we think Infra needs to be discussing? 14:29:39 (arch@ topics are for another meeting) 14:30:03 oh, by discussing ... 14:30:08 sgordon, you took only devel, take them all 14:30:20 I don't mean just in this meeting, more of, "what are we talking about on the mailing list, IRC, etc." - meaning, what do we care about :) 14:31:09 apevec, ah right 14:31:10 quaid, Maybe what is Arch and what is infra responsible for? 14:31:42 right 14:31:52 * quaid trying to frame that in to an action 14:31:55 apevec, hmm i dont actually get the options on the others 14:32:14 hmm 14:32:15 oh wait there they are 14:32:19 had to verify login again 14:32:50 apevec, collected them all ;) 14:32:54 14[[07Special:Log/newusers14]]4 create210 02 5* 03Danken 5* 10created new account User:YanivBronhaim 14:32:55 excellent! 14:33:26 sgordon, I find that formal docs are harder to maintain without a structured team. A wiki is not easy either, granted, but still, I've found them usable, with (as RobertM said) a wiki editing team 14:33:32 #action Define what is the purview of infra@ and what is purview of generic arch@ 14:34:06 dneary: there is evidence that a wiki gets 10x to 20x the collaboration levels of DocBook in a VCS 14:34:29 dneary, the fact that they are harder does not make them less desirable to have 14:34:31 dneary: which is why Fedora Docs has tried many ways to do "write in MediaWiki, convert and publish in DocBook" 14:34:47 quaid, Yes. Anne Gentle (Conversation & Community) documents her experiences moving formal docs teams to wiki-based documentation and has similar results, IIRC 14:35:05 i dont suppose anyone looked at fedora-reloaded.com ? 14:35:07 dneary: I think sgordon accepts that he may be doing all the heavy lifting for using Publican for guides, but a community of writing help can grow to support that - it does in Feodra, too 14:35:21 oh wait that's not the address 14:35:24 maybe josh took it down 14:35:35 quaid, I believe that the only way to maintain docs which correspond to a given version of the source code is to maintain them with the source code. 14:35:37 http://fedorareloaded.com/ 14:35:50 dneary: right, Mozilla's experience with that 8+ years ago is what made me accept wiki for Fedora Docs when I was a Big DocBook Weenie 14:36:13 http://www.fedorareloaded.com:8080/TopicIndex/Books/index.seam 14:36:14 quaid, My apologies if it comes across like I'm arguing against Publican, I also have some questions re the barrier to starting doc writing with it 14:36:21 sgordon: not lately 14:36:39 dneary, it comes across like the same discussion we had 6 months ago 14:36:40 there are a bunch of new web-editing book writing tools we can look at, too 14:36:41 Anyway, I seem to have derailed the discussion again - sorry 14:36:47 dneary: sorry, I'm just giving context that I know 14:36:58 dneary: I have the same questions :) 14:37:01 ok, my fault too 14:37:03 * quaid moves on 14:37:13 #topic 07:24 < skvidal> quaid: in the private git repo 14:37:13 07:25 < skvidal> quaid: that's assembled so puppet can access it 14:37:13 07:25 < skvidal> quaid: sysadmin-main has access to it 14:37:16 whoops 14:37:29 ops 14:37:31 #toic How are we going to be more open & enable others to do Infra work? 14:37:36 #topic How are we going to be more open & enable others to do Infra work? 14:37:42 3rd time's a charm 14:37:56 this is a topic that RobertM & I have discussed a bit, mostly onlist 14:38:31 I'm VERY in favor of making it easy to trust each other, and I really want to give out ssh and sudo access to people who are interested and want to help 14:38:38 I have lots of idea and a few of them might actually be good :) 14:39:45 fwiw, the only reason I didn't set RobertM up is i) we don't have a process so that would be a bit presumptious, and ii) I sensed RobertM didn't want to be just given keys without some trust sharing first, which I agree with - same with ewoud, we never resolved this 14:39:50 RobertM: tell us! 14:40:19 quaid: ping 14:40:29 quaid, I am under the trust but verify kind of mind set. This is my though for adding someone to the team. 14:40:45 my prefered solution is a puppet repo on gerrit where we can review it, and when you have sufficient trust and/or need you can give more permissions 14:40:58 quaid, Can we have sort of tiered services - T1 - must work (needs proven sysadmins) and T2 - if we have some downtime or performance issues, it ain't so bad, we can go back to last night's backup (where we liberally hand out root) 14:41:10 RobertM: how do we verify? 14:41:13 rgolan: pong 14:41:17 Plus a puppet repo where people can propose specific changes for T1 and T2 14:41:37 right 14:41:40 quaid: could you install syntax highlighting plugin for our wiki? 14:41:47 Get phone, address, email address, and do a least one phone call and maybe confirm there mailing address. To verfy we have at least some info about the person 14:41:50 public puppet repo with review as T0 :) 14:41:56 and I see that Fedora Infra uses groups cleverly - sort of like having backup admins, etc. 14:42:34 rgolan: sure; can you send an email to infra@ with whatever details you have on how to, etc.? URL to Wikimedia page is probably sufficient 14:42:39 ewoud, Then we should swap T1 and T2 14:42:44 rgolan: I'll pass your message through if you are not on the list 14:42:46 How do other groups handle it? What can we learn for them? 14:43:01 ewoud, T0 - no permission needed, T1 - low barrier to entry, T2 - high barrier to entry 14:43:13 ewoud: here's a general question - note RobertM's suggested verification, is that something you'd be comfortable and able to do? 14:43:14 dneary: something like that 14:43:20 i.e., have a phone call with e.g. me? 14:43:23 quaid: I'm almost sure I'm not on the list. subscribe and you'll appove me? 14:43:37 rgolan: self-approval 14:43:51 personally I dislike phones, but some verification would be nice 14:43:52 rgolan: 10x 14:44:15 how do other solve it? (looking at fedora again) 14:44:24 ewoud, quaid Could do snail mail as well but the phone is easier and faster. 14:44:32 best would be meeting at a conference or something 14:44:48 Email is to easy to fact. Phone and mailing addresses at least in the US is harder 14:45:08 Fedora does something like this: 14:45:08 ewoud, Yes face to face would be best. 14:45:55 But that would greatly limit the pool since how many people are going to fly someplace to join a voluntary team. 14:45:58 * interested people given ssh to bastion host with r/o access to logs, etc. - nothing they can do, lots to see 14:46:05 ** usually also get the nagios firehose turned on 14:46:06 I visit FOSDEM every year, but you are US-based so that might be harder 14:46:18 * people express interest in helping in an area, e.g. sysadmin-web 14:46:36 * given ability to make changes that are reviewed in just sysadmin-web areas, via Puppet 14:46:57 * having ssh+r/o+nagios = abiliyt to find and diagnose other problems, so show capability and trust 14:47:06 * over time, just increase what people can do 14:47:13 so it's trust over time, done via irc and email 14:47:37 so IMO I've built that trust over time with ewoud, and currently feel very strongly that way about RobertM - without a phone call, etc. 14:48:05 another idea - have a Google video conference hangout 14:48:42 quaid: what's the bastion host? 14:49:22 quaid, I have been burned in the past with people who were ghosts. They looked real but weren't. I still think getting and confirm one real world piece of info ontop of the trust of time aspect is important. 14:49:51 ewoud: it's the single host that has ssh accessible from outside of the private network 14:50:03 quaid: ok 14:50:04 Something that is hard to spoof like a mailing address or a phone number not linked to google voice. 14:50:05 ewoud: so you ssh to the bastion host, which only has port 22 open, etc. - then ssh to others 14:50:19 ah, a jump host 14:50:38 I think we should finish the topic of how to trust each other via mailing list, since we're getting close to the end of the hour :) 14:51:01 #action Finish building trust on the mailing list 14:51:13 or do I need to assign that to someone? 14:51:21 for now we're somewhat unassigned 14:51:41 but does our bot get that? 14:51:49 quaid, The important thing is to have a situation where people can do useful work with no permission at all, and then earn trust in an intermediate stage before getting total trust 14:52:01 dneary: I think that would preferable 14:52:05 dneary: +1 14:52:08 but to get us to that stage 14:52:20 here's my catch: 14:52:33 I need other people to have sudo so we can build that system we just described 14:52:45 aka catch 22 :) 14:52:49 if we have to do that first before we can include e.g. ewoud and RobertM, it's going to take a while 14:52:53 14[[07Vdsm Developers14]]4 !10 02http://www.ovirt.org/w/index.php?diff=3747&oldid=3414&rcid=3841 5* 03YanivBronhaim 5* (+15) 10/* Building a Vdsm RPM */  14:53:01 RobertM: yep! 14:53:21 is there anyone internally we could add who is not already added to assist with that? 14:53:42 sgordon: the challenge there is, why should anyone trust dneary (for example) just because he's @redhat.com? 14:53:51 sgordon: so I'd like to apply the same paintbrush, if we can 14:53:55 sure 14:54:01 but then you are stuck with catch 22 14:54:04 doesnt phase me :p 14:54:37 I'm going to propose (on the mailing list) that we use me as a trust circle for just the first round 14:54:52 quaid: sounds decent to me 14:54:54 but why should we trust you! ;) 14:54:55 let me establish a personal trust via any method with anyone interested right now, and use that to build the first circle of trusted sudo folken 14:54:56 jks 14:54:59 we have to start somewhere 14:55:01 Well for one thing a @redhat.com address means Red Hat has confirmed he exists and is an employee so that apply a certain amount of exist of him 14:55:02 sgordon: seriously! I know :) 14:55:11 RobertM: good point 14:55:14 well i guess the main thing is 14:55:26 we know where to go to get a @redhat.com person beaten up 14:55:29 ;p 14:56:01 true that 14:56:14 ok, like I said, propose and decide on mailing list is our method, so I'll take this there 14:56:25 #action quaid to propose himself as seed for a trust circle on infra@ 14:56:27 i guess this is really what FAS is for in fedora land, where as you earn trust you are added to groups that grant various rights/abilities 14:56:38 sgordon: +1 yep 14:56:39 but the actual gaining of trust is still via irc email afaik 14:56:44 FUDs if you are lucky/able to attend 14:57:16 right, but we can't wait for in person, and people are sometimes unable to talk or videochat, and may need to hide employer, or even their work from their government 14:58:09 I think that's why it's vital to have a puppet git repo where they can suggest changes, but we can still review them 14:58:20 ok, so we didn't quite get to the last topics, which is fine 14:58:26 ewoud: +1 that does seem to be the vital bit 14:58:46 yeah, i for example probably wouldn't want direct access 14:59:00 but if i could submit patches to get what i need to support docs if or when we do that 14:59:06 it would speed things up 14:59:33 plus i think it helps a lot just understanding what is actually going on 14:59:38 it would also help people who are uncertain about their work 14:59:48 and provide a change log 14:59:49 * RobertM Has spent that last 8 years building systems like that so programmers can get there job done without need such access. Well they were kicking and screaming each step of the way :) 15:00:18 ewoud: +1 people feeling that Puppet is a safety net helps people feel confident 15:00:21 RobertM: that's how you know you're doing a good job ;) 15:00:39 ewoud, amen bother 15:00:46 does anyone need to finish meetng now? 15:01:03 quaid, You definitely shouldn't trust dneary 15:01:05 well, I should be good an finish anyway to be disciplined :) 15:01:14 dneary: I'll trust you with that opinion then 15:01:14 quaid, I am a passable sysadmin, at best 15:01:27 dneary: thus the perfect foil for an example! 15:01:52 (I do maintain several boxen, but wouldn't trust myself with anything where 100s of people depended on it being up) 15:02:11 any objections to closing the meeting? 15:02:15 we can keep talking of course :) 15:02:27 * quaid waits 30 seconds or so 15:02:36 The 1h meeting is a symptom of "we are in corporate land" you know 15:02:41 true 15:02:52 but I recognize that ewoud and RobertM are both likely at corporate-land-work :) 15:02:58 I think we're getting somewhat informan, which is often a sign we're done 15:03:02 quaid, I think you would have liked my Ignite talk 15:03:06 heh 15:03:10 ewoud: +1 well said 15:03:14 I've written it up, will post speaker notes later this week 15:03:21 I can stay on for a while 15:03:31 quaid: $employer is not that corporate and rather informal 15:03:53 #topic Formal meeting at an end, informal chat which we'll log for now since we're on the same topics :) 15:03:56 how about that :) 15:05:39 5 pm here, people starting to go home so I get more time 15:05:51 heh 15:06:16 I think this meeting we sort of got to know eachother, in the next meeting I think we should decide what is something we want to run and what not to 15:06:31 that would be me, going home. sorry for taking up your time :) 15:06:48 tjikkun_work: go home, your wife is waiting for you ;) 15:06:51 tjikkun_work: ha! 15:07:16 tjikkun_work: if you're not careful I'll drag you into infra as well 15:07:31 too late :) 15:07:45 I was about to say that it always works 15:09:07 but anyway, on http://www.ovirt.org/wiki/Category:Infrastructure we have a list of infra 15:09:37 has anybody looked at e.g. http://www.ovirt.org/governance/adding-a-subproject/ and see any reason we shouldn't try to get Infra recognized as a formal sub-project? 15:10:03 does the definition of a sub-project match? 15:10:04 I think the list could be expanded, so a list of services, servers 15:10:55 let me make sure I have that explictly as a task, since much of it is trapped in my brain 15:11:21 #action quaid to do first draft of services, servers, and infrastructure design for http://ovirt.org/wiki/Design_of_oVirt_project_infrastructure 15:12:09 maybe with that list we could see what to focus on when puppetizing 15:12:23 I'd guess that jenkins would be the first, but maybe there's more 15:12:48 yeah, even though we don't change the main wordpress, mediawiki instances, we might do that 15:13:07 although I am going to propose moving those services to OpenShift, to reduce our workload 15:13:23 (which also gives us a built-in way to do patch review and such for T0 - T2) 15:13:48 I have no experience with openshift, but since it's based on git I think it is good 15:14:27 as you said, built-in way for patch review 15:15:13 so for next week a list of servers and services, maybe then set some goals? 15:16:06 +1 15:16:19 ok, I'll close the meeting log so we don't forget :) 15:16:47 #endmeeting