<div dir="auto">Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.<br><br><div data-smartmail="gmail_signature"><br>Phillip Smith<br>954-558-5022</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 7, 2023, 5:27 PM Phil Dibowitz <<a href="mailto:phil@ipom.com">phil@ipom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">re-sending with Smith on CC.<br>
<br>
On 10/7/23 11:38, Phil Dibowitz wrote:<br>
> Restarted. Added cron to restart. Will find time to pair debug with <br>
> Smith this week.<br>
> <br>
> Get BlueMail for Android <<a href="https://bluemail.me" rel="noreferrer noreferrer" target="_blank">https://bluemail.me</a>><br>
> On Oct 7, 2023, at 1:50 AM, Ilan Rabinovitch <<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a> <br>
> <mailto:<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a>>> wrote:<br>
> <br>
> Looks like we were down again for a few hours this morning.<br>
> <br>
> On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch<br>
> <<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a>> wrote:<br>
> <br>
> <br>
> (resending with offlist folks)<br>
> <br>
> Looks like we had another outage this morning. I'm online for<br>
> the next<br>
> 45-60 min if someone wants to chat about it. Otherwise, I would<br>
> greatly appreciate it if someone on the infra team could keep an eye<br>
> on monitoring and at least restart things.<br>
> <br>
> On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
> Phil Smith is probably the best person since it seems to be<br>
> a Drupal<br>
> issue, best I can tell (I scoured logs a few months back and<br>
> found<br>
> nothing other than what you just described: apache threads<br>
> just stay<br>
> around, busy, until they eat up all the memory).<br>
> <br>
> I'm not sure he's on this list so I'm CC'ing hom.<br>
> <br>
> Phil - I'm happy to get you root access (send me an SSH<br>
> pubkey), or<br>
> anything else you may not already have to help you debug. I<br>
> spent quite<br>
> a while last SCALE trying to figure this out, Ilan has spent<br>
> time on<br>
> this, and it seems to be drupal-specific, best I can tell.<br>
> <br>
> <br>
> <br>
> On 10/5/23 14:49, Ilan Rabinovitch wrote:<br>
> <br>
> Theres definitely something continuing to go on with<br>
> drupal and/or<br>
> httpd. Something continues to eat up all the worker<br>
> threads from<br>
> httpd. Before we set maxworkers to something reasonable<br>
> it would eat<br>
> up all the memory, now it just consumes all the workers<br>
> and the site<br>
> becomes unresponsive.<br>
> I dont think this relates to the upgrade, as it was<br>
> happening before.<br>
> <br>
> We've had the site down three times in the past 2 days<br>
> as a result.<br>
> Monitoring detected it and I was able to respond to it<br>
> fairly quickly,<br>
> but I'm about to board a 30 hour flight, so I'm likely<br>
> not going to be<br>
> able to respond moving forward.<br>
> <br>
> Given the importance of the CFP this month I hope<br>
> someone has a chance<br>
> to dig in.<br>
> <br>
> On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz<br>
> <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
> Copy that, I'll keep an eye out. Nice work, and thanks!<br>
> <br>
> On 10/3/23 18:06, Ilan Rabinovitch wrote:<br>
> <br>
> This is done. Here's the specific ids for any<br>
> roll backs:<br>
> <br>
> 1) start scale-web1 up again (i-017df11978f41d9ae)<br>
> 2) sync<br>
> /home/drupal/scale-drupal/httpdocs/sites/default/files/ from<br>
> scale-web2 (i-0bf1dd453774f2c3b) to scale-web1<br>
> (i-017df11978f41d9ae)<br>
> 3) move EIP (23.21.71.118 <<a href="http://23.21.71.118" rel="noreferrer noreferrer" target="_blank">http://23.21.71.118</a>>)<br>
> from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.<br>
> <br>
> Assuming no issues the next change to think<br>
> about is:<br>
> - enable the s3fs plugin work Phil Smith did<br>
> recently so static files<br>
> aren't stored on box anymore.<br>
> - move scale-web2 behind an ELB and auto scaling<br>
> group.<br>
> - remove the cron jobs that back up static files<br>
> <br>
> This will enable some reasonable failover that<br>
> doesn't require human<br>
> intervention.<br>
> <br>
> <br>
> On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz<br>
> <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
> Easy-peasey. Feel free to do it as your<br>
> leisure today. I have a few<br>
> errands to run, but will never be more than<br>
> 30m from home and will be<br>
> able to roll back today, and will be around<br>
> all day tomorrow and Thursday.<br>
> <br>
> <br>
> On 10/3/23 16:08, Ilan Rabinovitch wrote:<br>
> <br>
> Cutover:<br>
> - RDS Database snapshot<br>
> - sync<br>
> /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both<br>
> hosts in case users have uploaded static<br>
> assets<br>
> - Move EIP from scale-web1 host to<br>
> scale-web2<br>
> <br>
> Rollback:<br>
> - re-sync<br>
> /home/drupal/scale-drupal/httpdocs/sites/default/files/ on<br>
> both hosts in case users have uploaded<br>
> static assets<br>
> - Move EIP from scale-web2 host to<br>
> scale-web1<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> On Tue, Oct 3, 2023 at 3:41 PM Phil<br>
> Dibowitz <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
> Feel free to do it today.<br>
> <br>
> What's the migrate/failback process?<br>
> It's somewhere in cloudflair?<br>
> <br>
> If not I can do it next week.<br>
> <br>
> <br>
> On 10/3/23 11:09, Ilan Rabinovitch<br>
> wrote:<br>
> <br>
> Sounds like we should do this<br>
> today then, or you can grab next<br>
> week.<br>
> I'm going to be offline until<br>
> November after Thursday morning.<br>
> <br>
> On Mon, Oct 2, 2023 at 9:35 PM<br>
> Phil Dibowitz <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
> I'm out of town Fri-Sun, and<br>
> will only be around a laptop<br>
> sporadically.<br>
> I'm in town all week next<br>
> week though.<br>
> <br>
> <br>
> <br>
> On 10/2/23 13:22, Ilan<br>
> Rabinovitch wrote:<br>
> <br>
> Phil Smith, myself and a<br>
> few others have banged<br>
> on scale-web2 and we<br>
> think we're all set to<br>
> do a cutover there.<br>
> However, I'm going to be<br>
> out of town for a few<br>
> weeks starting Thursday<br>
> and do not yet know what<br>
> my connectivity will be<br>
> like.<br>
> <br>
> Would love to cutover<br>
> this week, but only want<br>
> to do it if others will<br>
> be available to roll<br>
> back to the old server<br>
> should something go awry<br>
> while I'm on vacation.<br>
> <br>
> On Sun, Sep 24, 2023 at<br>
> 9:46 PM Ilan Rabinovitch<br>
> <<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a>> wrote:<br>
> <br>
> <br>
> I believe Ive<br>
> addressed everyone's<br>
> feedback.<br>
> <br>
> scale-web2 is online<br>
> on running off the<br>
> scale_web_centos8<br>
> branch.<br>
> it's using a test db<br>
> running from a<br>
> snapshot of prod 2~<br>
> weeks ago.<br>
> you can change it as<br>
> you see fit without<br>
> worrying about<br>
> impacting production.<br>
> <br>
> To test you'll need<br>
> to update /etc/hosts<br>
> to point at the new<br>
> server.<br>
> following to your<br>
> /etc/hosts file.<br>
> <br>
> 54.205.247.164<br>
> <<a href="http://54.205.247.164" rel="noreferrer noreferrer" target="_blank">http://54.205.247.164</a>> <a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">www.socallinuxexpo.org</a> <<a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">http://www.socallinuxexpo.org</a>><br>
> <br>
> Ideally would like<br>
> to merge and migrate<br>
> product this week<br>
> <br>
> <br>
> On Fri, Sep 22, 2023<br>
> at 7:19 PM Phil<br>
> Dibowitz<br>
> <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
> Sorry I was in<br>
> Mexico.<br>
> <br>
> Left a review to<br>
> DRY it up. But I<br>
> don't see any<br>
> critical things.<br>
> <br>
> - Phil<br>
> <br>
> On 9/22/23<br>
> 15:23, Ilan<br>
> Rabinovitch wrote:<br>
> <br>
> Ping? Would<br>
> like to wrap<br>
> this up as<br>
> the non-stop<br>
> daily<br>
> outages are no<br>
> fun, Managed<br>
> to mitigate<br>
> it somewhat,<br>
> but given<br>
> none of us<br>
> are on call<br>
> getting to<br>
> some<br>
> reasonable<br>
> sense of HA<br>
> would be<br>
> appreciated<br>
> so folks<br>
> can go on<br>
> vacation<br>
> without<br>
> worrying<br>
> about who<br>
> will reboot<br>
> the servers.<br>
> <br>
> On Tue, Sep<br>
> 19, 2023 at<br>
> 8:59 PM Ilan<br>
> Rabinovitch<br>
> <<a href="mailto:ilan@linuxfests.org" target="_blank" rel="noreferrer">ilan@linuxfests.org</a>> wrote:<br>
> <br>
> <br>
> Thanks<br>
> Davide<br>
> for the<br>
> initial<br>
> review.<br>
> I think<br>
> I<br>
> addressed the<br>
> remaining feedback.<br>
> <br>
> Anything<br>
> else we<br>
> want to<br>
> test<br>
> before<br>
> merging?<br>
> <br>
> On Fri,<br>
> Sep 15,<br>
> 2023 at<br>
> 7:52 PM<br>
> Ilan<br>
> Rabinovitch <<a href="mailto:ilan@linuxfests.org" target="_blank" rel="noreferrer">ilan@linuxfests.org</a>> wrote:<br>
> <br>
> <br>
> Hi,<br>
> <br>
> I<br>
> took<br>
> a<br>
> stab<br>
> at<br>
> updating the cookbooks we use for the website to use<br>
> centos8. Motivation was getting a new version of php so we can use<br>
> some<br>
> drupal plugins that dont support php5.4, but i think this was on<br>
> the<br>
> backlog already for a while.<br>
> <br>
> <a href="https://github.com/socallinuxexpo/scale-chef/pull/283" rel="noreferrer noreferrer" target="_blank">https://github.com/socallinuxexpo/scale-chef/pull/283</a> <<a href="https://github.com/socallinuxexpo/scale-chef/pull/283" rel="noreferrer noreferrer" target="_blank">https://github.com/socallinuxexpo/scale-chef/pull/283</a>><br>
> <br>
> To<br>
> test<br>
> this<br>
> out,<br>
> I<br>
> spun<br>
> up<br>
> scale-web2 with a clone of the production<br>
> scale-drupal database, and then pointed my host file at it for<br>
> <a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">www.socallinuxexpo.org</a> <<a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">http://www.socallinuxexpo.org</a>>:<br>
> <br>
> I<br>
> then<br>
> tested the following:<br>
> <br>
> -<br>
> chef<br>
> runs<br>
> complete end to end<br>
> -<br>
> Backing up / restoring static assets (see backup scripts in<br>
> /usr/local/bin or in the scale-drupal cookbook)<br>
> -<br>
> browsing legacy static website (eg<br>
> <a href="https://socallinuxexpo.org/past/2002" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2002</a> <<a href="https://socallinuxexpo.org/past/2002" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2002</a>>/<br>
> <a href="https://socallinuxexpo.org/past/2003" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2003</a> <<a href="https://socallinuxexpo.org/past/2003" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2003</a>>/)<br>
> -<br>
> Registering as a speaker, confirming that I got the email and that<br>
> my<br>
> account worked.<br>
> -<br>
> publishing the submitted talk<br>
> -<br>
> creating sponsors/exhibitors/blog post/events/etc<br>
> <br>
> Pending things:<br>
> -<br>
> Centos8 seems to default to using php-fpm instead of mod_php. Not<br>
> opposed but it's different. Seems to work fine, but I've not load<br>
> tested it or anything.<br>
> <br>
> -<br>
> drush (<a href="https://www.drush.org" rel="noreferrer noreferrer" target="_blank">https://www.drush.org</a> <<a href="https://www.drush.org" rel="noreferrer noreferrer" target="_blank">https://www.drush.org</a>>/) packages no longer seemed to be<br>
> present in centos or epel. remi seems to have it but the drush docs<br>
> suggest we should be installing this in our drupal code base via<br>
> composer instead of as a package. I dont have strong opinions, but we<br>
> should figure out the right path before merging.<br>
> <br>
> Cheers,<br>
> Ilan<br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra<br>
> mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
> --<br>
> Phil Dibowitz<br>
> <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
> Open Source<br>
> software and<br>
> tech docs<br>
> Insanity Palace<br>
> of Metallica<br>
> <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/ <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
> "Be who you are<br>
> and say what you<br>
> feel, because<br>
> those who mind don't<br>
> matter and those<br>
> who matter don't<br>
> mind."<br>
> - Dr. Seuss<br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra<br>
> mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
> --<br>
> Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
> Open Source software and<br>
> tech docs Insanity Palace of<br>
> Metallica<br>
> <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a><br>
> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
> <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a><br>
> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
> "Be who you are and say what<br>
> you feel, because those who<br>
> mind don't<br>
> matter and those who matter<br>
> don't mind."<br>
> - Dr. Seuss<br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
> --<br>
> Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
> Open Source software and tech docs<br>
> Insanity Palace of Metallica<br>
> <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a><br>
> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
> <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a><br>
> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
> "Be who you are and say what you<br>
> feel, because those who mind don't<br>
> matter and those who matter don't mind."<br>
> - Dr. Seuss<br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
> --<br>
> Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
> Open Source software and tech docs Insanity<br>
> Palace of Metallica<br>
> <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a><br>
> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
> <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
> "Be who you are and say what you feel,<br>
> because those who mind don't<br>
> matter and those who matter don't mind."<br>
> - Dr. Seuss<br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
> --<br>
> Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
> Open Source software and tech docs Insanity Palace<br>
> of Metallica<br>
> <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
> <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
> "Be who you are and say what you feel, because those<br>
> who mind don't<br>
> matter and those who matter don't mind."<br>
> - Dr. Seuss<br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
> --<br>
> Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
> Open Source software and tech docs Insanity Palace of Metallica<br>
> <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
> <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
> "Be who you are and say what you feel, because those who<br>
> mind don't<br>
> matter and those who matter don't mind."<br>
> - Dr. Seuss<br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> ------------------------------------------------------------------------<br>
> <br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
<br>
</blockquote></div>