<div dir="auto">Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.<br><br><div data-smartmail="gmail_signature"><br>Phillip Smith<br>954-558-5022</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Oct 7, 2023, 5:27 PM Phil Dibowitz <<a href="mailto:phil@ipom.com">phil@ipom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">re-sending with Smith on CC.<br>
<br>
On 10/7/23 11:38, Phil Dibowitz wrote:<br>
> Restarted. Added cron to restart. Will find time to pair debug with <br>
> Smith this week.<br>
> <br>
> Get BlueMail for Android <<a href="https://bluemail.me" rel="noreferrer noreferrer" target="_blank">https://bluemail.me</a>><br>
> On Oct 7, 2023, at 1:50 AM, Ilan Rabinovitch <<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a> <br>
> <mailto:<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a>>> wrote:<br>
> <br>
>     Looks like we were down again for a few hours this morning.<br>
> <br>
>     On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch<br>
>     <<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a>> wrote:<br>
> <br>
> <br>
>         (resending with offlist folks)<br>
> <br>
>         Looks like we had another outage this morning. I'm online for<br>
>         the next<br>
>         45-60 min if someone wants to chat about it. Otherwise, I would<br>
>         greatly appreciate it if someone on the infra team could keep an eye<br>
>         on monitoring and at least restart things.<br>
> <br>
>         On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
>             Phil Smith is probably the best person since it seems to be<br>
>             a Drupal<br>
>             issue, best I can tell (I scoured logs a few months back and<br>
>             found<br>
>             nothing other than what you just described: apache threads<br>
>             just stay<br>
>             around, busy, until they eat up all the memory).<br>
> <br>
>             I'm not sure he's on this list so I'm CC'ing hom.<br>
> <br>
>             Phil - I'm happy to get you root access (send me an SSH<br>
>             pubkey), or<br>
>             anything else you may not already have to help you debug. I<br>
>             spent quite<br>
>             a while last SCALE trying to figure this out, Ilan has spent<br>
>             time on<br>
>             this, and it seems to be drupal-specific, best I can tell.<br>
> <br>
> <br>
> <br>
>             On 10/5/23 14:49, Ilan Rabinovitch wrote:<br>
> <br>
>                 Theres definitely something continuing to go on with<br>
>                 drupal and/or<br>
>                 httpd. Something continues to eat up all the worker<br>
>                 threads from<br>
>                 httpd. Before we set maxworkers to something reasonable<br>
>                 it would eat<br>
>                 up all the memory, now it just consumes all the workers<br>
>                 and the site<br>
>                 becomes unresponsive.<br>
>                 I dont think this relates to the upgrade, as it was<br>
>                 happening before.<br>
> <br>
>                 We've had the site down three times in the past 2 days<br>
>                 as a result.<br>
>                 Monitoring detected it and I was able to respond to it<br>
>                 fairly quickly,<br>
>                 but I'm about to board a 30 hour flight, so I'm likely<br>
>                 not going to be<br>
>                 able to respond moving forward.<br>
> <br>
>                 Given the importance of the CFP this month I hope<br>
>                 someone has a chance<br>
>                 to dig in.<br>
> <br>
>                 On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz<br>
>                 <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
>                     Copy that, I'll keep an eye out. Nice work, and thanks!<br>
> <br>
>                     On 10/3/23 18:06, Ilan Rabinovitch wrote:<br>
> <br>
>                         This is done. Here's the specific ids for any<br>
>                         roll backs:<br>
> <br>
>                         1) start scale-web1 up again (i-017df11978f41d9ae)<br>
>                         2) sync<br>
>                         /home/drupal/scale-drupal/httpdocs/sites/default/files/ from<br>
>                         scale-web2 (i-0bf1dd453774f2c3b) to scale-web1<br>
>                         (i-017df11978f41d9ae)<br>
>                         3) move EIP (23.21.71.118 <<a href="http://23.21.71.118" rel="noreferrer noreferrer" target="_blank">http://23.21.71.118</a>>)<br>
>                         from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.<br>
> <br>
>                         Assuming no issues the next change to think<br>
>                         about is:<br>
>                         - enable the s3fs plugin work Phil Smith did<br>
>                         recently so static files<br>
>                         aren't stored on box anymore.<br>
>                         - move scale-web2 behind an ELB and auto scaling<br>
>                         group.<br>
>                         - remove the cron jobs that back up static files<br>
> <br>
>                         This will enable some reasonable failover that<br>
>                         doesn't require human<br>
>                         intervention.<br>
> <br>
> <br>
>                         On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz<br>
>                         <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
>                             Easy-peasey. Feel free to do it as your<br>
>                             leisure today. I have a few<br>
>                             errands to run, but will never be more than<br>
>                             30m from home and will be<br>
>                             able to roll back today, and will be around<br>
>                             all day tomorrow and Thursday.<br>
> <br>
> <br>
>                             On 10/3/23 16:08, Ilan Rabinovitch wrote:<br>
> <br>
>                                 Cutover:<br>
>                                 - RDS Database snapshot<br>
>                                 - sync<br>
>                                 /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both<br>
>                                 hosts in case users have uploaded static<br>
>                                 assets<br>
>                                 - Move EIP from scale-web1 host to<br>
>                                 scale-web2<br>
> <br>
>                                 Rollback:<br>
>                                 - re-sync<br>
>                                 /home/drupal/scale-drupal/httpdocs/sites/default/files/ on<br>
>                                 both hosts in case users have uploaded<br>
>                                 static assets<br>
>                                 - Move EIP from scale-web2 host to<br>
>                                 scale-web1<br>
> <br>
> <br>
> <br>
> <br>
> <br>
>                                 On Tue, Oct 3, 2023 at 3:41 PM Phil<br>
>                                 Dibowitz <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
>                                     Feel free to do it today.<br>
> <br>
>                                     What's the migrate/failback process?<br>
>                                     It's somewhere in cloudflair?<br>
> <br>
>                                     If not I can do it next week.<br>
> <br>
> <br>
>                                     On 10/3/23 11:09, Ilan Rabinovitch<br>
>                                     wrote:<br>
> <br>
>                                         Sounds like we should do this<br>
>                                         today then, or you can grab next<br>
>                                         week.<br>
>                                         I'm going to be offline until<br>
>                                         November after Thursday morning.<br>
> <br>
>                                         On Mon, Oct 2, 2023 at 9:35 PM<br>
>                                         Phil Dibowitz <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
>                                             I'm out of town Fri-Sun, and<br>
>                                             will only be around a laptop<br>
>                                             sporadically.<br>
>                                             I'm in town all week next<br>
>                                             week though.<br>
> <br>
> <br>
> <br>
>                                             On 10/2/23 13:22, Ilan<br>
>                                             Rabinovitch wrote:<br>
> <br>
>                                                 Phil Smith, myself and a<br>
>                                                 few others have banged<br>
>                                                 on scale-web2 and we<br>
>                                                 think we're all set to<br>
>                                                 do a cutover there.<br>
>                                                 However, I'm going to be<br>
>                                                 out of town for a few<br>
>                                                 weeks starting Thursday<br>
>                                                 and do not yet know what<br>
>                                                 my connectivity will be<br>
>                                                 like.<br>
> <br>
>                                                 Would love to cutover<br>
>                                                 this week, but only want<br>
>                                                 to do it if others will<br>
>                                                 be available to roll<br>
>                                                 back to the old server<br>
>                                                 should something go awry<br>
>                                                 while I'm on vacation.<br>
> <br>
>                                                 On Sun, Sep 24, 2023 at<br>
>                                                 9:46 PM Ilan Rabinovitch<br>
>                                                 <<a href="mailto:ilan@socallinuxexpo.org" target="_blank" rel="noreferrer">ilan@socallinuxexpo.org</a>> wrote:<br>
> <br>
> <br>
>                                                     I believe Ive<br>
>                                                     addressed everyone's<br>
>                                                     feedback.<br>
> <br>
>                                                     scale-web2 is online<br>
>                                                     on running off the<br>
>                                                     scale_web_centos8<br>
>                                                     branch.<br>
>                                                     it's using a test db<br>
>                                                     running from a<br>
>                                                     snapshot of prod 2~<br>
>                                                     weeks ago.<br>
>                                                     you can change it as<br>
>                                                     you see fit without<br>
>                                                     worrying about<br>
>                                                     impacting production.<br>
> <br>
>                                                     To test you'll need<br>
>                                                     to update /etc/hosts<br>
>                                                     to point at the new<br>
>                                                     server.<br>
>                                                     following to your<br>
>                                                     /etc/hosts file.<br>
> <br>
>                                                     54.205.247.164<br>
>                                                     <<a href="http://54.205.247.164" rel="noreferrer noreferrer" target="_blank">http://54.205.247.164</a>> <a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">www.socallinuxexpo.org</a> <<a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">http://www.socallinuxexpo.org</a>><br>
> <br>
>                                                     Ideally would like<br>
>                                                     to merge and migrate<br>
>                                                     product this week<br>
> <br>
> <br>
>                                                     On Fri, Sep 22, 2023<br>
>                                                     at 7:19 PM Phil<br>
>                                                     Dibowitz<br>
>                                                     <<a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a>> wrote:<br>
> <br>
> <br>
>                                                         Sorry I was in<br>
>                                                         Mexico.<br>
> <br>
>                                                         Left a review to<br>
>                                                         DRY it up. But I<br>
>                                                         don't see any<br>
>                                                         critical things.<br>
> <br>
>                                                         - Phil<br>
> <br>
>                                                         On 9/22/23<br>
>                                                         15:23, Ilan<br>
>                                                         Rabinovitch wrote:<br>
> <br>
>                                                             Ping? Would<br>
>                                                             like to wrap<br>
>                                                             this up as<br>
>                                                             the non-stop<br>
>                                                             daily<br>
>                                                             outages are no<br>
>                                                             fun, Managed<br>
>                                                             to mitigate<br>
>                                                             it somewhat,<br>
>                                                             but given<br>
>                                                             none of us<br>
>                                                             are on call<br>
>                                                             getting to<br>
>                                                             some<br>
>                                                             reasonable<br>
>                                                             sense of HA<br>
>                                                             would be<br>
>                                                             appreciated<br>
>                                                             so folks<br>
>                                                             can go on<br>
>                                                             vacation<br>
>                                                             without<br>
>                                                             worrying<br>
>                                                             about who<br>
>                                                             will reboot<br>
>                                                             the servers.<br>
> <br>
>                                                             On Tue, Sep<br>
>                                                             19, 2023 at<br>
>                                                             8:59 PM Ilan<br>
>                                                             Rabinovitch<br>
>                                                             <<a href="mailto:ilan@linuxfests.org" target="_blank" rel="noreferrer">ilan@linuxfests.org</a>> wrote:<br>
> <br>
> <br>
>                                                                 Thanks<br>
>                                                                 Davide<br>
>                                                                 for the<br>
>                                                                 initial<br>
>                                                                 review.<br>
>                                                                 I think<br>
>                                                                 I<br>
>                                                                 addressed the<br>
>                                                                 remaining feedback.<br>
> <br>
>                                                                 Anything<br>
>                                                                 else we<br>
>                                                                 want to<br>
>                                                                 test<br>
>                                                                 before<br>
>                                                                 merging?<br>
> <br>
>                                                                 On Fri,<br>
>                                                                 Sep 15,<br>
>                                                                 2023 at<br>
>                                                                 7:52 PM<br>
>                                                                 Ilan<br>
>                                                                 Rabinovitch <<a href="mailto:ilan@linuxfests.org" target="_blank" rel="noreferrer">ilan@linuxfests.org</a>> wrote:<br>
> <br>
> <br>
>                                                                     Hi,<br>
> <br>
>                                                                     I<br>
>                                                                     took<br>
>                                                                     a<br>
>                                                                     stab<br>
>                                                                     at<br>
>                                                                     updating the cookbooks we use for the website to use<br>
>                                                                     centos8. Motivation was getting a new version of php so we can use<br>
>                                                                     some<br>
>                                                                     drupal plugins that dont support php5.4, but i think this was on<br>
>                                                                     the<br>
>                                                                     backlog already for a while.<br>
> <br>
>                                                                     <a href="https://github.com/socallinuxexpo/scale-chef/pull/283" rel="noreferrer noreferrer" target="_blank">https://github.com/socallinuxexpo/scale-chef/pull/283</a> <<a href="https://github.com/socallinuxexpo/scale-chef/pull/283" rel="noreferrer noreferrer" target="_blank">https://github.com/socallinuxexpo/scale-chef/pull/283</a>><br>
> <br>
>                                                                     To<br>
>                                                                     test<br>
>                                                                     this<br>
>                                                                     out,<br>
>                                                                     I<br>
>                                                                     spun<br>
>                                                                     up<br>
>                                                                     scale-web2 with a clone of the production<br>
>                                                                     scale-drupal database, and then pointed my host file at it for<br>
>                                                                     <a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">www.socallinuxexpo.org</a> <<a href="http://www.socallinuxexpo.org" rel="noreferrer noreferrer" target="_blank">http://www.socallinuxexpo.org</a>>:<br>
> <br>
>                                                                     I<br>
>                                                                     then<br>
>                                                                     tested the following:<br>
> <br>
>                                                                     -<br>
>                                                                     chef<br>
>                                                                     runs<br>
>                                                                     complete end to end<br>
>                                                                     -<br>
>                                                                     Backing up / restoring static assets (see backup scripts in<br>
>                                                                     /usr/local/bin or in the scale-drupal cookbook)<br>
>                                                                     -<br>
>                                                                     browsing legacy static website (eg<br>
>                                                                     <a href="https://socallinuxexpo.org/past/2002" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2002</a> <<a href="https://socallinuxexpo.org/past/2002" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2002</a>>/<br>
>                                                                     <a href="https://socallinuxexpo.org/past/2003" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2003</a> <<a href="https://socallinuxexpo.org/past/2003" rel="noreferrer noreferrer" target="_blank">https://socallinuxexpo.org/past/2003</a>>/)<br>
>                                                                     -<br>
>                                                                     Registering as a speaker, confirming that I got the email and that<br>
>                                                                     my<br>
>                                                                     account worked.<br>
>                                                                     -<br>
>                                                                     publishing the submitted talk<br>
>                                                                     -<br>
>                                                                     creating sponsors/exhibitors/blog post/events/etc<br>
> <br>
>                                                                     Pending things:<br>
>                                                                     -<br>
>                                                                     Centos8 seems to default to using php-fpm instead of mod_php. Not<br>
>                                                                     opposed but it's different. Seems to work fine, but I've not load<br>
>                                                                     tested it or anything.<br>
> <br>
>                                                                     -<br>
>                                                                     drush (<a href="https://www.drush.org" rel="noreferrer noreferrer" target="_blank">https://www.drush.org</a> <<a href="https://www.drush.org" rel="noreferrer noreferrer" target="_blank">https://www.drush.org</a>>/) packages no longer seemed to be<br>
>                                                                     present in centos or epel. remi seems to have it but the drush docs<br>
>                                                                     suggest we should be installing this in our drupal code base via<br>
>                                                                     composer instead of as a package. I dont have strong opinions, but we<br>
>                                                                     should figure out the right path before merging.<br>
> <br>
>                                                                     Cheers,<br>
>                                                                     Ilan<br>
> <br>
>                                                             ------------------------------------------------------------------------<br>
> <br>
>                                                             scale-infra<br>
>                                                             mailing list<br>
>                                                             <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                                                             <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
>                                                         --<br>
>                                                         Phil Dibowitz<br>
>                                                         <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
>                                                         Open Source<br>
>                                                         software and<br>
>                                                         tech docs<br>
>                                                         Insanity Palace<br>
>                                                         of Metallica<br>
>                                                         <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/ <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
>                                                         "Be who you are<br>
>                                                         and say what you<br>
>                                                         feel, because<br>
>                                                         those who mind don't<br>
>                                                         matter and those<br>
>                                                         who matter don't<br>
>                                                         mind."<br>
>                                                         - Dr. Seuss<br>
> <br>
> <br>
>                                                         ------------------------------------------------------------------------<br>
> <br>
>                                                         scale-infra<br>
>                                                         mailing list<br>
>                                                         <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                                                         <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
>                                                 ------------------------------------------------------------------------<br>
> <br>
>                                                 scale-infra mailing list<br>
>                                                 <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                                                 <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
>                                             --<br>
>                                             Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
>                                             Open Source software and<br>
>                                             tech docs Insanity Palace of<br>
>                                             Metallica<br>
>                                             <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a><br>
>                                             <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
>                                             <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a><br>
>                                             <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
>                                             "Be who you are and say what<br>
>                                             you feel, because those who<br>
>                                             mind don't<br>
>                                             matter and those who matter<br>
>                                             don't mind."<br>
>                                             - Dr. Seuss<br>
> <br>
> <br>
>                                             ------------------------------------------------------------------------<br>
> <br>
>                                             scale-infra mailing list<br>
>                                             <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                                             <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
>                                         ------------------------------------------------------------------------<br>
> <br>
>                                         scale-infra mailing list<br>
>                                         <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                                         <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
>                                     --<br>
>                                     Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
>                                     Open Source software and tech docs<br>
>                                     Insanity Palace of Metallica<br>
>                                     <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a><br>
>                                     <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
>                                     <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a><br>
>                                     <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
>                                     "Be who you are and say what you<br>
>                                     feel, because those who mind don't<br>
>                                     matter and those who matter don't mind."<br>
>                                     - Dr. Seuss<br>
> <br>
> <br>
>                                     ------------------------------------------------------------------------<br>
> <br>
>                                     scale-infra mailing list<br>
>                                     <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                                     <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
>                                 ------------------------------------------------------------------------<br>
> <br>
>                                 scale-infra mailing list<br>
>                                 <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                                 <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
>                             --<br>
>                             Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
>                             Open Source software and tech docs Insanity<br>
>                             Palace of Metallica<br>
>                             <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a><br>
>                             <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
>                             <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
>                             "Be who you are and say what you feel,<br>
>                             because those who mind don't<br>
>                             matter and those who matter don't mind."<br>
>                             - Dr. Seuss<br>
> <br>
> <br>
>                             ------------------------------------------------------------------------<br>
> <br>
>                             scale-infra mailing list<br>
>                             <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                             <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
>                         ------------------------------------------------------------------------<br>
> <br>
>                         scale-infra mailing list<br>
>                         <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                         <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
>                     --<br>
>                     Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
>                     Open Source software and tech docs Insanity Palace<br>
>                     of Metallica<br>
>                     <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
>                     <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
>                     "Be who you are and say what you feel, because those<br>
>                     who mind don't<br>
>                     matter and those who matter don't mind."<br>
>                     - Dr. Seuss<br>
> <br>
> <br>
>                     ------------------------------------------------------------------------<br>
> <br>
>                     scale-infra mailing list<br>
>                     <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                     <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
>                 ------------------------------------------------------------------------<br>
> <br>
>                 scale-infra mailing list<br>
>                 <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>                 <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
>             --<br>
>             Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank" rel="noreferrer">phil@ipom.com</a><br>
>             Open Source software and tech docs Insanity Palace of Metallica<br>
>             <a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a> <<a href="http://www.phildev.net" rel="noreferrer noreferrer" target="_blank">http://www.phildev.net</a>>/<br>
>             <a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a> <<a href="http://www.ipom.com" rel="noreferrer noreferrer" target="_blank">http://www.ipom.com</a>>/<br>
> <br>
>             "Be who you are and say what you feel, because those who<br>
>             mind don't<br>
>             matter and those who matter don't mind."<br>
>             - Dr. Seuss<br>
> <br>
> <br>
>             ------------------------------------------------------------------------<br>
> <br>
>             scale-infra mailing list<br>
>             <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>             <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
>     ------------------------------------------------------------------------<br>
> <br>
>     scale-infra mailing list<br>
>     <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank" rel="noreferrer">scale-infra@lists.linuxfests.org</a><br>
>     <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>  <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
<br>
</blockquote></div>