<html><head></head><body style="zoom: 0%;"><div dir="auto">Restarted. Added cron to restart. Will find time to pair debug with Smith this week.<br><br></div>
<div dir="auto"><!-- tmjah_g_1299s -->Get <!-- tmjah_g_1299e --><a href="https://bluemail.me"><!-- tmjah_g_1299s -->BlueMail for Android<!-- tmjah_g_1299e --></a><!-- tmjah_g_1299s --> <!-- tmjah_g_1299e --></div>
<div class="gmail_quote" >On Oct 7, 2023, at 1:50 AM, Ilan Rabinovitch <<a href="mailto:ilan@socallinuxexpo.org" target="_blank">ilan@socallinuxexpo.org</a>> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<pre class="blue">Looks like we were down again for a few hours this morning.<br><br>On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch<br><ilan@socallinuxexpo.org> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #729fcf; padding-left: 1ex;"><br> (resending with offlist folks)<br><br> Looks like we had another outage this morning. I'm online for the next<br> 45-60 min if someone wants to chat about it. Otherwise, I would<br> greatly appreciate it if someone on the infra team could keep an eye<br> on monitoring and at least restart things.<br><br> On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz <phil@ipom.com> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ad7fa8; padding-left: 1ex;"><br> Phil Smith is probably the best person since it seems to be a Drupal<br> issue, best I can tell (I scoured logs a few months back and found<br> nothing other than what you just described: apache threads just stay<br> around, busy, until they eat up all the memory).<br><br> I'm not sure he's on this list so I'm CC'ing hom.<br><br> Phil - I'm happy to get you root access (send me an SSH pubkey), or<br> anything else you may not already have to help you debug. I spent quite<br> a while last SCALE trying to figure this out, Ilan has spent time on<br> this, and it seems to be drupal-specific, best I can tell.<br><br><br><br> On 10/5/23 14:49, Ilan Rabinovitch wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #8ae234; padding-left: 1ex;"> Theres definitely something continuing to go on with drupal and/or<br> httpd. Something continues to eat up all the worker threads from<br> httpd. Before we set maxworkers to something reasonable it would eat<br> up all the memory, now it just consumes all the workers and the site<br> becomes unresponsive.<br> I dont think this relates to the upgrade, as it was happening before.<br><br> We've had the site down three times in the past 2 days as a result.<br> Monitoring detected it and I was able to respond to it fairly quickly,<br> but I'm about to board a 30 hour flight, so I'm likely not going to be<br> able to respond moving forward.<br><br> Given the importance of the CFP this month I hope someone has a chance<br> to dig in.<br><br> On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz <phil@ipom.com> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #fcaf3e; padding-left: 1ex;"><br> Copy that, I'll keep an eye out. Nice work, and thanks!<br><br> On 10/3/23 18:06, Ilan Rabinovitch wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #e9b96e; padding-left: 1ex;"> This is done. Here's the specific ids for any roll backs:<br><br> 1) start scale-web1 up again (i-017df11978f41d9ae)<br> 2) sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from<br> scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae)<br> 3) move EIP (<a href="http://23.21.71.118">23.21.71.118</a>) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.<br><br> Assuming no issues the next change to think about is:<br> - enable the s3fs plugin work Phil Smith did recently so static files<br> aren't stored on box anymore.<br> - move scale-web2 behind an ELB and auto scaling group.<br> - remove the cron jobs that back up static files<br><br> This will enable some reasonable failover that doesn't require human<br> intervention.<br><br><br> On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"><br> Easy-peasey. Feel free to do it as your leisure today. I have a few<br> errands to run, but will never be more than 30m from home and will be<br> able to roll back today, and will be around all day tomorrow and Thursday.<br><br><br> On 10/3/23 16:08, Ilan Rabinovitch wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"> Cutover:<br> - RDS Database snapshot<br> - sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both<br> hosts in case users have uploaded static assets<br> - Move EIP from scale-web1 host to scale-web2<br><br> Rollback:<br> - re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on<br> both hosts in case users have uploaded static assets<br> - Move EIP from scale-web2 host to scale-web1<br><br><br><br><br><br> On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz <phil@ipom.com> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"><br> Feel free to do it today.<br><br> What's the migrate/failback process? It's somewhere in cloudflair?<br><br> If not I can do it next week.<br><br><br> On 10/3/23 11:09, Ilan Rabinovitch wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"> Sounds like we should do this today then, or you can grab next week.<br> I'm going to be offline until November after Thursday morning.<br><br> On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz <phil@ipom.com> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"><br> I'm out of town Fri-Sun, and will only be around a laptop sporadically.<br> I'm in town all week next week though.<br><br><br><br> On 10/2/23 13:22, Ilan Rabinovitch wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"> Phil Smith, myself and a few others have banged on scale-web2 and we<br> think we're all set to do a cutover there.<br> However, I'm going to be out of town for a few weeks starting Thursday<br> and do not yet know what my connectivity will be like.<br><br> Would love to cutover this week, but only want to do it if others will<br> be available to roll back to the old server should something go awry<br> while I'm on vacation.<br><br> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch<br> <ilan@socallinuxexpo.org> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"><br> I believe Ive addressed everyone's feedback.<br><br> scale-web2 is online on running off the scale_web_centos8 branch.<br> it's using a test db running from a snapshot of prod 2~ weeks ago.<br> you can change it as you see fit without worrying about impacting production.<br><br> To test you'll need to update /etc/hosts to point at the new server.<br> following to your /etc/hosts file.<br><br> <a href="http://54.205.247.164">54.205.247.164</a> <a href="http://www.socallinuxexpo.org">www.socallinuxexpo.org</a><br><br> Ideally would like to merge and migrate product this week<br><br><br> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"><br> Sorry I was in Mexico.<br><br> Left a review to DRY it up. But I don't see any critical things.<br><br> - Phil<br><br> On 9/22/23 15:23, Ilan Rabinovitch wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"> Ping? Would like to wrap this up as the non-stop daily outages are no<br> fun, Managed to mitigate it somewhat, but given none of us are on call<br> getting to some reasonable sense of HA would be appreciated so folks<br> can go on vacation without worrying about who will reboot the servers.<br><br> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch <ilan@linuxfests.org> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"><br> Thanks Davide for the initial review. I think I addressed the<br> remaining feedback.<br><br> Anything else we want to test before merging?<br><br> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch <ilan@linuxfests.org> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 1ex 0.8ex; border-left: 1px solid #ccc; padding-left: 1ex;"><br> Hi,<br><br> I took a stab at updating the cookbooks we use for the website to use<br> centos8. Motivation was getting a new version of php so we can use<br> some drupal plugins that dont support php5.4, but i think this was on<br> the backlog already for a while.<br><br> <a href="https://github.com/socallinuxexpo/scale-chef/pull/283">https://github.com/socallinuxexpo/scale-chef/pull/283</a><br><br> To test this out, I spun up scale-web2 with a clone of the production<br> scale-drupal database, and then pointed my host file at it for<br> <a href="http://www.socallinuxexpo.org">www.socallinuxexpo.org</a>:<br><br> I then tested the following:<br><br> - chef runs complete end to end<br> - Backing up / restoring static assets (see backup scripts in<br> /usr/local/bin or in the scale-drupal cookbook)<br> - browsing legacy static website (eg<br> <a href="https://socallinuxexpo.org/past/2002">https://socallinuxexpo.org/past/2002</a>/<br> <a href="https://socallinuxexpo.org/past/2003">https://socallinuxexpo.org/past/2003</a>/)<br> - Registering as a speaker, confirming that I got the email and that<br> my account worked.<br> - publishing the submitted talk<br> - creating sponsors/exhibitors/blog post/events/etc<br><br> Pending things:<br> - Centos8 seems to default to using php-fpm instead of mod_php. Not<br> opposed but it's different. Seems to work fine, but I've not load<br> tested it or anything.<br><br> - drush (<a href="https://www.drush.org">https://www.drush.org</a>/) packages no longer seemed to be<br> present in centos or epel. remi seems to have it but the drush docs<br> suggest we should be installing this in our drupal code base via<br> composer instead of as a package. I dont have strong opinions, but we<br> should figure out the right path before merging.<br><br> Cheers,<br> Ilan<br></blockquote></blockquote><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><br> --<br> Phil Dibowitz phil@ipom.com<br> Open Source software and tech docs Insanity Palace of Metallica<br> <a href="http://www.phildev.net">http://www.phildev.net</a>/ <a href="http://www.ipom.com">http://www.ipom.com</a>/<br><br> "Be who you are and say what you feel, because those who mind don't<br> matter and those who matter don't mind."<br> - Dr. Seuss<br><br><br><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote></blockquote><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><br> --<br> Phil Dibowitz phil@ipom.com<br> Open Source software and tech docs Insanity Palace of Metallica<br> <a href="http://www.phildev.net">http://www.phildev.net</a>/ <a href="http://www.ipom.com">http://www.ipom.com</a>/<br><br> "Be who you are and say what you feel, because those who mind don't<br> matter and those who matter don't mind."<br> - Dr. Seuss<br><br><br><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><br> --<br> Phil Dibowitz phil@ipom.com<br> Open Source software and tech docs Insanity Palace of Metallica<br> <a href="http://www.phildev.net">http://www.phildev.net</a>/ <a href="http://www.ipom.com">http://www.ipom.com</a>/<br><br> "Be who you are and say what you feel, because those who mind don't<br> matter and those who matter don't mind."<br> - Dr. Seuss<br><br><br><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><br> --<br> Phil Dibowitz phil@ipom.com<br> Open Source software and tech docs Insanity Palace of Metallica<br> <a href="http://www.phildev.net">http://www.phildev.net</a>/ <a href="http://www.ipom.com">http://www.ipom.com</a>/<br><br> "Be who you are and say what you feel, because those who mind don't<br> matter and those who matter don't mind."<br> - Dr. Seuss<br><br><br><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><br> --<br> Phil Dibowitz phil@ipom.com<br> Open Source software and tech docs Insanity Palace of Metallica<br> <a href="http://www.phildev.net">http://www.phildev.net</a>/ <a href="http://www.ipom.com">http://www.ipom.com</a>/<br><br> "Be who you are and say what you feel, because those who mind don't<br> matter and those who matter don't mind."<br> - Dr. Seuss<br><br><br><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote><br> --<br> Phil Dibowitz phil@ipom.com<br> Open Source software and tech docs Insanity Palace of Metallica<br> <a href="http://www.phildev.net">http://www.phildev.net</a>/ <a href="http://www.ipom.com">http://www.ipom.com</a>/<br><br> "Be who you are and say what you feel, because those who mind don't<br> matter and those who matter don't mind."<br> - Dr. Seuss<br><br><br><hr><br> scale-infra mailing list<br> scale-infra@lists.linuxfests.org<br> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></blockquote></blockquote><hr><br>scale-infra mailing list<br>scale-infra@lists.linuxfests.org<br><a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br></pre></blockquote></div></body></html>