Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
Phillip Smith 954-558-5022
On Sat, Oct 7, 2023, 5:27 PM Phil Dibowitz phil@ipom.com wrote:
re-sending with Smith on CC.
On 10/7/23 11:38, Phil Dibowitz wrote:
Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android https://bluemail.me On Oct 7, 2023, at 1:50 AM, Ilan Rabinovitch <ilan@socallinuxexpo.org mailto:ilan@socallinuxexpo.org> wrote:
Looks like we were down again for a few hours this morning. On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch <ilan@socallinuxexpo.org> wrote: (resending with offlist folks) Looks like we had another outage this morning. I'm online for the next 45-60 min if someone wants to chat about it. Otherwise, I would greatly appreciate it if someone on the infra team could keep aneye
on monitoring and at least restart things. On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz <phil@ipom.com>wrote:
Phil Smith is probably the best person since it seems to be a Drupal issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just stay around, busy, until they eat up all the memory). I'm not sure he's on this list so I'm CC'ing hom. Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent quite a while last SCALE trying to figure this out, Ilan has spent time on this, and it seems to be drupal-specific, best I can tell. On 10/5/23 14:49, Ilan Rabinovitch wrote: Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before. We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward. Given the importance of the CFP this month I hope someone has a chance to dig in. On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz <phil@ipom.com> wrote: Copy that, I'll keep an eye out. Nice work, andthanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote: This is done. Here's the specific ids for any roll backs: 1) start scale-web1 up again(i-017df11978f41d9ae)
2) sync/home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118 <http://23.21.71.118>) from scale-web2 (i-0bf1dd453774f2c3b) toscale-web1.
Assuming no issues the next change to think about is: - enable the s3fs plugin work Phil Smith did recently so static files aren't stored on box anymore. - move scale-web2 behind an ELB and auto scaling group. - remove the cron jobs that back up static files This will enable some reasonable failover that doesn't require human intervention. On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com> wrote: Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday. On 10/3/23 16:08, Ilan Rabinovitch wrote: Cutover: - RDS Database snapshot - sync/home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets - Move EIP from scale-web1 host to scale-web2 Rollback: - re-sync/home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets - Move EIP from scale-web2 host to scale-web1 On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz <phil@ipom.com> wrote: Feel free to do it today. What's the migrate/failback process? It's somewhere in cloudflair? If not I can do it next week. On 10/3/23 11:09, Ilan Rabinovitch wrote: Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning. On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz <phil@ipom.com>wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though. On 10/2/23 13:22, Ilan Rabinovitch wrote: Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like. Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation. On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch <ilan@socallinuxexpo.org>wrote:
I believe Ive addressed everyone's feedback. scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production. To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file. 54.205.247.164 <http://54.205.247.164%3E www.socallinuxexpo.org < http://www.socallinuxexpo.org%3E
Ideally would like to merge and migrate product this week On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com>wrote:
Sorry I was in Mexico. Left a review to DRY it up. But I don't see any critical things. - Phil On 9/22/23 15:23, Ilan Rabinovitchwrote:
Ping? Would like to wrap this up as the non-stop daily outages areno
fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers. On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch <ilan@linuxfests.org> wrote:
Thanks Davide for the initial review. I think Iaddressed the
remaining feedback.
Anything else we want to test before merging? On Fri, Sep 15, 2023 at 7:52 PM IlanRabinovitch ilan@linuxfests.org wrote:
Hi, I took a stab atupdating the cookbooks we use for the website to use
centos8. Motivation was getting a new version of php so we can use
somedrupal plugins that dont support php5.4, but i think this was on
thebacklog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283 < https://github.com/socallinuxexpo/scale-chef/pull/283%3E
To test this out, I spun upscale-web2 with a clone of the production
scale-drupal database, and then pointed my host file at it for
www.socallinuxexpo.org http://www.socallinuxexpo.org:
I thentested the following:
- chef runscomplete end to end
-Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
-browsing legacy static website (eg
https://socallinuxexpo.org/past/2002 <https://socallinuxexpo.org/past/2002
/
https://socallinuxexpo.org/past/2003 <https://socallinuxexpo.org/past/2003
/) -
Registering as a speaker, confirming that I got the email and that
myaccount worked.
-publishing the submitted talk
-creating sponsors/exhibitors/blog post/events/etc
Pending things:
-Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load
tested it or anything.
-drush (https://www.drush.org https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs
suggest we should be installing this in our drupal code base via
composer instead of as a package. I dont have strong opinions, but we
should figure out the right path before merging.
Cheers,
Ilan
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallicahttp://www.phildev.net http://www.phildev.net/ http://www.ipom.com < http://www.ipom.com%3E/
"Be who you are and say what you feel, because those who minddon't
matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don'tmind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace ofMetallica
http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E