On Sat, Oct 7, 2023, 5:27 PM Phil Dibowitz <phil@ipom.com> wrote:

re-sending with Smith on CC.

On 10/7/23 11:38, Phil Dibowitz wrote:
> Restarted. Added cron to restart. Will find time to pair debug with
> Smith this week.
>
> Get BlueMail for Android <https://bluemail.me>
> On Oct 7, 2023, at 1:50 AM, Ilan Rabinovitch <ilan@socallinuxexpo.org
> <mailto:ilan@socallinuxexpo.org>> wrote:
>
> Looks like we were down again for a few hours this morning.
>
> On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch
> <ilan@socallinuxexpo.org> wrote:
>
>
> (resending with offlist folks)
>
> Looks like we had another outage this morning. I'm online for
> the next
> 45-60 min if someone wants to chat about it. Otherwise, I would
> greatly appreciate it if someone on the infra team could keep an eye
> on monitoring and at least restart things.
>
> On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz <phil@ipom.com> wrote:
>
>
> Phil Smith is probably the best person since it seems to be
> a Drupal
> issue, best I can tell (I scoured logs a few months back and
> found
> nothing other than what you just described: apache threads
> just stay
> around, busy, until they eat up all the memory).
>
> I'm not sure he's on this list so I'm CC'ing hom.
>
> Phil - I'm happy to get you root access (send me an SSH
> pubkey), or
> anything else you may not already have to help you debug. I
> spent quite
> a while last SCALE trying to figure this out, Ilan has spent
> time on
> this, and it seems to be drupal-specific, best I can tell.
>
>
>
> On 10/5/23 14:49, Ilan Rabinovitch wrote:
>
> Theres definitely something continuing to go on with
> drupal and/or
> httpd. Something continues to eat up all the worker
> threads from
> httpd. Before we set maxworkers to something reasonable
> it would eat
> up all the memory, now it just consumes all the workers
> and the site
> becomes unresponsive.
> I dont think this relates to the upgrade, as it was
> happening before.
>
> We've had the site down three times in the past 2 days
> as a result.
> Monitoring detected it and I was able to respond to it
> fairly quickly,
> but I'm about to board a 30 hour flight, so I'm likely
> not going to be
> able to respond moving forward.
>
> Given the importance of the CFP this month I hope
> someone has a chance
> to dig in.
>
> On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz
> <phil@ipom.com> wrote:
>
>
> Copy that, I'll keep an eye out. Nice work, and thanks!
>
> On 10/3/23 18:06, Ilan Rabinovitch wrote:
>
> This is done. Here's the specific ids for any
> roll backs:
>
> 1) start scale-web1 up again (i-017df11978f41d9ae)
> 2) sync
> /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
> scale-web2 (i-0bf1dd453774f2c3b) to scale-web1
> (i-017df11978f41d9ae)
> 3) move EIP (23.21.71.118 <http://23.21.71.118>)
> from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
>
> Assuming no issues the next change to think
> about is:
> - enable the s3fs plugin work Phil Smith did
> recently so static files
> aren't stored on box anymore.
> - move scale-web2 behind an ELB and auto scaling
> group.
> - remove the cron jobs that back up static files
>
> This will enable some reasonable failover that
> doesn't require human
> intervention.
>
>
> On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz
> <phil@ipom.com> wrote:
>
>
> Easy-peasey. Feel free to do it as your
> leisure today. I have a few
> errands to run, but will never be more than
> 30m from home and will be
> able to roll back today, and will be around
> all day tomorrow and Thursday.
>
>
> On 10/3/23 16:08, Ilan Rabinovitch wrote:
>
> Cutover:
> - RDS Database snapshot
> - sync
> /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
> hosts in case users have uploaded static
> assets
> - Move EIP from scale-web1 host to
> scale-web2
>
> Rollback:
> - re-sync
> /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
> both hosts in case users have uploaded
> static assets
> - Move EIP from scale-web2 host to
> scale-web1
>
>
>
>
>
> On Tue, Oct 3, 2023 at 3:41 PM Phil
> Dibowitz <phil@ipom.com> wrote:
>
>
> Feel free to do it today.
>
> What's the migrate/failback process?
> It's somewhere in cloudflair?
>
> If not I can do it next week.
>
>
> On 10/3/23 11:09, Ilan Rabinovitch
> wrote:
>
> Sounds like we should do this
> today then, or you can grab next
> week.
> I'm going to be offline until
> November after Thursday morning.
>
> On Mon, Oct 2, 2023 at 9:35 PM
> Phil Dibowitz <phil@ipom.com> wrote:
>
>
> I'm out of town Fri-Sun, and
> will only be around a laptop
> sporadically.
> I'm in town all week next
> week though.
>
>
>
> On 10/2/23 13:22, Ilan
> Rabinovitch wrote:
>
> Phil Smith, myself and a
> few others have banged
> on scale-web2 and we
> think we're all set to
> do a cutover there.
> However, I'm going to be
> out of town for a few
> weeks starting Thursday
> and do not yet know what
> my connectivity will be
> like.
>
> Would love to cutover
> this week, but only want
> to do it if others will
> be available to roll
> back to the old server
> should something go awry
> while I'm on vacation.
>
> On Sun, Sep 24, 2023 at
> 9:46 PM Ilan Rabinovitch
> <ilan@socallinuxexpo.org> wrote:
>
>
> I believe Ive
> addressed everyone's
> feedback.
>
> scale-web2 is online
> on running off the
> scale_web_centos8
> branch.
> it's using a test db
> running from a
> snapshot of prod 2~
> weeks ago.
> you can change it as
> you see fit without
> worrying about
> impacting production.
>
> To test you'll need
> to update /etc/hosts
> to point at the new
> server.
> following to your
> /etc/hosts file.
>
> 54.205.247.164
> <http://54.205.247.164> www.socallinuxexpo.org <http://www.socallinuxexpo.org>
>
> Ideally would like
> to merge and migrate
> product this week
>
>
> On Fri, Sep 22, 2023
> at 7:19 PM Phil
> Dibowitz
> <phil@ipom.com> wrote:
>
>
> Sorry I was in
> Mexico.
>
> Left a review to
> DRY it up. But I
> don't see any
> critical things.
>
> - Phil
>
> On 9/22/23
> 15:23, Ilan
> Rabinovitch wrote:
>
> Ping? Would
> like to wrap
> this up as
> the non-stop
> daily
> outages are no
> fun, Managed
> to mitigate
> it somewhat,
> but given
> none of us
> are on call
> getting to
> some
> reasonable
> sense of HA
> would be
> appreciated
> so folks
> can go on
> vacation
> without
> worrying
> about who
> will reboot
> the servers.
>
> On Tue, Sep
> 19, 2023 at
> 8:59 PM Ilan
> Rabinovitch
> <ilan@linuxfests.org> wrote:
>
>
> Thanks
> Davide
> for the
> initial
> review.
> I think
> I
> addressed the
> remaining feedback.
>
> Anything
> else we
> want to
> test
> before
> merging?
>
> On Fri,
> Sep 15,
> 2023 at
> 7:52 PM
> Ilan
> Rabinovitch <ilan@linuxfests.org> wrote:
>
>
> Hi,
>
> I
> took
> a
> stab
> at
> updating the cookbooks we use for the website to use
> centos8. Motivation was getting a new version of php so we can use
> some
> drupal plugins that dont support php5.4, but i think this was on
> the
> backlog already for a while.
>
> https://github.com/socallinuxexpo/scale-chef/pull/283 <https://github.com/socallinuxexpo/scale-chef/pull/283>
>
> To
> test
> this
> out,
> I
> spun
> up
> scale-web2 with a clone of the production
> scale-drupal database, and then pointed my host file at it for
> www.socallinuxexpo.org <http://www.socallinuxexpo.org>:
>
> I
> then
> tested the following:
>
> -
> chef
> runs
> complete end to end
> -
> Backing up / restoring static assets (see backup scripts in
> /usr/local/bin or in the scale-drupal cookbook)
> -
> browsing legacy static website (eg
> https://socallinuxexpo.org/past/2002 <https://socallinuxexpo.org/past/2002>/
> https://socallinuxexpo.org/past/2003 <https://socallinuxexpo.org/past/2003>/)
> -
> Registering as a speaker, confirming that I got the email and that
> my
> account worked.
> -
> publishing the submitted talk
> -
> creating sponsors/exhibitors/blog post/events/etc
>
> Pending things:
> -
> Centos8 seems to default to using php-fpm instead of mod_php. Not
> opposed but it's different. Seems to work fine, but I've not load
> tested it or anything.
>
> -
> drush (https://www.drush.org <https://www.drush.org>/) packages no longer seemed to be
> present in centos or epel. remi seems to have it but the drush docs
> suggest we should be installing this in our drupal code base via
> composer instead of as a package. I dont have strong opinions, but we
> should figure out the right path before merging.
>
> Cheers,
> Ilan
>
> ------------------------------------------------------------------------
>
> scale-infra
> mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
>
> --
> Phil Dibowitz
> phil@ipom.com
> Open Source
> software and
> tech docs
> Insanity Palace
> of Metallica
> http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/
>
> "Be who you are
> and say what you
> feel, because
> those who mind don't
> matter and those
> who matter don't
> mind."
> - Dr. Seuss
>
>
> ------------------------------------------------------------------------
>
> scale-infra
> mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
>
> --
> Phil Dibowitz phil@ipom.com
> Open Source software and
> tech docs Insanity Palace of
> Metallica
> http://www.phildev.net
> <http://www.phildev.net>/
> http://www.ipom.com
> <http://www.ipom.com>/
>
> "Be who you are and say what
> you feel, because those who
> mind don't
> matter and those who matter
> don't mind."
> - Dr. Seuss
>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
>
> --
> Phil Dibowitz phil@ipom.com
> Open Source software and tech docs
> Insanity Palace of Metallica
> http://www.phildev.net
> <http://www.phildev.net>/
> http://www.ipom.com
> <http://www.ipom.com>/
>
> "Be who you are and say what you
> feel, because those who mind don't
> matter and those who matter don't mind."
> - Dr. Seuss
>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
>
> --
> Phil Dibowitz phil@ipom.com
> Open Source software and tech docs Insanity
> Palace of Metallica
> http://www.phildev.net
> <http://www.phildev.net>/
> http://www.ipom.com <http://www.ipom.com>/
>
> "Be who you are and say what you feel,
> because those who mind don't
> matter and those who matter don't mind."
> - Dr. Seuss
>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
>
> --
> Phil Dibowitz phil@ipom.com
> Open Source software and tech docs Insanity Palace
> of Metallica
> http://www.phildev.net <http://www.phildev.net>/
> http://www.ipom.com <http://www.ipom.com>/
>
> "Be who you are and say what you feel, because those
> who mind don't
> matter and those who matter don't mind."
> - Dr. Seuss
>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
>
> --
> Phil Dibowitz phil@ipom.com
> Open Source software and tech docs Insanity Palace of Metallica
> http://www.phildev.net <http://www.phildev.net>/
> http://www.ipom.com <http://www.ipom.com>/
>
> "Be who you are and say what you feel, because those who
> mind don't
> matter and those who matter don't mind."
> - Dr. Seuss
>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>
> ------------------------------------------------------------------------
>
> scale-infra mailing list
> scale-infra@lists.linuxfests.org
> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
>