<div dir="ltr">Looks like we had another outage for about an hour today. 10am - 11:36am PT. </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 8, 2024 at 9:53 AM Phil Dibowitz <<a href="mailto:phil@ipom.com">phil@ipom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">No, I woke up, saw the alert, bounced httpd, and had to run out the door.<br>
<br>
It's a reasonable guess that it's the same memory issue as before. We're <br>
currently bouncing apache in cron every-other-hour, which seems to <br>
mostly keep us up, but occasionally not, so my guess is that's RIGHT at <br>
the threshold.<br>
<br>
I don't have the bandwidth the go spelunking - the httpd.conf <br>
adjustments I made in... November or whatever that was seemed to help, <br>
but obviously they didn't solve everything. I still suspect that Drupal <br>
is causing connections to be held open somewhere, but that's pure <br>
speculation.<br>
<br>
We have the data, between the apache_status.log and DD to figure it out, <br>
I'd just need time to do it, and I don't have the cycles right now.<br>
<br>
On 1/7/24 17:11, Ilan Rabinovitch wrote:<br>
> Looks like we had another outage today. Any idea whats up?<br>
> <br>
> On Mon, Dec 25, 2023 at 1:17 AM Ilan Rabinovitch <<a href="mailto:ilan@linuxfests.org" target="_blank">ilan@linuxfests.org</a> <br>
> <mailto:<a href="mailto:ilan@linuxfests.org" target="_blank">ilan@linuxfests.org</a>>> wrote:<br>
> <br>
> Looks like another outage about 2 hours ago.<br>
> <br>
> On Sun, Dec 24, 2023 at 8:45 PM Ilan Rabinovitch<br>
> <<a href="mailto:ilan@linuxfests.org" target="_blank">ilan@linuxfests.org</a> <mailto:<a href="mailto:ilan@linuxfests.org" target="_blank">ilan@linuxfests.org</a>>> wrote:<br>
> <br>
> <br>
> <br>
> website was down again.. looks like for about an hour. ive just<br>
> restarted httpd.<br>
> <br>
> On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz <<a href="mailto:phil@ipom.com" target="_blank">phil@ipom.com</a><br>
> <mailto:<a href="mailto:phil@ipom.com" target="_blank">phil@ipom.com</a>>> wrote:<br>
> <br>
> Logs look good.<br>
> <br>
> <a href="https://github.com/socallinuxexpo/scale-chef/pull/302" rel="noreferrer" target="_blank">https://github.com/socallinuxexpo/scale-chef/pull/302</a><br>
> <<a href="https://github.com/socallinuxexpo/scale-chef/pull/302" rel="noreferrer" target="_blank">https://github.com/socallinuxexpo/scale-chef/pull/302</a>><br>
> drops restarts to<br>
> every 2 hours. Merged.<br>
> <br>
> On 10/21/23 13:47, Phil Dibowitz wrote:<br>
> > I dropped it to less often a few days ago, I haven't yet<br>
> looked at the<br>
> > logs to see what the status of httpd is at those times.<br>
> My plan is to do<br>
> > that, then drop it to less often, and rinse and repeat.<br>
> ><br>
> > Sorry new job is keeping me super busy.<br>
> ><br>
> > On 10/21/23 08:25, Ilan Rabinovitch wrote:<br>
> >> Are we able to remove the cron job that's restarting httpd?<br>
> >><br>
> >> On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz<br>
> <<a href="mailto:phil@ipom.com" target="_blank">phil@ipom.com</a> <mailto:<a href="mailto:phil@ipom.com" target="_blank">phil@ipom.com</a>>> wrote:<br>
> >>><br>
> >>> On 10/7/23 14:30, Phillip Smith wrote:<br>
> >>>> Yes, let's schedule something. I'm available tomorrow<br>
> morning and<br>
> >>>> Tuesday evening.<br>
> >>><br>
> >>> What can I provide you in the mean time? I thought I<br>
> saw an email from<br>
> >>> you saying you wanted Datadog access, but I can't seem<br>
> to find it.<br>
> >>><br>
> >>> Davide may have time Tuesday, I will have time Wed<br>
> evening. I'm out of<br>
> >>> town until Monday and have plans Mon/Tues evening.<br>
> >>><br>
> >>> - Phil<br>
> >>><br>
> >>><br>
> >>> _______________________________________________<br>
> >>> scale-infra mailing list<br>
> >>> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a><br>
> <mailto:<a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a>><br>
> >>><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> >> _______________________________________________<br>
> >> scale-infra mailing list<br>
> >> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a><br>
> <mailto:<a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a>><br>
> >><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> ><br>
> <br>
> -- <br>
> Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank">phil@ipom.com</a> <mailto:<a href="mailto:phil@ipom.com" target="_blank">phil@ipom.com</a>><br>
> Open Source software and tech docs Insanity Palace of<br>
> Metallica<br>
> <a href="http://www.phildev.net/" rel="noreferrer" target="_blank">http://www.phildev.net/</a> <<a href="http://www.phildev.net/" rel="noreferrer" target="_blank">http://www.phildev.net/</a>><br>
> <a href="http://www.ipom.com/" rel="noreferrer" target="_blank">http://www.ipom.com/</a> <<a href="http://www.ipom.com/" rel="noreferrer" target="_blank">http://www.ipom.com/</a>><br>
> <br>
> "Be who you are and say what you feel, because those who<br>
> mind don't<br>
> matter and those who matter don't mind."<br>
> - Dr. Seuss<br>
> <br>
> <br>
> _______________________________________________<br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a><br>
> <mailto:<a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a>><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a> <<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a>><br>
> <br>
> <br>
> _______________________________________________<br>
> scale-infra mailing list<br>
> <a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a><br>
> <a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br>
<br>
-- <br>
Phil Dibowitz <a href="mailto:phil@ipom.com" target="_blank">phil@ipom.com</a><br>
Open Source software and tech docs Insanity Palace of Metallica<br>
<a href="http://www.phildev.net/" rel="noreferrer" target="_blank">http://www.phildev.net/</a> <a href="http://www.ipom.com/" rel="noreferrer" target="_blank">http://www.ipom.com/</a><br>
<br>
"Be who you are and say what you feel, because those who mind don't<br>
matter and those who matter don't mind."<br>
- Dr. Seuss<br>
<br>
<br>
_______________________________________________<br>
scale-infra mailing list<br>
<a href="mailto:scale-infra@lists.linuxfests.org" target="_blank">scale-infra@lists.linuxfests.org</a><br>
<a href="https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra" rel="noreferrer" target="_blank">https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra</a><br>
</blockquote></div>