Investigating – I’m not able to find anything obvious in our logs, and it doesn’t look like requests from Jerboa are getting rejected per se. When it occurs again, could you try to access the same content via a browser and see if a message comes up from Cloudflare?

Paradoxically, Voyager is what I use myself, so it was allowlisted! There were a ton of rules added during the DDoS, so I’ve trimmed it down a bit 😅

Appreciate this… I’m wondering if there’s some kind of race condition that’s causing the DB (or your client) to retry posting. Will investigate once I’m back from a work trip early next week!

Good question, and I’ve not thought about it previously other than that we would remove content if the original artist requested us to do so. Though, we don’t have a formal list of DNP content, we just expect that users post content they’re permitted to or that is already otherwise shared on other sites (like e621).

I’m not against a DNP list, but I think I’d lean more towards respecting an existing DNP like https://e621.net/wiki_pages/85. Though, if we did end up getting to the point of having a lot of DNPs that don’t appear on their list, we’d probably move to making our own.

I’ll bring this up to the rest of the team too for feedback!

Phlox’s amazing smile absolutely sends me every time XD

I believe there was a change to some Lemmy communities where if the community itself is marked as “NSFW” there’s no ability to toggle it off anymore: https://github.com/LemmyNet/lemmy-ui/pull/2887 (https://join-lemmy.org/news/2025-04-08_-_Lemmy_Release_v0.19.11).

If the post isn’t flagging automatically as NSFW in a NSFW community, then there may be a backend bug that needs to be reported to the Lemmy developers via GitHub.

Restarting the federation pods appears to have at least lowered the failure queue a little, but it’s showing that while we’ve sent things to lemmit.online (as the example), it hasn’t seen anything from them since 4/14/2025, 10:14:27 PM. So I’m presuming the next time they try to sync and send something to us it SHOULD update…

The federation state checker seems to be indicating that we’re not behind in sending them posts anymore so outgoing should be fixed, but I’m unsure how to troubleshoot incoming further. I’m wondering if there’s a bug with the latest version of Lemmy?

😂 We’ve given it a stern talking to and helped it recover through a restart, and image uploads appear to be working as normal again!

via here (I’m setting up an RSS feed monitor), via email (network[at]pawb.social), or with a DM message to highlight it. It seems like mentions in the original post don’t alert the mentioned person, so I only got a notice when you replied to a message from me / mentioned me in a comment. >~<;

Media uploads should be working again. My apologies.

Media uploads should now work again!

We’re investigating! Appreciate the report!

Taking a peek now, appreciate the heads up!

The server hosting the database crashed for a still unknown reason. Sometime around 5 am MT, the database server had a kernel panic and locked up, it’s been like that since then until we woke up to the alert this morning.

We’ve got a watchdog on the server and an auto-restart in case it hangs, and we’re working to migrate it away to a known stable server while we troubleshoot.

Time (if it’s been persistent for a while), what you were trying to do, what content you were interacting with, etc.

If you’ve got network logs from it, that’d be a cherry on top! Though for the most part, it’ll be our local logs I’ll need to correlate your issues with in order to figure out what happened.

Hi! Apologies for the delay on seeing this; Yes, if this continues to happen, please also @-mention myself or @natebluehooves@pawb.social so we get alerted to it and can investigate.

I’m hoping performance has improved as we haven’t been doing any major work on the infrastructure in quite a while so I’d be concerned if we’re having slow downs again, though that may have been due to the beta version we were previously running.

Apologies for the delay – We’re up to date now!