Hey lemmings!
I wanted to share a quick update about our recent performance issues and how I have addressed them.
The last 24h have been a bit rough for lemm.ee.
Last night, I spent some time debugging federation issues with lemmy.world. We managed to significantly improve the situation - lemmy.world content is now reaching lemm.ee with a very high success rate - but this has had the effect of increasing incoming federation traffic on our servers significantly.
Additionally, we have been seeing steadily increasing normal user traffic over the past week, which is awesome from a community standpoint, but of course means that our servers have to do more work to keep up with all the new people.
To top things off, today there appeared a badly configured instance in the network, which was effectively launching a DoS attack against lemm.ee for several hours. Most likely it was unintentional, but unfortunately the end result was a sudden increase in our server load.
All these factors combined resulted in a really bad experience for most lemm.ee users today. Page load times have consistently been spiking into as much as 10 seconds or more for the whole day:
In fact, a lot of page loads just timed out with errors.
Fortunately, it seems I have managed to clear up the problems!
I have put a bunch of mitigations in place, and after monitoring the situation for the past hour, it seems that our performance issues have been resolved for now. So hopefully, you can enjoy browsing lemm.ee again without it feeling like torture!
Here are specific steps I took:
- I have doubled the hardware resources for our backend servers and database.
- I purchased a Cloudflare pro subscription for lemm.ee for 1 year. This took out a considerable chunk of my budget for lemm.ee, but in return it will allow me to analyze and optimize our cache usage to a far greater extent. I am already seeing vastly reduced load times for cacheable content (try opening https://lemm.ee a few times in a row as a logged out user - it should be blazing fast now!)
- I have configured a rate limiter which will prevent future DoS from the specific method that was used against us today.
Of course, all of the above is costly. Luckily, lemm.ee users have been very generous with donations in the month of June, and in fact a significant amount of donors have opted for monthly recurring contributions. This all gives me the confidence to increase our spending for now, and I am currently expecting to NOT increase my personal planned contribution of 150€/month, as the increased costs so far are entirely being covered by donations!
Let me take this opportunity to thank the sponsors who made the upgrades possible! All lemm.ee users are now enjoying better performance thanks to you, I could not have done it without you awesome people.
On a final note, I just want to say that I hope a lot of these issues can be solved by optimizations in Lemmy software itself in the future. I have been personally contributing several optimizations to the Lemmy codebase, and I know many others are focused on optimizations as well. Just throwing extra resources at the problem will probably not be a sustainable solution for very long 😅. But I am optimistic that we are moving in the right direction with the software changes, and we’ll be enjoying reduced resource needs before long.
That’s all I wanted to share today, I wish you all a great weekend!
What a phenomenal response. Every single day makes me happy that I chose lemme.ee as my home instance. Truly appreciate all of the hard work you are pouring into this. And we can see how difficult it is based off of your total transparency. Thank you!
Yo, I’m a simple Reddit refugee, just trying to figure out how to make my way in the fediverse, and I signed up to lemm.ee not long after this post went up. I honestly chose this instance on a whim, and after a bit of exploring and learning about how Lemmy works, boy does this post make me glad I landed here!
Thanks for running this instance, and for housing us reddit noobs :)
Thank you for using your time and resources to create this space! Reading updates on server performance in main, gives me a tingly homey feeling I haven’t felt in a loooong time
here’s to everlasting lemmy.ee tingles 🍻
I’ve just joined up to lemme.ee and subscribed to a community on another instance. I can see the posts in that community, but they all show up with zero comments. If I view the community on the web, there are several comments there. Is this a bug?
The community is !nrl@aussie.zone
It is a federation thing. Newly federated communities don’t load comments with their initial import. You’ll start to see comments as you go.
Just to add to the other reply, if there’s an old comment chain you really want to reply to you can grab the URL of the last comment, search for it through your home instance and that’ll force an index of the comment you searched for plus any parents.
Thank you @sunaurus@lemm.ee for all of your work. I am continually realizing that I chose the right instance with how transparent and responsive you have been.
Yeah the slowdown was a bit rough, been browsing off and on all day today. Thanks for fixing that. Seems to be working a lot better now. That’s a bummer you had to increase expenses though.
It’s feeling quite good at the moment so thank you for all your hard work.
Awesome instance. Really fast speed (compared to sh.itjust.works), there is no blocked instances/communities. Kudos❤️
Thank you for operating this instance!
I hope you are tracking some dollar amount for your time as well. Even if you’re not actually taking any pay it would be good to have a sense of labor as a component of a thing’s upkeep cost.
Thanks for the kind words! To be honest, I’m not tracking my time, as I’m not expecting to break even at all anyway, even before starting to account for the time I’ve spent 😄
Thanks!
I’m definitely seeing an increase in speed. I think at some point there will be a market for servers that users must pay to use. For the time being, I’m more than happy to contribute to a server that is open to all, but good speed needs to be there.
You broke the https://lemm.ee homepage, it returns a json.
It should be fixed now!
It is, only that I had to clear the cache :)
It does load that JSON very quickly!
Ah, yes, the optimist
And really, who doesn’t love json?
I mean protocol buffers might be more efficient, but json is nice and readable. Much nicer than XML for an example. And significantly more readable than protobuf!
Thank you very much. I’ve been made to feel very welcome and have joined on of the many support groups, that his helping me remove typing r/ from my muscle memory. Have a great weekend.
Not sure if this is related to the infra upgrade, but my earlier issue with not seeing all the posts in the meta community is now fixed.
Awesome news! I did make a small fix to a localization bug in Lemmy-ui, which was causing some people to not see posts, so it could have been that. But in any case, I’m glad it’s sorted for you!
I don’t mean to grossly oversimplify… But does this mean it’s time to upgrade to 0.18.1?
Generally I would recommend against running RC builds. I was willing to take the risk in this case for lemm.ee, because:
- I feel comfortable with debugging and fixing issues in code as they crop up
- I have been working myself on stabilizing 0.18.1 and am pretty well aware of its current state
- The set of problems in 0.18.1-rc4 is a bit better than the different set of problems we had with 0.17.4 😅
At the end of the day, you have to acknowledge the risks and see if they’re worth it for you.
You are a freaking hero!