LiveJournal Maintenance

Back online

2013-05-17T00:59:50Z

We've just brought User Cluster #9 back online, and the errors being caused by the maintenance should stop occurring. Notifications are sending again, but may be delayed as there is a backlog of notifications waiting to be sent. If you are still encountering any errors, please open a Support request so we can investigate the issue.

Maintenance still in progress

2013-05-16T20:39:36Z

We are still in the process of bringing User Cluster #9 back online, and it is unfortunately taking longer than we anticipated. We are making progress, but are still several hours away from this being fixed. To address a few common questions we are seeing:

How many user clusters are there?

There are 13 user clusters in total.

How can I find out what user cluster my account is on?

You can see which user cluster you are on at http://www.livejournal.com/misc/whereami.bml if you are logged-in. If you cannot login, your account is located on user cluster #9.

I am not on cluster 9, but still can't post or edit entries. What's happening?

Trying to update or edit posts may still fail even if you are not on user cluster #9. An Error 500 will appear when loading the update/edit journal page if you have posting access to a community which is located on this cluster. The update module at http://www.livejournal.com/portal/ may still allow you to post while maintenance is ongoing.

I'm not getting notifications. Is this related?

Subscription notifications are not currently being sent as a result of this maintenance. You may still receive other types of emails, such as pingbacks and password notification emails, but will not receive notifications of new entries or comments being posted.

What other things aren't working right now?

Twitter digest posts are not currently being imported as a result of this maintenance. Some other pages & features may display errors if they need to access information located on user cluster #9. The inbox and community management pages are both known to be showing errors for people affected by this.

We will post again either when user cluster #9 is back online, or if we have any additional information to post. Thanks again for your patience while we work to fully restore service to the site.

Maintenance update

2013-05-16T11:35:27Z

We successfully finished maintenance on cluster #7. All accounts’ owners from this cluster can now log into their journals.

We are working on restoring the user cluster #9, it will take approximately an hour. We will keep you informed. Please accept our apologies for the inconvenience.

Maintenance update

2013-05-16T04:44:34Z

To followup on the previous post, the same symptoms for user cluster #7 are also present for users on cluster #9, so we're in the process of fixing it as well. Having 2 clusters to work on rather than 1 unfortunately means that we expect it to take approximately 6-8 hours for everything to be resolved. We do, however, know how the problems with each cluster started and it is not something which will cause any additional clusters to have these issues. We'll post here again either when the issue has been resolved, or if we have any significant developments to update you on.

Emergency maintenance on user cluster #7

2013-05-16T01:46:10Z

We're doing some emergency maintenance on one of our databases (user cluster #7, there are 13 user clusters in total. You can see which user cluster your account is on here). The estimated duration of this maintenance is 4-5 hours.

If your account is located on user cluster #7, you will not be able to login to your account until this maintenance has been completed. If you are already logged-in, you will be unable to post, edit, or delete any material on LiveJournal until the maintenance is completed.

If your account is not on this cluster, LiveJournal will still be up, you will be able to login. You may be able to post, edit, and delete content, but if any communities you have posting access to are on cluster #7, you will see an Error 500 when attempting to load the update/edit journal page, or viewing your inbox if any messages have been received from a user on cluster #7. Other pages may also be similarly affected if they attempt to load usernames or data from this cluster. You may also encounter problems viewing journals, entries, comments, or private messages from accounts which are on this cluster.

We're working as quickly as possible to get everything back up & running, and appreciate your patience during this maintenance.

Maintenance today, some expected downtime

2013-04-19T18:16:36Z

We're planning on doing some database maintenance today, and LiveJournal will be down during this maintenance. The maintenance is scheduled to begin at 12:00 PDT (click to see other timezones) and is expected to last approximately 15 minutes.

You can keep an eye on the LiveJournal Status Page to see when we're back, but we'll also be posting to LiveJournal's Facebook page and LiveJournal's Twitter account to let you know when we're back and to provide any additional updates if we go beyond our planned maintenance window.

[Edit] We're back up again! Thank you for your patience.

Photohosting system maintenance

2013-04-02T22:03:07Z

We will be preforming some emergency maintenance on our photo hosting system this afternoon @ 3:30PM PST. This maintenance window is planned to last around 15 minutes.

There is a possibility that new uploads may encounter delays or time out. Existing photos will remain unaffected.

Thanks!
Andreas

Scheduled maintenance today

2013-02-26T19:57:19Z

Hi all!

We're going to be doing some network maintenance and are expecting approximately 15-30 minutes of downtime beginning at 4:00 P.M. PST (click to convert time). We'll keep our site status page updated during any downtime which occurs.

[edit] We have completed this maintenance! If you're having any problems with the site, please contact LiveJournal Support.

Intermittent issues/database maintenance

2012-06-25T01:54:01Z

Hi All,

Today intermittent code errors like "Can't call method "selectrow_hashref" were displayed as our application timed out to certain databases. This did not affect any data or cause any permanent issues -- it's just the result of a regular maintenance tool that wasn't running properly.

As of now, everything should be back to normal. We like to thank everyone for your patience, as well as for writing in reports of the site accessibility issues.

Thanks,
Andreas

Site access issues

2012-06-12T23:43:57Z

There was a problem earlier which prevented some people from being able to reach the site. We've made some temporary changes to address this, and everyone who was unable to load the site should be able to now.

What happened?

LiveJournal uses multiple providers to determine how to route traffic from the Internet to our data center. One of those providers went down, which meant anyone who was being routed through this provider couldn't reach LiveJournal. We've temporarily stopped routing any traffic through this provider until they've fixed the problem. Whenever things are back to normal on their end, we'll start routing traffic through them again.

We don't expect there to be any further accessibility issues from this, but if you find yourself unable to reach the site over the next few hours, please send an email to webmaster@livejournal.com and let us know.

Network maintenance today

2012-05-04T19:21:27Z

Hello LiveJournal users!

In about 40 minutes (1:00 PM Pacific Time), we are going to make system changes. This may disrupt service for some users for only a few seconds. We do not anticipate more than 1 minute of downtime due to this maintenance.

We apologize for the inconvenience.

Warm regards,
David Newhall
LiveJournal, Inc.

Site Availability Issues

2012-04-25T04:34:24Z

Hi Everyone,

This afternoon we experienced a technical failure of our Memcached system. Memcached encountered some hardware failures in just the right places (Murphy's Law), which caused issues with some related internal systems and a cascade effect that affected site availability.

Fortunately, we managed to come up with a permanent solution and everything is back up again. Memcached only affects temporary cached data, so no user data was lost or corrupted in any way.

In addition to dealing with these issues, our engineers are working with us to eliminate the few, extremely unlikely points of failure that we managed to encounter today. Better quality "downtime" pages and status updates are now a priority, and I apologize we didn't have something for this situation, as failures on this level are highly unexpected.

Thanks for your patience,
Andreas - LJ Operations

PayPal Availability & Maintenance Notification

2012-04-18T19:55:19Z

PayPal has informed us they will be performing site maintenance on April 19, between 9:00PM PDT and 11PM PDT. This means it is possible that PayPal payments attempted during this maintenance window may fail, even though we queue PayPal orders here at LiveJournal.

Therefore, you may wish to make any PayPal payments either before or after the maintenance window.

Thanks everyone,
Andreas - LJ Ops.

Site Maintenance

2012-01-31T21:46:03Z

We're going to be doing some general site maintenance today, beginning at 4:00 P.M. PST (Convert this to your time here). We'll edit this post to let you when the maintenance is complete, and will post any other updates as needed to our Status Page.

[edit] And we're done!

Site status

2011-11-11T17:26:55Z

The LiveJournal team is aware that some users may experience occasional difficulties with accessing the service. Unfortunately, the website has been under a DDoS attack for the last 18 hours. We are glad to say that despite the severity of the attack, we have been successful in dealing with it. Since previous attacks, we have increased our system capabilities, so for many users this has gone unnoticed or caused only minor, temporary issues. We are still dealing with this, and our system administrators will continue to closely monitor the site and work to resolve any problems which occur over the weekend. Thank you for your patience and we will keep you posted.

Site maintenance today

2011-11-03T17:35:24Z

We are scheduling a maintenance window for 12:00 P.M. PST today, November 3rd (Convert this to your time here) to make additional changes to our cookie handling process. The changes being made are intended to address the following:

1) The issue which is causing some people to be logged out frequently should be fixed; once the change is live, you should be able to remain logged in.

2) Clients will be able to use cookies to authenticate with LiveJournal again. This should cause most clients/browser plugins which stopped working or lost some functionality last week to work again.

We do not expect there to be any downtime as a result of this maintenance.

[Edit] And we're done! Please let us know if you're still having problems staying logged in.

Site Maintenance

2011-10-27T19:45:20Z

Our apologies for the delay in reporting these details and any inconvenience this has caused. We wanted to make sure we fully analyzed the extent of the situation before publishing details.

The following occurred - while updating the configuration of our internal caching system, Varnish, for a few minutes the system began to issue cached pages from the users who most recently visited the same page, as the system considered this the most relevant source of data. Thus, for 3 minutes, some users may have seen pages which appeared as though they were logged in as another random account, but it was actually just a snapshot of the page of the last visitor. It had no effect on security, as it was not possible to perform any actions on behalf of this other account. When attempting to load another page during these few minutes, another cached page was served in most cases.

This issue primarily affected people in the United States; the Russian-speaking audience was almost completely unaffected because the changes occurred very late at night in Russia. However, we are grateful to those of you who noticed this and quickly brought our attention to the issue, which gave us the opportunity to quickly understand the cause and resolve it.

The changes which were made are intended to improve site security, and reduce malicious activity on the site. It will make it more difficult to steal cookies from public locations, or spoof them for malicious attacks. We're also working on a few other things:

Better communication with our 3rd party developers

More thorough testing before rolling out changes

Finally, better communication with you about our development process

Again, please accept our apologies for any inconvenience.

Maintenance Completed

2011-10-07T17:55:59Z

Hi Everyone,

Although we completed the network maintenance, the site remained unavailable or slow for many users until Wednesday night. Unfortunately, we discovered many underlying issues with our network installation, and decided to take additional downtime to fix everything up.

There should be no site-wide network/reachability issues following this announcement. Although you might not notice any major differences, this upgrade is certainly a very good thing for us, and will let us handle increased traffic at faster speeds.

The LiveJournal team and I thank you deeply for your patience, and are working hard to keep everyone informed better before we make changes or experience unexpected problems.

Much appreciated,
Andreas - LJ Ops.

EDIT: Changed to plaintext for readability instead of the rte

Planned site-wide downtime

2011-10-05T06:06:55Z

Dear all,

LiveJournal will be unavailable for up to three hours starting at the following times, Wednesday, October 5. During this time, LJ might be unreachable, slow, or experience issues loading. This will affect all services, including LJ Talk & Scrapbook.

22:30 GMT

03:30 PM PST
04:30 PM MST
06:30 PM EST

The purpose of this downtime is to install new core routers, upgrade other network devices, and add more Internet uplinks to our primary datacenter. I expect the real downtime to be much shorter then stated above, but unforeseen problems could require up to three hours.

Thanks,
Andreas

Paid compensation for site downtime

2011-08-01T23:28:08Z

As many of you are aware, LiveJournal underwent an attack last week which resulted in some downtime and caused other accessibility issues during this time. While the attack is over, some intermittent issues remain which we are in the process of fixing.

We want to take this time to let paid members know that we will be extending your paid time by 2 weeks if you had a paid account at any time between July 25th and July 31st. This time will be added to your account a few days from now. Thank you for your continued patience while we get the site back to normal.

Sorry we broke the LJ again...

2011-07-28T00:48:07Z

Hi all,

For the last few days site performance has been pretty bad, however we're slowly making changes to get as many people access to the site as possible. We keep getting new equipment to beef things up, so the chances of this happening again are pretty remote. Sunday night we experienced a surge of increased junk traffic. Our site is already pretty fragile, so it caused a lot of cascade failures and I stayed up all night with ops to bring stuff back up.

Sorry guys, I hate being so serious, but we'll get everything sorted in a few days.

In the meantime, the most reliable way to access the site is logged out. Viewing certain journals while logged in may give you a timeout error due to some bugs we're working on. If you're constantly getting errors, try clearing out your cookies and trying again.

-Andreas

---

"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall

Login Session, Site Slowness

2011-05-12T19:59:18Z

If you go to http://www.livejournal.com/manage/logins.bml , you can see the recent login history for your account.

Yesterday a change was made that recorded incorrect IP addresses for a few hours. Probably a 172.something address. That is not our staff, just a screw-up on our part. [edit: The entries in your login history are not other people, we just recorded wrong information during a small window of time yesterday.]

As far as everything else goes, yesterday we pushed new code out to the website. I'm not completely sure of the full changes, but some things are buggy, and we've asked our developers to take some time to help us concentrate on performance.

Pics updates...

2011-05-11T01:38:47Z

Hey everyone,

We're pushing pics.livejournal.com (scrapbook) through some new servers, so performance will be much better. What that means for us is more reliability, faster uploads, and less headache. The service has been mostly left on its own for a while (2 years? more?), so it is due for an upgrade.

Sorry we broke things intermittently this afternoon. Everything should be well now, but over the next day or two we might make some minor changes and will update status.

Thanks!
Andreas

Suggestions, and yes, we still love Singapore

2011-04-14T22:44:44Z

This update is for users of SingNet/SingTel,

During our previous attacks, some SingNet users were inadvertently blocked as part of our mitigation. It turns out that some ISPs give the users a different IPs on every outbound request. You will not get "Your IP has been blocked" messages any more, and we've fixed the problem.

For everyone else,

Everything has been humming along nicely. We're still ironing out bugs and working on projects to improve things. It's not like we don't have enough to do, but I thought we'd make this fun.

We have a great suggestions community, but if you could sum it up in a sentence or two, what could we do better? More Twitter posts? Account creation limits? What's the trade off between spam and captchas? Should we make everyone do difficult math problems before posting? :)

EDIT: We're not actually going to limit accounts. The most we might do is limit the rate at wich you can create new accounts, like only a a few per *minute*.

-Andreas

DDoS Botnet

2011-04-08T22:27:49Z

As is obvious, we've been dealing with more DDoS issues. The latest attack has been focused on a single journal, in an obvious attempt to bait us into suspending it. If they were really upset about this journal, they would probably have more luck contacting abuse than attacking our service..

I have decided to post a map of the computers DDoS-ing us, and all the associated IP addresses are contained in the map. If you see yourself there, you might consider cleaning out your computer.

--> Large Map

Technically minded users will notice that we've started using a caching proxy to deliver content. We've had great success implementing open-source software like Varnish to upgrade our capacity and speed things up. While no single tool can stop a targeted DDoS, we will continue to use new tactics to fight them.

As usual, you can check our status and Support pages for information.

Thanks for being awesome :)
~Andreas - LJ Ops