Network maintenance today
[info]franky
Hello LiveJournal users!

In about 40 minutes (1:00 PM Pacific Time), we are going to make system changes. This may disrupt service for some users for only a few seconds. We do not anticipate more than 1 minute of downtime due to this maintenance.

We apologize for the inconvenience.

Warm regards,
David Newhall
LiveJournal, Inc.

Site Availability Issues
[info]oceanplexian
Hi Everyone,

This afternoon we experienced a technical failure of our Memcached system. Memcached encountered some hardware failures in just the right places (Murphy's Law), which caused issues with some related internal systems and a cascade effect that affected site availability. 

Fortunately, we managed to come up with a permanent solution and everything is back up again. Memcached only affects temporary cached data, so no user data was lost or corrupted in any way.

In addition to dealing with these issues, our engineers are working with us to eliminate the few, extremely unlikely points of failure that we managed to encounter today. Better quality "downtime" pages and status updates are now a priority, and I apologize we didn't have something for this situation, as failures on this level are highly unexpected.

Thanks for your patience,
Andreas - LJ Operations

PayPal Availability & Maintenance Notification
[info]oceanplexian
PayPal has informed us they will be performing site maintenance on April 19, between 9:00PM PDT and 11PM PDT. This means it is possible that PayPal payments attempted during this maintenance window may fail, even though we queue PayPal orders here at LiveJournal.

Therefore, you may wish to make any PayPal payments either before or after the maintenance window.

Thanks everyone,
Andreas - LJ Ops.

Site Maintenance
Carved logo
[info]theljstaff
We're going to be doing some general site maintenance today, beginning at 4:00 P.M. PST (Convert this to your time here). We'll edit this post to let you when the maintenance is complete, and will post any other updates as needed to our Status Page.

[edit] And we're done!

Site status
Carved logo
[info]theljstaff
The LiveJournal team is aware that some users may experience occasional difficulties with accessing the service. Unfortunately, the website has been under a DDoS attack for the last 18 hours. We are glad to say that despite the severity of the attack, we have been successful in dealing with it. Since previous attacks, we have increased our system capabilities, so for many users this has gone unnoticed or caused only minor, temporary issues. We are still dealing with this, and our system administrators will continue to closely monitor the site and work to resolve any problems which occur over the weekend. Thank you for your patience and we will keep you posted.

Site maintenance today
Carved logo
[info]theljstaff
We are scheduling a maintenance window for 12:00 P.M. PST today, November 3rd (Convert this to your time here) to make additional changes to our cookie handling process. The changes being made are intended to address the following:

1) The issue which is causing some people to be logged out frequently should be fixed; once the change is live, you should be able to remain logged in.

2) Clients will be able to use cookies to authenticate with LiveJournal again. This should cause most clients/browser plugins which stopped working or lost some functionality last week to work again.

We do not expect there to be any downtime as a result of this maintenance.

[Edit] And we're done! Please let us know if you're still having problems staying logged in.

Site Maintenance
Carved logo
[info]theljstaff
Our apologies for the delay in reporting these details and any inconvenience this has caused. We wanted to make sure we fully analyzed the extent of the situation before publishing details.

The following occurred - while updating the configuration of our internal caching system, Varnish, for a few minutes the system began to issue cached pages from the users who most recently visited the same page, as the system considered this the most relevant source of data. Thus, for 3 minutes, some users may have seen pages which appeared as though they were logged in as another random account, but it was actually just a snapshot of the page of the last visitor. It had no effect on security, as it was not possible to perform any actions on behalf of this other account. When attempting to load another page during these few minutes, another cached page was served in most cases.

This issue primarily affected people in the United States; the Russian-speaking audience was almost completely unaffected because the changes occurred very late at night in Russia. However, we are grateful to those of you who noticed this and quickly brought our attention to the issue, which gave us the opportunity to quickly understand the cause and resolve it.

The changes which were made are intended to improve site security, and reduce malicious activity on the site. It will make it more difficult to steal cookies from public locations, or spoof them for malicious attacks. We're also working on a few other things:

  • Better communication with our 3rd party developers

  • More thorough testing before rolling out changes

  • Finally, better communication with you about our development process


Again, please accept our apologies for any inconvenience.

Maintenance Completed
[info]oceanplexian
Hi Everyone,

Although we completed the network maintenance, the site remained unavailable or slow for many users until Wednesday night. Unfortunately, we discovered many underlying issues with our network installation, and decided to take additional downtime to fix everything up.

There should be no site-wide network/reachability issues following this announcement. Although you might not notice any major differences, this upgrade is certainly a very good thing for us, and will let us handle increased traffic at faster speeds.

The LiveJournal team and I thank you deeply for your patience, and are working hard to keep everyone informed better before we make changes or experience unexpected problems.

Much appreciated,
Andreas - LJ Ops.

EDIT: Changed to plaintext for readability instead of the rte

Planned site-wide downtime
[info]oceanplexian
Dear all,

LiveJournal will be unavailable for up to three hours starting at the following times, Wednesday, October 5. During this time, LJ might be unreachable, slow, or experience issues loading. This will affect all services, including LJ Talk & Scrapbook. 

22:30 GMT

03:30 PM PST
04:30 PM MST
06:30 PM EST

The purpose of this downtime is to install new core routers, upgrade other network devices, and add more Internet uplinks to our primary datacenter. I expect the real downtime to be much shorter then stated above, but unforeseen problems could require up to three hours.

Thanks,
Andreas

Paid compensation for site downtime
Carved logo
[info]theljstaff
As many of you are aware, LiveJournal underwent an attack last week which resulted in some downtime and caused other accessibility issues during this time. While the attack is over, some intermittent issues remain which we are in the process of fixing.

We want to take this time to let paid members know that we will be extending your paid time by 2 weeks if you had a paid account at any time between July 25th and July 31st. This time will be added to your account a few days from now. Thank you for your continued patience while we get the site back to normal.

You are viewing the community [info]lj_maintenance