Carved logo

Site maintenance today

We are scheduling a maintenance window for 12:00 P.M. PST today, November 3rd (Convert this to your time here) to make additional changes to our cookie handling process. The changes being made are intended to address the following:

1) The issue which is causing some people to be logged out frequently should be fixed; once the change is live, you should be able to remain logged in.

2) Clients will be able to use cookies to authenticate with LiveJournal again. This should cause most clients/browser plugins which stopped working or lost some functionality last week to work again.

We do not expect there to be any downtime as a result of this maintenance.

[Edit] And we're done! Please let us know if you're still having problems staying logged in.
Carved logo

Site Maintenance

Our apologies for the delay in reporting these details and any inconvenience this has caused. We wanted to make sure we fully analyzed the extent of the situation before publishing details.

The following occurred - while updating the configuration of our internal caching system, Varnish, for a few minutes the system began to issue cached pages from the users who most recently visited the same page, as the system considered this the most relevant source of data. Thus, for 3 minutes, some users may have seen pages which appeared as though they were logged in as another random account, but it was actually just a snapshot of the page of the last visitor. It had no effect on security, as it was not possible to perform any actions on behalf of this other account. When attempting to load another page during these few minutes, another cached page was served in most cases.

This issue primarily affected people in the United States; the Russian-speaking audience was almost completely unaffected because the changes occurred very late at night in Russia. However, we are grateful to those of you who noticed this and quickly brought our attention to the issue, which gave us the opportunity to quickly understand the cause and resolve it.

The changes which were made are intended to improve site security, and reduce malicious activity on the site. It will make it more difficult to steal cookies from public locations, or spoof them for malicious attacks. We're also working on a few other things:

  • Better communication with our 3rd party developers

  • More thorough testing before rolling out changes

  • Finally, better communication with you about our development process


Again, please accept our apologies for any inconvenience.

Maintenance Completed

Hi Everyone,

Although we completed the network maintenance, the site remained unavailable or slow for many users until Wednesday night. Unfortunately, we discovered many underlying issues with our network installation, and decided to take additional downtime to fix everything up.

There should be no site-wide network/reachability issues following this announcement. Although you might not notice any major differences, this upgrade is certainly a very good thing for us, and will let us handle increased traffic at faster speeds.

The LiveJournal team and I thank you deeply for your patience, and are working hard to keep everyone informed better before we make changes or experience unexpected problems.

Much appreciated,
Andreas - LJ Ops.

EDIT: Changed to plaintext for readability instead of the rte

Planned site-wide downtime

Dear all,

LiveJournal will be unavailable for up to three hours starting at the following times, Wednesday, October 5. During this time, LJ might be unreachable, slow, or experience issues loading. This will affect all services, including LJ Talk & Scrapbook. 

22:30 GMT

03:30 PM PST
04:30 PM MST
06:30 PM EST

The purpose of this downtime is to install new core routers, upgrade other network devices, and add more Internet uplinks to our primary datacenter. I expect the real downtime to be much shorter then stated above, but unforeseen problems could require up to three hours.

Thanks,
Andreas
Carved logo

Paid compensation for site downtime

As many of you are aware, LiveJournal underwent an attack last week which resulted in some downtime and caused other accessibility issues during this time. While the attack is over, some intermittent issues remain which we are in the process of fixing.

We want to take this time to let paid members know that we will be extending your paid time by 2 weeks if you had a paid account at any time between July 25th and July 31st. This time will be added to your account a few days from now. Thank you for your continued patience while we get the site back to normal.

Sorry we broke the LJ again...

Hi all,

For the last few days site performance has been pretty bad, however we're slowly making changes to get as many people access to the site as possible. We keep getting new equipment to beef things up, so the chances of this happening again are pretty remote. Sunday night we experienced a surge of increased junk traffic. Our site is already pretty fragile, so it caused a lot of cascade failures and I stayed up all night with ops to bring stuff back up.

Sorry guys, I hate being so serious, but we'll get everything sorted in a few days.

In the meantime, the most reliable way to access the site is logged out. Viewing certain journals while logged in may give you a timeout error due to some bugs we're working on. If you're constantly getting errors, try clearing out your cookies and trying again.



-Andreas

---

"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall

Login Session, Site Slowness

If you go to http://www.livejournal.com/manage/logins.bml , you can see the recent login history for your account.

Yesterday a change was made that recorded incorrect IP addresses for a few hours.  Probably a 172.something address. That is not our staff, just a screw-up on our part. [edit: The entries in your login history are not other people, we just recorded wrong information during a small window of time yesterday.]

As far as everything else goes, yesterday we pushed new code out to the website. I'm not completely sure of the full changes, but some things are buggy, and we've asked our developers to take some time to help us concentrate on performance.

Pics updates...

 Hey everyone,

We're pushing pics.livejournal.com (scrapbook) through some new servers, so performance will be much better. What that means for us is more reliability, faster uploads, and less headache. The service has been mostly left on its own for a while (2 years? more?), so it is due for an upgrade.

Sorry we broke things intermittently this afternoon. Everything should be well now, but over the next day or two we might make some minor changes and will update status

Thanks!
Andreas








Suggestions, and yes, we still love Singapore

This update is for users of SingNet/SingTel,

During our previous attacks, some SingNet users were inadvertently blocked as part of our mitigation. It turns out that some ISPs give the users a different IPs on every outbound request. You will not get "Your IP has been blocked" messages any more, and we've fixed the problem. 

For everyone else,

Everything has been humming along nicely. We're still ironing out bugs and working on projects to improve things. It's not like we don't have enough to do, but I thought we'd make this fun. 

We have a great suggestions community, but if you could sum it up in a sentence or two, what could we do better? More Twitter posts? Account creation limits? What's the trade off between spam and captchas? Should we make everyone do difficult math problems before posting? :)
 


EDIT: We're not actually going to limit accounts. The most we might do is limit the rate at wich you can create new accounts, like only a a few per *minute*.  
 
-Andreas 

DDoS Botnet

As is obvious, we've been dealing with more DDoS issues. The latest attack has been focused on a single journal, in an obvious attempt to bait us into suspending it. If they were really upset about this journal, they would probably have more luck contacting abuse than attacking our service..

I have decided to post a map of the computers DDoS-ing us, and all the associated IP addresses are contained in the map. If you see yourself there, you might consider cleaning out your computer.

--> Large Map





Technically minded users will notice that we've started using a caching proxy to deliver content. We've had great success implementing open-source software like Varnish to upgrade our capacity and speed things up. While no single tool can stop a targeted DDoS, we will continue to use new tactics to fight them.

As usual, you can check our status and Support pages for information.  

Thanks for being awesome :)
~Andreas - LJ Ops