Go to mobile version
theljstaff ([info]theljstaff) wrote in [info]lj_maintenance,

Site Maintenance

Our apologies for the delay in reporting these details and any inconvenience this has caused. We wanted to make sure we fully analyzed the extent of the situation before publishing details.

The following occurred - while updating the configuration of our internal caching system, Varnish, for a few minutes the system began to issue cached pages from the users who most recently visited the same page, as the system considered this the most relevant source of data. Thus, for 3 minutes, some users may have seen pages which appeared as though they were logged in as another random account, but it was actually just a snapshot of the page of the last visitor. It had no effect on security, as it was not possible to perform any actions on behalf of this other account. When attempting to load another page during these few minutes, another cached page was served in most cases.

This issue primarily affected people in the United States; the Russian-speaking audience was almost completely unaffected because the changes occurred very late at night in Russia. However, we are grateful to those of you who noticed this and quickly brought our attention to the issue, which gave us the opportunity to quickly understand the cause and resolve it.

The changes which were made are intended to improve site security, and reduce malicious activity on the site. It will make it more difficult to steal cookies from public locations, or spoof them for malicious attacks. We're also working on a few other things:

  • Better communication with our 3rd party developers

  • More thorough testing before rolling out changes

  • Finally, better communication with you about our development process


Again, please accept our apologies for any inconvenience.

  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    switch

    Your reply will be screened

  • 361 comments
Previous
← Ctrl← Alt
Next
Ctrl →Alt →

[info]ilea

October 27 2011, 19:49:50 UTC 7 months ago

Comments aren't being mailed out.

[info]heinous_bitca

October 27 2011, 19:52:09 UTC 7 months ago

It had no effect on security,

How can you say that when users were able to see other's locked posts, from what those who reported the problem said?

[info]moreteadk

October 27 2011, 19:59:44 UTC 7 months ago

Yes. This. People tend to post under lock excactly so that random other users can NOT see it. It was a crystal clear 100% security problem.

It may not have had any other consequences than that, but that doesn't stop it being a security problem.

[info]spoofmaster

7 months ago

[info]ashley_y

7 months ago

[info]franzi1981

7 months ago

[info]ashley_y

7 months ago

[info]ron_newman

October 27 2011, 19:52:10 UTC 7 months ago

Will you please fix whatever you did that broke ljdump, ljmigrate, ljarchive, etc. ?

[info]scarlet_discord

October 27 2011, 19:52:40 UTC 7 months ago

Seconded!

[info]alobar

7 months ago

[info]budgie_uk

7 months ago

[info]twoeleven

October 27 2011, 19:54:06 UTC 7 months ago

some users may have seen pages which appeared as though they were logged in as another random account, but it was actually just a snapshot of the page of the last visitor. It had no effect on security
No, of course not: viewing other people's locked entries with random amounts of privilege isn't a security problem at all.

The truth: it's not just for breakfast any more.

[info]cpsings4him

October 27 2011, 19:55:17 UTC 7 months ago

Ever notice these posts always seem to start with an apology? :p

[info]druid_va

October 27 2011, 20:01:56 UTC 7 months ago

I was always told never follow an apology with an explanation, it invalidates the apology.

[info]rax

7 months ago

[info]ruakh

7 months ago

[info]druid_va

7 months ago

[info]ruakh

7 months ago

[info]intrepia

7 months ago

[info]salviolog

7 months ago

[info]snailbones

October 27 2011, 19:56:04 UTC 7 months ago


Thank you for letting us know what's happening, but I really worry that you don't seem to understand how much damage you do to your customers' confidence and loyalty every time something like this happens and you fail to let everyone know immediately that you're working on it.

[info]kittydesade

October 27 2011, 20:03:31 UTC 7 months ago

+1

Customer Service: It's not just for old fuddy-duddies anymore.

[info]elionwyr

7 months ago

[info]ladycat777

October 27 2011, 19:57:06 UTC 7 months ago

Thank you for the update. I appreciate how fast you worked on this; internet time being what it is.

[info]slinkslowdown

October 27 2011, 19:58:40 UTC 7 months ago

Thank you for finally acknowledging the problem and good luck with fixing this all quickly.

[info]pretty_panther

October 27 2011, 20:01:30 UTC 7 months ago

The fact that you don't see people seeing any sort of locked post that they were not meant to as a security breach says a hell of a lot.

[info]dungenessmaster

October 27 2011, 20:19:28 UTC 7 months ago

+1

I ended up in some stranger's PM RP when I went to check my own inbox. I logged right back out without touching anything, but how is seeing someone else's private business not a security issue?

[info]ketsundere

7 months ago

[info]cake_and_dreams

October 27 2011, 20:02:13 UTC 7 months ago

Finally, an apology. Thanks for the hard work and all, but in the future, you guys could tell us you're working on it instead of just having Mark make things worse with us, or maybe ask for input about changes from us, or compensate those of us with paid accounts. I've got 4 paids. I don't want to renew them anymore after having multiple accounts with you guys for years because your business sense has just started plummeting into the negatives.

And no security problem? Please! One of my friends ended up in the inbox of another person who was selling things through her journal. She saw everything - paypal address, home addresses... how is this not a security issue?

[info]laurahonest

October 27 2011, 20:14:32 UTC 7 months ago

this! "She saw everything - paypal address, home addresses... how is this not a security issue?"
My address is stricken from all public records, there is NO way for anyone to find my address if i don't give it to them and this is a HUGE security issue.

[info]seraphic

7 months ago

[info]shigeharu

October 27 2011, 20:04:47 UTC 7 months ago

"There was no effect on security" sounds really misleading to me, given that people were reportedly able to view private content, even if they had no ability to actually edit it. It may not be as bad as people had thought, but to completely write off the problem is inaccurate.

Thank you for this post, however. I'm glad we finally know what's going on.

I hope that you carry through your promises that you're working on better testing and better communication; saying you're working on it is well and good, but restoring your userbase's faith is of course going to come down to what we see in action, not what promises we see made when the storm has passed.

[info]moreteadk

October 27 2011, 20:05:40 UTC 7 months ago

We wanted to make sure we fully analyzed the extent of the situation before publishing details.

That's great. It's good that you try to get the full picture before making any sweeping statements which might then have to be corrected or taken back.

But could you at the very least, as soon as you become aware of the problem, just make a quick post saying that you know there is a problem and that you're working on it as fast as you can or something.

Just so that, you know, people can see that they're being heard and not just ignored. Because us out here, the users, we don't know what you lot are doing! We can't see you working on it unless you tell us that you are. So if users are reporting problem and nothing happens for two days(!!!), how are they supposed to know they're being taken seriously at all?

Please try to remember this next time? Please? It would make it so much easier for the users, and in turn I imagine for you lot as well.

[info]mab_browne

October 27 2011, 22:10:03 UTC 7 months ago

This. I acknowledge that LJ can't make a specific comment until it's had a chance to investigate, but ffs, please, please immediately tell your user base that you know that you have a problem and that you're looking at it. Because that is what kills what pathetic trust I had left that you guys are fit to entrust things like my credit card details to.

[info]ldymusyc

October 27 2011, 20:06:53 UTC 7 months ago

How about putting this apology in [info]lj_releases, which people are checking for updates and is "A running list of all changes to the site, bugfixes and enhancements.", or in [info]news, for "Regular postings about major news and events happening on LiveJournal: new features, new options, and major happenings."? Pretty sure this qualifies as both a bug fix and a major happening.

Dumping this in a relatively minor community? What, are you hoping people won't see it? Frankly, I'm surprised this didn't get posted to [info]lj_biz, which hasn't been updated in three years. It's like you're trying to look shady and uncommunicative.

[info]ladyvox

October 27 2011, 22:35:45 UTC 7 months ago

Yep!

[info]citymusings

7 months ago

[info]lizzie9208

7 months ago

[info]eyesofpity

October 27 2011, 20:07:00 UTC 7 months ago

Just have a few issues with this:

1. The issue with seeing people's logged in pages for accounts that didn't belong to them happened for far longer than 3 minutes.

2. It DID affect security since people could view information on other's accounts.

3. How hard is it to say "We're aware of the issue and are working on a resolution"? We don't need to know all of what caused it, just that you are aware of it and working on it so we can stop reporting it.

4. And still haven't fixed the fact that we can't stay logged in. One of my friends was logged out yet again while I was typing this comment. All your update seemed to do was make it hard for your legit users to stay logged in but spammers seem to have no problem, plus there was a security hole (or may still be? this isn't clear on if you fixed it or not).

5. And with a breach this bad, what should have been done was roll back the changes as soon as the first reports came in.

I admit, I'm very disappointed in you, LJ staff. You need to also add to that bullet list "Better customer service" because people who are paying for a service like to be treated with respect or they will go elsewhere. People will pay more for goods and services with better support.

[info]vixyish

October 27 2011, 20:38:48 UTC 7 months ago

All of these points, a thousand times.

[info]oudeteron

October 27 2011, 20:07:32 UTC 7 months ago

Well, now I wonder who got to view my highest-security friends group without my consent or knowledge. While it's a relief they wouldn't have been able to modify anything, this is pretty much a nightmare scenario for anyone who needs to retain essential control over their (near-)private information. This is a hell of a confidentiality issue.

[info]storyfan

October 27 2011, 20:17:32 UTC 7 months ago

I was thinking that, too. While I don't post a great deal of very personal stuff on the internet (too old and too much of a fuddy duddy to trust it), I do have some filtered things posted about my family and real-life friends. Those posts are filtered for good reasons.

Plus, it would certainly be awkward for a person to see a filtered page from a friend's LJ and realize she wasn't included on that filter. From reading this explanation, that could have happened.

And who's to say these random pages couldn't be saved by the unintended viewer? Taking a snapshot of a page is quite easy.

You're correct — this is one hell of a confidentiality issue.

[info]oudeteron

7 months ago

[info]anjak_j

7 months ago

[info]mercilynn

October 27 2011, 20:08:44 UTC 7 months ago

We're also working on a few other things:

Better communication with our 3rd party developers
More thorough testing before rolling out changes
Finally, better communication with you about our development process


.........

.........

.........

Really? Yeah. I appreciate the apology, but as a business, those are basic customer service relations.

[info]iron_valkyrie

October 27 2011, 20:56:16 UTC 7 months ago

THIS.

[info]laurahonest

October 27 2011, 20:10:13 UTC 7 months ago

How is that not a security issue? I was one of those that was able to see other people's private entries and I can honestly say that even though it was out of my control i still feel badly for the person whose privacy YOUR actions caused me to invade. I had to log out 3 times to actually be logged out. I logged out of one account that wasn't me and found myself logged into someone else's account immediately.
That to me, and I am sure to the users whose privacy was invaded, is a rather large security issue.

I would appreciate it if you could stop rolling out these updates to stuff that isn't broken, and thereby breaking new stuff, and focus on fixing the stuff that you broke a year ago. I have been having problems with notification emails for the last year and you still haven't fixed it. I have changed email accounts twice at your suggestion and it is still an ongoing error.

The banners at the top of the page are another thing you don't need to change every month. Every time you change those i can't access lj for a couple of hours and then when it finally works again i see that the banner has changed.

TL;DR
Being able to see others/have others see my private entries is a huge security issue.
Stop trying to upgrade stuff that functions and focus on fixing the stuff that is actually broken.

[info]darth_eldritch

October 27 2011, 22:20:44 UTC 7 months ago

+1,000

Stop trying to upgrade stuff that functions and focus on fixing the stuff that is actually broken.

I wish they would do this!!!!!

[info]pwalkeri

October 27 2011, 20:10:31 UTC 7 months ago

I'm worried less about your ability to eventually fix these kinds of flubs as I am your understanding of "security breach". If users were able to see others' personal posts/PMs/etc, then that definitely qualifies as a security breach. Otherwise, why bother with passwords at all?

[info]triadruid

October 27 2011, 23:11:20 UTC 7 months ago

It's not a site security breach because all data is intact and unable to be damaged. It's a personal confidentiality breach, which is a little different animal, from a business/computer science standpoint.

[info]karadin

7 months ago

[info]kengr

7 months ago

[info]lorax

October 27 2011, 20:14:21 UTC 7 months ago

As already said "there is no effect on security" is misleading. More accurately, if they couldn't edit entries/view profile or account information, or view things like billing info, then the result was that security wasn't MORE GREATLY compromised.

Deleted comment

[info]scolaro

October 27 2011, 20:19:35 UTC 7 months ago

Thank you for the explanation, but I second what others have already said: It wouldn't have killed anyone if you'd posted a quick entry (maybe in [info]news, as it sounds like...y'know...actual NEWS) at the beginning of this occurrence.

Instead of being informed by you, I (and probably many others) waded through other users' entries full of reports about the issue - all private users who collected information and tried to make sense of this, all by themselves.

Why is it so difficult to keep the users informed about vital things connected to this site?

[info]wetdryvac

October 27 2011, 20:21:29 UTC 7 months ago

Much of this duplicates the above, but in order:

1) Viewing of private information is a security breach of very high order.
2) Failure to notify is a breach of trust.
3) Failure to place notifications in news.livejournal.com, paidmembers.livejournal.com, and so forth means many who need to know won't.
4) Inability to back up journal/comments set via ljarchive toolset and similar means those of us who would back up our information are unable to do so.

The most basic requirements for security and trust have been violated here, and that's a pretty serious problem. That this problem falls in with a pattern of lack-of-information in event of site issues diminishes trust further. Rendering backups easier instead of more difficult would be a beginning to restoring that trust, but truth and full disclosure in all applicable areas is an absolute requirement.

[info]aizome

October 27 2011, 21:07:23 UTC 7 months ago

All of the above.

[info]pbristow

7 months ago

[info]sinfulslasher

October 27 2011, 20:25:53 UTC 7 months ago Edited:  October 27 2011, 20:26:17 UTC

Excuse me, but how can this not have any effect on security when I'm suddenly able to see some random other user's F-LOCKED entry?! Or, even worse, personal information? Whether or not I'm able to perform any action is beside the point. We're talking about a serious security and privacy breach! *boggles*

Also? Not that I'm surprised by this, but acknowledging the issue for the very first time more than 24 hours after it happened? Not. Cool. At. All.

[info]neapeaikea

October 27 2011, 20:33:55 UTC 7 months ago

LOL! Lj, no! This is not an adequate apology. In addition to all the reasons stated by those above (this was a MAJOR security breach!) I'd like to add that it is very annoying to be told the problem didn't affect Russians because of the timezone. That is complete hogwash. It did affect them, and the many, many other nationality users in similar timezones, there weren't just as many logged in at that moment. To say it wasn't a problem because it was late at night when it occurred... come on.

But I do appreciate that you have made this post, and I hope that this will be the drop that make you finally learn how to treat your customer base.

[info]falenandawn

October 28 2011, 03:28:40 UTC 7 months ago

I'm kind of insulted they included that line, period. What the hell does nationality matter in something like this?

People's privacy was violated. End of story.

[info]amaranthine3

7 months ago

[info]spiffynamehere

October 27 2011, 20:33:59 UTC 7 months ago

Yes, I find the issue of people being able to access others' locked entires, private messages, and private information to be not at all a security issue.

What planet do you live on?

[info]resonant

October 27 2011, 23:38:15 UTC 7 months ago

Let's see if their logic works with other things

I have had a paid account since 2001. When it comes time to renew, there may be an issue with transferring funds from my account to yours. This will have no impact on payment, as no strangers will actually be able to do anything with the money while it stays in my pocket.
Previous
← Ctrl← Alt
Next
Ctrl →Alt →
Create an Account
Forgot your login or password?
Facebook Twitter More login options
English • Español • Deutsch • Русский…