Allan Goes to War | TechSNAP 5

Allan Goes to War | TechSNAP 5

We’ll cover the dirty details of a Facebook flaw that exposes your private account info to snoops, look into the privacy issues around “Smart Meters” and discuss a few big tech rivals coming together to fight a bad law.

Plus Allan shares one of his many war stories in our first installment of our continuing series!


Show Notes:

Topic: Facebook app platform flaw exposes personal data

  • Older form of user authentication used by some facebook apps returned a token that could be used for both read and write access to a users account
    • Access to personal information
    • Write wall posts and private messages
    • Send invites and RSVPs
    • Access photographs and other objects
  • These tokens can be leaked via the referrer field or given to advertisers and other third parties who you did not authorize, when the tokens should only be able to be used by the application you did authorize
  • The flaw was fixed when facebook switched to the standardized OAUTH API that works differently and requires public and private keys, but the old API is still supported to avoid breaking existing apps
  • Facebook is phasing out the old method, by September 1st all apps must use OAUTH 2.0 and by October 1st all apps must have an SSL Certificate.
  • Changing your password invalidates all old tokens

Topic: Google and Facebook oppose California Law against tracking cookies

  • would create an unnecessary, unenforceable and unconstitutional regulatory burden on Internet commerce
  • Would require sites to allow users to opt-out of storing information such as:
    • date and hour of online access
    • the location from which the information was accessed
    • The device and its operating system used for access
    • IP addresses
  • Would kill google analytics and other such products
  • Google also claims it would reduce online safety and increase fraud
    • Many e-commerce providers use details such as where the user is located, previous access patterns, referral information etc, to help determine the validity of online transactions.
  • The Federal Trade Commission and Department of Commerce have also promoted the idea of self-regulation

Topic: Is your electric meter spying on you?

  • The California Public Utilities Commission proposes new privacy regulations to prevent utilities from sharing or selling your smart meter data
  • Such data could be used to build a complex behavioural profile on you
    • when you wake up
    • when you shower
    • when you leave for and return from work
    • when you go on vacation
  • Where is this information stored? How is it protected? could it be compromised like PSN/SOE was?

War Story: When it goes wrong, it goes very wrong

  • Background: We were co-located with the local cable company, they had been offering data center services for years and were rapidly expanding and becoming a major force in the area. We particularly liked the data center because it was a block from my house in the event of a problem. However, after a few years there, the local cable company was bought out by a major national cable system. This new company decided that data center services were outside of their business focus and requested that all customers find a new home.
  • So we signed a contract with a new data center and arranged with our customers to physically move half of our equipment over two consecutive weekends. In preparation for this we shifted load away from the gear we were going to move and adjusted our router configuration, leaving the redundant router as the primary at the old location. However, to save reconfiguration time during the final phase of the move, the new configuration on the backup router was only the ‘running’ config, not the ‘boot’ config. The boot config was setup so that when the router was physically moved next weekend, it would be read to go online immediately.
  • Things were all going according to plan until about an hour before we were scheduled to being our move. A massive power event knocked out utility power to about a 40 block radius that included the data center. This was obviously nothing to be concerned about, this was a real data center, with redundant battery backups and a generator system.
  • The servers kicked over to the battery backup, and things went well for the first few minutes. Once the system decided that the utility power was not going to be restored quickly, it spooled up the generator, which came online cleaning and started generating power. Once the generator was online, the system attempted to switch to generator power, however this failed, so the system resumed off of battery power, and sent out an alarm to the data center staff. The system attempted the transfer again, but again power was not getting from the generator to the battery backup system.
  • Eventually, after about 15 minutes, the batteries ran flat and the entire data center went dark. The catastrophic loss of power also took out my home Internet connection from the same provider. Now, all of my servers were down, but I was unaware of the issue because my own Internet connection was down as well. I quickly became aware of the situation when my offsite monitoring starting pinging my phone.
  • I called the ISP/Data Center and they said they had suffered a power loss, this explained why my home internet was down, but surely the data center should be fully functional. It wasn’t.
  • Once utility power was restored after a total of about 30 minutes, things started to come back online, however because our routers and servers had modified ‘boot’ configs in preparation for the move that was going to happen later that day, things did not come back online cleanly. We rushed to the data center and started reconfiguring the gear that was to remain in the old data center for another week.
  • Root Cause Analysis:
    • Due to lack of maintenance by the new management of the data center, the transfer cables between the generators and the battery banks had corroded and failed to transfer power
    • Due to the closing of the business unit, the data center was understaffed to deal with the catastrophic event
  • Lessons Learned:
    • Don’t change your boot config until the last possible minute. The idea was to save time at the far end during the move, but it ended up costing us when the power prematurely changed our configuration
    • Internal directories services need to be redundant, reconfiguring the servers took an excessive amount of time because the servers were unable to lookup user to uid mappings
    • Make sure your SSH server does not wait for DNS
    • When a data center is going down hill, GET OUT. We were one of the first customers to leave, but we wish we had left 2 weeks sooner.



Question? Comments? Contact us here!