A Simple Mistake | TechSNAP 4

A Simple Mistake | TechSNAP 4

The guys focus on the recent major network compromises, and outages – and what was at the core of their failure. Like Sony’s PSN and SOE attacks, and the recent Amazon EC2 outages. What do these very separate events have in common?

Find out what simple mistakes snowballed into full-on network meltdowns. Plus the EU’s nutty plans to convince websites to prompt every user to sign a EULA for their cookies!


Show Notes:

Topic: SOE Breached as well, 24 million records stolen


  • Old database from 2007 compromised, 12,700 credit cards with expiry dates and 10,700 direct debit accounts
    • Old data was not destroyed, why?
    • Was this data not encrypted, as sony claims the PSN credit card database was?
    • most of these cards are likely expired, but some banks use extended expiration dates
    • direct debit accounts are likely more at risk, although harder to exploit
  • Sony says that PSN and SOE are isolated systems, but it seems the attacks are related
  • Data was stolen as part of the original compromise on April 16-17th (earlier than previously reported), not a separate compromise
  • If the data is separate, how were both databases compromised?
  • If the data is not isolated, why were SOE customers not notified weeks ago when the breech was discovered? More attempted cover-up by Sony.
  • SOE passwords are hashed (no specifics on algorithm or if they were salted)
  • Data includes: name. address, e-mail, birthdate, gender, phone number, username name, and hashed password
  • Unconfirmed rumours that the credit card lists have been offered for sales or to Sony
  • Sony offering customers from Massachusetts free identity theft protection service, as required by state law in the event of such a breech
  • It later came to light in congressional hearings in the US (which Sony declined to attend) that Sony was using outdated, known vulnerable software, and that this fact had been reported to them by security researches months before these attacks
  • Sony says that it has added automated monitoring and encryption to its systems in the wake of the recent attacks.

Topic: Wikileaks may have forced the US Government’s Hand


  • US knew that someone was hiding in the compound since at least last summer
  • US was unsure who was in the compound, believed it was UBL but were unsure, and unwilling to risk disclosing the depth of their penetration of the oppositions security
  • Classic Intelligence Paradox, what use is having the information if you cannot use it, but using it will expose your sources and methods.
  • The wikileaks release of Guantanamo documents exposed the US’s penetration of UBL’s courier network
  • US likely decided to move immediately to avoid squandering the opportunity

Topic: Stupid EU law of the week


  • Basically will result in users being met with mini-EULA asking you to opt in to cookies in order to enter every site on the internet
  • Law has a specific provision to allow cookies to be used to track the contents of your shopping cart
  • Cookies are an important part of web applications. HTTP is stateless, and cookies are the easiest and most convenient way to maintain state
  • Controls for cookies are best left to the browser, which decides and enforces policies on cookies
  • There already exists the ‘same-domain’ policy in all browsers, cookies can only be read by the site that set them
  • There exists a better alternative already supported by Google and Mozilla, the DNT (Do Not Track) opt-out system asks advertisers to not use or not collect behavioural data. Google’s system works slightly differently but accomplishes the same goal.
  • This is yet another example of governments passing laws without considering the technical implications of their implementation. Governments seem to purposefully avoid consulting actual experts and instead hire consultants that will agree with their position.

Topic: Image authentication system cracked




  • Digital SLR camera technology that signs photos with a private key when they are taken to allow their originality to be verified.
  • The image and the meta data are both hashed with SHA-1 (this is possibly insufficient, SHA-256 or better should be used for cryptographic security and future proofing)
  • The two hash values are then encrypted separately using a 1024-bit RSA key (again, insufficient key size, even SSL requires 2048 bit keys now) and stored in the EXIF data
  • The verification software then validates the signature and compares the hashes
  • Very similar system with similar flaw found in the Canon Original Data Security system. Neither Canon or Nikon have responded nor indicated they will address the issues
  • ElcomSoft managed to extract the private key and sign forged images that then passed verification
  • It seems all Nikon cameras use the SAME key, not separate keys per camera, so once the key is exposed, the entire system is compromised, not just the single camera

Topic: Amazon Post Mortem, some data loss


  • Original failure was caused by network operator error
  • Failure caused some data loss, a small portion but still significant
    • Online cloud services such as Chartbeat lost data
  • Replica system had no rate limiting, so when a large number of EBS volumes failed, the creation of replicas to replace them overloaded the centralized management system (the only shared part of the EBS infrastructure)
  • All Availability zones ran out of capacity, new replicas of data could not be created
  • EBS nodes that needed to create replicas as well as EC2 and RDS nodes backed by them became ‘stuck’ waiting for capacity to store replicas. Effected about 13% of all nodes in the availability zone.
  • Create Volume API calls have a long timeout, caused thread starvation as the requests continued to back up on the shared centralized management system (EBS Control Plane)
  • The overload of the control plane caused all EBS nodes in US-EAST to experience latency and higher error rates
  • To combat this, amazon disabled all ‘Create Volume’ API calls to restore service to the unaffected Availability zones
  • EBS control plane again became overwhelmed with other API calls caused by the degradation of the effected availability zone, all communications between the broken EBS volumes the control plane were disabled to restore service to other customers
  • Lessons going forward:
    • Rate limiting on all API calls
    • Limit any one availability zone from dominating the control plane
    • Move some operations into separate control planes in each availability zone
    • Increase stand-by capacity to better accommodate growth and failure scenarios
    • Increase automation in network configuration to prevent human error
    • Additional intelligence to prevent and detect ‘re-mirroring storms’
    • Increase back off timers more aggressively in a failure scenario
    • Focus on re-establishing connections with existing replicas instead of making new ones
    • Educate customers about using multiple-AZ (Availability Zone) setups to reduce the impact of partial failures of the cloud
    • Improve communications and Service Health Monitoring tools


Question? Comments? Contact us here!