I just received the following e-mail the ASN
On Tuesday, March 22, 2011, your HighWire sites began receiving a flood of unusual requests for PDFs. What we observed was that a single click to request a PDF of an article would result in multiple requests for the PDF being sent to your sites. In some cases a single request was being multiplied more than 100 times. This caused an overload on the systems responsible for serving PDFs. Early impacts, around 5:00 to 6:00 AM (PDT), were felt as slow response for PDF serving but as the day moved on and traffic on the sites rose the impacts increased until the whole sites were slow as the PDF requests backed up, in the 8:00-9:00 AM (PDT) hours.Wow, that's cool. Both that they fixed it so fast and that they cared enough to send a note. Go Team ASN!
Our first response was to turn off access to the PDFs so that the sites would continue to serve other pages well. This was effective and allowed us the breathing room to analyze the traffic more carefully, and we were able to localize the large majority of these multiple requests as coming from users of the most recent version of Google's Chrome browser, which was just updated in the last few days.
At approximately noon on Tuesday we blocked the use of the most recent version of Chrome from all H20 sites. We regretted taking this action (I use Chrome as my browser, too) but it was the way we could restore full service to more than 90 of your users and still allow the other 10 the option to choose a different browser and receive full service as well. Chrome was blocked from approximately noon until 5:15 PM, when we were able to deploy a fix for the Chrome bug.
During the time that Chrome was blocked we were able to perform diagnosis and to determine that for many PDFs, Chrome was requesting the file 32Kbytes at a time. What this mean is that for some large PDFs a single click of the mouse was generating 11,000 requests for the PDF. We were also able to determine how to prevent Chrome from making these requests and we built a fix that was deployed t0 all H20 sites at 5:15 PM.
At the time of this writing all sites have been restored to full function and all users are receiving full service. We have no reason to believe this problem will recur and we deeply regret the impact that the Chrome update had on you and your readers.
