This is the local data center connection option I was referring to, the Coresite data centers on Stender Way (in Sunnyvale) had this option.
There is a latency spike between local and Amazon infrastructure so it would be critical to build your system so that putting this spike in the mix didn't impact your own flow path.
I had thought a bit about how we might do that at Blekko and I would probably shift crawling into the cloud (user's don't see that compute resource) and move crawler hosts over to the frontend/index side. But I'm sure there would have been a bunch of ways to slice it.
There is a latency spike between local and Amazon infrastructure so it would be critical to build your system so that putting this spike in the mix didn't impact your own flow path.
I had thought a bit about how we might do that at Blekko and I would probably shift crawling into the cloud (user's don't see that compute resource) and move crawler hosts over to the frontend/index side. But I'm sure there would have been a bunch of ways to slice it.