We a short while ago experienced a consumer who is a multi-national retailer with both equally a actual physical and Online presence. The consumer essential a way to obtain certain business intelligence (BI) data from the Internet on a each day basis. Soon after many unsuccessful tries to develop this features themselves, they came to us for a resolution.
On the surface the specifications appeared to be difficult and it was easy to see why their very own IT workforce experienced unsuccessful to locate a alternative. They were thinking “inside of the box”, nevertheless, and hadn’t regarded 3rd-celebration alternate options. google index download expected that the software perform all of these responsibilities:
Retrieve new product or service listings on competitor’s website web sites.
Retrieve present-day pricing for all goods shown on competitor’s internet web pages.
Retrieve entire textual content of competitor’s Push Releases and community money stories.
Keep track of all inbound one-way links pointing to competitor’s net web sites from other internet websites.
The moment the information was obtained it essential to be processed for reporting purposes and then saved in the details warehouse for future accessibility.
Following reviewing present-day world wide web-dependent details acquisition technological know-how, including “spiders” which crawled the Web and returned details which then had to be processed by HTML filters, we determined that the Google API and Web Providers available the ideal alternative.
The Google API delivers distant obtain to all of the lookup engine’s uncovered features and offers a communication layer which is accessed by means of the “Simple Item Access Protocol” (Soap), a web providers standard. Since Soap is an XML-based engineering it is very easily built-in into legacy website-enabled apps.
The API satisfied all of the requirements of the application in that it:
Furnished a methodology for querying the World wide web working with non-HTML interfaces
Enabled us to agenda frequent look for requests designed to harvest new and current information on the focus on subjects.
It presented details in a format which was able to be quickly built-in with the client’s legacy programs.
Employing the Google API, Soap and WSDL, our developers were being capable to determine messages that fetched cached pages, searched the Google document index and retrieve the responses with out possessing to filter out HTML or reformat the details. The resulting details was then handed off to the client’s legacy systems for validation, reporting and even more processing in advance of achieving the facts warehouse.
Throughout the Proof of Concept stage we ran tests wherever we ended up in a position to reliably establish and retrieve up to date public relations and investor relations details that exceeded the client’s anticipations.
In our future test we retrieved the most presently out there solution webpages which have been mentioned in Google and then ran a different query to retrieve the Google “cached page” versions. We ran these two data sets by big difference filters and had been in a position to create correct rate boost and reduce experiences as well as identify new solutions.
For our last take a look at we utilized the Google API’s potential to accessibility the “url:” attribute to speedily develop lists of inbound back links.
These restricted tests shown that the Google API was able of making the BI details that the shopper asked for as perfectly as demonstrating that the details could be returned in a pre-defined structure which eradicated the have to have to utilize submit retrieval filters.
The consumer was pleased with the results of our Evidence of Strategy phase and approved us to move forward with setting up the resolution. The software is now in day-to-day use and is exceeding the client’s efficiency expectations by a vast margin.