Write some Software

We are seeking a computer programmer to build a custom web scraper for the World News Connection database that is hosted by East View. The scraper would be using a custom list of inputs and the output would be the number of hits (the number of unique articles that mentions the unit of analysis) for each input for a specified time interval. The output would need to include results by each time unit (Day, Week, Month, Year) . We would prefer the programmer to use a publicly available, open source scraper, such as Scrappy, so that the scraper/crawler could be adjusted if there were changes in our needs or to the World News Connection database. We would provide access to the backend code of World News Connections. The scraper would require the following features: 1) The ability to uploaded two separate lists of search terms to the scraper This first list are the units of analysis (UOA) and the second are keywords. For each UOA, a search must be conducted with each keyword. Both lists should be included in quotes (""). An example below shows two lists, one of units of analysis and one of keywords. UOA Keywords Obama Iran Putin Syria China Using these units of analysis, each item should be searched as follows: “Obama” AND “Iran” “Obama” AND “Syria” “Obama” AND “China” “Putin” AND “Iran” “Putin” AND “Syria” “Putin” AND “China” A user can search World News Connections using only Units of Analysis but CANNOT search using only keywords. 2) The user should be able to select the time start/end date in days, weeks (Monday to Sunday), months or years from January 1st, 1995 to December 30th, 2013(These are the dates for the entire database). The source date NOT the insertion date should be searched. A user could input these time intervals as a third list if needed, however we would prefer the user to be able to select only three items and the scraper would output these units itself. For example, if a user wanted to output a search for “British Petroleum” for each day from June 25th, 2012 to July 7th 2012, the user should only need to select three things: 1. Start Date: June 25th, 2012 2. End Date: July 7th, 2012 3. Interval: Day (Could be week, month, year). The output should then be shown as below: "Year" "Month" "Day" 2012 6 25 2012 6 26 2012 6 27 3) A GUI for the scraper so that we would be able to run the scraper at the time of our choosing and without reliance on the programmer. 4) A sustainable program that could be executed at any time by users. 5) The scraper must be able to search for the treaty names in the order the words appear “Charter of the United Nations” vs. “Charter”…”United Nations” 6) The output should be shown as follows: Unit of analysis "Keyword" "Count" "Year" "Month" "Day" Ukraine protest 0 2014 4 1 Ukraine protest 0 2014 4 2 Ukraine protest 3 2014 4 3 Ukraine protest 0 2014 4 4 Ukraine protest 0 2014 4 5 Ukraine crimea 8 2014 4 1 Ukraine crimea 7 2014 4 2 Ukraine crimea 7 2014 4 3 Ukraine crimea 5 2014 4 4 Ukraine crimea 9 2014 4 5 Thailand protest 1 2014 4 1 Thailand protest 4 2014 4 2 Thailand protest 5 2014 4 3 Thailand protest 0 2014 4 4 7) The scraper must be able to access WNC at varying intervals, determined by the user, so that we do not disturb the World News Connection server. The user can set how often to wait in between search terms in seconds or minutes. The developer should have proven experience in Python and Webscraping. Data science expertise or previous experience with an open source scrapping program is a plus. The source code of the search and results page and the associated links will be provided. If you have any questions, please feel free to reach out.