Deep Web Search – A How-To Site

Deep Web Search Tools for Everyone
  • What is the Deep Web?
    • How much of the Deep Web can be explored for free?
    • Deep Web vs Surface Web
  • How to Research
    • Deep Web Research Tools
      • Reverse Image Lookup
    • Data vs Information vs Analysis
    • Maximize your Shallow Web first
    • How to Research Right – Sources
    • Search engine for academic scholarship and thesis
    • Vetting Sources
      • Vetting Sources Tutorial
      • Vetting Authors of Secondary Sources
      • 10 Steps for Vetting Internet Sources
    • Deep Web Search Engines
  • Commercial Deep Web
  • OSint
    • OSInt 101 – Videos
    • Intelligence Analyst Resources and White Papers
    • OSInt example
    • Free Bayesian Software
  • Blog
  • About
RSS

Deep-Web on Facebook!

Relavant Ads

Categories

  • Deep web
  • deep web resources
  • news
  • OSInt
  • Research methods
  • Uncategorized

Pages

  • About
  • Blog
  • Commercial Deep Web
  • How to Research
    • Data vs Information vs Analysis
    • Deep Web Research Tools
      • Reverse Image Lookup
    • Deep Web Search Engines
    • How to Research Right – Sources
    • Maximize your Shallow Web first
    • Search engine for academic scholarship and thesis
    • Vetting Sources
      • 10 Steps for Vetting Internet Sources
      • Vetting Authors of Secondary Sources
      • Vetting Sources Tutorial
  • OSint
    • Free Bayesian Software
    • Intelligence Analyst Resources and White Papers
    • OSInt 101 – Videos
    • OSInt example
  • What is the Deep Web?
    • Deep Web vs Surface Web
    • How much of the Deep Web can be explored for free?

New site added for OSInt video

Feb15
by admin on February 15, 2013 at 5:46 pm
Posted In: deep web resources

While researching on the current massive meteorite strike in Russia this morning, one deep web resource has shown how useful it is for extracting out videos for local OSInt.

http://www.liveleak.com

It is being added to the deep web list

 Comment 

Using OSInt research and Bayesian stats to predict Turkey imposing a No-Fly zone in Syria

Oct14
by admin on October 14, 2012 at 4:10 am
Posted In: OSInt

The title says it all.  We’re going to try to give a meaningful probability of the Turks imposing a No Fly zone for the Syrian Air Force over part or all of Syria.

Now usually, when people here the word statistics, they seek out the nearest bathroom and try to hang themselves. But hold on, my curious friend, this is going to be an interesting trip…

Methodology

  1. Research and gather OSInt (open source intelligence) which are indicators (events that tend to hint) of Turkey imposing a No-Fly zone
  2. Outline the possible indicators with probability of happening before a No-Fly zone is imposed, and happening anyway.  This is normally done by an analyst with expert knowledge, but I’ll take a crack at it.
  3. Line out givens, such as the starting probability.
  4. Use Bayesian math to quantify expert opinion into stats

Research

Factors

Shallow and Deep Web research pulls up these current factors-

  1. Syria shot down a Turk jet fighter over international airspace. Turkey called for UN condemnation:  P(e|h) = 0.80  & P(e|nh)=.85
  2. Turkish Popular support at home is against conflict with Syria: P(e|h) = 0.2  & P(e|nh)=0.2 (not an indicator)
  3. Turkey army placed on high alert 10/11 after imposing a commercial no fly zone over syria: P(e|h) = .70  & P(e|nh)=.50
  4. Syria closes its airspace to Turkish flights. 10/13:  P(e|h) = .80  & P(e|nh)=.40
  5. Syria shelled a little into Turkey, the Turks didn’t take kindly to it and shelled back heavily.: P(e|h) = .85  & P(e|nh)=.80
  6. Military sources in Ankara also disclosed that at least 25 F16 fighter jets had been transferred to the Diyarbakir air base near the Syrian border. (Debkafile): P(e|h) = .85  & P(e|nh)=.65
  7. The Turks intercept a civi airliner from Moscow to Damascus based on intel there was military contraband onboard. Tensions heated up with the Russians and increase probability Russia may take action against Turkey. P(e|h) = .80  & P(e|nh)=.50

After each item item I have 2 numbers – P(e|h) = the event happens given Turkey plans  a no flight zone  & P(e|nh) that the event happens given Turkey does not plan a no-fly zone

Factoring all of these into an intuitive estimate I currently give the estimate to be a 40% probability of Turkey imposing a military no fly zone for the Syrian AF in the next 60 days.  This is just a gut instinct number. We’ll compare it to real math later.

I’ve also got to be careful not to fall under the intel analyst fallacy of ‘recency’ which is to overweight recent evidence like the civil air no-fly ban. Which I really really think should be heavily weighed.

Again, this part is where an good analysts expert opinion really helps the accuracy.  I’m not an intel analysis, I just play one on TV.

Hypotheses

>> What we seek! – P(h|e) The probability that h (our hypothesis) is true GIVEN (the | means ‘given’) that the evidence/event has happened.

P(h)= That probability the Turks will impose a no fly zone on the Syrian AF in the next 90 days.

P(e) = The probability that the event/evidence would happen whether the hypothesis is true or not.

P(e|h) = The probability that the evidence would happen given the hypothesis is true.

I’m going to put my neck out on a limb and put down an estimate of P(h) of the probability Turkey would make a no fly zone in any conflict. This is without knowing any current events, just that a conflict is underway.  If I had time, I’d go back and research all conflicts Turkey has fought since 1960. Then basically count the times it has done a no-fly zone divided by conflicts fought.  In this case, I’ll just throw down 10% as a SWAG. Thus P(h)=0.10

Using Bayesian statistics for Prediction

Basic Bayes theorem is

P(h|e)=    (P(h) * P(e|h))    /  P(e)

For prediction purposes and ease of use, the intel community has reordered the thing for easy updating to:

P(h|e)/P(nh|e)=[P(h)/P(nh) x P(e|h)/P(e|nh)]

and replace the awkward equation with symbols to get

R = P x L

So P is merely P(h)/P(nh) = 0.1/ (1-0.1) =  0.1111

Just to be clear here, my estimation of a no fly during a conflict without any events is a swag of 0.1 — therefore the P(nh) is (1-0.1)=0.9  because they either DO impose a no-fly or they DON’T.  It has to add up to 1.0.

where L= P(e|h)/P(e|nh)  which, as you notice is a ratio (Called the Likelihood). And you may notice that those numbers are estimated up top under my section called Factors.

Now here we go. I’m going to compute all the ratios under the section Factors in order from #1 – #7

0.8/0.85 x 0.2/0.2 x 0.7/0.5 x 0.8/0.4 x 0.85/0.8 x 0.85/0.65 x 0.8/0.5 = 5.858

So back to R = P x L

R = 0.1111 x 5.858

R = 0.6508

Remember R = P(h|e)/P(nh|e) is just a ratio aka Odds.   This is saying there is a ~2/3:1 chance that Turkey will put the kabosh on Assad.

So don’t think this is a percentage aka 65%. Nope, that is wrong. The answer could just as easy been 2.23 and you can’t say 223% (not only a no-fly zone over Syria but Iran, too!). Therefore, we must convert to a percentage

Percentage = odds/ (odds +1)

= 0.6508/(0.6508 +1)

= 39.4233 % and to compensate for significant figures (we’re making guesses, duh) we round to 1 decimal.

=  40 % chance that Turkey will impose a no-fly zone inside Syria on Syria’s air force.  Take note that my intuitive estimate was 40% so it passed my reality check.  But the cool thing here, is analyst can argue not about the final number, but the P(e|h) & P(e|nh) given to each factor.  It becomes a sort of accounting and audit  method for predictions.

The cool thing about Bayes, is if more Factors come to light we can update our Ratio.

So, I reach out to all interested parties.  If you think we should change some of our probabilities on the Factors, then put a comment below.  If  a new factor arises, post your P(e|h) & P(e|nh)  for that factor.

+++++++++++++++

Update 16OCT12, yet again

I’ve been give lots of feedback to take in new factors. Main flaw is that I have ignored negative factors, so I will update my Likelihood factor.

Last left at L=5.858

Additional Factors

  • Syrian Air Defences are superior and will require neutralizing: P(e|h) = .20  & P(e|nh)=.40
  • Political Will by Turkey to get involved in Syria : P(e|h) = .4  & P(e|nh)=.6

new R= 0.11 x 5.858x  .12/.4 x .4/.6

R=0.21479

Percentage = .21470/(1+.21479) = 17%

With 1 sig fig it rounds to 20%  which I think is more believable.

++++++++++++++++++++

Update 17OCT12

New Factors

  • Russia has sent  Technical Advisers to set up S-300 anti-aircraft missiles to aid Syria and decrease the chances of Turkey or the West will attempt airstrikes or impose no-fly zones. Source is  London-based Arabic language Al Quds-Al Arabi, according to News7. P(e|h) = .4  & P(e|nh)=.5
  • Russia has installed advanced radar systems at key Syrian military and industrial installations. Radar will also cover as the Incirlik military base in Turkey, from the Syrian side. Same source as above. P(e|h) = .40  & P(e|nh)=.45
  • Moscow veiled threat to become involved to protect Syria. Source same as above. Unvetted. For sake of exercise, assumed accurate. The P() on this is more subjective than others.  P(e|h) = .8 & P(e|nh)=.85

Last R=0.21479 call it R1

R = R1 x .4/.5 x .4/.45 x .8/.85

= 0.14375

Percentage R/(1+R) =  12.56%  and for 1 sig fig becomes…

= 10% chance Turkey will impose no-fly zone on Syria in the next 90 days.

FTR – Now would be  a good time to review all the factors and rank the ones that affect R the most to the least. In this way, I could update my percentage of P() to more valid numbers and reduce the bias caused by recency on some events. ie. Syria closing it’s airspace to Turkish flights was given P(e|h) = .80  & P(e|nh)=.40 which I believe now to be too much weight on that event.

Now I see that my original gut instinct was way to high. But this one seems a bit low. However, Bayesian forecasting is proven to be better than the expert who gives the P() for factors. And I am definitely no expert on Syria.

└ Tags: bayesian statistics, prediction
 Comment 

Fighting the system – Trying to take a Deep Web asset into Shallow Web

Oct07
by admin on October 7, 2012 at 12:24 am
Posted In: news

Some brave, bold academic decided to stand up to a commercial pay-fee deep website that holds academic articles. The result you will read about below, but in essence, bad things happened.

This highlights a moral battle. There is a a great deal of financial gain to the gatekeepers of wonderful data behind fee-pay portals of Deep Web sites.  Controversy erupts when the data can be argued to be public domain.

 

From Aljazeera News –

On July 19, 2011, Aaron Swartz, a computer programmer and activist, was arrested for downloading 4.8 million academic articles. The articles constituted nearly the entire catalogue of JSTOR, a scholarly research database. Universities that want to use JSTOR are charged as much as $50,000 in annual subscription fees.

Individuals who want to use JSTOR must shell out an average of $19 per article. The academics who write the articles are not paid for their work, nor are the academics who review it. The only people who profit are the 211 employees of JSTOR.

Swartz thought this was wrong. The paywall, he argued, constituted “private theft of public culture”. It hurt not only the greater public, but also academics who must “pay money to read the work of their colleagues”.

For attempting to make scholarship accessible to people who cannot afford it, Swartz is facing a $1 million fine and up to 35 years in prison. The severity of the charges shocked activists fighting for open access publication. But it shocked academics too, for different reasons.

 

http://www.aljazeera.com/indepth/opinion/2012/10/20121017558785551.html

└ Tags: aljazeera, jstor
 Comment 

OSInt makes the grade

Sep18
by admin on September 18, 2012 at 1:31 am
Posted In: news

http://www.strategypage.com/htmw/htintel/articles/20120917.aspx

From Strategypage – September 17, 2012: Last July the U.S. Army issued a manual, Army Techniques Publication 2-22.9, on using open source (mainly searching the Internet) intelligence. Also called OSINT, the troops have been using the Internet for intelligence work for over a decade. The publication of ATP 22.9 is a way for the senior army leadership to say, “message received and understood.” ATP 22.9, despite all the useful tips it contains, won’t go far in helping the many soldiers already using the Internet but it will be useful in convincing their bosses that much useful stuff can be obtained from the Internet.

While the U.S. intelligence community officially recognized the importance of OSINT (Open Source Intelligence) back in 2004, it was years before there was a lot of enthusiasm from the top brass for using this growing source of information.

The Internet has made OSINT a really, really huge source of useful intelligence. It’s not just the millions of gigabytes of information that is placed on the net but the even more voluminous masses of message board postings, blogs, emails, and IMs (instant messaging) that reveal what the culture is currently thinking. It was corporate intelligence practitioners who alerted the government intel people to the growing usefulness of Internet based data. Corporations have developed, over the last few decades, a keen interest in gathering intel on competitors, new markets, and all manner of things that might affect them. This “competitive intelligence” (or corporate spying) became big business. The Internet has made this a much more useful exercise.

However, corporate intel specialists were concerned that government agencies, especially the CIA, were not taking sufficient advantage of OSINT. Part of the problem was cultural. The intelligence agencies have always been proud of their special intel tools, like spy satellites, electronic listening stations, and spy networks. Most of these things are unique to government intelligence operations. People who use this stuff tend to look down on a bunch of geeks who simply troll the web. Even when the geeks keep coming up with valuable stuff, they don’t get any respect. The fear was that some foreign countries were exploiting OSINT more effectively than the United States. No foreign intel agency will admit to this but there are indications that some nations are mining the Internet quite intensively and effectively.

This fear grew as China, Russia, and other nations were caught using the Internet for direct espionage (hacking into other nations’ networks). While examining that threat it was discovered that the heavy use of OSINT was part of the hacking operations. Thus over the last five years the CIA and other major intel agencies got more enthusiastic about OSINT, and this made it easier for Internet-savvy army leaders to get ATP 22.9 into print.

j

 Comment 

Deep Web Lectures added

Sep17
by admin on September 17, 2012 at 1:34 am
Posted In: deep web resources

Ok, so maybe I missed a few lectures in college. Perhaps quite a few.  But via deep web research, I’ve found an excellent deep web science lecture resource for high authority, peer reviewed, lectures covering science and technology.  Known at videolectures.net and provided free of charge to the public. The library contains 15,000 lectures of which 66% are in English. Provided to us for free courtesy of the Center for Knowledge Transfer subsidized by Slovenia. Hats off to those guys.

└ Tags: deep web, video lectures
 Comment 
  • Page 3 of 7
  • «
  • 1
  • 2
  • 3
  • 4
  • 5
  • »
  • Last »

Sponsored Links

Below are advertisers that we have vetting and we believe in their product.

Research Companies

Intelius - Public Records Information

Recent Posts

  • Back up after hack
  • Removed dead links added Online Archive of CA
  • Added Shodan
  • Slideshare.net added to engines
  • Findsmarter.com added

Archives

  • August 2020
  • June 2016
  • June 2015
  • March 2015
  • September 2014
  • June 2014
  • May 2014
  • April 2014
  • May 2013
  • March 2013
  • February 2013
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • February 2012
  • January 2012
  • December 2011

©2011-2020 Deep Web Search - A How-To Site | Powered by WordPress with Easel | Subscribe: RSS | Back to Top ↑