I first heard of this free tool by Christian Wojner while reviewing a project done by a couple graduate students from the SANS Technology Institute. It is popular among the forensics population. There is a nice, but older posting on the SANS Digital Forensics blog. Densityscout runs on Windows and also Linux and it worth playing around with, you might just find a gem or two. I put it to use yesterday when a neighbor contracted a bad case of Scareware. Densityscout knows the tricks malware authors use to try to hide their programs from antivirus or other methods of detection. So instead of looking for things like the image of the checksum they look for things like run time packing, or encryption.
In the screenshot, notice the difference between the TOR files and the other files. They stand out and would be good candidates for further inspection.
I have used a much cruder technique to look for encrypted network traffic. Get rid of the header information and cat the hexadecimal payload into sort | uniq -c if there are almost the same number of each hex character they payload is probably encrypted.