URL Expander
Overview
Details
Core Features:
- Expanding links masked by URL shortening services.
- Following/limiting redirect chains (ie. redirects that lead to more redirects)
- Low-level proxy capabilities for hiding your origin (if desired)
- Specifying a useragent string for use in retrieving URLs
URL Expander was built with security analysts in mind, specifically for use with log data containing shortened links where the actual location isn't revealed in the logs. For example, some email appliances may scan incoming emails for hyperlinks, but if any of those hyperlinks are short URLs, the real location may not be specified in the logs.
Usage
| expandurl field=<string> [redirect_limit=(<# of times to expand a daisy redirect of shortened links>)]
field: the name of the field containing the short URL.
redirect_limit: Defaults to 0, meaning redirects are not followed. Some short URLs will lead to other short URLs, which lead to other short URLs, etc. "redirect_limit" allows you to follows these redirects while specifying the max # of redirects to follow (avoiding an endless loop). This field is not required.
NOTE: The "expandurl" command works best if you narrow your search results to only URLs masked by shortening services. This can be done easily with a lookup table.
Output
expandurl will produce a new field for each row named "expanded_url".
Sample:
url,expanded_url
http://bitly.com/[short_link1], http://someexpandedurl1.com/stuff
https://goo.gl/[short_link2], http://anotherlongurl.com/stuff2
Examples
Example 1: Expand short URLS, don't follow redirects
index=your_index [inputlookup short_services.csv | return 10000 $short_service] | expandurl field=url | table url, expanded_url
In this example, short_services.csv is a lookup table containing a list of shortening services like "tiny.cc", "ow.ly", "bitly.com", etc. This narrows the search results. You should provide this table.
The expandurl command is applied to the "url" field of every row, outputting a new field named "expanded_url".
Example 2: Expand URLs and follow up to 3 redirects
index=your_index [inputlookup short_services.csv | return 10000 $short_service] | expandurl field=url redirect_limit=3 | table url, expanded_url
In this example, short_services.csv is a lookup table containing a list of shortening services like "tiny.cc", "ow.ly", "bitly.com", etc. This narrows the search results. You should provide this table.
The expandurl command is applied to the "url" field of every row, outputting a new field named "expanded_url". If the URL leads to a daisy chain of redirects, up to 3 redirects will be followed (hence the redirect=3 attribute).
Lookup Table
As stated above, it is best to narrow your search results by matching on a list of URL shortening services. Here is a look uptable you can upload as "short_services.csv" for use with the examples above:
short_service
7.ly
adf.ly
admy.link
al.ly
bc.vc
bit.do
doiop.com
fur.ly
good.ly
is.dg
lnked.in
lynk.my
mcaf.ee
ouo.io
ow.ly
ph.dog
qr.net
s.id
shrinkee.com
shrinkurl.in
sptfy.com
thinfi.com
tiny.cc
tinyurl.com
tny.im
urlkr.com
v.gd
y2u.be
zi.ma
zzb.bz
adfoc.us
bitly.com
coinurl.com
git.io
goo.gl
hec.su
shorte.st
tldrify.com
tr.im
Release Notes
Version 1.1
Fixed bug in requests