IP to ASN lookup, using Redis

Hello all,

Just recently, I've had the need to map an IP address to an AS number in a quick manner for use in another project. I started by looking for methods of obtaining the data neccessary for creating the IP to ASN map. I do not personally own a router than is running full BGP tables and did not want to abuse one of the looking glass routers for obvious reasons. On my search, I came across two well formatted indexes. One mapping masks to ASNs and the other mapping ASNs to owner names.

Netmask to AS number

AS number to AS owner

This is great, because I could create to arrays or hashes, including the keys/values. Now since I am new to Ruby, I started out by searching for the suitable libraries to scrape the data from the two above maps. I found a nice ruby-curl library on GitHub, which had some nice features, that would allow me to Curl directly into an array using regex with it's scan() method... how nice! After creating my initial script and pulling the data, for some reason the Curl wouldn't fully populate the array... it would stop between 80.x.x.x and 90.x.x.x every time. After messing around with it a bit, I couldn't get it to work how I wanted, so I switched to using curb, which didn't have the nice scan() method, but I was able to play with the data just how I wanted in no time. I was able to get the script working by putting the key/values into a Ruby hash and it was easy to pull the values out how I wanted but this would mean scraping the data from the above tables every time I ran the script. After some searching, I decided to go with Redis due to the simple model of data I was playing with (just a bunch of key/values). Now the fun part begins; storing the data for persistence. The Redis Ruby client is awesome and very easy to use. You import the values with redis.set and call them with redis.get... nice and simple. For example:

irb(main):007:0> redis.set("8.8.8.0/24", "15169")
=> "OK"
irb(main):008:0> redis.set("15169", "Google Inc.")
=> "OK"
irb(main):009:0> puts "Hey! Don't touch ASN #{redis.get("8.8.8.0/24")}! That's owned by #{redis.get("15169")}"
Hey! Don't touch ASN 15169! That's owned by Google Inc.
      

Since this script needs to support ANY IPv4 address but the table consists of only the netmask used by the specific AS, which may not be in the same /24 or whatever mask easily identified by simple string matching, makes this a little bit more difficult. What we need to do, is convert the IP address into a value that can easily be compared to any other and if there is no match, check all masks until there is a match (Example: for a /24, ... /32, /31, /30, /29 ... until there is a match, which eventually you will run into). This is accomplished by converting the IP to long (each octet using 8 bits, 32 bits total). We would never do this for normal subnet math, because after 8 bits, our bit positions are larger than we would see in an octet i.e. 256, 512, ... (2^31). But this allows us to store the IP in a single 32-bit unsigned integer, instead of four 8-bit integers like we see in the original delimited address. Ruby does not have a built-in function for this, so after researching and testing, I was able to come up with:

ip = 8.8.8.8
ipAry = ip.split(/\./)
long = ipAry[3].to_i | ipAry[2].to_i << 8 | ipAry[1].to_i << 16 | ipAry[0].to_i << 24
      

For 8.8.8.8, this should effectively return 134744072. The script would then take that long IP and search redis for all 32 bits first and if no match is found, 31 left-most bits, and so on, until we find the correct network boundary match. It will continue searching until redis.get(?) isn't nil. For example:

# Stripping 0 right-most bits, /32 bit strictest match fails
# (134744072 >> 0) << 0 = 134744072
redis.get(134744072) = nil

# Continuing, we get a match stripping 4 right-most bits, /28 bit match
# (134744072 >> 4) << 4 = 134744064
redis.get(134744064) = 15169

# Even though this is a /24 network, we match because bits 25-28 are also zeroes, so the ip to long conversion is the same
134744072.to_i.to_s(2) = 00001000000010000000100000001000
      

Now that the logic part is finished, now time to make the script usable and make some final housekeeping changes. The ip2long code was put into a method, along with the search (bit strip 0-31) for a matching long netmask. I then added argument support, so you can pass the IP address to the script. Since the Redis import would occur every time, I had to find a nice way to stop this. Counting the keys was an option, but the "redis.keys('*').count" would take up to 10 seconds to complete, which wouldn't give me the fast lookup I want to achieve. I then decided to just use optparse and choose an option to initiate the Redis import, otherwise, it would just skip it and run the script as if the necessary Redis keys were already there. So loading the data once with quick subsequent searches. The final code I ended up with is:

#!/usr/bin/env ruby

require 'curb'
require 'redis'
require 'optparse'

def ip2long(ip)
  ipAry = ip.split(/\./)
  long = ipAry[3].to_i | ipAry[2].to_i << 8 | ipAry[1].to_i << 16 | ipAry[0].to_i << 24
  long
end

def getMask(ipLong)
  redis = Redis.new
  (0..31).each {|msk|
    ipRef = (ipLong >> msk) << msk
    if redis.get(ipRef).nil?
    else
      return ipRef
    end
  }
end

def redisImport
  redis = Redis.new
  redis.flushall
  http = Curl.get("http://thyme.apnic.net/current/data-raw-table")
  http.body_str.each_line {|s|
    s.scan(/([0-9.]+)\/[0-9]+\s+([0-9]+)/) {|x,y|
      z = ip2long(x)
      redis.set(z, y)
    }
  }
  http = Curl.get("http://thyme.apnic.net/current/data-used-autnums")
  http.body_str.each_line {|s|
    s.scan(/([0-9]+)\s+(.*)/) {|x,y|
      redis.set(x, y)
    }
  }
end

userIp = ARGV[0]
redis = Redis.new

ARGV.options do |opts|
  opts.on("--charge") { puts "Charging Redis..." ; redisImport ; exit }
  opts.parse!
end

ipLong = ip2long(userIp)
zMask = getMask(ipLong)
zAsn = redis.get(zMask)

puts "(" + userIp + ") belongs to ASN " + zAsn + " - " + redis.get(zAsn)
      

Results

The initial Redis import is initiated with the --charge option when running the script. The import usually takes around 1 minute to complete but the IP to ASN/owner queries only take around 200ms afterwards, which is pretty fast. When running the script after the initial "charge", the script is getting the values directly from Redis and isn't using Curl to receive the values anymore. This is the nice thing about using a persistent cache, is that you can run the script over and over without losing the data as you would if you stored the data in a Ruby array or hash. If you want to use this script, make sure you have redis-server running and make sure you have the Ruby libraries installed, which you can install with gem install. Here is how the script functions:

[rsty@home ~]$ ruby ip2asn.rb --charge
Charging Redis...
[rsty@home ~]$ ruby ip2asn.rb 4.2.2.2
(4.2.2.2) belongs to ASN 3356 - Level 3 Communications, Inc.
[rsty@home ~]$
      

Leave any comments if you have any suggestions or tips to enhance this script or make it more efficient. Or just for general discussion.. Here is the GitHub repo: https://github.com/nckrse/ruby-redis-ip2asn