From my IPV4 database, here are the the Registrar and Country statistics as of April 10 2009.

Since the last update, ARIN has crossed the 1.6B IPV4 address boundary, adding nearly 5M IPV4 Addresses. APNIC matched this growth by adding an additional 5M new addresses of its own.

Registrar      Number of IPV4 Adresses
---------      -----------------------
arin                    1,600,640,000
ripencc                   564,850,616
apnic                     528,362,496
lacnic                     79,931,136
afrinic                    21,526,272

The majority of the growth in APNIC continues to be driven by China, which added an additional 4M IPV4 addresses since March 14.

Country                                     Number of IPV4 Adresses
------------------------------------------  -----------------------
UNITED STATES                                            1472741888
CHINA                                                     191119104
JAPAN                                                     152412672
EUROPEAN UNION                                            114155680
GERMANY                                                    85174424
CANADA                                                     75846144
KOREA, REPUBLIC OF                                         72142592
UNITED KINGDOM                                             70675288
FRANCE                                                     68370880
AUSTRALIA                                                  36573184
ITALY                                                      32202432
BRAZIL                                                     29754880
TAIWAN, PROVINCE OF CHINA                                  24680704
RUSSIAN FEDERATION                                         24529224
SPAIN                                                      21794976
MEXICO                                                     21504000
NETHERLANDS                                                21290792
SWEDEN                                                     18982304
INDIA                                                      18285568
SOUTH AFRICA                                               14009344
POLAND                                                     13869704
TURKEY                                                     10417600
DENMARK                                                     9281632
FINLAND                                                     8932864
ROMANIA                                                     8640512
SWITZERLAND                                                 8249320
HONG KONG                                                   8206848
NORWAY                                                      7425584
AUSTRIA                                                     7290592
ARGENTINA                                                   7239424
INDONESIA                                                   6997248
VIETNAM                                                     6707456
BELGIUM                                                     6412416
NEW ZEALAND                                                 6115072
CZECH REPUBLIC                                              6040960
UKRAINE                                                     5507904
THAILAND                                                    4743936
CHILE                                                       4731136
PORTUGAL                                                    4473952
SINGAPORE                                                   4410880
COLOMBIA                                                    4261632
IRELAND                                                     4203680
MALAYSIA                                                    4147456
PHILIPPINES                                                 4070656
ISRAEL                                                      3949760
GREECE                                                      3834624
HUNGARY                                                     3716480
VENEZUELA                                                   3693056
BULGARIA                                                    3334912
EGYPT                                                       2731008
SAUDI ARABIA                                                2703616
UNITED ARAB EMIRATES                                        2286848
LITHUANIA                                                   2009216
IRAN, ISLAMIC REPUBLIC OF                                   1894400
PERU                                                        1715968
PAKISTAN                                                    1637632
CROATIA                                                     1623136
SLOVAKIA                                                    1611520
COSTA RICA                                                  1502208
LATVIA                                                      1384448
SLOVENIA                                                    1277952
PANAMA                                                      1130240
MONTENEGRO                                                  1083648
ESTONIA                                                     1021464

As China’s Internet growth continues, the number of addresses assigned to ISPs in that nation will continue to grow. However, it is unlikely that China will equal or exceed the US mainly due to the movement to mobile devices, which will be managed using private IP space rather than public IPs.

Tonight, I figured out how to add the Resolved IP Addresses for a host to measurement data and store that information for further debugging. It was very simple – I was trying to find complex solutions to this issue.

Turns out the solution is built right into PERL: The Socket module.

My thought is that I will update the table with the test config with three new columns:

• HTTPS/HTTP
• Hostname
• Page information

There will likely be a new table that joins with the raw data on

• Date
• Agent_id
• Test_id

And contains a comma-delimited list of all the IP addresses that the agent resolved the hostname to at the test time. This lookup will be run after the measurement, so the DNS lookup component of the measurement is not compromised.

I don’t have an ETA on this, as I want to test it fairly thoroughly before I expose the data. Adding the columns to the test config table will be transparent, but agent modification will need to be verified and then rolled out to all of the folks hosting measurement locations.

What problem does this provide a solution to?

It is vital for firms who use geographic load balancing and CDNs to verify that their data is being served from location appropriate IP addresses. I will be able to tie the information collected here into the IP-Location data I collect for other purposes and help companies ensure that this is being done.

On a daily basis, I update the Geographic IP database that I created many years ago. Although not as powerful as some of the commercially available Geographic databases, it has more than served my purposes over the years.

One of the benefits of collecting this data is being able to extract substantial metrics on the distribution of IPV4 addresses. This post is the latest in a series of descriptions of the distribution of addresses at the moment.

Here are the Registrar Stats for IPV4 addresses as of March 14 2009.

The ARIN IPV4 address space (which includes the US) is still the largest by far, with nearly 3 times the allocated IPV4 addresses of the two next largest registrars, RIPE and APNIC. The dominance of the US is even more noticeable in the IPV4 addresses by Country table.

Belying its growing importance on the Internet stage, China has grown from fourth place in the first of these analyses to second place in this study. However, it still has a long way to go before it catches up with the US.

An interesting concept that comes out of this data is that China is making do with substantially fewer public IPV4 addresses than the US is. This means that they have wholeheartedly embraced IPV6 (unlikely) or are using the private IP space for most communications.

The five top-level IP Registrars (ARIN, RIPE, APNIC, LACNIC, AFRINIC) manage tens of million IPV4 and IPV6 addresses, assigning them to networks around the world. These addresses are key to accessing the Internet for all end-users.

About once a year, I generate the statistics out of the IP address database I maintain. The last one was published back in August 2007,  and, as can be imagined, there have been substantial changes that have occurred in the last 12 months.

At the country level, there has been a substantial change in the top five as China has superseded Japan in the total number of IPV4 addresses. In fact, in the last 12 months, Japan has seen a net loss in the total number of allocated IPV4 addresses.

The adjustment in China is completely expected, as China has substantial room to expand its IPV4 population, compared to a country such as Japan which is likely approaching saturation in this area.

In the total number of IPV4 addresses assigned by registrar, ARIN (US, Canada, and parts of the Caribbean) still has a substantially larger population than the other regions. However, this region only grew by 3 million IPV4 addresses in the last 12 months, compared to 42 million for RIPE (Europe and the Middle East), and 74 million for APNIC (Asia-Pacific). LACNIC (Mexico, Latin and South America) grew by 13 million IPV4 addresses and AFRINIC (Africa) by 4 million.

Some of this growth in APNIC, AFRINIC, and LACNIC can be attributed to adjustments in the Registry structure itself. Now that there are five registrars with responsibility for a clearly defined set of nations, a number of adjustments have had to occur in the registrar of record for IPV4 ranges.

In the past, for nations that were not explicitly covered by a regional registrar, the IPV4 range may have been assigned by the registrar now responsible for the region, most likely ARIN or RIPE.

Now that there are registrars specifically designated to cover these formerly grey areas, these historical artifacts are being corrected as registry information is renewed.

However, this cannot directly account for the noticeable growth in AFRINIC, APNIC, and LACNIC relative to the two older registrars. This is yet another indication of the strong growth of the Internet outside of its established base of the United States, Canada, Western Europe, and a small number of industrialized nations in the Asia-Pacific region.

Updated September 16 2008 to reflect the numerous changes that have resulted since the original article was posted in 2005 – smp

IP to Country Mapping

The idea for IP to Country mapping is one that has started to appear more frequently on the Internet in recent months. All GeoIP systems do warn users that they are not 100% accurate. The accuracy of GeoIP mapping can be affected by things such as large corporate or ISP networks where traffic is routed out a small number of public access points, regardless of the traffic’s point of origin.

Making IP Addresses Searchable

The first issue that needs to be addressed is how to determine if an IP Address is in one of the ranges that is defined as originating from a distinct country. The simplest way to range-match IP Addresses is to abandon the dotted-quad notation we are all familiar with, and convert the IP Address to an IP Number.

All IP Addresses can be converted into decimal numbers that fall into a known range between 0 and (2^32)-1 (4294967295). In reality, the range is even smaller than that, as public IP Addresses fall between 0.0.0.0 (IP Number: 0) and 223.255.255.255 (IP Number: 3758096383).

A quick search of the Web showed me that there is a way to create functions to convert IP Addresses to their numerical (IP Number) equivalent and reverse the process.

sub ip2long {
	return sprintf("%u",unpack("l*", pack("l*", unpack("N*", inet_aton(shift)))));
}

sub long2ip {
	return inet_ntoa(pack("N*", shift));
} 

<--snip-->

if ($start_ip =~ /\d+\.\d+\.\d+\.\d+/) {
         ## my $ip_address = shift($start_ip);
         chomp($start_ip);
         $long_start = ip2long($start_ip);
         ## print "$ip_address converts to $ip_number\n";
 }

If you are using PHP in your applications, this conversion process is made even easier by native function calls.

Convert IP Address to IP Number
$ip_number = sprintf("%u",ip2long($ip_address));

IP Address Location Data

Now that we have settled on a format for the IP data to be used in the database, we now have to find IP data that allows us to map IP Addresses to countries. This is easier than it sounds, as this data is centrally held by the 5 regional IP Registries — ARIN, RIPE, APNIC, LACNIC, and AFRINIC. After poking around in the depths of their Web sites, I found that they actually provide text formatted versions of the allocated and assigned IP ranges that they are responsible for. All of the registries use the same format, which makes parsing the these files a simple process.

As of March, 2005, the distribution of IPV4 networks in the database by registry is:

+----------+--------------+
| registry | num_networks |
+----------+--------------+
| arin     |        36073 |
| ripencc  |        14813 |
| apnic    |        10474 |
| lacnic   |         1460 |
| afrinic  |          443 |
+----------+--------------+

5 rows in set (0.52 sec)

I chose to use the PERL module WWW::CURL to retrieve the files. You could re-write the application to use LWP or some other method on systems where cURL is not supported, as it is a simple file download over FTP. I update the data once a day, which may at first appear excessive. However, I have seen upwards of 40-50 new rows added to the database in a single day.

Some may ask why I chose to write the downloaded files to a file rather than immediately inserting them into the database. Using this two-step process gives me the ability to manually rollback to an older database if there is a problem retrieving one of the registry files. I have set an arbitrary limit of 75,000 lines for the entire aggregated file; if the file is less than that, the remainder of the process is aborted.

The data retrieved from the registries is in the following format.

Registry Raw Data Format

<snip>
apnic|CN|ipv4|202.127.4.0|256|19950610|assigned
apnic|BN|ipv4|202.160.0.0|2048|19950610|allocated
apnic|NP|asn|4613|1|19950611|allocated
apnic|LK|ipv4|203.143.0.0|1024|19950612|allocated
apnic|MO|asn|4609|1|19950615|allocated
apnic|KR|asn|4670|1|19950616|allocated
apnic|SB|ipv4|202.63.254.0|512|19950618|assigned
apnic|JP|ipv4|202.232.0.0|262144|19950618|allocated
apnic|SG|ipv4|203.127.192.0|8192|19950618|allocated
apnic|PK|asn|4615|1|19950629|allocated
apnic|HK|asn|4614|1|19950704|allocated
</snip>

The fields are all “|” (pipe-character) separated, and are described below.

COLUMN		VALUES
---------------------------------------------------------------------
REGISTRY:	apnic,arin,ripencc,lacnic,iana
COUNTRY_CODE:	One of 240 unique 2-character country codes or "*"
ADDRESS_TYPE:	asn,ipv4,ipv6
ADDRESS:	Either the starting IP Address or AS Number or "*"
NUMBER:		Number of IPs in range or "1" if ADDRESS_TYPE is "asn"
DATE:		Date IP range or AS Number was added to database or "*"
RANGE_TYPE:	"allocated" -> borrowed; "assigned" -> owned

Storing the Data in MySQL

To store the data, I created a two-table MySQL database named “ip_registry”, using the script below.
Database creation statement for ‘ip_registry’

CREATE DATABASE ip_registry;

Table structure for table ‘country_code’

CREATE TABLE ip_registry.country_code (
code char(2) default NULL,
country varchar(50) default NULL,
UNIQUE KEY code (code)
);

Table structure for table ‘ip_map’

CREATE TABLE ip_registry.ip_map (
code char(2) default NULL,
registry char(10) default NULL,
ip_from double default NULL,
ip_to double default NULL,
UNIQUE KEY registry (registry,ip_from,ip_to)
);

As of March 2005 September 2008, the “ip_map” data table for my system runs to 63,263 88,000+ rows. This value will change daily, and may decrease suddenly at times. The registries make an effort to aggregate as many IP networks as possible into the largest possible contiguous block, and this aggregation process will reduce the number of individual entries by 2,000 – 3,000 rows in a single day.

The recognized standard for country codes is ISO 3166. In this standard, each nation is assigned a unique, two-character code. The ONLY exception I found to this rule is that, for historical reasons, the IP registries have entries for the United Kingdom listed with two country codes (GB and UK). I could have corrected this in the Perl script by standardizing on a single country code, but I preferred the solution of adding another row to the “country_code” table.

From the raw Registry data, I determined that only four of the fields useful for the project that I was working on: REGISTRY, COUNTRY_CODE, ADDRESS, and NUMBER. I then wrote PERL code to read the raw IP Registry data from the data file I created previously, convert the starting IP address to a number, use this starting IP Number that to generate the end IP Number, and then insert the rows into a database.

PERL: IP Number conversion and database insert

<--snip-->

if ($line_count >= 115000) {

## Establish Database Connection

print "\n\nOpening database connection";

my $dbh = DBI->connect("DBI:mysql:host=[host];database=logger","[username]","[password]",{PrintError=>0});

## Remove existing values
my $sth = $dbh->do("TRUNCATE TABLE ip_map");
print " --> Data from ip_map table dropped";

$sth = $dbh->prepare("INSERT into ip_map values (?,?,?,?,?,?)");

$count = 0;

print " --> Completed\n";

## Insert Data Into Database

print "Inserting data into the database";

open (PROCESS, "<$file");
  while ($line =
) {
        chomp ($line);
        if (($line =~ m/\|ipv4\|/) and ($line !~ m/\|\*\|/)) {

                ($registrar,$country_code,$item_type,$start_ip,$num_ip,$entry_date,$registry_type) = split(/\|/, $line);

                $long_start = 0;

                if ($start_ip =~ /\d+\.\d+\.\d+\.\d+/) {
                        ## my $ip_address = shift($start_ip);
                        chomp($start_ip);
                        $long_start = ip2long($start_ip);
                        ## print "$ip_address converts to $ip_number\n";
                        $long_end = $long_start + ($num_ip-1);
                        $count += $sth->execute($country_code,$registrar,$long_start,$long_end,$num_ip,$start_ip);
                }
        }
  }

}

<--snip-->

The “TRUNCATE” statement in the script has the affect of dropping the table and re-creating it using the column names and types defined in the initial create statement. It is easier to rebuild the data table each time new data is inserted to ensure that duplicates and overlaps do not enter into the database.

Why is the value of “$long_end” defined by “$long_start + ($num_ip-1)”? The IP address ranges delivered by the registries count the starting value as one of the items in the set — i.e. counting using ordinal numbers.

START_IP:   12.236.236.0
END_IP:     12.236.236.255
NUMBER_IP:  256

If cardinal numbering is used to calculate the address range, incorrect IP addresses will be generated.

IP Number Calculations
WRONG!	216853760 = 216853504 + 256 	-> END_IP = 12.236.237.0
RIGHT!	216853759 = 216853504 + (256-1)	-> END_IP = 12.236.236.255

I have also added a sanity-check that stops the insertion process if the number of lines in the data file is less than 75,0000 115,000. This would prevents the creation of a truncated database if one of the registries does not update their data files or the script is unable to retrieve the data files. The value of 75,000 115,000 appears high, but the data files that the data is extracted from also contain autonomous system and IPV6 data, as well as the IPV4 data that is inserted into the database. Currently, the data file runs to over 90,000 130,000 lines, so the 75,000 115,000 line barrier seems very reasonable to prevent inserting a broken dataset.

Querying the Database

Now that the database is constructed, we can start to run queries against it.

mysql> select ip.code,ip.registry, ip.ip_from, ip.ip_to, co.country
-> from ip_map ip, country_code co
-> where (ip.code = 'IS') and (ip.code = co.code);

+------+----------+------------+------------+---------+
| code | registry | ip_from    | ip_to      | country |
+------+----------+------------+------------+---------+
| IS   | ripencc  | 1049722880 | 1049731071 | ICELAND |
| IS   | ripencc  | 1359937536 | 1359970303 | ICELAND |
| IS   | ripencc  | 1383088128 | 1383096319 | ICELAND |
| IS   | ripencc  | 1385447424 | 1385455615 | ICELAND |
| IS   | ripencc  | 1390215168 | 1390280703 | ICELAND |
| IS   | ripencc  | 1403846656 | 1403863039 | ICELAND |
| IS   | ripencc  | 1433681920 | 1433690111 | ICELAND |
| IS   | ripencc  | 1439023104 | 1439039487 | ICELAND |
| IS   | ripencc  | 1440481280 | 1440514047 | ICELAND |
| IS   | ripencc  | 2644312064 | 2644377599 | ICELAND |
| IS   | ripencc  | 3238264832 | 3238330367 | ICELAND |
| IS   | ripencc  | 3245150208 | 3245154303 | ICELAND |
| IS   | ripencc  | 3261718528 | 3261726719 | ICELAND |
| IS   | ripencc  | 3264217088 | 3264282623 | ICELAND |
| IS   | ripencc  | 3556884480 | 3556892671 | ICELAND |
| IS   | ripencc  | 3558785024 | 3558793215 | ICELAND |
| IS   | ripencc  | 3565084672 | 3565092863 | ICELAND |
| IS   | ripencc  | 3584524288 | 3584532479 | ICELAND |
| IS   | ripencc  | 3585114112 | 3585122303 | ICELAND |
| IS   | ripencc  | 3585433600 | 3585441791 | ICELAND |
| IS   | ripencc  | 3586023424 | 3586031615 | ICELAND |
| IS   | ripencc  | 3587538944 | 3587547135 | ICELAND |
| IS   | ripencc  | 3587981312 | 3587997695 | ICELAND |
| IS   | ripencc  | 3641278464 | 3641282559 | ICELAND |
| IS   | ripencc  | 3642535936 | 3642540031 | ICELAND |
| IS   | ripencc  | 3650592768 | 3650596863 | ICELAND |
| IS   | ripencc  | 3650596864 | 3650600959 | ICELAND |
| IS   | ripencc  | 3651915776 | 3651919871 | ICELAND |
+------+----------+------------+------------+---------+

28 rows in set (0.09 sec)

So the database structure is sound. It is important to build the file using all four five registries; even though Iceland is now covered by RIPE, older IP allocations and assignments were been handled by both RIPE and ARIN.

Having the registry information helps build in the flexibility to add a WHOIS functionality using this database, something that I have done on for my own IP Tracking and logging tool. This allows for further drilldowns on the data, beyond the scope of this article.

The main item that will be of interest to most Web site administrators is that they can now build dynamic pages using a data source which tracks their visitors’ announced IP address to the country of origin with a high degree of accuracy. This is particulary useful if you are attempting to distribute users to geographically diverse mirror sites. You can also do fun things, such as displaying the flag of the country that the visitor is coming from.

A generic copy of the registry retrieval and database insertion script can be yours just by contacting me.

Joi Ito, at the ICANN meetings in Brazil, has posted some interesting comments and links regarding the exhaustion of the IPv4 Space. [here]

The growth in private IP Spaces has helped stave off this exhaustion, even through the massive growth in the home broadband and mobile markets. I forsee a compromise coming shortly where IPv6 is used between Network Transit Providers, while IPv4 is reserved exclusively for customer-facing usage.

Just another of my hare-brained ideas…