Today's ARIN estimated depletion date:

IPv4 / IPv6 and TCAM memory


When I met Mack McBride at the Rocky Mountain IPv6 Conference about a month ago, he brought to my attention an upcoming depletion issue with router memory that I thought I would write a few words about.

For small companies and home user, a default route is used to route all traffic off the network into the service provider’s network for further transportation to the destination. Default routing is not an option for service providers. The they peer with multiple other providers and wants to route packets to the path with the lowest cost (fewest hops). Instead of default routing, they share the routing table via BGP. However, with more and more networks connecting to the internet, this table is growing rapidly. This post examines one of the future problems that the internet is facing due to this growth and due to deployment of IPv6.

The routing table in routers is using a specialized memory called TCAM. This memory is very fast so that the next hop destination for each packet can be found quickly. The TCAM is limited to 1 million entries in the Cisco 7600 (Catalyst 6500) routers. This type of router is very commonly used by service providers. An issue in such router will affect most, if not all service providers around the world. The amount of TCAM memory has been the same for several years. Even if you buy one of those routers today, you will not get more than 1 million TCAM entries.

The TCAM is designed in such way that IPv4 takes up one entry and IPv6 takes up two (that might sounds strange since IPv6 addresses are 4 times larger than IPv4 addresses, but that is the way it is). The default configuration is to allocate 512k entries to each of the protocol resulting in 512k possible IPv4 routes and 256k possible IPv6 routes.

The IPv4 routing table
The current full BGP routing table consists of about 450,000 IPv4 routes (see Note that this number is from a typical backbone network and has more routes than an enterprise might see. Other sources might give a slightly smaller value. Nevertheless, a year ago that number was only about 400,000 IPv4. So when will we fill up the TCAM memory? Well, it depends on how fast the routing table will grow moving forward.

Is a linear growth of IPv4 routes realistic? Historically we have had a slight exponential growth as can be seen in this image ( A rough estimate is that the routing table grows exponentially and that it double in size every 5 or so years. But perhaps a linear extrapolation gives us a reasonable scenario; we will hit the 512k mark for IPv4 routes in about a year. I have a hard time to see how we wouldn’t reach the 512k in less than two years even if I use a very conservative perdition.
No new IPv4 networks are available for delegation from the APNIC RIR. RIPE as well as other RIRs are soon to face the same faith. For many large IPv4 users the alternative is to use their current inventory of IPv4 address and slice it up in smaller chunks. This might make the IPv4 table to grow even faster than what we historically have seen.
Once the limit is reach, the routers will have to start to use slower memory for routing. This will create an increase in latency for packets traversing the internet. What happens is that routes that are hardware switched become software switched. This generally causes significant router issues and can even crash the router.
An option that service providers can consider is to filter smaller routes. A service provider can basically throw away routes smaller than a certain size. For example in IPv4 you can setup filters to throw away routes smaller than /24. Most providers already do this so no new routes would become unavailable nor would it save space. What is likely to happen is providers will start filtering deaggregates where a covering prefix exists, at least for some time until this problem is resolved. This might create a suboptimal path for packets resulting in an increased latency.

The IPv6 routing table
The current IPv6 routing table is only about 10,000 IPv6 routes inside of a service providers network. With IPv6 you normally get very large chunk of IPv6 space assigned to you so fragmentation of IPv6 space is not as big of an issue as fragmentation of IPv4 space. However, IPv6 is just starting to be deployed and we can expect the IPv6 routing table to grow pretty hefty in the next couple of years. But reaching the default TCAM settings of 256k IPv6 routes will probably take quite a while.
However, the full 256k routes might not be available for IPv6. It is possible to reconfigure the TCAM memory. Service providers might be forced to do so to make sure that they still fit all the IPv4 routes. Two alternative configurations are to fit 768k IPv4 routes and 128k IPv6 routes or 640k IPv4 routes and 192k IPv6 routes. The 768/128 setting will give the IPv4 routing table enough space to grow for several years. But it will on the other hand limit the IPv6 table to 128k entries. That number number scares me because I’m not sure that will be enough to fit the full IPv6 routing table when IPv6 is fully deployed. The 640/192 setting might be a better option, but might pose a limit on IPv4 instead. I’m not sure 640k IPv4 routes it will be sufficient to cover the need for that many year considering that we exponential growth and a fragmented IPv4 space due to lack of the availability of new addresses. Any of those two settings are in any case better than the default 512/256.
A ratio of 4:1 between IPv4 and IPv6 routes is in my opinion not unrealistic. That would imply that we have 128k IPv6 routes when we hit 512k IPv4 routes mark. I base this on the fact that.

  • With IPv6 fully deployed, each AS would advertize at least one IPv6 prefix. Currently about 40,000 AS numbers are being used and each AS is advertizing around 10 IPv4 prefixes each.
  • A lot of organizations are multi homed to increase redundancy. They peer with more than one provider and this creates two routing entries in the routing tables. If they are multi homed in IPv4 then we can expect them to have the same requirement for their IPv6 connections.
  • A lot of network administrators are using the old IPv4 thinking when they deploy IPv6. They use a 1 to 1 mapping between how IPv4 is deployed and how IPv6 is deployed. This might create more than necessary IPv6 routes.
  • Cisco, at some point thought that a 2:1 ratio between IPv4 and IPv6 routes was the best guess. Otherwise they wouldn’t have created this as the default setting for the TCAM memory. Shouldn’t we believe Cisco? They should know what they are talking about, right?

If you have an opinion on the ration between IPv4 and Ipv6 prefixes that is different from mine, then please post a comment.

The growth of the IPv4 routing table combined with the deployment of IPv6 has an interesting side effect in terms of router TCAM memory. The TCAM memory must be balanced in such was that both the IPv4 and the IPv6 routes fit. Routers must be reconfigured and there is a point if the future where there will be a problem fitting both tables into the available memory. Cisco is rumored to be working on a solution, they better hurry. In the mean time, please login to your router and change the TCAM settings to allow for more IPv4 routes.

Tags: , , ,

6 Responses to “IPv4 / IPv6 and TCAM memory”

  1. JB says on :

    Most ISP currently filter on /24.
    People I have spoken to about this TCAM problem says we could filter on /23 or /22 to solve this problem. You could, but according to pootaroo approx 53% of routing entries are /24.

    Replacing a /24 with default route? Possible, but suboptimal

  2. Michiel says on :

    [i]“The TCAM is designed in such way that IPv4 takes up one entry and IPv6 takes up two (that might sounds strange since IPv6 addresses are 4 times larger than IPv4 addresses, but that is the way it is).”[/i]

    That’s not so strange considering that for the routing only the network part of the address – the higher 64 bits – is relevant. The host part – the lower 64 bits – need not be stored in the global routing table.

  3. Michiel says on :

    “Cisco is rumored to be working on a solution, they better hurry.”

    The solution seems pretty obvious: more TCAM memory. Moore’s law is still valid, so I wonder what takes them so long..

  4. JMV2009 says on :

    Not so relevant here, but there are now AAAA ipv6 records from Facebook, even without white-listing!
    That will flush out almost all broken configurations.
    In turn, this will enable many servers to turn on their system for ipv6 too.

  5. Mack McBride says on :

    The 2:1 ratio in TCAM is based on the chip design used by Cisco that uses 72 bits, 144 bits, or 288 bits. Since IPv6 addresses are 128 bits they fall in the second bucket size. IPv4 addresses fall in the first bucket size. The ratio of the two bucket sizes is 2:1.

    TCAM does not scale according to moore’s law. Doubling the size of tcam takes more than 2 times as many transistors. This means TCAM is exponential vs linear.

    An alternative to larger TCAM (which is expensive) is using TCAM compression, this can be done strictly in software.

    Another alternative would be specialized blades to act as forwarding ‘helpers’ so the DFC on each blade doesn’t need the full TCAM.

  6. Ted K. says on :

    One possible solution would be dedicated TCAM banks – one for IPv4 and another for IPv6. This would also pave the way for a future where one has more than 512K IPv6 entries. Costly in terms of hardware, but (I suspect) with only minimal changes to the software in return.

Leave a Reply