TLS over DOCSIS... So the MSO's now want to offer layer2 services over the DOCSIS network. I think this sounds like a awesome idea but with any new ideas come technical problems. Most of the issues seen so far are back office stuff like VLAN management and routing over the MSO's metro. Other issues is how to auto provision the VLAN tags info a modem config file and the new need for a bigger MTU size from 1500 to 1524-1532. You need the bigger MTU size not just on the CMTS but also on the rest of the metro network to make sure large IP packets also have room for the VLAN tags and room for QinQ (VLAN tags on top of VLAN tags). The end idea is to be able to provide Layer2 VPN services for business customers so they can have Frame Relay like services for there customers. Once the MSO's start providing layer2 services it will allow them to expand there business models and services to many more customers. If you are not following me here think about it like this. Business A has a location on the other side of town and this smaller office need to access layer2 services like NetBIOS or an AS400 and the customer wants to keep the remote office on the same subnet so there main office can provide all IP servers like DHCP, VoIP, Printing, SMS, Patch Management. Well if you link the office via layer2 you do not need a router between and the remote office would use the main offices site for internet and you would only need one firewall and internet connection. Anyway I feel this is a good step for the MSO's and the vendors are already working on the router code to support this new idea.

I wanted to post a quick field note on the Cisco 10012Ubr when swapping MC5x20S linecards with MC5x20U cards. When you swap the S card to replace a U card you may lose you IOS config for the line card being replaced. You will need to backup you running config to a tftp or notepad before swapping the cards.

Here is my preferred process when swapping any kind of 5x20 line card with HA (HCCP). Now the way I am explaining here is assuming that you are using HCCP bitmaps (will also work with HCCP global).

- Do a "show running-config" and copy/paste the configuration for the cable line card that needs to be replaced onto a text editor.

- Do a "show cable modem summary total" this will give you a baseline of the number of DOCSIS devices before you begin.

- Do a "show cable call" this will let you see if any active 911 calls are in progress (you should not perform any kind of maintenance when 911 calls are active).

- hccp (group) switch (member) (i.e. router# hccp 1 switch 50) This will force the active card to fail over to the protect card so you do not interrupt service. you will repeat this command till all groups and members on the card to be replaced are failed over to protect.

- Do a "cable power off <slot/port>" to power off the line card that needs to be replaced.

- Remove the old line card from the uBR10K chassis

- Insert the new line card into the uBR10K chassis

- Do a "cable power on <slot/port>" to power on the newly inserted line card.

- Do "show running configuration" to make sure the config on the replaced line card is persent.

-  (This step is if any config is lost) Paste the saved IOS configuration for the line card from the text editor

- Do a "show hccp brief" to verify the groups and member to fail back to the replaced line card.

- hccp (group) switch (member) (i.e. router# hccp 1 switch 50) This will fail the from the protect card back to the normal working card. Repeat this command till all the protected groups & members are back to there normal working card.

- Do a "show cable call" to make sure you PacketCable calls or SIP traffic is back up and running (you may want to make test calls).

- Do a "show cable modem summary total" and compare it to the one you ran before the card swap. Now you might have less devices online now but as long as the difference is less than 5% you should be fine since some of the device might take longer to come back online.

- Make sure all cable downstream / upstream interfaces are up and operation. You might have to do a "no shutdown" on some interfaces.

- Do a "wr mem"  or "copy running configuration start-up configuration" to save the IOS configuration

 Now some engineers will wonder why I went though the process of powering down the slot and not just pulling the linecard and letting the normal process of protection take place. Well I do it this way because I do not want to tempt fate when dealing with equipment that cost more than I make in a year and also you can risk static electricity shorting out the old or new card if you do not power down the slot. Cisco added the option for a reason. If you are planning to replace any line cards and are worried about the process feel free to drop me an e-mail and I will assist if time permits. Good luck and also feel free to share you CMTS war stories with me too.

Well I just got back from Defcon 15 in Las Vegas and I had a great time. I was in the area for work and happen to notice that Defcon was happening the same time I was in town and knew I had to make it. I was amazed by the number of attendants that was there. Once I got into the con I was able to find some folks I knew and was able to catch some interesting talks on VoIP and SIP. Since my background is DOCSIS I did not find anything directly related to the MSO world but I did get my fix on network security training which is very important for any MSO engineer. I am also still studying for my CISSP which I will take in September at Chicagocon. Lately I have noticed an increase in traffic to this site and if anyone coming by here and has a CMTS, DOCSIS, or MSO related question feel free to e-mail me (slimjim100(at)slimjim100.com) and I will try to help you. I have also picked up more side work but if you have a project you need a consultant on feel free to contact me as I do side jobs but also have a few friends that also do MSO work too. In other news I have been running into some interesting OSPF issues on my of my vendors platforms and the only fix so far is to statically add the router and them remove it to force the route to reload though OSPF. Now the issue could be a peer we have since it is a very specific route that has issues so once I get more info I will now where the true routing issue is. Once I have more info on the fix and the true issue I will post what CMTS I have seen this on and hopefully I can list the fix or root cause. Cable Labs is this week in Colorado and most of the DOCSIS/CMTS vendors are out trying to sell there solution. I have an associate out there and will report anything here is it's worth reporting. As of now I have not really seen anything new on the market worth talking about but soon we may see real DOCSIS 3.0 modems.

I have been busy updating this site with more vendor links and information to help anyone out looking for troubleshooting info on different CMTS's. I added Arris, Cisco, Motorola, Bigband Cuda, and other info tab links to the banner. My goal is to get all the information one might need while troubleshooting a CMTS in one place. I am trying my hardest to make sure I credit anyone's work I link of post here so if some thing you wrote is listed on this site and you would like it updated or removed just let me know. On that note if you work for a vendor and would like to share some kind of technical info about your product let me know. I will post technical documents but I do not want a whole bunch of marketing crap as this site is not here to advertise any CMTS but only to help support the engineers that have to maintain them. Anyway I am still looking for side work so if you would like to hire me or one of my engineer friends just drop me a line.

Well it has been interesting in the last month for me. I have been getting contacted from a lot of different people that I guess read this blog. I have been asked to review a router (CMTS's), Verify or Certify equipment for a fee, do side jobs trouble shooting DOCSIS issues in other countries, and a few other basic questions. I started this blog as a place for me to vent and explain the issues I run into in hope it would save someone else time. I have now noticed that if you Google "DOCSIS Engineer" or "CMTS Engineer" you will get my blog or my Resume listed on the first page. I guess since I am getting this kind of attention I should mention that I do have a Resume section on my site with my Resume and a few Resumes of skilled engineers I personal know and have worked with for years. If you are looking for a skilled DOCSIS/VoIP/VideoIP/Data Engineer let me know as I am on the market and so are a few skilled engineers I know. On a side note the industry is buzzing with DSG, Switch digital Video, DOCSIS 3.0{D} and HD. If you have any question about MSO technology or would like me to share some kind of industry news let me know via my contact page.

June 20 2007 (6:45PM)   ChicagoCon September 17 - 23

I have modified the banner on this index page to advertise a very important event coming up in Chicago. The banner is for "Chicago Con" which is a Security convention focused on Security & Network Training (boot camps) with many important industry speakers/guest. Now I did not just post the banner and give them free advertising for nothing. One of the guest speakers at this event is Me. I have not talked much on this Blog about my security background or interest but I do write a lot of papers under my real name discussing important security issues in the industry today. I invite you to join me for training this September in Chicago for Chicago Con. I will be there as a speaker and also to take the CISSP 7 day Boot Camp. Drop me an e-mail if you think you can make it and we can meet up if you like. In other news I maybe on the job market soon so if you have a Security or MSO position in or around the Atlanta area send me a line and feel free to view my resume.

Well the battle is on over I-CMTS and M-CMTS some of the vendors are focusing on the M and others the I. So which way are you going on your network? This is the question you need to ask yourself as this might decide what vendor you are going to grow with. With all the vendors supporting both the real questions is how much will it cost and what is the impact on your network. You will have to know if the new design with have HA (High Availability) for your VoIP subscribers and what will your network look like in 2-4 years. Other things to think about is your Engineers training and if they will need to learn anything new to support you growing network. The big question is also where will it all be in 5 years and how long will it really take to get to DOCSIS 3.0. It is a given that the cable industry is has the high ground on supplying customers with all there communication needs so I guess the next question is "What is next after DOCSIS 3.0 and every thing over IP"? I am still wondering when we will see real DOCSIS 3.0 CPE and the Security that will come with it. I know there is a lot of questions on where and what to go with but the good news is that MSO's have options and with options comes lower cost and new technology.  SCTE Cable Trade show is in Orlando this week so this might give us a chance to see "What's Next". As always if you have comments or questions feel free to e-mail me @ Slimjim100@slimjim100.com.

It has been a short time since I have post but I have not been sitting idle. There has been a lot going on in the MSO industry. Lots of new code for all the CMTS's and everyone is talking about DOCSIS 3.0. It's kind of funny you hear so much about DOCSIS 3.0 when most of the providers out there are still on DOCSIS 1.1 and there plants are in need of maintenance to support the higher modulation profiles needed to support the newer wideband and higher QAM's. I think the industry is getting there but DOCSIS 3.0 is not really on the market yet like the vendors are saying. To be 100% DOCSIS 3.0 compliant you have to also have the 3.0 modems and the thing that is killing all the vendors is the 256bit AES encryption chipset needed to make a modem full DOCSIS 3.0. Now all of the vendors will tell you they have 3.0 modems or 3.0D (D = Downstream load balancing ability) modems. But none of the modems I have seen to date have the required AES encryption chipset in them. Next big obstacle will be to get the modems cost effective around $100 or less per modem.  Other key words in the industry right now is Edge QAM and M-CMTS. I feel (this is my option) that the vendors (not all of them but 99% or so) are just not prepared today for how quick the DOCSIS 3.0 took off so the M-CMTS is there answer. An M-CMTS is basically just an external box of downstreams/QAMs to give the MSO's the ability to have many load balanced downstreams to compete with Fiber to the home. They need the downstreams to reach the 100Mbs down load speeds (with 3 downstreams they can get about 150Mbs down). Now the idea of more downstreams is a great idea but why not just make newer line cards for the exciting CMTS's? It would be cheaper and better to have more down streams in the same current CMTS instead of just piling more equipment outside of the CMTS's. There is one vendor that has a multiple downstream card coming to market but it won't be ready soon enough to beat all the M-CMTS vendors to the market. The funny thing about this whole M-CMTS thing is that all the vendors pushing this idea are moving to the line card of Downstreams but just do not have that ready for some time.  So I wonder is the M-CMTS going to be a fad or will it be one more piece of equipment stacked on top of our CMTS's and when will it support carrier class redundancy. From what I have seen so far all this external QAM/Downstreams are communicating with the central CMTS via Ethernet so if for some reason the switch you have in place for all of the devices to communicate goes down your network goes with it. For all the MSO's running VoIP now this is also an issue because you have now added another device to possibly fail into you voice network. Anyway I am sure there will be growing pains and lessons learned with this is all said and done.

Sorry I have added any new entries here for awhile. Well it's almost time for the new DST (daylight savings time) change. This new change imposed by Congress in 2005 is starting to look like the next Y2K scare. Hardware and software vendors are rushing to patch things so the clocks will not be off by an hour for 2 weeks. Cisco has a patch but only for CMTS's running IOS 12.3(17b)BC3 or later. What this means is that time sensitive applications such as encryption and other critical processes could have issues if your routers clock is off on it's timing. Now this is not just an issue with routers. Any IT appliance can be affected even your home alarm clock can be affected if it auto time changes and was made prior to 2005. I have also seen issue with other Vendors CMTS's. I can not list the documents here as each vendor has the documents listed in there logon section of there site but I can say is that you best off running NTP (Network Time Protocol) for your timing references on any network router or switch.  One fear I have heard so far from the DST issue that applies to CMTS's is that you TOD sever, DHCP server, and provisioning platform needs to be patched. Also you will need to think about VoIP and billing if you are using VoIP.  Someone had asked me about BPI+ and if it would have problems and I have tested it by changing the clock on a working CMTS (without NTP) by an hour, day, and even a year. In all cases BPI+ still worked because the time was still after the date stamp on the root certificate. I did see before in a CMTS where DOCSIS 1.0 was upgraded to DOCSIS 1.1 (with BPI+) where BPI+ had issues cause the Routers clock was set to the year 2001 and the root cert was date stamped in 2002. All the modems where in reject (PK) and they would not come online till the system clock was set of NTP was added. I personally feel the biggest issue we as the MSO industry will see because of this on the CMTS's is that the syslog and logging will be off for about 2 weeks till the older DST setting takes affect. But I guess it's a guessing game till then.