Early this morning I work another maintenance window and upgraded the IOS software on a Cisco 10012 router. I was careful as this little router was running over 17,000 broadband devices. There's something about killing the service of 17,000 users voice and data services that kind of makes me take my time before a reboot. I checked the config 3 times and made sure everything was in place before I sent the "reload" command. The next 5 minutes was stressful as I seen my pings to the loopback interface go unanswered. I finely started to get reply's from the router and took a smoke break. After the little smoke break was done I started checking to see how many of the 17,000 modems & eMTA's where back online. It can take up to an hour to get them all back online so I was just sending the command "show cable modem summary total" over and over again to see the device count climb as the CPE devices came online. When I was at about 80% online I started to make test calls to some eMTA's. Good news is that the calls went thought and all was looking good. In 1 hour I managed to upgrade both primary and secondary processors and get all but 0.1% of customer back online. I was happy with the work and decided to call it a night. Later that day I made it into the office (about 1:30PM) and made sure the call center did not see any customer issues. Well it all went well and now I am ready for the next router on my list of 4 Cisco 10012's to upgrade. I also have some more node move I plan on doing this week to insure our customers have the bandwidth they need. While I was working early this morning I took a few pictures (they are posted below).

Well last week I got the first of the IOS upgrades done and so far I have not seen any issue so I am ready to move forward with he rest of the Cisco CMTS upgrades this week. I put in the maintenance tickets this morning. I also went over the node utilization reports and we are looking real good. Out of over 300 nodes I have less than 10% at or over 50% utilization and none over 75% at peak hours. This means customer should never see a bottle neck in there bandwidth. I also have been reviewing the IP utilization reports and all is well there too. I am now looking at a few upcoming projects to further split nodes and increase plant capacity. I also heard that a big sale contact went though today so I will have to work on upgrading one of the backbone links to an OC-48. this is always good and I am glad my company dose not wait till the last minute to upgrade backbone circuits. With this new upgrade we will still be way below 40% utilization on the backbone links to the internet. I am also looking at stream lining the CMTS configs and reviewing the ACL's to see if I need to add or remove and of the lines of config. Oh by the way I wrote another article for www.ezinearticles.com and I have added a copy to here till ezinearticles.com approve it. The article is named "How Cable Modems Work" I hope to add more content to this site soon. here is more pictures too..

(4) 7246VXR                          Cisco 15454 ONS                RiverStone RS 8000

    IMC Media Converter                  Cable system Per Amps

Today was cool I got news that my Company is willing to send me off to the $4,000 Cisco Optical Class. This is good news since I figured they would not pay for it. I also go word that we are going to support Backbone MPLS so our system can sell services to customers in other systems. It's basically a Privet Line Ethernet to anywhere we have network in the USA. Other good news is that our Junipers are ready to go for the MPLS setup. I also went over the network utilization and seen I have a few more nodes to speed up. I hope to include an ISO Upgrade this week too so I can burn in an image before mass roll out of a new ISO image. After the node work and ISO burn-in I will also look at the Cisco IAD product to see if it's worth using to sell T-1's.  Since we already had VoIP and a soft switch with MTA's our next need is to push PRI T-1's over IP to the soft switch. In other news we might need to upgrade one of our OC-12's to an OC-48 or just add a OC-48 as another router to our peering site. This OC-48 upgrade is just a possible precaution to make sure our bandwidth over 40% utilization. Well I have a few other work pictures for you guys/gals.

Cisco 7246 (We replaced all of them).                             Cisco ONS 15454(top) Terawave PON (bottom)            Two Cisco 15454's top one has OC-48's in it

                   My Work truck with a Cisco 10012 Ubr Router loaded in it. This is just the router not the cables or RF switches.

Well it's time to move some nodes! Well it's really not that fun. I have some much per-work to do. I first have to enable all the interfaces and add descriptions to them. Then I have to assign the frequency to the downstreams/upstreams make sure that the QAM is 256 on the downstream and a lot of little other fun interface commands. I next have to lock out the protection RF switch so I am not dealing with protection card issues all morning. After all that I get to start getting the combiners ready and go over the node move plan with my fellow engineer. Now I am ready to call the NOC and tell them I am starting the maintenance and for them to ignore all alarms out of the router till I am finished with my maintenance. All the pr-work took about an hour or so and we where ready to start moving nodes. I picked the non Virtual Node Splits (VNS) first because they are easier and I like to get them out of the way. Let me quickly explain what a VNS is... On a node you normally have 1 upstream and 1 downstream (downstream is shared with 4 upstreams on the Cisco 5x20 cards) per node. In some cases you can have multiple nodes on 1 upstream if they are small nodes. But the VNS is where you give one node 2 upstreams so there is more upstream bandwidth (upload) for that node. Downstreams bandwidth with 256QAM is about 40Mbs useable and the upstream with QPSK is about 5 Mbs useable bandwidth. When you add 2 upstreams to a node that node gets about 10 Mbs dedicated bandwidth and it shares the 40 Mbs downstream band with the other upstreams. Basically the work being done to night is to increase the bandwidth for the customers in the nodes being moved. We monitor the bandwidth in the nodes and try to keep it below 50% utilized at the busiest hours of the day (6:00PM-11:00PM). I have to say my company dose work hard to keep as much bandwidth as possible for the customers. Well I finished the single nodes moves fine but when I went on to move the VNS (I had about 8 to do) I noticed that we did not have the right pads (this is a resistor used to balance the RF signal) to do the splits in the STC. I asked the Headend Tech to drive to the other hub site to get the pad box. Well this was an unexpected delay and we waited form him to drive to a different city and drive back to finish our work this took about an hour. After he got back I told him there was no reason for him to hang around and told him he could leave if he wanted. The Headend tech was onsite just incase we needed him for support and is we found any bas cable that had to be replaced. Now it's after 3:00AM and we are just starting the VNS moves. We successfully moved 2 out of 8 of the VNS node moves when my fellow engineer was not feeling well and left. He offered to stay but from the look on his face he was not feeling good and I only have about 6 more move to do. I told him to drive safe and continued the moves. Well just my luck the next node I tried to move would not come online. I then spent about and 40 minutes trouble shooting and decided to try the next VNS on the same downstream (I move the other VNS back to it's old combining and it worked). Well the second move also failed so I then started looking at the combining to see if something was mixed up. Well now about and hour+ into trouble shooting I looked to the CMTS's RF switch and found the issue to be a crossed RF cable on the RF switch. After fixing the cabling issue I moved forward with the node moves. About now I am running out of maintenance window and I still have a lot of nodes to move. Well to shorten this post a bit I finish the node moves at about 7:00AM and then had to unlock the Protection switch and then about another hour adjusting the routers config and updating all the paper work to show the node moves. About the time I finished all the work and paper work it was around 8:45AM and I decided I should go home. I got a call from the only other engineer that was not working and he told me that he would review the nodes and make sure there was no issue during the day as I sleep. Well I did run my web cam for a few seconds during the maintenance window and have included a little video file here. I hope to post back soon.

Well I have not posted till now because nothing was really happening but tonight (or this morning) I have my maintenance window for the mass node moves. It kind of started like this I went to work around 8:00AM as I always do. I checked e-mail and created a few NOC tickets to add IP ranges for devices on the routers (i.e. had to add more IP's for cable modems and eMTA's). I chatted with my fellow engineers and was on a couple conference calls. Boss took us out to lunch ( he paid!) and we discussed what projects we where doing and also what needs to be looked at. He also informed us that our job titles might change soon but our work will remain the same "Jacks of all trades". We are the know it alls for our system so basically we have to be able to fix any kind of technical problems that any other department can't figure out. This means experience and Google are you life line. Now back to the job title change thing. Since I have worked with this company (3 + years) I have had my title change a few times. I was hired as a Data Network Engineer then it was changed to Broadband Engineer and now I think they are looking at Systems Support Engineer. The job titles are getting a lot more open sounding and less technical sounding and it kind of sucks. I guess if the title comes with a small pay raise I will have no problems with it (I do not expect a raise as from my experience with this company I have only gotten raises annually and it is link from 1%-4% kind of cheap). I am not bashing the company I like working here and I am happy with how my manager treats me, I can even goes as far as saying my current manager is one of the best managers I have had. I just wish the pay was better a lined with the work and amount of work done. I guess I have myself to blame since I took and job and I have the option to quit if I am not happy. Anyway I have gotten off track... Well I left the office at 4:00PM and went home to sleep a little before coming back in to work the node moves.