My attention deficit disorder is going nuts so I’m going to veer off Triadland a little bit more. We have defined a reference low-cost system and we clearly discussed the fact it’s definitely not the best system for a public safety mobile environment. So I’m going to call this article the prequel. How do we determine what equipment to use or in other words, what is the best system? Actually, most of them have some unique value depending on the application.
Ok, that’s a copout and allows me to avoid the question. However, there is no easy answer. I’ll go through the process that I use when asked to design a system. Keep in mind there are way more variables than I can cover here. The other thing that is highly important is that you know as many products on the market. I’m not just talking about mesh products, but PTP products, PTMP products, mobile products, virtual IP software, VPN products, antenna designs, etc… In addition to reading and keeping data sheets from everybody, get your hands on and get as much field experience with as many manufacturers as possible.
Almost every design I create involves products from multiple manufacturers. Most designs have some type of budget restraint but every design has some function that has higher priority than others. In some cases, the project was referred to me by a supplier or manufacturer. When that happens, I have a personal obligation to work with their particular product lines. That hasn’t been an issue so far but sometimes it means reaching deep into the product toolbox to find other products that enhance a design. Being vendor agnostic is very important for a consultant.
For example, one manufacturer came to me to help design a PTMP system using their APs. The problem was that they didn’t really have an outdoor product line available yet. They were working on one but in the meantime, they designed a NEMA enclosure and cabling structure that allowed the indoor units to be used outdoors. The basic problem with most indoor units was that the power output on the transmitter is usually underpowered compared to most outdoor units. FCC certifications with the manufacturer antennas were also clearly going to limit the range of the equipment.
In this case, I first had to do a little legwork to understand the rules I was limited by. I then had to analyze in detail, the bandwidth requirements at each location to determine what PTP or PTMP design options I had and what modulation rates I was going to be able to achieve on each link. This was to ensure that I didn’t overload the links. Since this design was going to start with 5.8GHz links, I was going to be limited to MCS rates up to the magical 300Mbps depending on distance. As we discussed in earlier articles MCS(15) rates are hard to get but in reality, I’m not getting more than 80Mbps TCP/IP in one direction. Even that was going to be difficult. The other problem is I didn’t know the APs in terms of actual throughput with video streams so that’s another variable where I have to err on the side of caution. It’s also probably a good idea to know the chipsets the AP manufacturers are using and the FCC AP filing information to review manufacturer data to get the actual facts that aren’t in the brochure.
Given the distances, FCC regulations, bandwidth requirements, and a few other unknowns, it was evident there was no way this equipment was going to be able to handle all the traffic. That’s when the design changes and you have to go to different manufacturers for at least some of the project. The important thing to me is to still use as much as the referring supplier or manufacturers equipment as possible.
For example, let’s say that we have a single primary building connecting to 2 other buildings that are within 30 degrees of each other from the primary building. We will call them A, B, and C. The A building is the primary, the B building is 1 mile away and the C building is 2 miles away.
I have 3 options.
1)One AP with a sector antenna on building A with 1 AP on each of the other 2 buildings with directional panel antennas (maximum gain legally certified antenna) on the other 2 buildings
2)Two APs on building A on different frequencies, one pointing at each of the other 2 buildings
3)One AP on the building A pointing to one AP on the closest building with a second AP then relaying to the farthest building.
Which option depends on how the network is going to be used, which direction is the traffic flowing to, and how much bandwidth do we need through each building. We also have to understand what the modulation rates are going to be at 1 mile links and 2 mile links. There may also be Fresnel zone issues or obstructions between buildings that may limit my options. Keep in mind FCC legal antenna requirements that may not give you enough gain to overcome a bad Fresnel issue. If for example, you used option 2, you may get MSC(15) rates from A to B but MCS (9) rates from A to C. The MCS(9) rates are clearly unacceptable and may come with a higher packet error issue. That might force you into option 3 to keep rates up. At the same time, you have just increased the amount of traffic through the A to B link. It was this type of situation that forced me to design the network using 3 high-bandwidth licensed options and using twenty 5.8GHz APs to feed into these. I used the maximum amount of the referring party’s equipment without compromising the integrity and price model of the design and still met the required specifications of the system. I also created redundant paths using the 5.8GHz radios to connect the entire critical infrastructure. This supported use of additional 5.8GHz APs which were cheaper than the licensed radios for backup.
The other part of this design utilized a technique that seems to becoming more common. Don’t take manufacturers suggested deployment strategies as gospel. There are many different ways to deploy a radio regardless of the original concept. Nothing says that a mesh radio has to be deployed in mesh designs only. Nothing says that a municipal deployment has to be mesh as demonstrated by TriadLand. There are many ways to deploy the various products out there. For example, I used mesh equipment to design a system that really ended up as a 98% PTMP system. The difference was that instead of 10 APs sharing 100Mbps, I ended up with 10 APs sharing 300Mbps. This is a huge difference.
Now I’m going to step into the shoes of some of you who have called me about projects. Let’s pretend I’m Joe IT manager (yes, I’m being sexist but I don’t have time to make this completely gender neutral and it screws up one of my points later) and the City Manager just came to me and said he wants WiFi downtown. He wants to make the city look progressive and it may enhance some of the business opportunities. More business means more taxes, bigger city budgets, and the City Manager’s name on driving this type of project. However, as the IT manager, my experience is in wire line infrastructure and I’ve never installed anything more than a few APs around the building.
To add to the pressure, as much as a successful project makes careers, unsuccessful ones do just the opposite. I know of IT managers that have left cities after municipal projects failed. Therefore, as the IT manager, I have to first make sure I define what the City Manager wants. Do they want ubiquitous coverage through most of the downtown area so that everybody can connect everywhere? Does the city want to charge for the service or offer it for free? Does the city want to offer connectivity to local businesses that may be limited to one incumbent local internet provider? These are just a few of the questions that have to be asked to the City Manager before the project even starts. They need to clearly define the expectations before you take the project even further. It’s kind of hard to hit a target when you don’t know what it is. If it isn’t in writing, I wouldn’t go one step further as there is no way to be successful if I don’t know what the goal is. There is no defined measuring stick. It’s also a good way to let mission creep into the project and you will never hit the final goal.
In most cases, the city manager usually needs some guidance to help define the goals. The goals may not be within the budget range and compromises might have to be made. If you have helped them define the goals, when you have to go back and tell them the budget restraints you will have more support. In other cases, there might be other departments such as city utilities that might have to be involved. Knowing that up front and having a City Manager mediating between departments is very helpful. Let’s just say that although some departments work great together in the best interest of the city, other departments are well, kind of like asking Randy Johnson to let you get a hit.
Once I’ve defined the goals of the system, then you need to start bringing in people that have that level of experience. Some of that may be in-house, some of it will have to be brought in from the outside. This is where the penny-wise and pound foolish idea comes in. It’s also where many of us computer people don’t want to admit to the people around us that we really don’t know everything. This is where thing may start going awry. Having someone with experience who has been through the war of early deployments can save a lot of hassle and expense later.
We all have a bias towards something. Ford Mustang people would rather eat dirt than drive a Chevy Camaro. Apple users would rather give up their latte’s than use a Windows computer. We all gravitate to our comfort zone. It’ also why many companies use attractive salespeople (all you IT guys know exactly what I’m talking about), have promotions, and take many an IT person to lunch to open the door and help create that product loyalty. Once that loyalty is established, it’s hard to get people to change their mind. Try to tell a Cisco person that there may be a better router product for this application and you might as well have asked them to give up their first born. They will look at you like you came from another planet. Obviously I’m stereotyping in the interest of humor but there is a significant amount of truth in there.
Brochures, free demo units, and great advertising shouldn’t be the criteria used for a deployment. It should be what fits for the project. For example, if you only have 1-2 wired connections to a 1 mile square deployment, you are going to have lots of hops. In that case, depending on the design, you really need 2 backhaul radios in an AP to minimize the bandwidth loss. If there is only 1 hop, it’s not that important. That changes though if a radio is a central point with several APs feeding into it. A 100Mbps AP handling 4 other AP chains under full load is limited to 25Mbps per chain. That’s where a 2, 3, or 4 radio backhaul design becomes very important.
If you want indoor coverage, there is a lot of vegetation, high vehicle traffic load, or guaranteed connectivity to even the lowest powered WiFi devices, then you will need a lot of access points to stay within a short distance of the target client. Having fewer APs which have to go farther also mean more noise is picked up since you are capturing a larger area. That also means that lower powered devices will have trouble connecting. Several poor quality connections will also cause the AP to slow down significantly. There is a saying that a happy customer will tell a friend, an unhappy customer will tell 10 friends. In the world of Internet, an unhappy customer may be telling thousands of friends.
Make no mistake, even if the service is free, it’s public and reflects on your city. If you put up a system that does not work correctly for its target client, that’s worse than if you didn’t put up any system at all. People will try to make the system do things it was never designed for and when it fails, they will make that public. For example, the system gets turned on, the local newspaper writes an article, and one guy can’t connect because he has some old, early production 802.11b laptop sitting directly behind his refrigerator in his apartment. That’s the guy who will be posting in the comment in the newspaper under the Internet article how the system doesn’t work and then will be calling City Hall. Forget the 100 people who are using it successfully, you will never hear from them.
All this information and we still haven’t figured out what AP to use. This process is simply to start weeding through which equipment might be used based on goals and design limitations. We didn’t even cover site-lease agreements for private property, insurance issues, Internet Gateway points, security, zoning, and a few other details which can derail a project. Some of this isn’t directly tied to the AP obviously, but it’s a good idea to be working on this parallel to the design. For example, if I find out that none of the electrical poles have power during the day, then my design changes drastically. Instead of a mesh design, I may be looking at a lot of hub and spoke designs with longer range. Either that or I have to find 20 hour battery backup units for each pole. If I need private property, then I have to worry about aesthetics, 24 hour access, etc… Designs can change depending on many factors. However, these factors dictate which equipment is to be used. Get through all that, re-evaluate the checkbook, and then we can figure out which APs to start looking at.
Sunday, July 18, 2010
Monday, July 5, 2010
Chapter 9: I’m from the Government, I’m here to help you
Now let’s talk about the real-world in deploying this type of system. Realistically, light poles or traffic lights are the obvious deployment locations. They are usually spaced pretty optimally. The problem is they are either owned or operated by the local municipality or the local power company which in many cases, is overseen by a local governing board. Either way, the key word here is government.
I have dealt with municipalities that are both easy to work with and others that are very difficult to work with. It always comes down to the individuals within the departments. However, it also always takes a consensus of staff or departments to get a project through. The ability to get a project passed goes down as more people are required to make a decision and the more money the department has to expend. So getting approval to use municipal facilities generally takes a pretty long time, especially if involves the budgetary cycle.
Now you have to throw in the legalities of a deployment. You need some type of site lease for any facilities you are mounting equipment on. In addition, you might also need insurance in case something gets damaged. We have been required to carry a $2 million dollar policy for some locations. If you have to deal with property owners for buildings, the contracts get more complicated as there is the issue of sale of the building, acess, etc… Even though technically Triadland looks pretty easy, the devil is in the details.
When you go back to build financials for a deployment and you start adding in all the back end costs, you can see that unless you are expecting a lot of paying users, in the thousands, that a multi-million dollar Capex is going to kill the project unless you accurately predict the costs. If you are utilizing a lot of vertical assets, such as some of the models using 60 APs per square mile, and you have to pay for the rights to each of these poles with fees such as $10 per month per pole, then the monthly nut goes way up. Of course knowing you need 60 APs up front instead of thinking you need 30 APs and then later finding out you really needed 60 to begin with is far worse. The days of building a system and the users will come went away with the cell phone companies providing better bandwidth options. Although even Apple knows that WiFi can provide the bandwidth it needs for video applications, the cost and difficulty of managing a few towers versus thousands of APs makes it a no-brainer for cellular providers.
Although some of the cable companies have moved into the WiFi area with hot-spots to enhance their security, it’s not really a national standard for a business model. There is even discussion of cellular companies paying WiFi providers for access to their network. However, it’s evident that although WiFi has high-bandwidth capacity, defining the target client is key to profitability unless Verizon throws some petty cash your way.
As with any good business plan, in complete contrast to former companies that have failed in high-profile examples, the key is keeping Capex down, keeping monthly expenses down, providing a good quality service to retain and add clients, defining a realistic revenue stream, defining system expectations, and making sure you have control of as many facets of the product environment as possible. Sometimes this results in compromises in technical capabilities to balance out profitability and success. Building a system that costs $10,000 per square mile but generates $3000 per month in revenue with a $1000 per month of expenses is far better than building a $100,000 per square mile system that generates $6000 per month in revenue and costs $5,000 per month to support (keep in mind the cost of money needed to build out the system and manufacturer warranty costs) is not a better deal. There is nothing wrong with saying we can’t give 100% indoor coverage if 40% makes it profitable and 80% indoor coverage makes it unprofitable. If the system fails financially, it’s irrelevant what percentage gets covered.
I bring these examples up to show that it takes different designs for success in different applications. Although the industry idea of a municipal wireless system has traditionally been stick APs up every few hundred feet and build a system that is everything to everybody, the best design is the one that is self-sustaining financially which motivates the operator to keep it running.
Municipal Wireless by definition used to mean 100% coverage everywhere, similar to cell phones, with a universal product that was everything to everybody. We need to redefine it so that Municipal Wireless means any 802.11 service protocol deployed in a Wide Area model to provide bandwidth using a unique methodology that can’t be provided by existing cellular or satellite models. Start there and we have a whole new set of opportunities. Throw in 802.11N, MIMO, new 3.65GHz and 900MHz equipment, very low cost wireless equipment, and the possibilities are endless.
Instead of going to a municipality and asking them to be an anchor tenant, why not ask them to pay $10 per month to place a camera on your network or even better, you own the camera and lease viewing time to the city. It’s far easier for the police department and water department to find the funding to pay $100 a month for access to your camera than to Capex $1000 for a camera. Not only is the source of the funding different meaning different rules for purchasing, it also relieves them of hardware obsolescence, technical support costs, and internal training. Expand this idea out to other areas as well.
Many governmental departments are paying $40-$60 per month for cellular data cards. I can tell you for a fact that they are a mixed blessing. The lower bandwidth doesn’t work well for video uplinks. There are dead spots or areas where many of the systems slow down to lower rates and in many smaller cities, they are limited to 100K or less systems. Yes I know LTE and Wimax are here in many cities but try uploading a 3-6Mbps video stream and tell me how that works. Put up a system that can provide better outdoor coverage and they would gladly pay you instead of Verizon or Sprint. Assuming you have outdoor coverage taken care of to the vehicle, you don’t need laptop coverage everywhere because in reality, the city vehicle is a mobile hotspot itself. Use 2 AP’s in the car, $200 instead of $100, and you have a mobile hotspot with about 200’-400’ of coverage around the car. For improved performance in a mobile environment, a dedicated router product such as a Peplink Mobile Max or CarFi, or something similar might be better. I like the Peplink Mobile Max product because it has a load-balancing hardware VPN tunneling feature coming out for secured communications. In mission critical environments, I’ve even looked at using 2.4GHz and 900MHz in the same car simultaneously.
I know that it’s hard to get rid of the old ways. There are lots of manufacturers still building the same equipment they sold 5 years ago with minor changes to support 802.11N. Unfortunately, not even all of them have the ability or design to upgrade to 802.11N. The AP manufacturers need to figure out how to build a product that can make a deployment profitable if they want to get back into that market. If municipalities are the only clients, there is a limited market for too many manufacturers. What is needed is a model where private venture capitalists, many of whom have already been burned, see a business model that is not based on advertising, but based on reasonable costs and has solid income potential. It can’t be based on the idea that government officials are needed to make a business viable. We as consultants need to suggest that manufacturers get entry level prices way down, find additional sources of WISP revenue, and create an expectancy of profitability to drive the market. The manufacturer who does that will own the market.
I have dealt with municipalities that are both easy to work with and others that are very difficult to work with. It always comes down to the individuals within the departments. However, it also always takes a consensus of staff or departments to get a project through. The ability to get a project passed goes down as more people are required to make a decision and the more money the department has to expend. So getting approval to use municipal facilities generally takes a pretty long time, especially if involves the budgetary cycle.
Now you have to throw in the legalities of a deployment. You need some type of site lease for any facilities you are mounting equipment on. In addition, you might also need insurance in case something gets damaged. We have been required to carry a $2 million dollar policy for some locations. If you have to deal with property owners for buildings, the contracts get more complicated as there is the issue of sale of the building, acess, etc… Even though technically Triadland looks pretty easy, the devil is in the details.
When you go back to build financials for a deployment and you start adding in all the back end costs, you can see that unless you are expecting a lot of paying users, in the thousands, that a multi-million dollar Capex is going to kill the project unless you accurately predict the costs. If you are utilizing a lot of vertical assets, such as some of the models using 60 APs per square mile, and you have to pay for the rights to each of these poles with fees such as $10 per month per pole, then the monthly nut goes way up. Of course knowing you need 60 APs up front instead of thinking you need 30 APs and then later finding out you really needed 60 to begin with is far worse. The days of building a system and the users will come went away with the cell phone companies providing better bandwidth options. Although even Apple knows that WiFi can provide the bandwidth it needs for video applications, the cost and difficulty of managing a few towers versus thousands of APs makes it a no-brainer for cellular providers.
Although some of the cable companies have moved into the WiFi area with hot-spots to enhance their security, it’s not really a national standard for a business model. There is even discussion of cellular companies paying WiFi providers for access to their network. However, it’s evident that although WiFi has high-bandwidth capacity, defining the target client is key to profitability unless Verizon throws some petty cash your way.
As with any good business plan, in complete contrast to former companies that have failed in high-profile examples, the key is keeping Capex down, keeping monthly expenses down, providing a good quality service to retain and add clients, defining a realistic revenue stream, defining system expectations, and making sure you have control of as many facets of the product environment as possible. Sometimes this results in compromises in technical capabilities to balance out profitability and success. Building a system that costs $10,000 per square mile but generates $3000 per month in revenue with a $1000 per month of expenses is far better than building a $100,000 per square mile system that generates $6000 per month in revenue and costs $5,000 per month to support (keep in mind the cost of money needed to build out the system and manufacturer warranty costs) is not a better deal. There is nothing wrong with saying we can’t give 100% indoor coverage if 40% makes it profitable and 80% indoor coverage makes it unprofitable. If the system fails financially, it’s irrelevant what percentage gets covered.
I bring these examples up to show that it takes different designs for success in different applications. Although the industry idea of a municipal wireless system has traditionally been stick APs up every few hundred feet and build a system that is everything to everybody, the best design is the one that is self-sustaining financially which motivates the operator to keep it running.
Municipal Wireless by definition used to mean 100% coverage everywhere, similar to cell phones, with a universal product that was everything to everybody. We need to redefine it so that Municipal Wireless means any 802.11 service protocol deployed in a Wide Area model to provide bandwidth using a unique methodology that can’t be provided by existing cellular or satellite models. Start there and we have a whole new set of opportunities. Throw in 802.11N, MIMO, new 3.65GHz and 900MHz equipment, very low cost wireless equipment, and the possibilities are endless.
Instead of going to a municipality and asking them to be an anchor tenant, why not ask them to pay $10 per month to place a camera on your network or even better, you own the camera and lease viewing time to the city. It’s far easier for the police department and water department to find the funding to pay $100 a month for access to your camera than to Capex $1000 for a camera. Not only is the source of the funding different meaning different rules for purchasing, it also relieves them of hardware obsolescence, technical support costs, and internal training. Expand this idea out to other areas as well.
Many governmental departments are paying $40-$60 per month for cellular data cards. I can tell you for a fact that they are a mixed blessing. The lower bandwidth doesn’t work well for video uplinks. There are dead spots or areas where many of the systems slow down to lower rates and in many smaller cities, they are limited to 100K or less systems. Yes I know LTE and Wimax are here in many cities but try uploading a 3-6Mbps video stream and tell me how that works. Put up a system that can provide better outdoor coverage and they would gladly pay you instead of Verizon or Sprint. Assuming you have outdoor coverage taken care of to the vehicle, you don’t need laptop coverage everywhere because in reality, the city vehicle is a mobile hotspot itself. Use 2 AP’s in the car, $200 instead of $100, and you have a mobile hotspot with about 200’-400’ of coverage around the car. For improved performance in a mobile environment, a dedicated router product such as a Peplink Mobile Max or CarFi, or something similar might be better. I like the Peplink Mobile Max product because it has a load-balancing hardware VPN tunneling feature coming out for secured communications. In mission critical environments, I’ve even looked at using 2.4GHz and 900MHz in the same car simultaneously.
I know that it’s hard to get rid of the old ways. There are lots of manufacturers still building the same equipment they sold 5 years ago with minor changes to support 802.11N. Unfortunately, not even all of them have the ability or design to upgrade to 802.11N. The AP manufacturers need to figure out how to build a product that can make a deployment profitable if they want to get back into that market. If municipalities are the only clients, there is a limited market for too many manufacturers. What is needed is a model where private venture capitalists, many of whom have already been burned, see a business model that is not based on advertising, but based on reasonable costs and has solid income potential. It can’t be based on the idea that government officials are needed to make a business viable. We as consultants need to suggest that manufacturers get entry level prices way down, find additional sources of WISP revenue, and create an expectancy of profitability to drive the market. The manufacturer who does that will own the market.
Subscribe to:
Posts (Atom)