Chapter 15 - Thinking is cheaper than doing

I said we would figure out how to compete with cable and it’s time to put up or shut up. I’m not writing the entire business plan here and I’m going to leave some key pieces out to protect some of what I’m working on, but the basic concept is here and it’s solid. We will put it all together later but let’s just think it through first and add some more tools.

Early on while developing this idea, I realized that while anybody could create a system that delivered bandwidth, the retail price of bandwidth was going to probably go down, making a Capex recovery more difficult. I have realized now that I looked at it the wrong way and that there would never be enough bandwidth, regardless of the cost. I also felt that there weren’t enough apps out there to drive the need for more bandwidth so that was the first part of the equation that had to be solved. This problem started fixing itself with the advent of smart phones, YouTube, video conferencing, VoIP, etc… and bandwidth needs started increasing very quickly. More recently I have to personally thank Google for their new search engine and Direct TV with the NFL package. Just keep em’ coming boys.

Small businesses need better options than T-1 circuits. They are slow and expensive. DSL circuits are slightly faster down and almost slower up. Of course, if they could keep them working or keep the performance at the purchased rate, that would be even better (can you tell I had another Saturday morning wasted calling Qwest tech support for a circuit bouncing like a superball) . Cable is also starting to expand again. However, in any area that only has one local loop provider, let’s just say you aren’t getting any coupons in the mail.

If you really want to see the opposite of capitalism in action in this area, just visit Mexico and try to start a wireless internet business. You have to hand your business plan over to Telmex, all the details and technologies, and hope they approve you opening up a company to compete with them. Yea, like that’s going to happen. It costs $1200-$2500 for a simple T-1 circuit down there. Imagine Cricket having to ask Verizon if they can open up a competing cell phone company. Mexico will always be playing catch up technically to the rest of the world until they allow open competition for small businesses. For those who want to tell me that Telmex is providing low-cost DSL service, try to make a VoIP International call over the service and amazingly, the quality is really bad. Wonder if that has anything to do with Telmex charging exhorbitant rates for international calling. If any country needed wireless options to compete with Telmex, Mexico does.

It’s also evident that the chokehold held by local loop provides wasn’t going to change without a huge investment in fiber or cable. I still believe that fiber is the best technical way to deliver massive bandwidth. It is by far the best long term solution based on what I see in the future for wireless, but it’s not the most cost-effective unless you happen to be digging up the street for some reason and laying new conduit. The power companies had the best chance to break that monopoly but the Bandwidth over Powerlines idea was simply a bad idea. They had a better solution on the table and still do, but I don’t see any of them playing that card. They could have easily delivered 30Mbps to the home for a lot less money than BoP but but I haven’t seen anyone deploy that idea, nor do I expect them to.

Investment in fiber is difficult to make for residential because of the long term ROI. That’s why most companies have abandoned it. The population density in the U.S along with the lack of new home construction and infrastructure mean that almost the only group willing to throw money at FTTH is the federal government since hey, it’s not their money and they need more votes. Whether it’s the most efficient use of our tax money is irrelevant. In the meantime, wireless costs have come way down while capacity has increased. Of course, cable companies haven’t been sitting on their hands and their capacity has increased simultaneously. DSL, well let’s just say that the DSL companies are still evaluating expanding into the Realty and Moving and Storage businesses. I really like the new marketing plan in Phoenix calling DSL “Heavy Duty or HD Internet. I almost crashed my car I when I saw that billboard I was laughing so hard. I was thinking HD stood for Howdy Doody Internet after several more crashes of my service again this week.

Being the wireless technical gurus that we all are, we just simply say throw up AP’s everywhere which is what municipal wireless and mesh systems tried to do. This would cover about half the calls I have gotten from people interested in these types of projects. One minute of explanation concerning getting access rights and insurance along with pesky little details like ROI, and the conversation ends. As I pointed out in the last article, as cheap as wireless is to deploy compared to fiber, it’s just as difficult to recoup the ROI because it’s difficult to deliver a triple-play solution which has the highest revenue. It’s also logistically difficult, expensive, and a very long term project to deploy hundreds of vertical assets around a city, assuming you can even get access to them. Nowadays interference, as pointed out in Chapter 13, is also a way of life.

If you have read all the articles, the blueprint is already there for alternative municipal deployments or my favorite phrase, Guerilla Wireless. The devil is in the details and implementation. We needed an inexpensive source of Internet which we found for as little as $1 per Mb. We need a backhaul infrastructure that can support up to 2Gbps or more. That’s off the shelf today with all the licensed and unlicensed equipment out there and throughput ranging up to 4Gbps or more. Our last mile equipment on the client side is less than $100. We also know that we can build systems capable of delivering last mile up to 30Mbps or more to residential or business systems with AP’s costing as little as $100.

Part of the formula in calculating ROI in an area depends on whether you are using PTMP, open WiFi, or a hybrid model to provide service. It’s also important to decide if you are going to compete on price alone or on service levels. Going head to head with comparative pricing has been the battle that wireless hasn’t been able to fight. We can fight that now. However, you are going to have to make sure that your system is rock solid. In addition, if you can get $40 dollars or more, regardless of the deliverable bandwidth, your numbers start to look better. With all the people who are dropping cable TV and going to Internet TV only, if you can deliver the bandwidth, you can compete and be profitable. Keep the TV thing in mind as you calculate other revenue streams also. Also keep in mind the billing structure for your local power company. These are hints to maximize your revenue per bandwidth.

Now you are probably thinking that I’ve left out an important piece such as vertical assets. Again, I point out that I gave the blueprint for how to install systems in many environments. What I haven’t covered is how to get local assets, get the support of the community behind you, and do it for little or no monthly costs. You aren’t getting around the Capex part but if you can get a lot of your vertical assets and contribute to the community on many levels, that’s a huge advantage. WiFi is a game of inches (thank you Vince Lombardi) that has to be played at that level. It also has to be played at the political level. However, put the right players in the game to handle these, coupled with a solid technical team and the numbers work. Your team has to be fast, ambitious, and willing to take risks.

Let’s add a few more tools into our toolbox first. The biggest tool is using single building supporting a 4 square mile area for less than $3000 in equipment. We know that large brick buildings block 2.4GHz and above pretty well (assume from the back of the antenna, not the front). Instead of putting antennas on the tops of buildings, let’s look at putting the radios on the side of the buildings. I can see property managers cringing already because they don’t want their buildings looking like NASA. However, our equipment is fairly small. For example, a Ubiquiti Nanostation 2M is about 11 inches tall and 3 inches wide. This AP supports up to 100Mbps+ of real world throughput with a 20MHz wide channel using 802.11N. It’s basically the size of a brick. Paint it red (assuming red brick), hang it 30-100+ feet in the air and nobody would even see it. Throw three of them on a wall and there is 300+Mbps shooting one direction. It’s easy to hit a laptop with these at 1000 feet LOS. This fits pretty much every school out there and since most of the high schools already rent their light poles for cellular phones, it’s not much of a stretch to get the building.

Now throw in the 5GHz versions of the same radio on the same wall using four 20MHz channels in 5.8GHz, and you have 700MHz of bandwidth shooting in one direction. The brick wall eliminates interference from behind the behind or on the sides. Expand this out to 4 walls and 2.8Gbps of bandwidth is now available all over the surrounding area of the building. Assuming the building is 60’ and the houses around the area are 30’ with few trees above 40’, outdoor 5.8GHz CPE’s could easily connect back up to 2-5 miles away with less than $150 in equipment. So for $3000 in radio equipment, you are delivering a massive amount of bandwidth from a central location.

If a student could get direct access to school computer assets at a high bandwidth rates, think about what other services the schools could offer. How does the idea of “Education Everywhere” sound? This was one of my earlier ideas to bring better assets to students and more control to their online experience. Also think in terms of low cost access, breaking the digital divide, and political capital. This idea has many legs.

Don’t stop at schools though. Any brick building with 30’ of height or more is a potential vertical asset. In some cases, you can make a deal to provide internet for a fee inside the building to tenants. In other cases, you might have to give something away for free but you still get the asset. Compared to hundreds or thousands of dollars that cellular or data companies pay for vertical assets, this isn’t a bad idea. You just have to scale the cost to the potential market. In this game, you don’t get the advantage of amortizing losing areas across the big picture.

I leave the details of physically running cabling (through the wall or conduit outside the wall) and bolting against the wall to each individual installer. For example, if there is metal flashing on the roof, it might be easier to use a strap over the roof and paint the radio the same color as the flashing. If you are in the middle of the wall, it might be easier to drill through and bolt in from the back while running the cable through the same hole. There are a hundred ways to make it aesthetically pleasing, but there is no denying the performance capability. If you are really ambitious, you can build a metal box around each one of the radios to reduce noise and tighten up the beam patterns. Of course the installation, switches, and trying to figure out where to get 3Gbps of back end bandwidth is still an issue. I would estimate a full deployment like this around $10K in Arizona.

Take this down to a smaller scale also. One of the complaints for residential deployment is that there may not be vertical assets in every area. Wrong. There are vertical assets in every area. Tell me that there isn’t at least 1 person every square mile that would trade free internet for roof rights to their house. My guess is that there is 10 times that. Because the coverage zone is short, ½ mile every direction, it’s extremely cheap (less than $1000 installed) and quick to install this type of system. If you sell to 10 people per square mile, your monthly vertical asset cost is 1/10^th the monthly revenue or $50. Even if you don’t trust the people in the house to pay the electrical bill, it’s not that expensive to add a meter. Who needs lights when every house on the planet is a potential vertical asset?

On the house, think chimney with some type of metallic shield if you want the signal isolation. Four AP’s with 400Mbps of bandwidth up there cost less than $300 for 360 degrees of short range coverage. If it’s a smaller area, even a single omni-directional antenna can deliver 40 Mbps. If there area is covered with trees, chimneys aren’t high enough, or the Home Owner’s association is run by Ghengis Kahn, then use your imagination. In areas where you aren’t sure that your AP site is going to be stable, use 1x1 802.11N radios with omni-directional antennas on the surrounding client areas so that if you have to move the AP, you don’t have to reposition a bunch of client radios.

Plan for your system to be dynamic and budget for that. Wireline companies generally don’t have that issue. However, the ability to be dynamic and spontaneous is also the advantage to WiFi. As long as you plan on that type of environment instead of being surprised, then this model become significantly more successful.

All of these ideas and techniques are to provide the tools to make WiFi competitive. However, these are still just the technical ideas without getting into the financial details. I wouldn’t propose them if I didn’t know they could be financially successful. If your focus is 100% technical, this doesn’t work. You will absolutely need someone with business and marketing experience to make this happen. I keep learning about new ideas that can generate more revenue daily if the infrastructure is in place, so don’t limit yourself to simply being a bandwidth providers. Associations and strategic partnerships are just as important to the creation of a growing system.

With the economy in a slump, there is always a place for someone to come in and do something cheaper, better, and more efficiently. Southwest isn’t the leader in the industry because they made the best martinis in First Class. They used their people more efficiently and delivered what the client wants. WiFi has the same option as demonstrated by Triad Wireless and other companies. Let’s get it moving people.

Chapter 14 - Money Talks

I have gotten calls from many people that want to start a WISP company, whether using a mesh/muni model or a PTMP system. Although I believe there is no better time than now, that doesn’t mean it’s easy but I hope to prove that it’s financially feasible everywhere. Competing against satellite or cellular services like EVDO, WiMax, or even LTE is a no-brainer in areas that have no wireline services. I’ll cover the new Sprint/Clearwire LTE service just announced for Phoenix in another article. I can build small, profitable financial models all day that can provide superior services even at subscriber/bandwidth models of 10-1. There are also other services like VoIP that can be provided but for the beginning of this analysis, we are going to focus on Internet services only. However, let’s take this into the professional corporate/investor level and see what happens.

Breaking this down into 3 areas and 2 types of design models to cover areas. These areas are a gross oversimplification because each area has different RF models that also have to be considered based on how RF friendly they are. However, for the sake of argument, let’s use these numbers:

1. 
   
   Rural – 20 or less potential subscribers per square mile
2. 
   
   Suburb – 600 potential subscribers per square mile (2700 people per square mile in Phoenix)
3. 
   
   City – 3,000 plus potential subscribers per square mile (Boston 12000 people per square mile)

The 2 basic types of models are PTMP and WiFi. So, let’s look at how each of these models can be deployed profitably. We covered inexpensive WiFi systems early on and I still stand by that model, but let’s analyze it a little more critically and see where it fits in the big picture. More important than anything, is it possible to build a system that has the financial strength to be a growing and profitable company versus Joe Technical’s weekend hobby?

As many WISPs have successfully demonstrated, clearly it’s possible to be profitable if you market in some rural areas. Assuming a starting company of 4 people, the revenue generated has to be around $40,000 per month to be profitable at the low end. I’m just summarizing some of the spreadsheets that I have used so you will have to take my word on the cost structure. With an average monthly rate of $30 per month per client, we calculate that it will take 1333 clients to make that type of revenue each month. By all accounts, that’s a pretty impressive size to jump right into. Keep in mind this doesn’t include your original Capex and how much investment you need to get to 1333 clients. It’s not going to be cheap.

If you are a hot-spot provider, you get revenue from hourly, daily, and weekly rates. Those markets are also shrinking in the U.S. for the most part but some new ideas for phone users are coming forth. In this discussion, we are going to focus on the monthly subscribers. There are also other revenue sources such as installation revenue, business rates that are higher than personal subscriber rates, and other normal ISP types of services. Additional services might require additional staff with more expertise, thus raising the associated monthly costs.

Rural markets are easy. Set up a PTMP system on whatever inexpensive vertical assets you can get access to. If there is not a lot of vegetation, these systems can handle36 square miles easily per tower and up to 300+ users per tower. This assumes you don’t already have competition in the area and there is no interference. You just need 7 towers with 200 users on each tower cover your monthly expenses and be profitable. If there are any of these areas left in the U.S., let me know but I’m not going to hold my breath.

However, there are some rural markets that are underserved simply because of distance, vegetation, and costs of deployment. They may not have 1300 clients, but find a few of them with a couple hundred clients per area and the plan still works. Some of these areas were just not reasonable to do with 2.4GHz or 5.8GHz. They may have been a good 900MHz radio option but limits in equipment, the band, and interference mean that most of the 900MHz products either only delivered 1.5Mbps or the cost of deployment was too high. Newer 802.11N 2x2 MIMO equipment that is hitting the market now should allow for an improvement in throughput in these areas with client bandwidth similar to anything available in the 2.4 and 5.8GHz bands and cost far less. Depending on the design, it should be reasonably easy to promise 10Mbps to a client on the wireless link depending on back end bandwidth. 900Mhz isn’t an ideal band due to limited spectrum width and interference in city or even suburban deployments. However, it extends out the range of clients in dense vegetation areas. In most rural deployments without too much interference, I would expect bandwidth off a centralized tower to max out around 160Mbps with the right setup in 900MHz.

Let’s move into the suburbs now. Most suburbs are served by cable, DSL, or a combination of both. To be honest, DSL is simply not living up to the hype of the marketing division. In Phoenix, Qwest advertises speeds up to 20Mbps when they are lucky to hit 3Mbps, even within 1 mile of Sky Harbor Airport in the middle of the city. Move 300 feet and not only can they barely deliver 640Kbps, they can’t keep it running more than a month or two before it to crashes again. In my case, after I complained for the umpteenth time, they told me with one more complaint they would pull out of our business complex leaving me with no wireline high-speed bandwidth. I had to move my office just to get 3Mbps after several years of subpar service. Even if you are satisfied with your DSL service, it only takes one bad technician adding one more client in your area to cause it slow down or crash again. If this is the main provider of service in your area, I say let the best technology win. I will take WiFi based wireless over any area where DSL is the only technology available. That alone means that major cities still have opportunities.

Cable is another matter. Cable companies are promising huge amounts of bandwidth today and for the most part are very technically stable. In Phoenix, I have a 20Mbps circuit. However, I get around 11Mbps on Speedtest or Speakeasy most of the time. Other times I have seen it down to 1Mbps. Even though you may pay for a specific level of bandwidth, there is no SLA that you will be delivered that level as opposed to a business level SLA agreement. The reality is that the bandwidth advertised is the burst speed or web browsing speed. If you download or try to move large files such as video or a file transfer, that speed will be reduced significantly. However, other than fiber to my house which I doubt I will see in my lifetime, it’s the usually the fastest option.

In either case, this isn’t a market where 1Mbps is going to fly. Even Grandma and Grandpa are watching NetFlix. Google TV is no longer an urban legend either. Throw in every game machine out there downloading movies and unless you are willing to run with the big dogs, this is not a battle ground you want to enter. I’m talking about a wireless service where you plan in advance on delivering 5Mbps average to everyone with peaks up to 50Mbps or more and no more than an 8-1 bandwidth to user ration. 802.11 b/g/a won’t fly here. You have to be ready to invest in bringing a large bandwidth pipe in and the subsequent costs of deploying 1333 users. I have numbers on both PTMP and Muni-WiFi models but to simplify this model and since the area is generic, let’s assume a 50-50 model.

Since we are estimating 600 households per square mile (houses and multi-dwelling unit) and assuming we get 20% of that market, we need about 10 square miles to meet our $40,000 revenue total. That also means we need about 4 towers or building assets to ensure we have LOS to all the locations. Estimating a cost of $25K for the backhaul (assuming the entire system is wireless) gives you 2Gbps. However, it might make more sense for a local fiber, MPLS, or even cable backhaul although using a company for your backhaul that you plan on competing against may not be the best idea. Unless you consider bringing an anti-trust lawsuit against multiple telcom providers while they drive you out of business a fun time, you probably want to find other local carriers. In the end, we have determined that the tower installations will cost about $125,000.

Since half the clients are WiFi, we are still going to have to install 16 APs or more per square mile. I’m going to use that number along with some numbers on designs I’m working on now to come up with $30,000K and 360Mbps per square mile. Each AP will deliver up to 100Mbps to start with and if you go back to previous articles, can deliver several hundred Mbps if needed. Since we have the vertical assets, backhaul to the street lights is taken care of. The problem here is that it still costs us $300,000K and is the largest part of the Capex. Additional revenue sources have to be found here to justify this but we will discuss this later.

Our numbers assume that half of the clients will require truck rolls. That will cost $195,000 to deploy. Considering that each technician can do about 4 installations a day, it will take 160 man days to install enough clients. Since there is only about 20 workdays per month, it will take 8 crews to get up to this number within a month. Most of us would have to outsource this unless you have several friends with days off who also happen to work as installers for Direct TV. The revenue on installations will range from $0 to $200 depending on the market. Assume $100 and the net cost of deployment is going to run about $60K.

The other 650 clients can be installed any time since they are WiFi based. Either they can connect directly or some type of CPE device can be made available. Let’s call that a no Capex cost since $50 will come pretty close to the actual cost.

To summarize, the Capex for this model costs $530,000 to deploy. Even if we net $10K per month which is reasonable, this isn’t going to fly financially with a 4.5-year ROI. This is also a maximum bandwidth system than can compete directly with Cable/DSL/LTE or anything else out there for the near future. Scaling it out further doesn’t improve the ROI but it sure increases the revenue stream. So, how can we make it more cost effective? There are two ways, reduce the cost or find more revenue.

Start with the idea that we only need 100Mbps per square mile in the beginning. This reduces tower costs down to $15K per square mile for a backhaul system that will support 720Mbps instead of 2Gbps. We just saved $75K. It also drops the per mile costs of the WiFi system to about $20K per square mile. That’s another $100K on the savings side.

There isn’t much that can be done on the truck rolls other than to consider the option of doing the installations in house. That saves about $50 per installation but it may take 3-6 months to get up to 650 installations. That’s also more reasonable in most models that I have developed. What I didn’t take into account is the advertising necessary for this speed of penetration but there are several cost-effective ways to do this. I’m leaving this number alone for that reason.

The cost is now down to $360K. The ROI is down to 3 years but is still not that reasonable except for this, cable companies are not reducing their rates. In fact, they keep going up along with the bandwidth. One of the reasons is that they are being squeezed on the TV side by content providers demanding more revenue per users. At the same time, it’s much more difficult to raise TV cable rates due to competition from satellite providers and local ordinances. People are also dropping land lines faster than a Bugatti Veyron because of cellular phones. As much as I complain about cruddy DSL service, the reality is that it has also taken many of the Internet clients from cable due to low cost, which helps to drive the prices down. You do get what you pay for however. Hey DSL companies, here is an idea. Instead of trying to bring fiber to the home which you clearly can’t cost justify, how about just bringing it down the street so my little modem doesn’t have to connect 3 miles away across some 20 year old wires. If anybody needed a wireless option, the DSL companies do and they have boxes on almost every street. Having those assets for a wireless design would be my wildest dreams but for some reason, they can’t get off the wired mentality.

There are other revenue sources for this that I have covered in previous articles. Think through some ideas with cell phones, hot spots, multi-dwelling buildings, backhaul, VoIP, other ISP and business services, and the ROI actually starts dropping to about 2 years or less. If you have an existing company that already has sales people, project managers, and office staff in place, then the costs of getting to this point is pretty reasonable. Peg that at about $50K. Starting this project from scratch probably means about $150K. That adds 6-18 months on the ROI. Scaling the system and deploying more slowly puts out the ROI but reduces the Capex as the system starts paying for itself in about a year. There are many ways to play with the numbers but the bottom line is this, it’s now possible to compete with wireline services in any market. I’m also basing my numbers on what equipment can be bought today. I am pretty sure tomorrow may bring many more surprises that will change the financial and technology foundation of WiFi and that tomorrow is far closer than we think.

Chapter 13 - Interference isnt a Hockey Penalty

Unlicensed frequencies mean that interference is a way of life in most major cities. The question is what to do when all the frequencies you plan on using or even radios that are in operation start having errors. I just came from an installation like that. In addition, then I was asked to design an expansion to the system. One of the clues was that interference is a problem is when things work for a couple weeks and packet errors in the log jump from nothing to tens of thousands.

By default, most of us set up our networks with 20MHz wide channels. That’s the default for 2.4GHz WiFi and usually the default for most 5.8GHz deployments. However, what happens when we do a site survey and 500 APs show up on the list? If the design is already deployed, your options are limited. You basically have three:

1. 
   
   Find the least interfered channel. I usually don’t hold much hope out for that.
2. 
   
   Increase power at the possible expense of reduced modulation rates – which should also be titled “How to make new friends”
3. 
   
   Reduce channel width down to 10 or even 5 MHz – not an option with WiFi hotspots
4. Play FCC Russian Roulette and and use channels you aren’t supposed to be on with way more power than are legally allowed in those bands anyway. It seems that for those of people violating FCC rules, they must figure if you are going to break one rule, might as well break two. I have even seen systems set up by consultants and utilized by police departments on these frequencies. If the FCC ever drove through towns with sniffers they way Google did with cameras, the fines alone would solve the National Debt.

If you control both sides of the radio equipment equation, a combination of these together might squeeze out better performance and minimize the interference with a little patience. Sometimes you just have to take the best option out of bad choices. For example, is it better to have a 20MHz wide channel with a few errors or a 10MHz wide channel with no errors? That same argument applies with a 20MHz wide channel with more power which might reduce modulation versus a 10MHz wide channel with lower power and higher modulation. Keep in mind that you are now running your design on the edge so that any new interference will probably degrade your system even further. On the other hand, you might be disrupting someone else enough that they move to another band.

Designing the system from the beginning provides more options than having to fix an existing system. Let’s get something out right now. If tomorrow you decided to deploy a PTMP system with wide angle sector antennas in an unlicensed frequency band in a major city, you are not going to be the first one in those bands. This means expect interference and plan for it. Use the first three ideas and let someone else who should be working at McDonalds instead of consulting in the wireless industry, use option 4. If you get bored, site survey the band and give the FCC a little heads up on what you found in the 5.0-5.6GHz bands.

The strategy you develop from here is going to depend on the type of application. If the application is WISP services where you are trying to cover a general area, good luck. I’m not saying it can’t be done, it’s just going to be about as easy as washing a cat in the tub. Throwing up 90 degree or more sector antennas or even worse, omni-directional antennas is simply inviting problems. Not only are you going to tick off everyone around you, they are going to return the favor out of pure self-preservation.

Let’s start with the idea that deploying anything with 802.11a/b/g today is also simply a waste of bandwidth. Go right for 802.11n and try to go for 2x2 MIMO. However, keep in mind that as hard as the standards body tried, 802.11n simply doesn’t work well with legacy devices. It’s not that they won’t function, it’s just the legacy devices slow down 802.11n radios and simultaneously 802.11n will interfere with 802.11a/b/g radios. Throw in some manufacturer proprietary settings and things get even more interesting. Other than the basic ideas above, is there a way to work in a high-interference environment?

Start by thinking of your English 101 class in college. Charlemagne said, “let my armies be the rocks and the trees and the birds in the sky”. Let’s change that to “let my shields be the trees and the buildings”. Even though RF engineers in 2.4GHz frequencies and above look at vegetation and buildings as the enemy, they can also be allies in a design. Instead of fighting RF interference in the open air 10 floors or more above the city, take the battle to the ground where the buildings and trees block interference. Taking an RF survey 300’ in the air is a whole lot different than taking one standing on a corner block surrounded by trees and buildings.

So now you must be thinking that we are back to metropolitan WiFi. Actually no. WiFi assumes 2.4GHz and even at ground level is probably pretty congested. However, there is an FCC rule called the 3-1 rule pushed through by Vivato with the FCC. We pretty much know the current omnidirectional rule for WiFi, 30dBm radios with 6dBi omni-antennas. However, the 3-1 rule means that in a PTP link we can increase the antenna gain by 3dB if we reduce the power output by 1dB. If we are going PTP at ground level in 2.4GHz where we know interference is going to be an issue, some really directional equipment might work pretty well. It shouldn’t be too hard to get a 52dBm signal to go 2 blocks with directional antennas 15’ off the ground. This can be achieved with a 30dBm antenna and a power output of 22dBm. Obviously I’m joking but the point I’m trying to make is that highly directional antennas at ground level in any band will be very effective, even in high-noise environments. Drop the channel width down to 5MHz with a 2x2 MIMO 802.11N radio and your interference issues will probably become a memory. Some poor schmuck in the middle of this link might get hammered however, especially if he is using a legacy device. Let your conscience be your guide here.

5.8GHz works even better in this environment if the trees aren’t in the way. Testing has shown that 2.4GHz 2x2 MIMO dual-polarity will punch through trees but 5.8GHz works about as well as a political candidate does once he gets elected. It might work or it might not. Either way it’s not reliable. The buildings will also kill 5.8GHz signals from above, leaving the ground wide open. There aren’t a lot of indoor APs using 5GHz bands yet but with up to 53dBm of signal output, interference will be a memory. In reality, you really don’t need or actually want 50+dBm of signal to go 2 blocks unless you are shooting through dense rainforest but the concept of using highly directional antennas at ground levels might solve some problems.

Taking that further, if you are using a proprietary polling scheme like Motorola or Ubiquiti or frequency hopping, the noise those APs will generate will be massive. Using this technique in a financially feasible manner, assuming you don’t have the government’s open pocketbook backing you, requires either a mesh type radio with directional antennas or a generic radio that can be either an AP or a CPE device, depending on settings. There are also many variations of this that can work quite well also.

I am finding that almost every environment that I’m going into is challenging me to come up with new variations of designs to be successful either technically or financially. The old hub and spoke model is much more difficult to deploy in a city today when there are thousands of radios already deployed with many of them not following the FCC rules in either frequency or power output. Rising or high internet costs coupled with lower equipment costs are now opening up new competitive markets for wireless that require new financial models that actually work.

This design is specifically oriented towards a fixed location model in a high interference environment with talk buildings. Dynamic location designs have to be done differently because you don’t have fixed CPE locations. We covered this in some of the previous articles. Omnidirectional antenna use in a municipal environment just doesn’t fly any more. In lower population density environments, things are different. In the middle of a major city however, use the environment to your advantage.

On a side note, it’s been pointed out to me that I should probably proofread my posts a little further. I notice a few “minor” typos that I miss in my haste to get these done. Unfortunately this isn’t my day job and my wife who clearly proofed the first few, just doesn’t have time to keep me from looking like I never took a technical writing class before. She also hates the first person style in a technical document and gave me the Rollie Eye every time she took out a “you” and “we”. So, I’m down to proofing myself and I’m hoping that content is more important than grammar. Proofing these takes twice as long as writing them. So, if you the reader are willing to let the grammatical errors slide by, I’ll try and get them out a little quicker.

"Rory Conaway" - Triad Wireless

Chapter 6 – Free is not a Business Plan

Our system is installed and our credit card maxed out. Now, we have to either pay for it or figure out how it’s going to save what we invested in it. As an income based system, it’s pretty easy to figure out a direct correlation between expenses and revenue. If there is some kind of defined savings, we need to try and make that objective and measurable.

Let’s talk about the profit scenario. These are just the direct costs:

1. 
   
   We spent $10K putting the system in.
2. 
   
   50Mbps costs $450 per month (data center plus roof rights)
3. 
   
   Pole rental costs $5 per month per pole (16 poles) or $80 per month

On the income side, you are going to have daily, weekly, and monthly clients. Let’s say you charge $5 per day, $15 per week, and $30 per month. It’s fairly easy to calculate your income/revenue to put a profitable scenario together. However, let’s go back to the original premise of a low cost system.

A municipal WiFi system has the basic problem of reduced range due to simply physics limitations. I plan to share additional ideas along this area in the near future but for now, let’s assume all clients are 2.4GHz and we still need 802.11G compatibility. This means that we either spend the money on an expensive, all-encompassing infrastructure, ala the sixty AP 2x2 MIMO design, or put that cost on the client side. Having the clients cover part of the Capex not only means a lower initial investment, but costs can scale upward with income.

This design took the original Muni AP concept, added 6dBi or better on the antenna gain, and had the benefit of 802.11N improvements in receiver sensitivity that adds another 10dBm. It doesn’t take advantage of 2x2 MIMO so we left 3-6dBi on the table of signal quality and bandwidth. However, we spent $10K instead of $100,000-$150,000. For 10% or less of the cost, we got 50% of 2x2 MIMO performance and 120% of the performance of legacy 802.11b/g systems. Don’t worry, there is a lot of capacity still left on the table that we can add later.

We now have to deal with the problem of not being able to connect to 60% of the indoor clients. This isn’t unique as most of the Muni-Wireless systems recommended some type of high-power indoor repeater device. Unfortunately, it was an afterthought when they determined that a high percentage of users couldn’t connect or basically that the system was grossly oversold. The indoor repeater balanced the power equation between high-power AP’s and weak laptop transmitter. The problem with these devices is that they create more interference on the channel due to that combination of high-power and omni-directional signal pattern. A better solution for the network is a directional client radio with higher gain antenna and lower power. There are many products but I suggest Ubiquiti Nanostation 2M or Nanostation 2M Loco radios. They have an optional window mount for indoor coverage and cost less than $100. They are also dual-polarity 2x2 MIMO in case the network gets updated later (hint, hint). The radios may need to be mounted outdoor for longer range or to get over the tops of houses or trees which means truck roll. These devices are not repeaters all you get is Cat-5 to the computer. Indoor wireless coverage will require a separate indoor wireless router .

How does this affect our profitability? Assuming 200 potential clients in 1 mile area, we need to get 18 clients at $30 per month to break even on the direct bandwidth costs, not including the payback on our Capex. That’s less than 10% of the potential clients in our 1 mile area, assuming all residential housing. Not an unreasonable number. There won’t be a lot of profit on residential truck rolls but at $200 per install, at least it won’t be a loss.

With 50Mbps per square mile and 70 clients, the system can be cost competitive with most wire line services. What happens however, if there isn’t a data center down the street? We have to figure out how to backhaul from a data center much farther away and probably within a LOS shot for a direct wireless. That could cost anywhere from $500 to $15,000 depending on distance, interference, and frequency availability on the roof. Although you could contact the local loop carrier and ask for a quote on bandwidth, the reality is you will pay $300-$3000 for 1.5Mbps to a 45Mbps DS-3 circuit. Some areas have MPLS and other data options but if you can get 10Mbps for less than $1000 per month from a local carrier, you are doing well.

Another option is to look for wholesale carriers for DSL. Although DSL usually ranges from 512Kbps to 7Mbps average, this goes up or down in an area based on distance to the Central Office or DSL switch. Assuming you can get 7Mbps down and 1Mbps up and your DSL wholesale carriers allows you to resell the bandwidth, you will probably spend about $60. Order 7 of them, put a Peplink 710 router on your network and you have 49Mbps down and 7Mbps up of available bandwidth. No individual gets more than 7Mbps down and 1Mbps up, but the router will load balance the users to get them the best bandwidth available. You are still below your $450 per month budget but the router will cost $4000. Peplink and other companies have smaller routers for fewer DSL lines starting at $300, so you can budget based on expected system needs. Keep in mind your oversell rate of about between 10-1 and 20-1 and that means 70-140 clients getting close to full bandwidth 100% of the time. 70 clients would generate about $2100 per month in revenue compared to your direct costs of $030 per month. The DSL idea can scale starting from 1 circuit keeping monthly costs in line with revenue.

The previous scenario is basically worst case. Assuming you have apartment complexes in the area, not only does the revenue potential increase, so does the percentage of temporary users. These are users that need 1 day, 1 week, etc… The revenue per day for 1 day users is 5 times higher than monthly users. Anything you can do to attract those users is a huge increase in revenue. Throw in areas that include business users, and the revenue potential goes up even further. Business users can be charged 40% more than residential users so there is more potential there also. Hot-Spots like restaurants, parks, etc… will add more revenue.

Here is where we are going to diverge from the original concept of mesh systems and open up the opportunity to make significantly more revenue. It’s been mentioned that the only way to really guarantee 100% performance of a mesh network is to install 60 AP’s per square mile. The reality is that it’s extremely difficult to recoup the kind of capital expenditure at $2500 to $3000 per installed AP (parts, labor, back end, and other miscellaneous costs) you need for this coverage and the monthly costs. Even our design, scaled out to its maximum potential down the road, will cost $1400 per AP installed (but it will it move some serious bandwidth). If it was easy to make a profit, companies would be throwing up municipal systems so fast; it would make your head spin. Throw in monthly costs of pole rental, backhaul or local loop costs, support, business expenses, etc…, and this model fails unless you get the following:

1. 
   
   The local government pays for use of the network thus supplementing the cost or by being the anchor tenant.
2. 
   
   Sprint, AT&T, Verizon, or some other carrier pays to hand off some of their subscriber bandwidth needs since their purse strings are slightly deeper than most of ours
3. 
   
   You find the 1% area in the country where wired carriers use their monopoly’s to make it easy to compete, there are lots of free vertical assets, and there are very few trees.

The system we designed achieves the strategy of 100% street coverage which meets most of the needs of public safety and municipalities. This opens up the government market. We have determined that some users will need indoor subscriber units. However, the one area that hasn’t been covered directly is the idea of the system simultaneously being used as a Point-To-Multipoint (PTMP) system. Basically we need a hybrid muni system. A PTMP system has the advantage of range but doesn’t provide street level coverage and usually won’t cover indoor. With an outdoor antenna on the client side, the system can support clients up to 2 miles away LOS. Our upgraded system will support up to 5 miles or more. This greatly multiplies the potential revenue of the system. Clients purchasing indoor units are creating a mini-PTMP system already. The only difference is that as the provider, you will have to provide staff that can go on-site and install a radio in a residential location. On the positive side, it can also be another source of revenue since the cost of equipment will be less than $110 for the install. Keep in mind that every subscriber we add brings in another $360 per year or more. This design with that addition, keeps the best of both worlds.

The focus of municipal networks has historically been high-density areas. The obvious advantage is having a market potential of 10,000 clients or more. These are the kind of numbers that are needed to cover a multimillion Capex. The budget model we created allows for much lower density deployment while still creating a design that creates a product that has value for public safety, water meter, parking, video surveillance, and other options that create value for a municipality to become a client. That provides two potential markets. Throw in the PTMP market, and we have not only created 3 markets, we can provide a more reliable, stable product with higher bandwidth capacity per client, and a larger coverage area.

Does this change the model of a true municipal network? Not really. Besides cellular, the most profitable wireless networks are PTMP. They cover many of the areas that wired never moved into. For example, I have an area of 50 homes that was never profitable for wired due to the length of runs. Put up a single AP with an omni-directional antenna, feed it with a T-1, charge $60 per month, and everybody wins. It beats satellite hands down and people still watch Netflix. Some part of the municipal network usually has roof rights for backhaul, usually on unlicensed frequency, to the AP’s on streetlights if they aren’t attached to fiber. Those locations are providing a PTMP system already for the AP’s. I’m suggesting that this same model be used for clients. In later articles, I will show you how this part of the network can be upgraded to easily deliver 20Mbps to residential and 50Mbps or more to businesses.

I’ve taken some heat for the fact that this system isn’t 2x2 MIMO. Keep in mind that this was first designed to create an inexpensive and/or profitable network. It will perform better than an 802.11b/g system due to better receivers, higher antenna gain, and better protocol. It can also support a PTMP design that can cover a couple extra miles around the 1 mile area for additional customers. The network is better controlled with more users using directional antennas for indoor coverage which reduces interference in improves s/n ratio. It doesn’t have 100% indoor ubiquitous coverage but it also doesn’t cost $150,000 per square mile, although it can be upgraded. We will next cover how to increase the bandwidth at each AP up to 80Mbps or so and expand the total capacity.

Chapter 5 - Reality, what a Concept

In the last article, AP’s got hung, WDS links were set up for hopping, and we were ready to attach to Internet. Although TriadLand is ready to rock, we now need to connect the users. First we need to attach connect the network to some type of Internet service and then come up with a way to authenticate the users that want to use it. After that we will cover the details on some of the system management and how to overcome them.

WISP operators are typically forced to work with local bandwidth providers who typically have some type of monopoly for the area. A WISP can’t resell most of the business services over DSL or Cable due to the local provider not allowing that. WISPs are generally resigned to T-1 circuits or more expensive business options. Assuming that you order a T-1 for your system, you now have 1.5Mbps of bandwidth for your network. If you plan on using cable or DSL services, check with your local provider to see if that is allowed. Other bandwidth options are also available but you will have to check for each area.

Assume that one of the 4 center APs out of 16 are the Internet connection point. We will set up the WDS links so that now end point is no more than 4 hops (meaning we may have to skip one) which will keep the last AP with around 5Mbps at the end point. If we can skip more than one AP, we can keep the hops to 3 or even 2 if we have LOS between the AP’s. We have enough signal to hold very high modulation rates with ½ mile links between APs. Keep in mind that APs between the end point and the egress point will be handling users while simultaneously passing WDS backhaul for other APs down the chain. This will directly affect throughput for users down the chain. It is one of the limits of this design but no different than any of the other earlier mesh designs. The end result is that the entire square mile will eventually be routed through one AP.

One key issue I received a couple of emails on involved security. The drawback of inexpensive radios is that you may have to give up something in return for the reduced price point. This system provides no encryption over the WDS links. This problem gets resolved with additional hardware as part of an upgraded system. The system can run security between the laptop and the AP but there isn’t much use if the AP hops aren’t secure. If you plan on upgrading later with more bandwidth, then it might be a good idea to get the users to use WPA2 on the APs from day one so they won’t be confused later. Just make sure your EULA clearly states that the system is not secured over the wireless link.

The second problem with this network is that it only supports a single SSID. I do not know if that is going to change in the future. There is third party firmware that will run on the Bullets that may offer more options but that typically comes with additional costs which gets away from the original premise. If you need security, then VPN tunnels are the only option with the basic system. Phase 2 resolves most of the security issues.

There are many good products out on the market for authentication of users. Our sites use Patronsoft Firstspot for user authentication and management. FirstSpot runs on Microsoft Windows XP or Windows Server, can support SQL Server for extending site deployment and centralized user management, uses PHP for the web pages, and can run fail-over servers for offline management. Since our company has years of experience with Windows, it works for us. Those of you with more experience with Linux have many other options. We ran tens of thousands of users through our servers over 5 years so I’m pretty comfortable with it. However, it’s not CALEA compliant yet but they are looking at it now.

Triadland is up and running. Users attach to the broadcast SSID, get a login page, diligently read the EULA word for word in which they agree to follow the rules, and then they get online. Now what happens? This is stuff normally planned out in advance but the article focus was determining the basic wireless technology first. It’s time to deal with the actual functionality of operating the system.

This is where every WISP’s worst nightmares start. It begins with the Federal government getting access to your system in the name of Homeland Security and ends with junior making it his personal mission in life to download the entire Sony movie collection. Your first job is to file your CALEA paperwork. CALEA is an entire article by itself and I may cover it much later. Go to http://www.wispa.org/?page_id=2022 for more information. After that, you need to figure out how you are going to keep control of junior. You are also going to have to deal with the users that move large amounts of spamware or spyware without even knowing it. These same users can get you disconnected from your Internet circuit if it’s bad enough.

Let’s move on to the first issue which is keeping control of your network. Several things will stress both you and your network. Let’s start with junior’s desire to fill up that new 2 Terabyte hard drive he just got for his birthday. File-sharing is one of the biggest problems faced by most ISP’s. Fortunately or unfortunately, depending on which side of the equation you are on, recent rulings by the FCC allow operators to limit file-sharing. There are 2 basic ways to handle this. You can use either an authentication server that keeps track of bandwidth used or a web application firewall that allows you to block file-sharing applications. Limiting users to 10GB – 20GB per month is a good start.

Early on, we had an incident where some users on our network had been infected and turned into spam servers. The bandwidth provider started blocking Internet for the entire system until we got it resolved. With 200 plus users and some of them still running Windows 98, the battle to keep viruses and spam under control is difficult at best. We purchased a Barracuda Web Application server which not only blocked the offending users, it redirects them to run spam removal software and won’t let them on Internet until the computer is cleaned. A Web Application filter also allows blocking of specific websites that might cause unwanted legal attention and file-sharing applications.

Now that we have built our 1 square mile network in Triadland, our next step is to make either make it profitable or find a way to give it purpose. We will cover those ideas next article.

Chapter 4 - You Should Never Hang APs With Your Spouse

The low-budget Muni-wireless system is ready to be deployed. We have identified the perfect one square mile area to be covered. The streetlights are 25’ tall and exactly 660 feet apart with full time electric power. There are no trees and the houses are 20’ tall and made of wood. Nobody in the area owns a microwave oven or any indoor WiFi routers. This will be known as TriadLand. It’s my make believe city. I get to name it. The goal for the first square mile is complete indoor coverage to a laptop. Let’s find out if that is realistic.
First calculate the ideal distance that a laptop can connect to an AP. To do that, get the technical information on the AP, the antenna, and the WiFi card in the laptop. The AP we are going to use for this design is a product manufactured by Ubiquiti, the Bullet M2 HP (http://www.ubnt.com/bulletm). The antenna is a 15dBi omni-directional (tested closer to 14dBi) collinear vertical polarity design sold by L-Com (http://www.l-com.com/productfamily.aspx?id=6414). Laptop specifications will have to be an educated guess, since they will vary. We will use 15dBm for the power output and an antenna gain of 1dBi for all calculations. I’m going to assume the receive sensitivity is the same as the AP. Although the Bullet 2M HP is certified for only a 6dBi antenna, I have already addressed the issue and this combination will pass FCC rules in a few weeks.
Astute readers will notice that the firmware included with this radio doesn’t support mesh. However, it does support WDS. Basically, each AP will have to know the Mac address of the AP before and after it. This had to be entered manually into each AP. WDS then creates a layer 2 connection between each AP. By using WDS instead of mesh, we lose 2 things. The first is that if a radio fails, that path fails. There is no recovery option other than to drive out and replace the AP. However, it’s also possible to connect to the next AP wirelessly and manually bypass it. The second is that the system does not provide any type of load balancing. Not every mesh system does so that anyway. We also have an upgrade path if there are clusters of high-bandwidth users that we will cover later.
The Ingress/Egress point has to be located so as to minimize the number of hops. We will have a useable 50Mbps TCP/IP drop to about 10Mbps at 4 hops (I know that doesn’t calculate right but it’s processor limited, not bandwidth limited). I will cover an upgrade later that delivers an additional hop and expands the first hop to 100Mbps or more. Keep in mind UDP is much higher.
Mounting this AP/antenna combination requires some ingenuity in terms of bracketing and power. I am working in a simplified mount for it now. The antenna is provided with a simple U-Bolt for up to a 2.0” vertical pole. The radio simply screws into the bottom of it. Power requires 12-24dc volts. Interestingly enough, the power issue is nice for a solar application. However, it requires a PoE that although less than $10, is not designed for outdoor mounting and is 110V. If the radios will be mounted on horizontal poles, the system will require additional bracketing for this installation. The power problem will require a small Nema enclosure to seal the AC/DC converter along with a Metropole power adapter or an electrician can mount a power outlet inside the pole.
A very simple formula for calculating the link path budget is (Power Output of the AP) + (Antenna Gain in dBi) + (Receiver Sensitivity at the speed we need). The signal loss equation tells us what we are going to lose in signal level over distance or Signal Loss = (20 x Log10 (Frequency in MHz)) +( 20xLog10 (Distance in miles)) +36.6. Since all of the connections are very short range, LOS or NLOS, and well below 10GHz, we can sort of ignore a lot of other variables that are used for longer range and higher frequency calculations. So how far can we get 2 AP’s to talk to each other and keep a maximum modulation rate with a power level that matches the laptops?
Since the EIRP of the laptops is about 15dBm, we will start there. The link path budget between APs is -
802.11N: 15dBm (radio output) +28dBi (antenna gain is a summary of both sides) + 74dBm (speed at MCS7, 65Mbps, for the Bullet M2 HP) = 117dB
The link path budget for legacy laptops is -
802.11b/g: 15dBm (radio output) +15dBi (antenna gain is a summary of both sides, assumes 1dBi on the laptop) + 74dBm (speed at 54Mbps for the Bullet M2 HP) = 104dB
The second equation needed is the free space loss. I’m not going to go into the formulas here, but the result calculated at 660 feet was -83dB for the laptops. That means that we have an expected signal level of -56dBm. From AP to AP, we should see a signal level of -49dBm. So in TriadLand, we could install APs at 1320 feet with signal to spare at the full 802.11N or 802.11g bandwidth while delivering the same signal level on both sides.
The reality with an urban deployment is that the farther down the street the AP is located, the more houses the signal has to pass through. We will assume that the AP is mounted on a street light. Houses built of brick, stucco, or aluminum siding along with trees, bushes, and Uncle Bob’s motor home in the driveway, all add attenuation to the signal quality. Therefore, for the sake of this design, we will assume that each house adds about 10dBm of attenuation to the signal. The Bullet 2M, at the lowest connection speed in 802.11N mode, has a minimum sensitivity level of -96dBm. A good rule of thumb for noise in 2.4GHz is about -85dBm. That means we have -29dBm of attenuation room to make a connection or about 3 houses. However, since we want a 10dB difference between signal and noise, we now are down to penetrating 2 houses at maximum speed and dropping off from there.
Attenuation is the biggest problem in an urban setting which means that LOS is more important than distance. To guarantee connectivity in the worst locations with trees and brick houses, it will take APs about every 1/8th of a mile which means no client is further than 330 feet. Cities such as Mountain View, St. Cloud, Scottsdale, and others learned this the hard way after setting up the system with fewer APs per square mile than really needed. In addition, those APs used 7-9dBi. We are adding up to 6-8 more dB gain on the antenna side which should add about another house of penetration.
Being more realistic, what are the options? Let’s say that we assume 16 APs per square mile and all clients are 802.11N. Also assume 14 houses per block and 2 blocks for 1/4 mile. The light poles will be about 30’ from the front of the house. The farthest house is 660’ away and will not have LOS except for the first couple of feet into the front of the house. In the worst case, users may have to penetrate 8 or more houses at various angles, plus trees or vehicles may be in the way. If the houses are brick, stucco, or have aluminum siding, the signal will definitely not go through it. In this environment, we are installing 25-49 APs. The total cost for a 16 AP install is about $10k per square mile. As mentioned before, $1000 upgrades the bandwidth to 60Mbps for the square mile and 10Mbps at 4 hops. If you need more bandwidth than that, the cost jumps to $14K per square mile per 16 APs and carries a 100Mbps pipe through every AP.
Based on this analysis, you now have to make the decision as to the purpose of the network. If it’s ubiquitous coverage, then you have no options. It’s going to be a minimum of 25 APs up to 49 APs or even more in the worst environments. Even though the cost of each radio and antenna is $200, the cost of installation will drive this to about $14K-$42K per square mile with that many APs. That sort of defeats the idea of a budget system. If you were planning on this type of budget, then there are additional changes that should be done for many other reasons that we will cover in future articles.
If we just wanted outdoor coverage only to laptops on the main streets, 16 APs are probably more than sufficient. If the goal is simply street level coverage for cameras on light poles and vehicles, 2-4 APs may cover that. By eliminating the 100% coverage for indoor 802.11g laptops with their limited power and antennas, the system has more options and range goes up significantly. There are so many variables in a metropolitan deployment due to the physics of microwave frequencies, that $3000 per square mile in Arizona might become $18,000 per square mile in California.
Now that we know the realities of RF, let’s stay with the original concept and change the expectations of the system to meet our budget.
1)100% street level coverage for cameras and public safety vehicles
2)100% street level coverage for laptops and portable WiFi devices (this does not mean back yard and between houses)
3)20-40% of the residential locations will have 100% coverage
4)20-40% of the residential locations will have varying coverage within the house
5)20-40% will require indoor equipment or a professional outdoor installation of a client device.
6)The system will support legacy 802.11g equipment
7)Minimum residential bandwidth from 1Mbps to 5Mbps
8)20 Mbps per square mile total available bandwidth (upgradeable to 60Mbps per square mile for an additional $1000)
9)Expandable up to 300Mbps per square mile (for an additional $4000 per 16 APs)
Our second problem is user density. Given that most APs are limited to 20-30 users per AP and each AP covers up to 80 homes, this isn’t an issue. However, in some cities with apartments, density could reach into the 10,000 people per square mile range. That could mean hundreds of people per AP. If you find a place like that, don’t worry, we have plan C.
We now have to go back to the noise issue we discussed earlier. Typically you want the signal to be at least 10dB better than your noise. In Triadland, our noise figure is about -96dBm. In the real world, it’s going to range from -65dBm (bad) to -85dBm (reasonably good). Since our signal level at 1320 feet between APs with a 15dBm output is going to be -50dBm, we should be able to handle even higher noise areas. Laptops should be connecting LOS at about -62dBm at 660 feet.
In some areas of the country, and this is the part I like, the scaled system could deliver 20Mbps - 50Mbps to a residential or business location. Based on some numbers I received a few days ago, Internet bandwidth can be purchased for as little as $1 per 1Mbps per month at local data centers. If a residential 10Mbps rate is sold for $30 and the over-subscription rate is 10-1, the potential revenue on a 1Gbps circuit is $30,000 month assuming you have a way to transport it to your WiFi area. There are obviously a lot of capital costs here. This will also take serious engineering and planning involved for this type of revenue stream, but it’s now possible. 4 years ago, nobody was making a profit at this. IMHO, this now means that wireless can directly compete with cable and DSL services.
This design also solves another problem that WISPs are running into. If you read the last article, it’s evident that WISPs using unlicensed frequencies are running out of bandwidth. This design shortens up the distance from the client radio to the AP. Instead of connecting at a few miles and increasing the risk of interference, all clients are within a few hundred feet of an AP; thus, minimizing the interference issue. It also lowers the required altitude path.
Our basic network is defined. The budget is figured out. Our users in the first square mile of TriadLand have connectivity to the network. Now, how do we get them connected to the Internet and keep track of them? We will cover that in the next article.

Chapter 3 - Share and Share Alike

Although this was the week that we were going to discuss putting APs on the poles, an incident occurred last week that I think is worth discussing before we go any farther. Always keep in mind that unlicensed bandwidth is a shared commodity. It’s also a good idea to have a relationship with your competitors or anyone in the area that is using unlicensed bandwidth. Sometimes that’s a little hard to do, but it’s definitely in your best interest to try.

Traffic lights and cameras are a perfect application for 4.9GHz or 5.8GHz radios where fiber isn’t installed. For example, we connected several traffic lights in Glendale, AZ a month before the city hosted the 2007 SuperBowl. The entire network consisted of 5.8GHz SkyPilot equipment with Pelco cameras. The system is used to monitor the intersections and also as the SCADA system for the traffic lights. This allows the city to manually adjust the traffic lights before and after events to optimize traffic flow. The city just recently closed on a bid that ADOT managed for the next phase. For some reason that I still can’t explain, they removed the compatibility clause with the existing system and simply went for low bid. Due to the cost of the SkyPilot equipment, it was obvious that low bid would not be SkyPilot or a compatible system but something else. This means the city will now be supporting multiple mesh networks along with the extra long term costs. He who controls the funding; writes the rules but apparently doesn’t have to support it.

That led to another project to connect 13 traffic lights with 50 High-Definition cameras at resolutions up to 1600x1200 pixels. We settled on Axis cameras for this project. The ideal cameras, fixed and PTZ, would be 1080p, 24 frames per second, and H.264. The project came close with most of these specs, but there were some compromises. We either got PTZ and H.264 at 720p, or we got 1600x1200 at reduced frame rates with MPEG-4. Future intersections may get 1080p cameras. This was sufficient to meeting the system requirements for the application which was traffic light management, license plate recognition, video analytics, and future facial recognition software.

Either way, we needed a lot of bandwidth at the City Hall to make this work. There were no other antennas on City Hall. A quick site survey showed additional equipment in the area with fairly low signal levels. The AP chosen was Ubiquiti 802.11N Rockets with 90 degree, 20dBi dual-polarity sector antennas. The system has 4 APs and sector antennas on City Hall for 360 degree coverage. Each light depending on distance to City Hall gets either a Nanostation 5M (integrated antenna) or Rocket 5M (external flat panel sector antenna). We are also adding 2.4GHz 802.11N to all the intersections for future use. It only adds $200 to the cost of each intersection and has an 800’ range to a laptop.

The first AP was turned on about 6 months ago and covered one 3-way intersection with 3 cameras and a second client station for data backhaul to one of the City Parks. The park had maintenance staff and a camera for the skate area. The second AP was scheduled to be turned on this week to cover the next 90 degrees off City Hall. That was when we discovered that between the time we turned on the first base AP and then went back to turn on the second base AP a couple of months later, the IT department had a local WISP put a Motorola Canopy radio on the roof without notifying the traffic department. Even more fun, it was in the 5.8GHz band.

Our traffic system was designed to use all 100MHz of the 5.8GHz band and was designed with signal levels for each link between -50 and -65dBm. Since the general rule is at least 10dB headroom on the link path and the minimum signal for APs with an MCS15 link (130-144Mbps modulation rate), 2x2 MIMO is -75dBm, you try to design for your worst signal at the receiver to be -65dBm. At those signal levels with directional antennas on the client side at 2 miles or less, it’s fairly hard to interfere with this system.

As a courtesy to the vendor, I thought that giving them a call and asking if they could change the Canopy to 5.1-5.4GHz before I turned on any more APs would be the professional thing to do. Unfortunately, when you run into inexperienced WISP technical support staff who think that whatever product they use is significantly better than whatever product you use, there is a problem. The conversation started with me asking if there was any chance that they had other frequency options. The technician asked me what equipment I was using. I replied. He then made the comment, “Our equipment will stomp all over your equipment.” I quickly established that this wasn’t the person to which I needed to speak to resolve this issue.

Every wireless product handles interference in different ways. Better filtering, multiple polarity, diversity, beam-forming, better firmware and many other techniques are utilized to improve the quality of a connection. Obviously some equipment works better than others in high-interference environments. However, 802.11n doesn’t always play very well with 802.11 b/g or 802.11a radios. In addition, physics are physics. Throw in a 2x2 MIMO signal with very a high gain antenna and a total EIRP of 42dBi minimum on both polarities, and there is going to be interference. This is especially true if the competitor’s base station is 1000’ away and their system is based on a 30dBm EIRP signal level from the APs. Since the traffic system was designed with a very high level signal path with multi-polarity 2x2 MIMO, there wasn’t much concern that the local WISP was going to interfere with the traffic system. On the other hand, it was clear that the traffic system was going to cause some interference to the WISP operation. I was wondering if that was already the case.

The last thing I wanted to do was start a war between a city government and a local WISP. Since the WISP was already serving some of the registered voters, this was probably not a good idea. The ego in me wanted to turn the remaining two sector antennas directly at the WISP base station across the street, program the two radios with a channel width of 40MHz to cover most of the 5.8GHz band, turn on 802.11N mode only, start multicasting “Transformers 2” in UDP mode to my other laptop and phone, and crank the power to 48dBm EIRP. I felt there might be a lesson to teach on RF, signal-to-noise ratio, and how to be polite when other companies are trying to do the right thing. Instead, I called back and talked to upper level management to arrange a meeting to see what can be done. That’s when I found out as I had suspected, that the traffic system’s first AP possibly might already been causing some interference issues. Unfortunately that traffic system AP also serves the City Park and has been running for several months. As a courtesy, the best I could do was to turn the power down until we plan out a more cooperative strategy with the WISP.

Now we can come back to our design. The AP combo we are considering will have an EIRP of up to 36dBm and will be 802.11 b/g/n compatible in 2.4GHz. That is far more powerful than most indoor systems. There are ways to limit the interference and everyone sharing those bands should consider this. It’s good policy to always engineer for the least amount of power that you need. Also, you should consider using the least amount of bandwidth. If you don’t need a 40MHz channel, don’t use it. You gain 3dB on the receiver side with no more power if you use a 20MHz channel. Drop to 10MHz and you pick up 3dB more, etc… The tighter the channel, the better chance you have of getting a signal through.

There are many of you that may not even want b/g compatibility. If the system is only used where you control both the APs and the client radios, then you have some options. In that case, using 802.11N only is the best way to go. This results in fewer APs and better performance. It’s also possible in this same scenario to get as many as 100 users on an AP or as many as 300-400 on a single pole. If all the equipment is the same manufacturer, consider a 5-10MHz channel which will still deliver 12-25Mbps per AP. Running 2x2 MIMO doubles the throughput to 25-50Mbps in a 5MHz channel. This is probably more than most of us need for a link. With a 5MHz channel, there are 6 channels to work with in 2.4GHz instead of just 3.

Whatever gets installed will probably interfere with other existing systems. We have to be cognizant of the fact that nearby systems might be critical for hospitals, voice systems, video surveillance, office networks, etc. For example, a few years ago, we saw an outdoor AP in Scottsdale stadium take down a wireless voice system at a nearby hospital. Since we put our phone number in the SSID, they could contact us quickly. We adjusted our antennas and resolved the issue. Site surveys have to be done everywhere to understand the ramifications of the install in addition to understanding your own coverage zones.

Based on all this, if I saw a little Motorola Canopy radio on the roof and thought it was a good idea to possibly contact that vendor, what was the installer thinking when he saw all the Ubiquiti equipment along with a Motorola PTP600 radio? He obviously didn’t do any kind of site survey. The reality is that this is standard procedure for most companies.

Many companies that rent roof space for APs do it from some type of property management company. There is usually no bandwidth management on the roof. When someone finally overlaps frequencies, an avalanche of problems will start. For example, administrators may turn up power to overcome the interference if they have no other options which causes even more problems (this gets into noise and channel filtering). The responsible thing to do would be to notify all parties involved and see if there is a way to work together. Obviously this sentiment is not shared by everyone, or some field installers aren’t trained well enough to watch for this situation. Then again, they just may not have cared.

Unless I run into another tech support person who hasn’t been taught the concept of cooperation and I need to vent, we will get back on our project next article. I’m still working on some FCC equipment certification issues with the design. Not that we can’t do it but I want maximum performance and the equipment I need isn’t quite yet available. I’m also waiting for new products to test that may enhance the system. Anybody can build a system with unlimited budget. The fun is creating a carrier class dependable system on a shoestring budget; since, that’s what most cities have in their coffers these days. Any new products that have the potential to help preserve that coffer is going to get their attention. Next article, we get back to work.

Chapter 2 - Access Point Secrets You Didn’t Share with your Parents.

Continuing the discussion of the budget Muni-Wireless network requires analyzing the front line component - the Access Point (AP). There are many variations of APs. They all have features and capabilities that provide enhancements in certain environments. Some of the APs break the practical limit of 20-30 users by using multiple radios in a single enclosure, proprietary polling systems (not compatible with other vendors), and advanced beam-forming techniques. However, the focus is from a budget standpoint and that means this design will start with a single 2.4GHz radio with an omni-directional antenna. Later articles will cover upgrading the design to support more users and a larger coverage area. The beauty of this design is that it’s cheap to get in the game and scalable.

The ideal inexpensive AP was covered in the first article. This article will next cover what protocol it should support. Obviously if it supports 802.11n, that’s huge. 802.11n can provide up to twice the range of 802.11 b/g with the same single AP, single antenna design. This is due to a combination of a more efficient transmitting modulation scheme and better receiver design than 802.11b/g/n. In 802.11n, this would be considered a 1x1 MIMO. If the world was perfect and everybody had 802.11n devices, theoretically it would take only ½ to ¼ as many APs per square mile for the basic system. Unfortunately, the expansion of WiFi phones and their lack of support of 802.11n means keeping legacy 802.11b/g compatibility, range, and load in mind. If the network doesn’t need to support 802.11b/g the savings are huge. The reality is though, if the network was built today for the general public, the system should still support 802.11b/g equipment.

Taking the AP design a little further, figure out what the client connectivity area is. If the goal is to connect to Joe or Jill Teenager, who lives in suburbia with maple trees covering the sky and every home is built like the Windsor Castle (brick), while he/she is lounging on the couch with his/her iTouch which doesn’t support 802.11n; then even with 16dBi omni antennas, the AP will have to have to be within a couple hundred feet. If they live in Stuccoville, Arizona, (yes, stucco attenuates signal, but go with me on the flora thing) the range will be much greater since trees are 60’ tall with leaves starting at 55’ (palm trees). The only other vegetation is 2’ tall and is considered a lethal weapon (cacti for the Northerners). This is a slight exaggeration, but the basic physics stand. If the clients are outdoor only such as surveillance cameras, mobile hot spots for police, utilities, etc…, then the number of APs is reduced significantly. In fact in Arizona, there are places where 2 APs per square mile are all that’s needed for mobile coverage or every traffic light corner.

The really interesting thing concerning small deployments, parks, mobile home parks, etc…, is that the back end Internet bandwidth options are rarely as fast as the radio capacity for cost reasons, access, or End-User-License-Agreements. For example, one park that Triad manages is really only under heavy use for two months during the year (Spring Training) in addition to 2-4 high-use specialty events. Bandwidth options are a T-1 for a whopping 1.5Mbps ($450 monthly), DSL for 12Mbps ($100 monthly), or cable for 4Mbps ($200 monthly). Assuming 12Mbps is enough (it can be expanded with multiple DSL circuits as needed), users are limited to 2Mbps, the radios support up to 20Mbps in 802.11g, and there is only 2 hops maximum meaning no need for multi-radio APs. Even at 2 hops, bandwidth is still 10Mbps on the perimeter. There are 11 radios total to cover about ½ square mile with 2 high-density cluster areas. The 2 high-density areas are each comprised of 4 separate radios at the perimeter locations. The remaining areas are covered by 3 other radios. Using high-gain omni-directional antennas, laptops can connect at 800-1200 feet. All radios are in AP+WDS mode for relay and coverage. The total cost for all the radio equipment and antennas for this deployment was $2300, not counting the tower mounting brackets. The system has been up for over 2 years and is scheduled for a planned 802.11n upgrade in the next month or so. The omni-directional antennas will stay, meaning that the 2 hop system will now deliver about 20-30Mbps on the second hop.

The project utilizes four vertical assets up to 150’ that also include a 5.8GHz PTMP system. The 5.8GHz PTMP system has a range up to several miles in a 360 degree pattern. Each vertical asset consists of one radio with a standard vertical polarity sector antenna. When the PTMP system is upgraded, it will consist of 5.8GHz 802.11n, 2x2 MIMO equipment. The cost of the upgrade will be approximately $250 for the radio and antenna and will support up to 100Mbps per radio. Throw in another $100 per pole for backhaul and the system has now expanded from a simple hot-spot project to a 100 square mile 400Mbps infrastructure for an additional $1400.

It’s time to step back and discuss the unadvertised secrets of 802.11n since it affects the expectations of the design. Here’s a big shocker - the processor and firmware of the AP affects the radio performance. For example, the AP manufacturer claims 300Mbps. That’s modulation, not real-world throughput. That number is also rounded up from the real number which is between 270-288Mbps, depending on what’s called the guard scheme (not within the scope of this article and makes my head hurt). Keep in mind that many of the 802.11n radios also have 100Mbps Ethernet jacks because manufacturers know that communication is half-duplex. The Ethernet port is full-duplex so they consider 100Mbps up and 100Mbps down, 200Mbps total, not 200-300Mbps in one direction. Strike one.

The second issue is that 300Mbps is only achievable by running 40MHz wide channels. That works great in the living room for 50’. It doesn’t work so well when the AP is perched on a light pole with 200 houses in range. It would be difficult at best to go 500’ in 2.4GHz and get maximum theoretical modulation rates with a 40MHz wide channel. Multi-radio APs typically use the 5GHz bands for backhaul for that reason. That means the real-world useable 2.4GHz bandwidth is 20MHz which translates to 150Mbps. Strike two.

The 5.8GHz band is the most commonly used with 40MHz channels. 5.1GHz to 5.3GHz is usable with some manufacturers, but the EIRP drops significantly. However, for 500’ and no vegetation, that is reasonable. There should be very little interference in those bands. Of course, many of the radios that are already in those bands are probably set up by a local WISP. Although most WISPs are knowledgeable and legal, there are a few WISPs that either have no clue about the rules or are intentionally broadcasting illegally due to congestion in the 5.8GHz band. There are manufacturers who have certified equipment in those bands, but because of the limited EIRP in those bands, the equipment isn’t as popular. 5.8GHz also isn’t supported by most laptops and smartphones.

Assume that there isn’t interference in whatever band with a 40MHz wide channel. Then the next question is how far can a 300Mbps modulation level be maintained? Well, here is the second problem. To quote a good friend of mine, “speed, distance, reliability, cost - pick 3”. The first 2 are the most critical. For just the radio, it should be “speed, distance - pick one”. Basically, to get the 300Mbps modulation rate, receiver sensitivity drops to around -72- to -74dBm and the power output of the radio drops from the fabled 26-30dBm to around 24-26dBm or less for outdoor equipment and 15-18dBm for the retail $30-$200 equipment. To get the magical 300Mbps speed in a 360 degree coverage pattern from a single radio with an omni-directional antenna, there should be total LOS, no interference, and at maximum distances of around 500’ or less with the 40MHz channel. Most multi-radio AP manufacturers recommend directional antennas but one could argue that defeats the concept of ubiquitous mesh architecture. Strike three.

I have not done testing with most of the new 802.11n APs so some of you can jump in here with some real-world values. I’m also not going to get into sector antennas here for pole installations because the size of the antennas which makes it difficult to get through city zoning. I’m going to also get some grief here from the beam-forming guys; but realistically, none of the beam-forming systems in 2.4GHz can match a 28” tall sector antenna in performance. It’s a simple matter of physical capture area. However, newer sector antennas also support multi-polarity which gives them even more advantage.

Although baseball doesn’t have a strike four, here it comes. 300Mbps modulation rates are a real-world throughput of 150Mbps through one radio under absolutely ideal conditions. This typically doesn’t exist in most suburban realities. In addition, the processor is also a bottleneck for the AP. On some 802.11n devices, a straight ftp transfer from one computer to another computer with a 20MHz wide channel (much more realistic) will result in a transfer of about 35Mbps. Put two computers on each side, and then do the same transfer computer to computer, and the throughput jumps to about 60Mbps with each stream about 30Mbps. Do a 3x3 transfer, bandwidth goes up to 72Mbps, and each unit drops to around 24Mbps. Further testing up to 10x10 transfers demonstrates 85Mbps maximum. All of these numbers are perfect signals at short range in the lab. So what happened?

Basically, some processors/chipsets get more efficient as multiple transfers of data occur but are fairly limited for a single transfer. For example, using a 20MHz channel and MCS7 modulation rates (65-72Mbps depending on guard scheme), will result in a 60Mbps with 65% CPU overhead with UDP video traffic from a single camera. However, change that to TCP/IP and the rate drops to 35Mbps and CPU processor overhead jumps to 100%. This test was done with a 400MHz Atheros processor and chipset in the radio. Other manufacturers also use the 300Mbps but really add in the combined throughput of 2 radios, not a single data transfer rate. Keep in mind that this isn’t for every single product out there, but it shows that real-world testing is definitely required before a design is signed off on for the particular application.

802.11 a/b/g APs exhibit this same behavior when compared to the marketing material so it’s not a new phenomenon. In some cases, manufacturers used UDP traffic numbers instead of TCP/IP real-world traffic. In other cases, manufacturers were using 40MHz channels for their specifications which again, aren’t realistic for most installations. In many cases, when 40MHz channels were used, the CPUs capped out so that the result was not a doubling of the transfer rate but maybe 80% more over a 20MHz channel. For example, 20Mbps at 20MHz translated to 35Mbps at 40MHz.

Another problem with processor overhead is how many packets per second (PPS) that can be jammed through at one time. A user opening up a web-site usually has a fairly low PPS requirement, thus low CPU overhead. Open up a file-sharing application and the number of sockets and PPS can go through the roof. Low-cost APs trying to handle this kind of traffic typically slow down drastically. More expensive and faster processors along with better firmware scale better under load.

There are many reasons that one AP costs $100 and another costs $6100. More expensive APs will use 2, 3, 4 or sometimes more radios within a single AP enclosure. The rest of the costs come in the form of firmware, management tools, and capabilities. Many of the outdoor units have firmware designed to optimize video transmission quality, fast-handoff between AP radios while vehicles are moving, mesh implementation, beam-forming, multiple SSIDs, multiple frequency, channel bonding, and many other advanced WiFi technologies. 802.11n radios are also start with a better specification foundation.

Switching back to the budget system, WDS communication is built in to almost all the chipsets and supported in firmware. This allows APs to connect to each other through Layer 2. It’s simple, sometimes not compatible between manufacturers, and requires manual setup with MAC addresses of each device. Mesh firmware, with a basic setup, simply finds all other units in an area, then sets up either a layer 2 and/or a layer 3 network between the radios dynamically, as part of its underlying mapped infrastructure. Mesh firmware is always proprietary between APs.

The goal of Tales from the Towers is to create a budget, scalable, municipal WiFi system. It will probably never have many of the capabilities that more expensive systems can do. The budget WiFi system will not initially and probably may never support fast handoff for moving vehicles, optimized video traffic, or possibly multiple SSIDs. Therefore it’s important to understand what its capabilities are before deciding that price is the most important factor. In some cases, its inexpensive nature, flexibility, and scalability allow the network to get around many of the advanced features that some of the more expensive APs have. In others cases, it simply isn’t going to happen and the proper design should be Motorola, SkyPilot, Firetide, Tropos, MeshDynamic, Ruckus, BelAir, Meraki, Meru, or whatever product fits the needs and budgets of the municipality. Every one of these companies has features and benefits that make them ideal for specific applications. I have designed and installed systems using many of these products.

However, the focus here is a starter system the meets the cost and scalability of really tight budgets. This makes it deployable in areas that can’t justify the ROI of more expensive equipment and can bring Internet to areas that may not be able to justify a large investment. The next article will cover the actual AP deployment design.

"Rory Conaway"