Chapter 24 – The Grass isn’t always Greener on the other side of the Contractor Fence

Being a small company with a wide variety of technical needs that vary from project to project, I use a lot of contractors. For example, I couldn’t bend a pipe without more dents than a Honda in a hailstorm. However, WISP operations and many different types of projects like video surveillance, access control, or even VoIP projects means that I had to bring in other people with special skills that I don’t have. However, a recent event in being the sub-contractor on a project with EarthCam has made me rethink how I’m going to treat my contractors in the future.

A friend of mine who does sub-contracting through a bid site was recently in a bind the night before he did a project that EarthCam desperately needed done. Since it was late at night, I told him I would take care of it the next morning for him so his bid status stayed positive. The project was to simply hang 2 cameras on a wall. These cameras are simply high-resolution still image cameras that upload the pictures to a website over a data connection. These images are then monitored by the clients and contractors to keep track of the construction process and probably to create a build movie at the end. Nothing special there and something most of us could easily build for between $500 to $2500. In fact, we will cover that later in the article for anybody that wants to do the same thing and save as much as $10,000 or more, depending on the cameras.

The contract provided by EarthCam is not much different than most of us use. You bid a job for a fixed cost until completion. That should have been my first clue to walk away from this project, favor or not. I have never used this type of contract and pay most of my contractors hourly. The problem with EarthCam is their definition of a completed project comes from the Enron book of business definitions. Apparently, the installing the camera on the wall and the techs tell you they are receiving pictures doesn’t constitute a completed project. A week later, when they realize that there are some problems with the installation, they expect you to sit for 4 hours waiting for a shipment they failed to get on-site in time for a 6am installation, and then move cameras around a couple times because they marked them incorrectly, and you should do all of that for free or at half rate after a significant discussion.

I hope that all of you who use contractors treat them better than EarthCam does with their contractors. If a contractor takes a project for a fixed bid and completes the project, any problems they have after the initial installation should have additional appropriate compensation. In my case, putting up EarthCam cameras isn’t my business nor will I ever take a call from them again other than to let them know I’m filing in court for non-payment of my invoices. I think they must have used the same accountant Bernie Madoff used.

I’ve got tower projects that are coming up and in every case; I’m paying the climbers daily rates. That’s fair to them since they aren’t going to be able to do other projects, even if they get done early. It’s up to me to estimate their time on the project. If I make a mistake, it’s on my dime. Ethical business practices should be the cornerstone of every business whether dealing with clients, contractors, or vendors. The few dollars you may save on the project will never fix your reputation and may financially damage your company. Surveys have shown that every unhappy customer will tell 10 people. Nobody wants that as part of their business as it’s a lot harder to convince someone you are now ethical after you have demonstrated otherwise. In addition, in this day of instant worldwide communication, the word gets around that your business doesn’t deal with its contractors fairly and nobody wants to do the work for you.

If you have ever been to Disneyland and visited the exhibit right inside the front gate showing you the history of the happiest place on earth, then you now know what EarthCam does. They place regular digital cameras of various high-resolutions in locations to take pictures of long term projects such as building construction for historical and management review purposes. The camera is basically programmed take snapshots at set intervals such as 1-5 minutes or longer. The camera is then attached to a router that connects to the internet and each snapshot is then stored and added to the previous pictures to create a simple stop action video.

With all the web enabled devices out there today, this is a relatively easy task to do. There are many low-resolution video cameras that can do that now. However, other than construction monitoring, why would you want to do this? I have 2 words, forensic analysis.

Let’s say you set up a video camera to monitor a public park. The camera is watching a fairly large area such as a skate park. Some vandalism occurs overnight and you review the video in the morning to catch the villain. Unfortunately, the villain looks like an extra from “The Man With No Face”. The reality is that even a 2MP video camera at 100’ is going to make it difficult to have enough quality for a jury to determine without a doubt who the culprit it. However, if you supplement the video camera or use a 10MP camera or higher shooting every 30 seconds, there is a good chance that you will be able to zoom the image enough to see if the vandal is still using Clearasil.

The speed cameras that Janet Napolitano slammed in under executive order for Arizona before bailing out from wrath of the voters of Arizona, work just like that. For those of you who don’t know, speed cameras are automated ticket issuers installed along the freeway that instantly snap your picture when you drive by them at some preset trigger speed above the posted speed limit. If you are caught, then they automatically send you your picture in the mail along with your license plate and a very nice letter telling you to pay the fine. One little detail all the politicians left out (Is an omission of fact a lie? I can’t imagine any politician ever doing that.) is the devices were actually video cameras recording all traffic until a car was caught speeding. At that point, a second camera would shoot a high-resolution picture to supplement the video stream.

Regardless of your political position on this device (do you seriously believe I’m going to let that statement stand without mentioning that I thought it was a cowardly act by the woman now in charge of defending our borders, IMHO of course), the technology was sound. In order to see the quality of the video, I got one of the tickets for research purposes (okay, I got it because they were smart enough to place the cameras where the freeway drops from 65 to 55) and I will tell you that it picked up the fact I needed a shave at 1am in the morning in pitch dark through a dirty windshield.

So, if you want to take your video surveillance system to the next level, supplementing the video stream with a megapixel snapshot isn’t a bad way to do it. Keep in mind that storing video snapshots can be significantly less storage that the video stream itself. There is a lot of software that lets you do that easily. Software such as “Canon Remote Capture”, http://download.cnet.com/Canon-RemoteCapture/3000-18489_4-199150.html, lets you even run the camera remotely through the USB port. You might also need a USB over IP converter but there are many ways to do this. The only question is what to do if the camera is placed in an area that doesn’t have an internet connection.

Assuming you don’t have some type of close Internet access, the next step is a cellular router/modems. Skyroute 4000 series modems/routers look pretty good for that although there are a lot out there. However, you have to be careful here because many of the internet cellular data plans have bandwidth caps. If your 10MP camera is shooting uncompressed pictures across the internet every couple minutes, the plan could cap out and cost big time. Assuming a 12MB image every 5 minutes, that’s 60MB per hour or 1.32GB per day. Over a month, that’s almost 40GB. Considering that cellular carriers are already capping things, unlike BP, I’m guessing that your cellular bill will start rivaling your mortgage. So, if you use this method, keep this information handy. If you get the right camera and router, they should all fit inside some type of standard outdoor video camera case.

So let’s start putting in video systems that when the bad guy is shown on the 6 O’clock news, his momma can recognize him and apply proper justice. With inexpensive cameras and software, anybody can do the EarthCam thing and make sure we catch em’ all. And tip your contractor, or at least pay him for an honest day’s work, especially if he is doing your company a favor at great inconvenience to himself after your company dropped the ball. You really don’t want him complaining that your company (Did I mention the company is Earthcam?) takes advantage of contractors and then nickels and dimes them for their efforts. Who knows, they might even write a column telling people that yours is not the company they should be doing business with.

Chapter 3 - Share and Share Alike

Although this was the week that we were going to discuss putting APs on the poles, an incident occurred last week that I think is worth discussing before we go any farther. Always keep in mind that unlicensed bandwidth is a shared commodity. It’s also a good idea to have a relationship with your competitors or anyone in the area that is using unlicensed bandwidth. Sometimes that’s a little hard to do, but it’s definitely in your best interest to try.

Traffic lights and cameras are a perfect application for 4.9GHz or 5.8GHz radios where fiber isn’t installed. For example, we connected several traffic lights in Glendale, AZ a month before the city hosted the 2007 SuperBowl. The entire network consisted of 5.8GHz SkyPilot equipment with Pelco cameras. The system is used to monitor the intersections and also as the SCADA system for the traffic lights. This allows the city to manually adjust the traffic lights before and after events to optimize traffic flow. The city just recently closed on a bid that ADOT managed for the next phase. For some reason that I still can’t explain, they removed the compatibility clause with the existing system and simply went for low bid. Due to the cost of the SkyPilot equipment, it was obvious that low bid would not be SkyPilot or a compatible system but something else. This means the city will now be supporting multiple mesh networks along with the extra long term costs. He who controls the funding; writes the rules but apparently doesn’t have to support it.

That led to another project to connect 13 traffic lights with 50 High-Definition cameras at resolutions up to 1600x1200 pixels. We settled on Axis cameras for this project. The ideal cameras, fixed and PTZ, would be 1080p, 24 frames per second, and H.264. The project came close with most of these specs, but there were some compromises. We either got PTZ and H.264 at 720p, or we got 1600x1200 at reduced frame rates with MPEG-4. Future intersections may get 1080p cameras. This was sufficient to meeting the system requirements for the application which was traffic light management, license plate recognition, video analytics, and future facial recognition software.

Either way, we needed a lot of bandwidth at the City Hall to make this work. There were no other antennas on City Hall. A quick site survey showed additional equipment in the area with fairly low signal levels. The AP chosen was Ubiquiti 802.11N Rockets with 90 degree, 20dBi dual-polarity sector antennas. The system has 4 APs and sector antennas on City Hall for 360 degree coverage. Each light depending on distance to City Hall gets either a Nanostation 5M (integrated antenna) or Rocket 5M (external flat panel sector antenna). We are also adding 2.4GHz 802.11N to all the intersections for future use. It only adds $200 to the cost of each intersection and has an 800’ range to a laptop.

The first AP was turned on about 6 months ago and covered one 3-way intersection with 3 cameras and a second client station for data backhaul to one of the City Parks. The park had maintenance staff and a camera for the skate area. The second AP was scheduled to be turned on this week to cover the next 90 degrees off City Hall. That was when we discovered that between the time we turned on the first base AP and then went back to turn on the second base AP a couple of months later, the IT department had a local WISP put a Motorola Canopy radio on the roof without notifying the traffic department. Even more fun, it was in the 5.8GHz band.

Our traffic system was designed to use all 100MHz of the 5.8GHz band and was designed with signal levels for each link between -50 and -65dBm. Since the general rule is at least 10dB headroom on the link path and the minimum signal for APs with an MCS15 link (130-144Mbps modulation rate), 2x2 MIMO is -75dBm, you try to design for your worst signal at the receiver to be -65dBm. At those signal levels with directional antennas on the client side at 2 miles or less, it’s fairly hard to interfere with this system.

As a courtesy to the vendor, I thought that giving them a call and asking if they could change the Canopy to 5.1-5.4GHz before I turned on any more APs would be the professional thing to do. Unfortunately, when you run into inexperienced WISP technical support staff who think that whatever product they use is significantly better than whatever product you use, there is a problem. The conversation started with me asking if there was any chance that they had other frequency options. The technician asked me what equipment I was using. I replied. He then made the comment, “Our equipment will stomp all over your equipment.” I quickly established that this wasn’t the person to which I needed to speak to resolve this issue.

Every wireless product handles interference in different ways. Better filtering, multiple polarity, diversity, beam-forming, better firmware and many other techniques are utilized to improve the quality of a connection. Obviously some equipment works better than others in high-interference environments. However, 802.11n doesn’t always play very well with 802.11 b/g or 802.11a radios. In addition, physics are physics. Throw in a 2x2 MIMO signal with very a high gain antenna and a total EIRP of 42dBi minimum on both polarities, and there is going to be interference. This is especially true if the competitor’s base station is 1000’ away and their system is based on a 30dBm EIRP signal level from the APs. Since the traffic system was designed with a very high level signal path with multi-polarity 2x2 MIMO, there wasn’t much concern that the local WISP was going to interfere with the traffic system. On the other hand, it was clear that the traffic system was going to cause some interference to the WISP operation. I was wondering if that was already the case.

The last thing I wanted to do was start a war between a city government and a local WISP. Since the WISP was already serving some of the registered voters, this was probably not a good idea. The ego in me wanted to turn the remaining two sector antennas directly at the WISP base station across the street, program the two radios with a channel width of 40MHz to cover most of the 5.8GHz band, turn on 802.11N mode only, start multicasting “Transformers 2” in UDP mode to my other laptop and phone, and crank the power to 48dBm EIRP. I felt there might be a lesson to teach on RF, signal-to-noise ratio, and how to be polite when other companies are trying to do the right thing. Instead, I called back and talked to upper level management to arrange a meeting to see what can be done. That’s when I found out as I had suspected, that the traffic system’s first AP possibly might already been causing some interference issues. Unfortunately that traffic system AP also serves the City Park and has been running for several months. As a courtesy, the best I could do was to turn the power down until we plan out a more cooperative strategy with the WISP.

Now we can come back to our design. The AP combo we are considering will have an EIRP of up to 36dBm and will be 802.11 b/g/n compatible in 2.4GHz. That is far more powerful than most indoor systems. There are ways to limit the interference and everyone sharing those bands should consider this. It’s good policy to always engineer for the least amount of power that you need. Also, you should consider using the least amount of bandwidth. If you don’t need a 40MHz channel, don’t use it. You gain 3dB on the receiver side with no more power if you use a 20MHz channel. Drop to 10MHz and you pick up 3dB more, etc… The tighter the channel, the better chance you have of getting a signal through.

There are many of you that may not even want b/g compatibility. If the system is only used where you control both the APs and the client radios, then you have some options. In that case, using 802.11N only is the best way to go. This results in fewer APs and better performance. It’s also possible in this same scenario to get as many as 100 users on an AP or as many as 300-400 on a single pole. If all the equipment is the same manufacturer, consider a 5-10MHz channel which will still deliver 12-25Mbps per AP. Running 2x2 MIMO doubles the throughput to 25-50Mbps in a 5MHz channel. This is probably more than most of us need for a link. With a 5MHz channel, there are 6 channels to work with in 2.4GHz instead of just 3.

Whatever gets installed will probably interfere with other existing systems. We have to be cognizant of the fact that nearby systems might be critical for hospitals, voice systems, video surveillance, office networks, etc. For example, a few years ago, we saw an outdoor AP in Scottsdale stadium take down a wireless voice system at a nearby hospital. Since we put our phone number in the SSID, they could contact us quickly. We adjusted our antennas and resolved the issue. Site surveys have to be done everywhere to understand the ramifications of the install in addition to understanding your own coverage zones.

Based on all this, if I saw a little Motorola Canopy radio on the roof and thought it was a good idea to possibly contact that vendor, what was the installer thinking when he saw all the Ubiquiti equipment along with a Motorola PTP600 radio? He obviously didn’t do any kind of site survey. The reality is that this is standard procedure for most companies.

Many companies that rent roof space for APs do it from some type of property management company. There is usually no bandwidth management on the roof. When someone finally overlaps frequencies, an avalanche of problems will start. For example, administrators may turn up power to overcome the interference if they have no other options which causes even more problems (this gets into noise and channel filtering). The responsible thing to do would be to notify all parties involved and see if there is a way to work together. Obviously this sentiment is not shared by everyone, or some field installers aren’t trained well enough to watch for this situation. Then again, they just may not have cared.

Unless I run into another tech support person who hasn’t been taught the concept of cooperation and I need to vent, we will get back on our project next article. I’m still working on some FCC equipment certification issues with the design. Not that we can’t do it but I want maximum performance and the equipment I need isn’t quite yet available. I’m also waiting for new products to test that may enhance the system. Anybody can build a system with unlimited budget. The fun is creating a carrier class dependable system on a shoestring budget; since, that’s what most cities have in their coffers these days. Any new products that have the potential to help preserve that coffer is going to get their attention. Next article, we get back to work.