OnQ

On Certainty

When you need to be 100% certain on an issue, it's best to be 110% certain instead. Yesterday I spent several hours troubleshooting an issue with MS CSS, because I was 100% certain that the issue we all thought it could have been up front was not the case. Only I needed to check one more time, because it actually was.

Yes, I got bit by the dreaded APC bug. A client was reporting all the classic signs of the problem, so when I was on site Tuesday, that was the first thing I looked at. I found version 5 running on the server, so I crossed that one off the list. Sure, I could have (and in retrospect should have) updated it at the time, but their backup issues were more pressing.

I found what I thought was the backup problem (and may still have been) but then I ran into a problem with IE (couldn't load to find updates fo thebackup software, etc) and other problems, so I started workin those. Eventually, because I was stumped, I decided to call in MS CSS on the case. Specifically the security team, because I knew the tools they have to help find bad stuff on the box are really good.

So I went back to the client site Wednesday night (had to do this after hours) and started work again. I did several virus scans while booted off a clean disk, found no problems. But when I booted normally, it took 40 minutes to boot and log in,Spybot, RootKitReveale and BlackLight would no run, etc. I got security on the line, and had them send me the scanning package. It wouldn't run correctly. Finally, after three+ hours of beating our heads on this, he asked me to look inthe Services panel to see what might not be loadin correctly.

That's when I saw it. The two APC services were hung in a STARTING state. I rebooted the box into Safe Mode, looked again at the APC services, and sure enough, versino 6, the bad seed. 10 minutes later, the services had been removed, the system rebooted, and Microsoft Update was being run to catch all the patches that hadn't been installed in a year. The backup problem was fixed, and the delay in logon times were gone.

The moral of the story? Even if you're 100% certain that you don't have the APC problem, when MS CSS asks you if you might have the APC problem, check it anyway. It's not going to hurt or take a long time to take a second look, and it could save you troubleshooting pain in the long run. [Obviously, the previous statements do not apply solely to APC software but anything that you know you know for sure.]

On Virtualization

I spend a great deal of time working in a test environment, both for my customers and in developing content for the book. Many small business IT specialists do. We also recognize that it's diffcult at best to have one physical machine for every combination of settings needed for testing. Hence, we use some form of virtualization technology so we can run multiple virtual machines on a single physical box.

This past weekend, I discovered a new (to me) tool that has already helped me become much more productive with my virtual setup.

I'm no stranger to virtual technology. I've been running Vrtual PC for the Mac since its second version, back when it was a Connectix product. The ability for a piece of software to completely emulate a different hardware environment and have it be usable was amazing back in the day. I looked at several similar products over the years, including SoftPC for the Mac, several PC and Mac emulators for Sun and Solaris UNIX systems, but I always kept up with Virtual PC for the Mac, becuase it just worked.

Then Microsoft jumped into the foray and bought out the Connectix products. Not too far after that came Virtual PC for Windows, which allowed you to run one or more virtual terminals from your desktop PC. This was a great help for testers, as we could now simulate multiple machine environments with significantly less hardware costs. Virtual PC for Windows did have its limitations, however, as even with a beefy PC, you could still only have a few active virtual terminals before all of them, including the host station, started bogging down.

Introduce Virtual Server. I learned about this product while working at Microsoft. In fact, everyone on my team had one workstation dedicated to running several server environments for comparison and testing. Virtual Server has a nice web interface and remote connection tools so you can view all of the terminals running under VS without working directly on the console.

Virtual Server still has its practical limitations, but the prodict is designed to be able to run multiple server sessios on a single, high-end server box. And it's not too bad about it, either.

Then came the book. To build a test environment for screen captures and process verification, I got a higher-end box (given my budget) with multiple CPUs and loads of RAM so I could set up Small Business Server and three or four workstations in a virtual network all on one piece of hardware. I got the 180-day trial version of Virtual Server (figuring it wouldn't take more than 6 months to finish my portion of the book) and built the environment. In the initial loads, it worked well, although I could tell I was taxing the box when I had the server and four workstations running simultaneously. But it worked well, until I applied SP1 to the SBS server build. Suddenly, I couldn't get things to work right on the SBS node, even with all of the other virtual machines turned off. I eventually broke down and got a separate PC to run a dedicated SBS installation, as the virtual workstations continued to run just fine.

Lo and behold, six months came and went, and I still was not quite finished with the book. The trial of Virtual Server expired, and I was looking at having to completely rebuild the environment. Around the same time, I was having an IM conversation with Susan Bradley, who is working almost as hard as I am on the book project, and she mentions another virtualization product that had been pointed out to her (and she posted about it on her blog as well). So this weekend, when I'm badly neededing to fire up a couple of virtual workstations to capture some data for the book, I checked out this alternate solution. I had actualy heard of VMWare before, back when it was first released as a Linux product and we were giving it a go at Nortel Networks. I'd not used it or even kept up with the product line since I last built a box in 1999. But after reading about the VMWare Subscription, I decided to take the plunge and give VMWare a shot.

Within two hours, I went from knowing nothing about using VMWare to having created a new Windows XP test box connected to my SBS test network and fully patched, all from scratch. And this is without using the really high-end sever product from the VMWare suite.

The real value I see in continuing to work with VMWare is their P2V technology, which captures an image of a physical machine and loads it as a virtual machine. With that, I can load a "copy" of a client's server in my virtual network and perform tests on it to see how it will hold up. This is going to be critical for testing security updates and service packs in a "real world" situation without risking clients' production environments. I already have plans to test Exchange 2003 SP2 on an SBS server using this method once it becomes available.

So for the time being, Hello VMWare, Goodbye Microsoft. At least for virtual technologies on the PC platform. I'm still dependent on MS for Virtual PC for Mac. How ironic is that?

On Networking

And no, I don't mean the ethernet kind. I mean the type of networking where you're going out and pressing flesh for your business or other organization that you feel strongly about. It is definitely a skill that most of us have to develop, although it is amazing to watch those who are naturals at work. Sometimes successful networking is making the effort to get your name recognized by key contacts who may be able to help you in your endeavors. Sometimes, success comes from being in the right place at the right time.

This past week, I happened upon two of these situations, and I decided to share them. Monday morning I got a call from a contact I had made at a Chamber of Commerce lunch function several months before. I arranged to meet this potential new client on Tuesday, right after my BNI meeting. They tell me that they are moving offices and want to set up a new server in the office and get all the computers networked correctly. After we discuss the specifics of the services I can provide fo them, I ask about a few items related to their move, including security systems. After I let them know that I can refer them to a reliable company to install a new security system for them, they ask me if I know anyone who does phone systems and could help them paint the offices before they move in. I told them yes, and I would arrange for all three to contact them later in the day.

After leaving their office, I immediately got on my cell phone and made three calls to arrange for these other vendors to contact my potential new clients. When I checked back later in the day, all three had made contact, and two had made arrangements to start work in the next two days.

Even if I hadn't had my proposal accepted, I provided three solid referrals to businesses I network with. Those businesses will remember those referrals.

Friday morning I was with a different client. While doing several small jobs for them, one of the employees asks me about copiers, as the one they have is a loaner, and they really need to get a new unit in quickly. I had been involved in a massive copier replacement project several years ago, so I was able to provide some insight about that project to them. Then, during my last appointment for the afternoon, a differnet client asks if I know anyone who is needing a new copier. As it turns out, if they can refer someone who purchases a new copier to their copier contact, the contact will donate a copier to my client's organization. Before I got back to the office, I had amde contact with my client and the copier rep and they had a phone call scheduled for Monday to discuss their needs with him. Since they're wanting to get a new unit ASAP, my other client may well get the copier donation they need so desperatly.

What networking really boils down to is two things: listening to all your contacts at all times, and thinking outside the box. I know a number of computer consultants who would have discussed a proposal with my potential client Tuesday morning and never brought up solutions to their other, unspoken needs. But becaus I was willing to think outside the box, I saved my newest client several hours of time he would have spent trying the old fashioned way to find people to help with his needs. I may never see any acknowledgement from them about that, but I have a good feeling inside knowing that I helped them save time and money. That's what customer service is really all about.

On Publishing

I've been working on a book for Sams Publishing since the beginning of the year. It's a book on SBS 2003, Microsoft Small Business Server 2003 Unleashed, and I've been collaborating with several other authors on the title. Since we're getting close to wrapping up the submission of all original content, I decided to peek at Amazon and see if they had the ISBN for the book registered yet. Needless to say, I was surprised to see what came up.

Not only did Amazon have a listing for Microsoft Small Business Server 2003 Unleashed, but they also had the front cover image, the expected publication date, and an author bio. I had no idea that they were this far along in marketing the book.

This morning, I decided to see where else there were mentions of the book on the web. Here's what I came up with:

• Wal-Mart
  
• Overstock.com
  
• BestPrices.com
  
• Bookpool
  
• ACM Portal
  
• elx.com.au
  

Interestingly, it's not listed on the Sams Publishing site just yet.

Even though it's going to appear that way on the book cover, I can't take all the credit for the book. A number of SBS community members stepped up to the plate to contribute to the title, including a couple of MVPs. They are, in no particular order: Amy Babinchak, Anne Stanton, Chad Gross, Ed Walters, Henry Craven, Javier Gomez, Susan Bradley, and Tim Barrett.

More news about the book as it's fit to print, so to speak...

On Verizon

A few months ago, Verizon enabled some server settings changes on their mail servers which effectively broke the ability to send outbound e-mail through their mail servers (see the discussions on Susan Bradley's blog postings We Do Stop Working For You and Hey Verizon) Several of us had clients who were significantly impacted by this change, and many of us had our own servers impacted. We came up with several solutions, and for the most part, Verizon's stupidity didn't cause much more pain.

For about a month. Then the slow painful downhill slide into abject horror began.

Verizon clients who have static IP addresses can generally avoid having to use Verizon's SMTP servers for mail delivery from Exchange, as you can set up reverse DNS entries and what-not with your static IP that will get your mail past most of the spam cops. However, Verizon does not offer static IP addresses in all of their service areas, even with Business-class accounts. Yes, that includes me.

My solution was pretty simple. Use DNS for mail delivery where I can, and route outbound e-mail through the MailHop SMTP service provided by DynDNS. Yes, there were some mail domains I knew I'd have to route through MailHop from the get-go (AOL, Yahoo, etc.) but most mail I was sending through DNS was still getting through, so no big deal.

It started slowly, but over the past month, it's been getting worse by the day - more and more blacklists and other anti-SPAM tools ahve started blocking messages from my Exchange server. Somtimes because I'm in a dynamic IP block, but the last 12 mail delivery failures I've received have all noted that my IP address has been blocked becuase of excessive SPAM emanating from my address. The real kicker? Each one of those delivery failure notices has identified a different IP address!

I went back and looked at my DynDNS logs, and sure enough, Verizon has ben forcing an IP change on me every day. And on one of the days that some spammer got one of the IPs in the dynamic pool, s/he sent out enough spam to get blacklisted, so now I can't send mail via DNS for reasons that are completely out of my control.

At least Verizon doesn't block me from sending mail on port 25 to other severs, like some ISPs have started enforcing recently. At that point I'll have no recourse but to switch ISPs from Verizon, which is not a bad idea anyway.

If I could see that Verizon's decision to change their SMTP server configuration to refuse connections from Exchange servers was actually a move to help prevent the spread of SPAM from their network, I'd have a little understanding and tolerance for this idiocy. But it wasn't. In fact, it almost looks like they made the change to the SMTP authenticaion process specifically to prevent Exchange servers from being able to use their SMTP servers as smarthosts. And they've done nothing else to prevent the flow of SPAM through other mechanisms, so the blacklists have had to take that on instead.

So, in summary, Verizon charges me monthly access for ISP services, but will not give me a static IP. Then they take away my mechanism for sending e-mail reliably since I don't have a static IP. If I treated my clients this way, I would expect to get fired. I guess that's my last couse of action with Verizon - fire them and go with a different ISP.

On Power

As I sit here and listen to the delay between the flash of lightning and the sound of thunder shrink rapidly, I'm reminded that I don't have UPSes on some of my critical pieces. So, before I sign off into that good night, I must shut down a few systems to protect the investment. Perhaps the new toy fairy will bring me a few UPS boxes in the near future...

On Blogging

Here I am with YADBlog and I wonder to myself, "How many blogs is too many blogs?" The simple answer is, "I don't know, but if I'm not confused yet, it must not be too many."

Wait, do I mean blogs I'm reading, or blogs I publish? It must mean blogs I publish, because I'm using bloglines to keep up with the blogs I read on a (somewhat) daily basis.

So if I'm wondering if I have too may blogs that I'm publishing, how many blogs is that? Well, I've got my personal personal blog, my Lessons Learned blog, and now this one. not to mention a couple of other blogs that started, ran their course, and now live in blog infamy, having been superceeded by other blogs, or just no longer applicable.

So why start this one? Because I needed another place to ramble aimlessly about technology, people, projects, and other stuff related to the work side of my life. Yeah, yeah, yeah, I've got the Lesson Learned pages, but that's established itself as more of a "how to" of the Macintosh and SBS side of my brain. It's not a good ramble point for business contacts, technology missives, and general mayhem. And I really don't want to muck up my personal blog with this stuff, either.

So what's a technophile to do? Start YADBlog, of course. After all, what else is the internet good for?