Crazy Consulting Work

[NETWizz]

Okay, so I put in a bid to troubleshoot an Active Directory problem for a small company (600 computers)... and got chosen.

Chief Issue: Can Join XP and Server 2003 Computers to Domain, but cannot join Vista, 7, Server 2008, Server 2008 R2 computers to domain.

Error: RPC Server unavailable.

Bid Amount: $8,000

******************************

Okay, so I get there and first look at an XP machine and a Windows 7 machine to try to find the differences.


1. I run IPCONFIG /ALL on both systems... Everything is configured the same from the same DHCP server (obviously the IP addresses are different)... No problems here.

2. Next, I take a look at the Operation's Masters (FSMO roles) with "netdom query fsmo" All the FSMO roles are running, and I can ping the servers.

3. Next, I took a look in Sites and Services and compared the AD site info with the sub-nets and Domain Controllers... no problem...

4. Took a quick peak at replication with REPLMON to ensure all the Domain Controllers are properly syncing their Global Catalogs. Overall, I confirmed Active Directory is not broken.


5. Next I thought the XP and Vista+ machines might be talking to a different Domain Controller or getting a different answer from DNS or that something third-party is installed... something strange..., so I decided to poke into DNS.

6. I pinged things like servers, and domain controllers from both XP and 7... Got the same responses... Great.

7. To find an RPC server & join a domain, it is going to need to query the SRV records from DNS, so I did an nslookup on the SRV records for things like LDAP and kerberos.

... Basically queries like this nslookup -type=SRV _ldap._tcp.dc._msdcs.addomain.com


Found the problem is DNS:

Windows XP would display the SRV records for the Domain Controllers of the Active Directory Domain.

Windows 7 would NOT display any SRV records..

i.e. I got something like:
*** dnsserver.addomain.com can't find _ldap._tcp.dc._msdcs.addomain.com: Non-existent do
main



I told them to take me to the DNS servers

8. First thing I noticed was a failed hard drive in the Array that they didn't notice. Okay, that is NOT the cause of this problem, so I let them know.

9. I took a look at the DNS console on that Domain Controller & DNS server... Everything was fine!

10. I said, "You have two DNS servers, let's just try to reboot this one."

11. I reboot it and it must have taken 20+ minutes to boot. I looked at their IT guy in disgust and said, "Does it always hang taking forever to start networking and apply computer policies?" He said, "Yeah"

12. Before logon, I am greeted with "Windows has detected an IP Address Conflict." This is a DNS server and a Domain Controller with an IP Conflict.

13. Their tech said, "It always does that; we just click OK."

14. I asked, "What else has the same IP address?" He didn't know!

15. Shutdown the DNS server, so I can track it down...

16. Logon to their Cisco Catalyst switch and then do the following:

switch>en

There was no password!

switch# ping 10.x.x.x. (the IP of their DNS Server)
switch# show arp

It basically listed a long list that scrolled...

So, I did...

switch# show arp | include 10.x.x.x (the IP address of their DNS server that I just pinged)

I got a response like

10.x.x.x 1234.5678.9abc (A Cisco formatted Mac Address)

Okay, so I ran:

switch# show mac-address-table address 1234.5678.9abc

I can't remember the next command, but it was a show cdp neighbors (some argument for the interface & for detail)


It told me that the DNS server was on Interface Gigabit 0/1, which was fiber, lol... So, I followed that Fiber to an LIU... I asked their tech where the other end of it is.

17. He took me to another server-room... I logged onto the switch. ping, show arp, show mac-address-table... blah blah blah.. Interface FastEthernet 0/24

18. Followed that cable and it went to their Cisco firewall!!!

19. Logged onto the Cisco Firewall and did a show-run. That IP address was set as a management IP address.

20. The firewall was also running a caching DNS server!!! That was why those nslookups on Windows 7 would work for things like nslookup google.com.... nslookup server.addomain.com... but no SRV records (those aren't cached)...



21. I changed the IP address and removed the caching DNS server functionality from the Cisco Firewall... Then I booted the Domain Controller/DNS server, which booted in like 3 minutes!

Problem Fixed!



Q: So, why was XP getting the SRV records from the other DNS server and Vista/7 getting it from t he Firewall?

A: Windows XP seems to do a Round-Robin using any of its configured DNS servers though it WILL pick on on its local subnet before using a remote DNS server... This is the same behavior on Vista/7. I.e. If you have DNS 1 and DNS 2 setup on all your systems, you don't want them to ALL hit DNS 1 every time... Microsoft knows this! That said, you don't want to go to DNS 2 if it is across a slow WAN with high latency... Again, MS knows this.

So why the difference?

A: Vista/7 WILL query a caching DNS server, first. Hence, they were seeing records from the Firewall NOT another, properly configured Domain Controller... Hence, there were no SRV records & Vista/7 couldn't find the RPC server.


Crazy Hugh?


This took about 4 hours and I got paid $8,000 for which I need to file a 1099-Misc with the IRS.

 

[seedubya]

You deserve it. That was brilliant troubleshooting. I would have been struggling after step 6 or so.

Who's Hugh, btw?

 

[controlfreak]

Congrats good troubleshooting! Sounded like questions from a MCTIP exam. You were obviously worth the money as there techs clearly didn't seem qualified to run that network.

 

[NETWizz]

I have done the MCSE 2003 and recently the MCITP as well as become a Microsoft Certified Trainer.

My favorite thing still is teaching on the side at my local Community College.

I am A+ & Net+, but I don't teach any of the CompTIA stuff. That would be fun to get into.

 

[dbdawn]

NO wonder your nick is NETWizz

But ya, you would think the in-house tech wouldn't be ignoring the errors on that DNS server on first boot.

 

[Elemdee]

13. Their tech said, "It always does that; we just click OK."

 

[wimwauters]

This just beggars belief!

600 computers and they don't haven't got a single IT professional employed!?

I bet their backup doesn't work either... have they ever even heard the concept of an IT sysadmin?

 

[NETWizz]

This just beggars belief!

600 computers and they don't haven't got a single IT professional employed!?

I bet their backup doesn't work either... have they ever even heard the concept of an IT sysadmin?

I am not sure I know what you mean the concept of an IT sysadmin?

 

[codegreen]

Are you saying they have a full-time IT person, but they had to call in a contractor to troubleshoot their network? If so, why does their current admin still have a job?

In any case, nice work! You were certainly worth the money.

 

[NETWizz]

Are you saying they have a full-time IT person, but they had to call in a contractor to troubleshoot their network? If so, why does their current admin still have a job?

In any case, nice work! You were certainly worth the money.

Yes, I am saying they have two (2) IT people though I think one just answers the phone and works helpdesk I don't know if they are full-time or not though. Yes, they had to call in a contractor to troubleshoot their Active Directory Infrastructure. It is actually surprisingly common.

What tends to happen (not necessarily in this place) is the person who gets the job often has no experience and gets moved from another department OR they get the job because the know some manager who hires his friends. Maybe that person is good with computers and a manager thinks that is all Enterprise Admins do not understanding the scope of the job.

I do at least 6 to 8 jobs a year like this one only most are for deploying things like Lynch Server or Migrating Exchange... this is in addition to my full-time job and adjuncting as an associate professor (part-time pee-on no benefits) i.e. teaching one (1) course via distance education each month.


I can't say authoritatively as I don't understand this company's IT culture, I don't know this IT guy's job title, I don't know his specific job requirements etc. Maybe he is hired for Helpdesk & paid $30k in which case he is in way over his head though not his fault.

Yes, he is still employed.

He seems to keep things running day-to-day in a re-active manner. I.e. If something says there is a problem he clicks "OK," but doesn't troubleshoot it. If someone needs a password reset, I am sure he can do that. If something breaks, they repair it. What they don't do is improve their infrastructure, upgrade their infrastructure, migrate to newer software... anything pro-active. If it is working, they don't touch it probably because they are scared/afraid of breaking something because they don't know they can fix it being they don't even understand how it works.

 

[Martyn]

Are you saying they have a full-time IT person, but they had to call in a contractor to troubleshoot their network? If so, why does their current admin still have a job?

In any case, nice work! You were certainly worth the money.

This quite common to have full time lesser experienced people handling the day to day stuff(resetting print queues or passwords) and have someone overseeing to whole network and it's problem either adhoc or more usually on a part time one or two days a month. When you look at the math Netwiz earned good money for those 4 hours but from the companies point they haven't had to pay a full salary for a higher level tech so $8000 to them is good business. Everybody is happy

 

[NETWizz]

This quite common to have full time lesser experienced people handling the day to day stuff(resetting print queues or passwords) and have someone overseeing to whole network and it's problem either adhoc or more usually on a part time one or two days a month. When you look at the math Netwiz earned good money for those 4 hours but from the companies point they haven't had to pay a full salary for a higher level tech so $8000 to them is good business. Everybody is happy

Exactly... The IRS is happy too. Not being one of their employees, they don't match my Social Security etc. Hence, I pay more taxes out-of-pocket. When all is said and done, I will owe an extra $2500 - $3000 in taxes.

*********************

That's okay though because my main job pays me $2590 (after taxes) every two weeks (actually twice a month 24 pay periods) + benefits: (health, dental, vision, life, & retirement). Then I work as a professor teaching either Intro to BIS, Network Operating Systems, Overview of Programming (really Visual Basic .NET), or Web & e-Business Design (Really HTML & CSS). After taxes this gets me about $3,100 per month (NO BENEFITS)! Oh, and I work that job only about 8 months a year.

That is $86,000 (but, neither job rates my taxes based on all the other work I do). They all presume I am in a lower tax bracket not aware of my other income. When tax time rolls around, I generally owe $11,000 by April 15th.


That is where those 1099-MISC forms really help out. My average consulting job is $10k (some are $8k for a day. If it is more than a day $12k). If it is a week, I quote about $16k. ANY longer than that and it is a huge project and I hire some professionals for about $400/day, and we work as a team. The longest project I ever did was 1 month and the bid was $48,000. That is like $34,000 after taxes! The taxes were like 2x higher on my 1099 money since I was 100% responsible to pay them... That said, I had to take time-off work (beyond my vacation), so I took unpaid leave... I lost about 75% of my pay for work that month i.e. $4,500 or so I never made after tax. I also had to pay my full rate for health insurance and benefits completely out-of-pocket. My employer only owed me about $1,500 for the whole month of which I had to pay $1,200 in benefits more than usual... Then I had to pay out about $16,000 to the two (2) people I hired for the project. I had to pay a tax preparer to properly complete two (2) 1099-MISC statements to the IRS for the employees (actually sub-contractors) I hired and to document how much they were paid for tax purposes . This was about $1,000. Then I had to pay $5,200 in Lodging at a hotel for these people... I had to pay $1,800 for rental cars for these people. I had to repay them for fuel expenses at 31 cents a mile (I provide the car)... if it was their car it would be 51 cents a mile... Then I had to provide subsistence: I paid $8 for breakfast + $12 for lunch + $18 for dinner (for 2 employees... for 30 days including weekends)... This was about $2300.

I literally actually made $7,000 for 30 days of hell! + $300 from my job for taking a week of vacation pay and paying my own dental, vision, health, life, & retirement out of it...

I literally got $7300 ish vs. $5,200 (I was getting at the time for just showing up to work)... Really, I did ALL that for $2,100 more that month! I thought I was going to rake in the cash on that job too, lol

 

[ljtechservices]

But ya, you would think the in-house tech wouldn't be ignoring the errors on that DNS server on first boot.

the in-house techs probably inherited the problem. It's probably always done that so they let it as long as it didn't cause any other problems.
It probably wasn't until they tried adding win7 pc's that it got worse.

I find this problem very common.
Last year I took over IT support for an accounting firm. After about a month the new owner called me to see why one of the monitors was fuzzy.
we tried TS over the phone but I finally did a site visit. they had VGA extenders on the monitors that were out of sight. removed them and the blurryness went away.
I asked the user how long it had been that way. 3 years, ever since they bought new monitors. She just thought that was how it was supposed to be.
The extenders worked fine for CRT's but not for LCD monitors.

/sorry for the hijack, nice job Netwizz

 

[anonymous Mac Tech]

Then I work as a professor teaching either Intro to BIS, Network Operating Systems, Overview of Programming (really Visual Basic .NET), or Web & e-Business Design (Really HTML & CSS). After taxes this gets me about $3,100 per month (NO BENEFITS)! Oh, and I work that job only about 8 months a year.

Not trying to split hairs, but either you really are a Phd. and can justifiably call yourself a professor or you are not. I just spent many years in academia and any instructors I had that were addressed as professor mistakenly by students were quickly corrected by the instructors themselves when they hadn't earned their doctorate.

But aside from the above, I had been strongly considering going back and teaching part time at the Community College I had graduated from teaching either CompTIA courses, networking, or CompSci. Just need a 4 year degree and knowledge of the subject matter which I have both. There actually was an opening for a part-time CompTIA instructor position for which I could have applied, but I got caught up in other work and didn't apply in time. I'd actually love to train some Apple repair or OSX courses if they offered them, but between Apple policies and school politics they probably won't offer anything like that any time soon.

 

[NETWizz]

Not trying to split hairs, but either you really are a Phd. and can justifiably call yourself a professor or you are not. I just spent many years in academia and any instructors I had that were addressed as professor mistakenly by students were quickly corrected by the instructors themselves when they hadn't earned their doctorate.

But aside from the above, I had been strongly considering going back and teaching part time at the Community College I had graduated from teaching either CompTIA courses, networking, or CompSci. Just need a 4 year degree and knowledge of the subject matter which I have both. There actually was an opening for a part-time CompTIA instructor position for which I could have applied, but I got caught up in other work and didn't apply in time. I'd actually love to train some Apple repair or OSX courses if they offered them, but between Apple policies and school politics they probably won't offer anything like that any time soon.

Point taken. I do NOT have my Doctorate. That said, I do have my Master's and I am an Instructor in that case... That said, everybody has been using the title Professor, which I have always assumed meant "College Teacher."

I am NOT referred to as Doctor. I generally try to stay informal and ask my students to simply call me by my first name; we are (after-all) peers. I respect many of my students very much; in fact, some have some really awesome life experience working high-level positions during the day.