Archive for the 'Computers' Category

Reversing airflow on Cisco 3945 routers

The Cisco 3945 router ships with the default airflow “backwards” (back-to-front) compared with all other standard Cisco routers and switches. While back-to-front is available on a number of Cisco producta, as either a factory option (for example, the Catalyst 4948E-F) or as a field conversion (for example, the Catalyst 4500-X-16SFP), the 3945 is the only Cisco device I’ve encountered which defaults to a back-to-front airflow. There is an optional fan assembly (3900-FANASSY-NEBS) which has front-to-back airflow, but it is hard to find and represents a large additonal expense, since your 3945 presumably has a fan tray already that is working perfectly (but backwards).

This article shows how to convert a fan tray from the standard 3900-FANASSY to 3900-FANASSY-NEBS with the only new part required being an inexpensive (pennies) jumper and your time to do the conversion. NOTE: While it is possible to do this (with practice) by removing the fan assembly from a running 3945 and converting / reinstalling it before the 3945 shuts down from overheating, I suggest that you power down the 3945 first to avoid the problem. If you are converting multiple units, you can shuffle converted and un-converted fan assemblies with no downtime.

You will need the following tools and supplies:

  • Small needlenose pliers
  • Small diagonal cutters
  • Phillips screwdriver
  • Several small cable ties
  • Fan mode jumper (more on this below)

The “fan mode jumper” is just a short (1″ or so) length of wire with the correct pins on each end. The pins are Molex part number 39-00-0039, 18 cents each. The tool to crimp them is Molex part number 64016-0200, which is quite expensive at $137.21. However, you can get creative and use the small needlenose pliers to manually crimp the pin onto the cable. If you do this, I suggest also soldering the pin to the cable (after crimping) to ensure a good mechanical and electrical connection.

This is the jumper wire and the small cable ties (each picture is clickable to show a higher-resolution version):

Following the Cisco instructions, remove the bezel and the fan assembly from the router. Place the fan assembly face-down on your work table and remove the 8 Phillips screws holding the two halves of the fan tray together, as indicated by the circled red areas in this picture. Your fan tray may or may not have screws in the circled yellow areas; you do not need to remove those. This is a new spare fan tray – your fan tray will likely be a lot dustier:

Carefully separate the two halves of the fan tray. One piece is only sheet metal – set that one aside for later. The other piece has the fans, wiring, and connector. That is the piece we will be working with. There is also a small clear plastic light pipe, as shown in this picture. Carefully remove it and set it aside for later:

Each fan is held in place with 4 blue silicone rubber stakes. Here is a view of one of the fans:

Starting with the rightmost fan (numbered 4 in the stampings on the tray), using the needlenose pliers, gently squeeze the expanding “V” part of the stake and carefully lift that corner of the fan up. Repeat with the 3 remaining corners of the fan and lift the fan out:

You may need to use the small diagonal cutters to cut a cable tie holding the fan wires in place if you don’t have enough slack to perform the next step. Flip the fan over so what was the top of the fan is now on the bottom (one side will have a part number sticker while the other side will be blank). Re-install the fan on the 4 blue silicone rubber stakes, and while gently pulling on the free “tail” of the stake, push the fan down against the metal of the fan tray until the expanding “V” part pulls through the fan. Repeat for each of the 4 remaining fans.

When you have the next-to-rightmost fan out, you will see the back of the connector that connects the fan tray to the router:

The red arrows in the previous picture show where the jumper needs to be installed. Bend the jumper into a “U” shape and carefully insert it into the connector until it clicks. You should end up with it looking like this:

Once you have all 5 fans flipped over, examine the underside of the fan tray to make sure there are no wires sticking out and that the fans are all fully seated on the blue silicone rubber stakes. An un-seated stake will generally appear slightly “popped out” when you look at the underside of the fan tray.

Use the small cables ties to replace any ties you had to cut to get enough slack to flip the fans over. Next, re-install the light pipe. The small U-shaped bend lines up with a matching protrusion on the sheet metal, shown with an arrow in this picture:

Carefully re-install the other sheet metal half of the fan tray that you set aside at the beginning. You may have to wiggle the light pipe a bit to get it to line up with the holes in the fan tray. Make sure that the two halves of the fan tray are fully seated on each other with no protruding pieces (there are interlocking metal tabs on the two halves). Also make sure that no wires are sticking out or being pinched. If all looks good, re-install the 8 Phillips screws. Give the fan tray another look-over to make sure everything is in place, then re-install it in the router and power up the router. Once the router has booted, use the “show environment all” command to verify that all 5 fans are operating correctly and that the router has detected the new jumper and is operating in front-to-back mode:

SYSTEM FAN STATUS
=================
 Fan Tray: Installed with Reverse Air Flow. Air Filter Supported.
 Fan 1 OK, Low speed setting
 Fan 2 OK, Low speed setting
 Fan 3 OK, Low speed setting
 Fan 4 OK, Low speed setting
 Fan 5 OK, Low speed setting

Technical minutiae: The only thing the jumper does is tell the router that a front-to-back airflow fan tray is installed. If you don’t install that jumper, the router will still operate with front-to-back airflow but the environmental readings will indicate that the unit is an “air conditioner” (exhaust air is cooler than intake air):

SYSTEM FAN STATUS
=================
 Fan 1 OK, Low speed setting
 Fan 2 OK, Low speed setting
 Fan 3 OK, Low speed setting
 Fan 4 OK, Low speed setting
 Fan 5 OK, Low speed setting

SYSTEM TEMPERATURE STATUS
=========================
 Intake Left temperature: 18 Celsius, Normal
 Intake Right temperature: 17 Celsius, Normal
 Exhaust Right temperature: 16 Celsius, Normal
 Exhaust Left temperature: 17 Celsius, Normal
 CPU temperature: 49 Celsius, Normal
 Power Supply Unit 1 temperature: 21 Celsius, Normal
 Power Supply Unit 2 temperature: 22 Celsius, Normal

As you can see, exhaust air is being reported as 1 degree cooler than entering air. This is because the router doesn’t know the airflow is reversed, so the sensors behind the fan tray are being treated as exhaust and the sensors by the rear I/O panel are being treated as intake. Installing the jumper lets the router know airflow is reversed and that it should report the rear I/O panel sensors as exhaust and the sensors behind the fan tray as intake:

SYSTEM FAN STATUS
=================
 Fan Tray: Installed with Reverse Air Flow. Air Filter Supported.
 Fan 1 OK, Low speed setting
 Fan 2 OK, Low speed setting
 Fan 3 OK, Low speed setting
 Fan 4 OK, Low speed setting
 Fan 5 OK, Low speed setting

SYSTEM TEMPERATURE STATUS
=========================
 Left Intake temperature: 20 Celsius,  Normal
 Right Intake temperature: 18 Celsius,  Normal
 Right Exhaust temperature: 20 Celsius,  Normal
 Left Exhaust temperature: 21 Celsius,  Normal
 CPU temperature: 61 Celsius, Normal
 Power Supply Unit 1 temperature: 23 Celsius, Normal
 Power Supply Unit 2 temperature: 25 Celsius, Normal

Note: the two “show environment all” reports above were performed at different times, thus the differing temperature readings. There is no difference in cooling efficiency when reversing the airflow direction.

FreeBSD – Restoring inappropriately removed lang/php56 port

FreeBSD removed the lang/php56 port from the ports repository (in commits r488887) through r488894, approximately). This is due to a misunderstood “2019-01-01 lang/php56: Security Support ends on 31 Dec 2018”.

However, php.net says (emphasis added by me):

PHP 5.6.40 Released – 10 Jan 2019

The PHP development team announces the immediate availability of PHP 5.6.40. This is a security release. Several security bugs have been fixed in this release. All PHP 5.6 users are encouraged to upgrade to this version.

For source downloads of PHP 5.6.40 please visit our downloads page, Windows source and binaries can be found on windows.php.net/download/. The list of changes is recorded in the ChangeLog.

Please note that according to the PHP version support timelines, PHP 5.6.40 is the last scheduled release of PHP 5.6 branch. There may be additional release if we discover important security issues that warrant it, otherwise this release will be the final one in the PHP 5.6 branch. If your PHP installation is based on PHP 5.6, it may be a good time to start making the plans for the upgrade to PHP 7.1, PHP 7.2 or PHP 7.3.

FreeBSD removed the php56 port and dependencies as of 5.6.39. Users may be depending on either PHP 5.6 semantics, or on a port that was removed (such as devel/pecl-intl) as “collateral damage”.

I needed to restore these kits for those reasons. While I will be migrating to a PHP 7.x release with the next major rebuild of the systems in the coming month, I needed to deploy the 5.6 security fixes before then. I created a kit that restored many (but not all) of the removed ports, which you can download here. Security conscious users should NOT blindly install kits from untrusted sources like me, but instead create their own kits by looking at the official FreeBSD ports tree here. However, if you just need a quick fix and trust me, feel free to use my kit:

cd /usr/ports
tar -xpvf ~/php56-restore.tgz
mv MOVED /usr/ports/
mv php.mk /usr/ports/Mk/Uses/
portupgrade -an (to see what ports will be updated)
portupgrade -ai (interactively approve/deny updating individual ports)

Note that my kit does modify 2 “global” ports files – MOVED and php.mk. You may wish to restore the official versions after updating your php56 ports to avoid possible issues with other ports (though restoring MOVED will also restore the warnings about php56 ports being EoL and removed, and will try to update your php56-extensions if you let it).

Net Neutrality isn’t the only problem

Today (July 12th, 2017) a large number of sites have joined together to raise awareness of the threats to network neutrality. For example, reddit has a pop-over window that slowly types a message beginning with “The internet’s less fun when your favorite sites load slowly, isn’t it?” This is certainly a valid concern, and many people, including myself, have legitimate concerns about how the Internet is regulated. But there are enough sites raising that point, so I’d like to talk about something different – how sites are “shooting themselves in the foot” with slow-loading (and often buggy) page content.

It all starts when a web site decides they want to track visitors for demographics or other purposes. There are a large number of “free”* tools available that will collect the data and let you analyze it in any way you like. Sure, it comes with some hidden Javascript that does things you can’t see, but hey – it is only one thing on a page of otherwise-useful content, right?

Next, the site decides they’d like to help cover the cost of running the site by having a few advertisements. So they add code provided by the advertising platform(s) they’ve selected. So their page now loads a bit slower, and users see ads, but the users will still come for the content, right? And the occasional malware that slips through the advertising platform and gets shown on their site isn’t really their fault, right? They can always blame the advertising platform.

Somewhat later, the site gets an “offer they can’t refuse” to run some “sponsored content”. The page gets even slower and users are having a hard time distinguishing actual content from ads. Clicking on what looks like actual content causes an ad to start playing, or triggers a pop-under, or any one of a number of things that make for an unpleasant user experience.

Once everyone is used to this, things appear to settle down. Complaints from users are infrequent (probably because they can no longer figure out how to contact the site to report problems). Everyone has forgotten how fast the site used to load, except for the users running ad blockers, cookie blockers, script blockers, and so on.

But one day a SSL certificate becomes invalid for some reason (expired, a site was renamed, etc.) and the users are now getting a new annoyance like a pop-up saying that the certificate for btrll.com is invalid. Most users go “huh?” because they weren’t visiting (or at least they thought they weren’t visiting) btrll.com. Clicking the “close” button lasts for all of a second before the pop-up is back, because that ad site is determined to show you that ad. In frustration, the user closes their browser and goes out to buy a newspaper.

By this point, perhaps 5% of the actual page content is from the site the user was intending to visit. The rest is user tracking, advertising, and perhaps a bit of malware. There is a free tool run by WebPagetest.org which will let you analyze any web site to see what it is loading and why it is slow.

Here is the result for the CNN home page:

Now, that’s too small to be able to read, so this is the first part of it (click on this image for a larger view):

The blue line at 21 seconds shows when the page finished loading, although you can see that Javascript from a number of advertising providers continues to run indefinitely.

Now, let’s take a look at Weather Underground. Surely just serving weather information would have far less bloat than CNN, right? Not really:

Now, that’s too small to be able to read, so this is the first part of it (click on this image for a larger view):

It does manage to load in less time than CNN, but it is still pretty awful.

In the spirit of full disclosure, here is the result for this blog page:

Since the entire report fits, I didn’t need to add an unreadably-small overview image.

If you manage a web site, I encourage you to try WebPagetest.org yourself and see why your site is slow. If you’re just a user, you can also use WebPagetest.org to see why the sites you visit are slow. If you’re using add blocking or site blacklisting software while you browse, the list of hosts that are serving advertisements or other unwanted content will probably be useful to you when added to your block / blacklist.

* As they say, “If you aren’t paying for it, then you are the product being sold”.

Is no crypto always better than bad crypto?

SSL (Secure Sockets Layer, the code that forms the basis of the https:// in a URL) can use any number of different encryption methods (protocols) and key strengths. While all of the protocols / strengths were presumed to be secure at the time they were designed, faster computers have made “cracking” some of the older protocols practical, or at least potentially practical. Additionally, concerns have been raised that some of the underlying math may have been intentionally weakened by the proponents (for example, NIST and the NSA) of those protocols. Perhaps an underlying flaw in the protocol has been discovered. Due to this, web browsers have been removing support for these older, insecure protocols.

Additionally, even if a protocol is still considered secure, a browser may start enforcing additional requirements for the SSL certificate used with that protocol. “Under the covers” this is a rather different situation, but for the purpose of this discussion I will lump them together, since the average user doesn’t care about the technical differences, only that a service that they used to be able to access no longer works.

In theory, this is a good idea – nobody wants their financial details “sniffed” on the way between you and your bank. However, the browser authors have decided that all usage of those older protocols is bad and should be prohibited. They make no distinction between a conversation between you and your bank vs. a conversation between you and another site (which could be a web server, UPS – battery backup, a water heater, or even a light bulb!) in your house or company. Instead, they force you to disable all encryption and communicate “in the clear”.

To add to the complexity, each browser does things in a different way. And the way a given browser handles a particular situation can change depending on the version of the browser. That isn’t too bad for Internet Explorer, which doesn’t change that often. Two other browsers that I use (Mozilla Firefox and Google Chrome) seem to release new versions almost weekly. In addition, the behavior of a browser may change depending on what operating system it is running under. Browsers also behave differently depending on when the host at the other end of the connection obtained its security certificate. A certificate issued on December 31st, 2015 at 23:59:59 is treated differently than one issued one second later on January 1st, 2016 at 00:00:00.

In the following discussion, the terms “site” and “device” are generally interchangeable. I sometimes use the term “device” to refer to the system the browser is attempting to connect to. “Site” might be a more accurate term, but for many users a “site” implies a sophisticated system such as an online store, while an intelligent light bulb is more a “device” than a “site”.

In a perfect world, people could just deal with the browser blocking issue by installing new software and / or certificates on all of the devices they administer. Sure, that would be a lot of work (here at home, I have several dozen devices with SSL certificates and in my day job, I have many hundreds of devices) and possibly expense (the companies that sell the certificates don’t always allow users to request updated certificates for free, and updated software to handle the new protocol may not be free – for example, Cisco requires a paid support contract to download updated software). However, it is not that “easy” – any given device may not have new software available, or the new software still doesn’t handle some of the latest protocols.

This leads to an unfortunate game of “whack-a-mole”, where a browser will change its behavior, a company will implement new software to deal with that new behavior, but by the time the software has gone through testing and is released, the browser has changed its behavior again and the updated software is useless. A number of vendors have just given up supporting their older products because of this – they have finite resources and they choose to allocate them to new products.

The browser authors seem to feel that this is just fine and that users should either turn encryption off or throw away the device and buy a new one. Since the “device” is often a management function embedded in an expensive piece of hardware, that simply isn’t practical. A home user may not feel that replacing a working device is necessary and a business likely won’t replace a device until the end of its depreciation cycle (often 3 or 5 years).

This strikes me as a very poor way for browsers to deal with the situation. Instead of a binary good / bad decision which the user cannot override, it seems to me that a more nuanced approach would be beneficial. If browsers allowed continued usage of these “obsolete” protocols in certain limited cases, I think the situation would be better.

First, I agree with the current browser behavior when dealing with “Extended Validation” sites. These are sites that display a (usually) green indication with the verified company name in the browser’s address bar. In order to purchase an EV certificate, the site needs to prove that they are who they say they are. For example, your bank almost certainly uses an EV certificate. Users should expect that sites with EV certificates are using secure methods to protect connections. If a site with an EV certificate is using an obsolete protocol, something is definitely wrong at that site and the connection should not be allowed.

Second, the current behavior is OK when dealing with well-known sites (for example, amazon.com). This is a little more difficult for browsers to deal with, as they would need to keep a list of sites as well as deciding on criteria for including a site on that list. However, there already is a “master list” of sites which is shared between various browsers – it is called the HSTS Preload list. It could be used for this purpose.

Now we get to the heart of the matter – how to deal with non-EV, non-well-known sites. Instead of refusing to allow access to a site which uses an insecure protocol, a browser could:

  • Display a warning box the first time a site is accessed via an insecure protocol and let the user choose whether or not to proceed.
  • Re-display the warning after a reasonable period of time (perhaps 30 days) and ask the user to re-confirm that they want to use the insecure protocol to access the site.
  • On each page, indicate that the page is using an insecure protocol. This could be done by displaying the URL in the address bar on a red background or similar. Google Chrome does something similar with its red strikethrough on top of the https:// in the address bar. Unfortunately, in most cases Chrome will simply refuse to access a site it deems insecure.
  • NOT require dismissing a warning each time the user accesses the site.
  • NOT require a non-standard way of specifying the site URL in the address bar, bookmarks, etc.

Security experts will probably be thinking “But… That’s insecure!” It certainly is, but is it less secure than using no encryption at all (which is what the browsers are currently forcing users to do)? I don’t think so. In many cases, both the user and the site they are connecting to are on the same network, perhaps isolated from the larger Internet. For example, most devices here are only accessible from the local network – they are firewalled from the outside world.

Technical note: I am only talking about insecure protocols in this post. There is a different issue of bugs (problems) in some particular implementation of SSL – for example, OpenSSL. However, those problems can usually be fixed on the server side by updating to a newer SSL implementation version and generally do not remove protocols as part of fixing the bug. My post is focused on servers that are too old and / or cannot be updated for some reason, which is a completely different issue from server implementation bugs.

What do you think? I’d like to see comments from end users and security experts – feel free to try to shoot holes in my argument. I’d love to see comments from browser authors, too.

Brother Printer Upgrade Follies

“Well, I’ve been to one world fair, a picnic, and a rodeo, and that’s the stupidest thing I ever heard…”
— Major Kong, Dr. Strangelove

That pretty much sums up my feelings about the firmware update “procedure” Brother provides for their printers. Some time ago I purchased a Brother HL-6180DW to replace an aging LaserJet 2200DN which had decided to either feed multiple sheets or no sheets from the paper tray.

I have no issues with the HL-6180DW as a printer – it has worked fine for over a year, does everything I ask it to, and successfully pretends to be the LaserJet 2200DN that it replaced so I didn’t have to update any drivers. However, I went to reconfigure it the other day to change its hostname and was greated by the dreaded https strikethrough in Google Chrome (the “Your connection is using an obsolete cipher suite” error):

“No problem,” I thought to myself “I’ll just download the latest printer firmware.” I discovered that it is nowhere near that simple.

The first thing I did was download the latest updater from the Brother support site. Running the updater produced an un-helpful “Cannot find a machine to update.” error. Searching on the support site, this is apparently because I did not have the Brother printer driver installed. Of course I don’t – the whole purpose of this printer is to emulate printers from other manufacturers so people don’t have to install drivers when replacing the printer.

I then downloaded the printer driver from the Brother support site and ran it. It self-unpacked into a directory tree which contained no documentation. Fortunately, there was only one .exe. Unfortunately, running it appeared to have no effect other than popping up the Windows “Do you want to let this program make changes to your computer” alert box. Back to the Brother support site, where this support document bizarrely states:

“Case A: For users who connect the Brother machine to their computer using a WSD or TCP/IP port

Connect your computer to the Internet.
Connect the Brother machine to your computer with a USB cable.
The driver will be installed automatically.”

So, in order to install a network printer driver I don’t want, I have to find a USB cable and connect the printer to a PC via a USB port? That is downright bizarre… Armed with a USB cable, I do that and lo and behold, a new printer shows up which claims to be the Brother, attached via USB.

Back to the firmware update utility. Hooray! My printer is detected, and after agreeing that Brother can collect lots of information I don’t really want to give them, I finally get to click on a button to start the firmware update. After a long pause, it tells me that it cannot access the printer (which it detected just fine a few screens back). It tells me that I should check my Internet connection, disable the firewall, sacrifice a chicken, and try again. I proceed to:

  • Disable Windows firewall on my PC
  • Disable the Cisco firewall protecting my network
  • Disable IP security on the printer
  • disable IPv6 on the printer
  • Disable jumbo frames on the printer

None of which has any effect whatsoever.

After more flailing around, I decide on a desperate measure – I will change the printer port from USB to TCP/IP in the printer properties. A miracle – running the update utility produces a request for the printer’s management password, after sending my personal data Yet Again to Brother (or is that Big Brother?). After an extended period of watching the progress bar move at a varying rate (and jump from 80-odd percent complete to 100% complete), the update has finished!

After making sure I can still print from the other computers who still think they’re talking to a LaserJet 2200DN, I go back into the PC I used for the updating and re-enable Windows Firewall. Then I re-enable the Cisco firewall protecting my Internet connection. Lastly, I restore all the settings that I changed on the printer.

“All is as it was before…”
— Guardian of Forever, Star Trek

Back to Chrome to make sure this fixes the https strikethrough… no such luck. Hours wasted for no gain.

I have NO IDEA why Brother thinks this is a good idea. Maybe they’re paranoid about people getting access to the firmware images (although anyone with access to the network and a copy of Wireshark could capture it “on the fly”). The update utility messages could be vastly improved, instead of the “Doh! (Homer Simpson) that it does now. The support documentation could also be improved to actually explain what the utility needs in order to update the firmware.

Of course, my decade-old HP LaserJet 9000DTN came with an add-in network card which has a simple “download firmware update from HP” button (which, amazingly, still works despite HP having rearranged their web site multiple times since that card was new).

In a corporate network where I would have to get IT support involved in disabling my PC’s firewall, or (good luck!) disabling the corporate firewall in order to satisfy the Brother update utility, I think people would simply give up and not update the printer firmware.

And don’t think you can cheat and tell Brother you’re running Linux – the downloads for Linux don’t include a method to update the firmware.

De-bloating the Dell Server Update Utility – Continued

Dell has released the 2014.12 SUU, and it continues the tradition of expanding:

12/18/2014 07:13 AM 10,589,175,808 SUU_14.12.200.69.ISO

It is no longer sufficient to simply delete all the .exe files in the \repository directory if you still want it to fit on a single-layer DVD. You should delete all of the files in \bin\Windows and \java\Windows as well. This will leave you with 4,467,253,896 bytes, which is small enough to fit on a single layer DVD.

At some point in the future, unless Dell deals with the SUU bloat by splitting the Windows and Linux discs, you will need to use a double layer DVD, even with the Windows executables removed.

Dell OptiPlex 9020 mini-review

I previously reviewed the OptiPlex 755 here, along with entries about upgrading them and installing Windows 7. Click here for those entries.

Since then, I upgraded to OptiPlex 960 systems, but I didn’t feel that it would be fair to review them since I had built them from spare parts (starting with “barebones” chassis from eBay, which are scratch & dent discards from Dell Manufacturing, and adding the necessary parts) and this wouldn’t paint a true picture of the 960. I will say that the OptiPlex 960 is the first Dell business tower system that I would consider truly attractive – they obviously spent a lot of time on the case aesthetics.

My 960’s are getting a bit long in the tooth, since I have been using them since late 2010. Windows has been getting twitchier over time, with things like Virtual PC not wanting to start, “Internet Explorer has crashed”, and so on. This is a still a record for a Windows install, since earlier versions tended to die from “bit rot” and need a wipe and reinstall every few years.

I decided to try the newest OptiPlex model, the 9020 Mini Tower, as on paper its specs looked quite good. It eliminated the floppy disk / multi-card reader bay (which I don’t use, anyway) and was rearranged internally to provide a more useful layout and selection of expansion card slots. I had hoped that this would avoid some of the hassles I’d had in the past with getting a video card to fit into the system. With two PCI Express x16 slots (one of which is only wired x4), I hoped I would be able to experiment with my Intel X540 10 Gigabit Ethernet cards.

Unfortunately, when I went to the Dell business site to configure and purchase a 9020, it seems that they only have pre-configured models available. You can’t specify which processor you want, or even if you want extra memory installed! None of the pre-configured systems were available with a set of options I felt comfortable starting with, so I ordered the 730-8285 configuration from an authorized reseller. This system’s specs are:

  • OptiPlex 9020 Mini Tower
  • Intel® Core™ i7-4770 Processor (Quad Core, 3.4GHz Turbo, 8MB, w/ HD Graphics 4600)
  • Operating System: Windows 7 Pro 64-bit (includes Windows 8.1 Pro License and Installation Disk)
  • Graphics Card: AMD Radeon HD 8570, 1GB DDR3, 1DP 1DVI
  • 8 GB 1600MHz DDR3 Memory (2 x 4 GB)
  • Keyboard: Dell KB212-B QuietKey
  • Mouse: Dell USB Optical Mouse MS111
  • Hard Drive: 1 Terabyte 7,200 RPM
  • Internal Audio Speaker
  • Intel vPro Technology Enabled
  • Resource DVD contains Diagnostics and Drivers
  • 16X DVD+/-RW Drive
  • Chassis Intrusion Switch
  • Dell 3-Year NBD Warranty

At the same time, I ordered a Samsung 840 EVO SSD (250GB) and a pair of 4GB memory modules (to upgrade the system to 16GB total). I planned on using a BDR-206BK Blu-ray burner and HIS R9 270 video card, along with an ASUS Xonar D2X (for digital audio output) from inventory to round out the system.

Upon opening the chassis, I discovered that Dell is using a new, 12V-only power supply. This is based on a concept by Fujitsu (PDF whitepaper here). Unfortunately, despite that paper ending with “The 12 V Only System is not an industry standard yet but a proprietary solution, which is currently implemented by all Tier 1 Systembuilders like e.g. Dell, HP, Fujitsu!”, each of those manufacturers seems to use a slightly different implementation. Thus, there don’t seem to be any 3rd-party manufacturers building compatible power supplies. A search for supplies only turned up people complaining about the problem, not any replacements.

The theory behind the 12V-only power supply is that most of the power requirements on the motherboard are for 12V (newer systems have had 12V rails dedicated to the processor for some time), and the remaining voltages can be more efficiently generated on the motherboard.

So, I’m stuck with the somewhat-anemic stock 290W power supply, and don’t have a good way to power the HIS video card I was planning on using. It might be possible to do something using a reverse SATA power adapter to convert the 2nd HDD SATA power connector into a pair of Molex 4-pin connectors, then use a PCI Express power adapter to convert the HIS card’s connector to a pair of mating Molex 4-pin connectors. However, this may lead to overloading something, as the HDD power connectors are supplied via a single pin from the motherboard. It doesn’t seem to be worth risking damage to the motherboard to try to make this work.

So I am now looking for an attractive case (attractive in the sense of the OptiPlex 960, not in the “Fast and the Furious” sense with neon lighting, see-through panels, etc.) and will buy a generic motherboard (probably from SuperMicro) and components to build a system from scratch. At least I won’t be limited to half-length single-slot video cards.

In summary, I would classify the OptiPlex 9020 as “Not Recommended” due to the inability to configure the system as needed. The power supply issue is probably not relevant for most business users (the primary target market for OptiPlex users). Dell originally designed the OptiPlex line “for customers who are traumatized by change” (actual quote from a Dell Marketing VP many years ago), with a guarantee that the same system configuration would continue to be orderable for a year. The limited number of packaged configurations available means that the customer may wind up with multiple versions of the 9020 if ordered in separate batches.

Bear in mind that this reviewer represents the “traditionalist” view. Articles in the trade press keep telling us that “the next generation of things (be they desktops, notebooks, or tablets) will be the last big update” because the world will have moved on to something else by the time they are due for replacement. If you take that viewpoint, the smaller form-factor OptiPlex 9020 models (which can be treated as non-upgradable) may be an appropriate fit for the business environment. But selling a “classic” mini tower form factor system with limited options and where add-in cards are limited by lack of power just doesn’t make a lot of sense.

De-bloating the Dell Server Update Utility (SUU) DVD Image

Dell issues a quarterly Server Update Utility (SUU) image which is used to update most firmware on PowerEdge servers (and some other Dell devices). As I use FreeBSD on my servers (which is not supported by Dell) I have to boot the Dell CDU CD to get a standalone Linux system suitable for launching SUU. Unfortunately, the SUU ISO image has become increasingly bloated over time, and is now too big to either burn to a double-layer DVD or upload to the 8GB vFlash card in the iDRAC. I suppose there’s some method for dealing with this if you’re running a Dell-supported operating system, but us FreeBSD users are left out. Here is a list of the last 4 quarters of SUU images, showing their sizes:

01/03/2014 08:08 AM 7,986,208,768 SUU_740_Q42013_A00.ISO
04/13/2014 08:00 AM 8,434,493,440 SUU_14.03.00_A00.iso
07/26/2014 06:36 AM 9,057,501,184 SUU_14.07.00_A00.ISO
10/21/2014 03:23 AM 9,922,859,008 SUU_14.10.200.117.iso

The main part of the bloat is that the disc contains two versions of every update utility, one for Linux systems and one for Windows systems. Since the CDU provides a Linux system, we can delete all of the Windows files with no impact. I found it easiest to copy the entire SUU DVD to a scratch directory and then delete all the .exe files from the \repository directory. There’s quite a few of them:

F:\repository>dir *.exe
Volume in drive F is SUU743_117
Volume Serial Number is 442E-5D5D

Directory of F:\repository

[snip]

400 File(s) 5,490,684,272 bytes
0 Dir(s) 0 bytes free

Once I deleted these un-necessary files, I burned the remaining files (preserving the directory structure) to a DVD (a single layer DVD is now sufficient) with ImgBurn. There are more Windows files in other directories (for example, a Java runtime) but it isn’t necessary to delete those to get the size below the limit of a single layer DVD. Booting CDU and then switching to my modified SUU disc worked fine, and installed the few updates I was missing on my PowerEdge R710.

I don’t know why Dell doesn’t create separate SUU ISO images for Windows and Linux – it would cut people’s download times in half. Until they decide to do something, the above method should give you a usable SUU DVD.

Troubleshooting Catalyst 4948-10GE red status LED and no console output

This is not intended as a complete DIY. It requires equipment most of my readers won’t have, such as a hot air PCB rework station with magnifier. I am posting it to give you an idea of what is involved in the repair of these devices, and to provide info to any readers who do have the necessary equipment and just need to know the repair procedure.

I have been encountering more and more dead Catalyst 4948-10GE switches lately. These usually have a solid red Status LED and do not display any messages on the console when power is applied. This means that the switch did not make any progress at all in booting (one of the first steps in the boot process is to change the Status LED from red to orange). Catalyst 4948-10GE switches with this type of fault are frequently listed on eBay in the $250-$350 price range (usually marked “For parts or not working”). When troubleshooting these, the problem is often defective memory. Unfortunately, this memory is soldered to the circuit board in the switch, so it isn’t simply a matter of removing a faulty memory module and replacing it with a known good one. The old memory needs to be de-soldered and new memory soldered in, and you need to have a source for the obsolete memory chips needed for replacements.

These switches have 256MB of ECC memory, implemented via 5 32MB x 16-bit memory chips such as the Micron MT46V32M16-6T F. Three of the chips are located on the top side of the motherboard next to the power supply, and another two are located on the underside of the board (all images in this post can be clicked for a larger version):

Top side of board

Top side of board

Bottom side of board

Bottom side of board

In each of the boards I have repaired, the fault has always been in one of the bottom two chips. This makes sense as there is no airflow across the bottom of the board, so those chips are more likely to overheat than the ones on the top of the board. Cisco has announced an issue with an unspecified memory supplier (often rumored to be Micron), and the Catalyst 49xx family is on that list. However, the switches that I am seeing failures on are not on a Cisco support contract, and I haven’t read anything about Cisco fixing equipment not on a support contract for free, so I’ve been repairing them myself.

The first step is to remove the two existing memory chips from the underside of the board and clean and prepare the board for the new chips:

Memory removed

The next step is to solder the replacement chips into place:

New memory installed

Of course, you need to ensure that the chips are installed in the correct orientation (of course!) and that all pins are soldered to their respective pads (66 pins per chip) and that there are no shorts between pins. You also need to avoid damaging any of the neighboring components or the circuit board itself while doing this.

If all goes well, when you reinstall the board in the chassis and apply power, you will be greeted with the appropriate console messages and the switch will boot up normally. If not, remove the board and examine the area around the replaced chips under a magnifier to double check for bad connections or solder bridges.

2.5″ enterprise hard drives and power savings

I admit it – I used to have an unwarranted prejudice against 2.5″ enterprise hard drives, considering them “toy” drives, or at best suited for notebook use, or non-critical use in enterprise systems. I used WD Velociraptor drives on my Dell desktops (before I upgraded to SSDs), but that particular model was discontinued, and the WD web site has this discouraging note about the current models: “Models WD1000CHTZ, WD5000BHTZ and WD2500BHTZ are available on a build to order basis, contact your WD Sales representative for more information.” which I interpreted as “people aren’t buying these, but if you want a bazillion of ’em, we’ll restart the production line”. I also used WD 2.5″ drives as the operating system volume on my RAIDzilla II file servers, but the actual data volumes were built with 16 x 2TB 3.5″ drives.

However, in an attempt to reduce power consumption here, I decided to test 2.5″ enterprise drives as a replacement for identical-capacity 3.5″ drives, and the results were surprising (to me, at least). I upgraded one of my Dell PowerEdge R710 systems (gate.glaver.org, the system that is serving this web page that you’re reading) from 6 x 146GB 15K RPM 3.5″ SAS drives (ST3300657SS-H*) to 6 x 146GB 15K RPM 2.5″ SAS drives (ST9146852SS). All other components remained the same*. The drives are in a 5-drive RAID5 array controlled by a Dell PERC H700 controller, with the 6th drive being a dedicated hot spare.

Power consumption on this busy system dropped from 237W to 204W and became much more even (apparently, seeking on the 3.5″ drives consumes much more power than on the 2.5″ drives):

gate.glaver.org power consumption

Click the picture for a larger view

The PowerEdge R710 is already a pretty efficient system – this particular box has 2 x X5680 6-core Xeon CPUs, 48GB of registered ECC RAM, hardware RAID controller, etc.

Even more surprising was the discovery that disk I/O was still very fast, at well over 600MByte/sec:

(0:1) gate:~terry# dd if=/dev/mfid0 of=/dev/null bs=1m count=102400
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 171.422439 secs (626371804 bytes/sec)

Based on this, I will certainly give serious consideration to using 2.5″ drives in future builds.

Seagate has announced 2.5″ enterprise drives with up to 2TB capacity (in both SAS and SATA variants). While that is lagging behind the announced capacity for 3.5″ drives (8TB at this time), you can fit a lot more 2.5″ drives in a given chassis. I expect to use one or two additional drive generations in my existing RAIDzilla II chassis (upgrading to 4TB drives at some point, and then in the future to 8TB or 10TB drives). After that, it will be time to design the RAIDzilla III.

* Yes, I know this is normally a 300GB drive. Seagate didn’t make a native 146GB drive in the Cheetah 15K.7 family, and the -H suffix indicates a half-capacity drive for OEMs who needed to match existing drive capacities.

* This is not a particularly easy conversion, as the Dell chassis for the R710 is not modular. However, various sellers on eBay are selling new or used 2.5″ chassis (part number 33P6Y). You can move just about all of the old components from the 3.5″ chassis over – the only item you will need (other than the actual 2.5″ drives and trays) are the appropriate cables from the drive backplane to the RAID controller. For a PERC H700, that is 2 x R145M mini-SAS cables.