Archive for the 'Computers' Category

Net Neutrality isn’t the only problem

Today (July 12th, 2017) a large number of sites have joined together to raise awareness of the threats to network neutrality. For example, reddit has a pop-over window that slowly types a message beginning with “The internet’s less fun when your favorite sites load slowly, isn’t it?” This is certainly a valid concern, and many people, including myself, have legitimate concerns about how the Internet is regulated. But there are enough sites raising that point, so I’d like to talk about something different – how sites are “shooting themselves in the foot” with slow-loading (and often buggy) page content.

It all starts when a web site decides they want to track visitors for demographics or other purposes. There are a large number of “free”* tools available that will collect the data and let you analyze it in any way you like. Sure, it comes with some hidden Javascript that does things you can’t see, but hey – it is only one thing on a page of otherwise-useful content, right?

Next, the site decides they’d like to help cover the cost of running the site by having a few advertisements. So they add code provided by the advertising platform(s) they’ve selected. So their page now loads a bit slower, and users see ads, but the users will still come for the content, right? And the occasional malware that slips through the advertising platform and gets shown on their site isn’t really their fault, right? They can always blame the advertising platform.

Somewhat later, the site gets an “offer they can’t refuse” to run some “sponsored content”. The page gets even slower and users are having a hard time distinguishing actual content from ads. Clicking on what looks like actual content causes an ad to start playing, or triggers a pop-under, or any one of a number of things that make for an unpleasant user experience.

Once everyone is used to this, things appear to settle down. Complaints from users are infrequent (probably because they can no longer figure out how to contact the site to report problems). Everyone has forgotten how fast the site used to load, except for the users running ad blockers, cookie blockers, script blockers, and so on.

But one day a SSL certificate becomes invalid for some reason (expired, a site was renamed, etc.) and the users are now getting a new annoyance like a pop-up saying that the certificate for btrll.com is invalid. Most users go “huh?” because they weren’t visiting (or at least they thought they weren’t visiting) btrll.com. Clicking the “close” button lasts for all of a second before the pop-up is back, because that ad site is determined to show you that ad. In frustration, the user closes their browser and goes out to buy a newspaper.

By this point, perhaps 5% of the actual page content is from the site the user was intending to visit. The rest is user tracking, advertising, and perhaps a bit of malware. There is a free tool run by WebPagetest.org which will let you analyze any web site to see what it is loading and why it is slow.

Here is the result for the CNN home page:

Now, that’s too small to be able to read, so this is the first part of it (click on this image for a larger view):

The blue line at 21 seconds shows when the page finished loading, although you can see that Javascript from a number of advertising providers continues to run indefinitely.

Now, let’s take a look at Weather Underground. Surely just serving weather information would have far less bloat than CNN, right? Not really:

Now, that’s too small to be able to read, so this is the first part of it (click on this image for a larger view):

It does manage to load in less time than CNN, but it is still pretty awful.

In the spirit of full disclosure, here is the result for this blog page:

Since the entire report fits, I didn’t need to add an unreadably-small overview image.

If you manage a web site, I encourage you to try WebPagetest.org yourself and see why your site is slow. If you’re just a user, you can also use WebPagetest.org to see why the sites you visit are slow. If you’re using add blocking or site blacklisting software while you browse, the list of hosts that are serving advertisements or other unwanted content will probably be useful to you when added to your block / blacklist.

* As they say, “If you aren’t paying for it, then you are the product being sold”.

Is no crypto always better than bad crypto?

SSL (Secure Sockets Layer, the code that forms the basis of the https:// in a URL) can use any number of different encryption methods (protocols) and key strengths. While all of the protocols / strengths were presumed to be secure at the time they were designed, faster computers have made “cracking” some of the older protocols practical, or at least potentially practical. Additionally, concerns have been raised that some of the underlying math may have been intentionally weakened by the proponents (for example, NIST and the NSA) of those protocols. Perhaps an underlying flaw in the protocol has been discovered. Due to this, web browsers have been removing support for these older, insecure protocols.

Additionally, even if a protocol is still considered secure, a browser may start enforcing additional requirements for the SSL certificate used with that protocol. “Under the covers” this is a rather different situation, but for the purpose of this discussion I will lump them together, since the average user doesn’t care about the technical differences, only that a service that they used to be able to access no longer works.

In theory, this is a good idea – nobody wants their financial details “sniffed” on the way between you and your bank. However, the browser authors have decided that all usage of those older protocols is bad and should be prohibited. They make no distinction between a conversation between you and your bank vs. a conversation between you and another site (which could be a web server, UPS – battery backup, a water heater, or even a light bulb!) in your house or company. Instead, they force you to disable all encryption and communicate “in the clear”.

To add to the complexity, each browser does things in a different way. And the way a given browser handles a particular situation can change depending on the version of the browser. That isn’t too bad for Internet Explorer, which doesn’t change that often. Two other browsers that I use (Mozilla Firefox and Google Chrome) seem to release new versions almost weekly. In addition, the behavior of a browser may change depending on what operating system it is running under. Browsers also behave differently depending on when the host at the other end of the connection obtained its security certificate. A certificate issued on December 31st, 2015 at 23:59:59 is treated differently than one issued one second later on January 1st, 2016 at 00:00:00.

In the following discussion, the terms “site” and “device” are generally interchangeable. I sometimes use the term “device” to refer to the system the browser is attempting to connect to. “Site” might be a more accurate term, but for many users a “site” implies a sophisticated system such as an online store, while an intelligent light bulb is more a “device” than a “site”.

In a perfect world, people could just deal with the browser blocking issue by installing new software and / or certificates on all of the devices they administer. Sure, that would be a lot of work (here at home, I have several dozen devices with SSL certificates and in my day job, I have many hundreds of devices) and possibly expense (the companies that sell the certificates don’t always allow users to request updated certificates for free, and updated software to handle the new protocol may not be free – for example, Cisco requires a paid support contract to download updated software). However, it is not that “easy” – any given device may not have new software available, or the new software still doesn’t handle some of the latest protocols.

This leads to an unfortunate game of “whack-a-mole”, where a browser will change its behavior, a company will implement new software to deal with that new behavior, but by the time the software has gone through testing and is released, the browser has changed its behavior again and the updated software is useless. A number of vendors have just given up supporting their older products because of this – they have finite resources and they choose to allocate them to new products.

The browser authors seem to feel that this is just fine and that users should either turn encryption off or throw away the device and buy a new one. Since the “device” is often a management function embedded in an expensive piece of hardware, that simply isn’t practical. A home user may not feel that replacing a working device is necessary and a business likely won’t replace a device until the end of its depreciation cycle (often 3 or 5 years).

This strikes me as a very poor way for browsers to deal with the situation. Instead of a binary good / bad decision which the user cannot override, it seems to me that a more nuanced approach would be beneficial. If browsers allowed continued usage of these “obsolete” protocols in certain limited cases, I think the situation would be better.

First, I agree with the current browser behavior when dealing with “Extended Validation” sites. These are sites that display a (usually) green indication with the verified company name in the browser’s address bar. In order to purchase an EV certificate, the site needs to prove that they are who they say they are. For example, your bank almost certainly uses an EV certificate. Users should expect that sites with EV certificates are using secure methods to protect connections. If a site with an EV certificate is using an obsolete protocol, something is definitely wrong at that site and the connection should not be allowed.

Second, the current behavior is OK when dealing with well-known sites (for example, amazon.com). This is a little more difficult for browsers to deal with, as they would need to keep a list of sites as well as deciding on criteria for including a site on that list. However, there already is a “master list” of sites which is shared between various browsers – it is called the HSTS Preload list. It could be used for this purpose.

Now we get to the heart of the matter – how to deal with non-EV, non-well-known sites. Instead of refusing to allow access to a site which uses an insecure protocol, a browser could:

  • Display a warning box the first time a site is accessed via an insecure protocol and let the user choose whether or not to proceed.
  • Re-display the warning after a reasonable period of time (perhaps 30 days) and ask the user to re-confirm that they want to use the insecure protocol to access the site.
  • On each page, indicate that the page is using an insecure protocol. This could be done by displaying the URL in the address bar on a red background or similar. Google Chrome does something similar with its red strikethrough on top of the https:// in the address bar. Unfortunately, in most cases Chrome will simply refuse to access a site it deems insecure.
  • NOT require dismissing a warning each time the user accesses the site.
  • NOT require a non-standard way of specifying the site URL in the address bar, bookmarks, etc.

Security experts will probably be thinking “But… That’s insecure!” It certainly is, but is it less secure than using no encryption at all (which is what the browsers are currently forcing users to do)? I don’t think so. In many cases, both the user and the site they are connecting to are on the same network, perhaps isolated from the larger Internet. For example, most devices here are only accessible from the local network – they are firewalled from the outside world.

Technical note: I am only talking about insecure protocols in this post. There is a different issue of bugs (problems) in some particular implementation of SSL – for example, OpenSSL. However, those problems can usually be fixed on the server side by updating to a newer SSL implementation version and generally do not remove protocols as part of fixing the bug. My post is focused on servers that are too old and / or cannot be updated for some reason, which is a completely different issue from server implementation bugs.

What do you think? I’d like to see comments from end users and security experts – feel free to try to shoot holes in my argument. I’d love to see comments from browser authors, too.

Brother Printer Upgrade Follies

“Well, I’ve been to one world fair, a picnic, and a rodeo, and that’s the stupidest thing I ever heard…”
— Major Kong, Dr. Strangelove

That pretty much sums up my feelings about the firmware update “procedure” Brother provides for their printers. Some time ago I purchased a Brother HL-6180DW to replace an aging LaserJet 2200DN which had decided to either feed multiple sheets or no sheets from the paper tray.

I have no issues with the HL-6180DW as a printer – it has worked fine for over a year, does everything I ask it to, and successfully pretends to be the LaserJet 2200DN that it replaced so I didn’t have to update any drivers. However, I went to reconfigure it the other day to change its hostname and was greated by the dreaded https strikethrough in Google Chrome (the “Your connection is using an obsolete cipher suite” error):

“No problem,” I thought to myself “I’ll just download the latest printer firmware.” I discovered that it is nowhere near that simple.

The first thing I did was download the latest updater from the Brother support site. Running the updater produced an un-helpful “Cannot find a machine to update.” error. Searching on the support site, this is apparently because I did not have the Brother printer driver installed. Of course I don’t – the whole purpose of this printer is to emulate printers from other manufacturers so people don’t have to install drivers when replacing the printer.

I then downloaded the printer driver from the Brother support site and ran it. It self-unpacked into a directory tree which contained no documentation. Fortunately, there was only one .exe. Unfortunately, running it appeared to have no effect other than popping up the Windows “Do you want to let this program make changes to your computer” alert box. Back to the Brother support site, where this support document bizarrely states:

“Case A: For users who connect the Brother machine to their computer using a WSD or TCP/IP port

Connect your computer to the Internet.
Connect the Brother machine to your computer with a USB cable.
The driver will be installed automatically.”

So, in order to install a network printer driver I don’t want, I have to find a USB cable and connect the printer to a PC via a USB port? That is downright bizarre… Armed with a USB cable, I do that and lo and behold, a new printer shows up which claims to be the Brother, attached via USB.

Back to the firmware update utility. Hooray! My printer is detected, and after agreeing that Brother can collect lots of information I don’t really want to give them, I finally get to click on a button to start the firmware update. After a long pause, it tells me that it cannot access the printer (which it detected just fine a few screens back). It tells me that I should check my Internet connection, disable the firewall, sacrifice a chicken, and try again. I proceed to:

  • Disable Windows firewall on my PC
  • Disable the Cisco firewall protecting my network
  • Disable IP security on the printer
  • disable IPv6 on the printer
  • Disable jumbo frames on the printer

None of which has any effect whatsoever.

After more flailing around, I decide on a desperate measure – I will change the printer port from USB to TCP/IP in the printer properties. A miracle – running the update utility produces a request for the printer’s management password, after sending my personal data Yet Again to Brother (or is that Big Brother?). After an extended period of watching the progress bar move at a varying rate (and jump from 80-odd percent complete to 100% complete), the update has finished!

After making sure I can still print from the other computers who still think they’re talking to a LaserJet 2200DN, I go back into the PC I used for the updating and re-enable Windows Firewall. Then I re-enable the Cisco firewall protecting my Internet connection. Lastly, I restore all the settings that I changed on the printer.

“All is as it was before…”
— Guardian of Forever, Star Trek

Back to Chrome to make sure this fixes the https strikethrough… no such luck. Hours wasted for no gain.

I have NO IDEA why Brother thinks this is a good idea. Maybe they’re paranoid about people getting access to the firmware images (although anyone with access to the network and a copy of Wireshark could capture it “on the fly”). The update utility messages could be vastly improved, instead of the “Doh! (Homer Simpson) that it does now. The support documentation could also be improved to actually explain what the utility needs in order to update the firmware.

Of course, my decade-old HP LaserJet 9000DTN came with an add-in network card which has a simple “download firmware update from HP” button (which, amazingly, still works despite HP having rearranged their web site multiple times since that card was new).

In a corporate network where I would have to get IT support involved in disabling my PC’s firewall, or (good luck!) disabling the corporate firewall in order to satisfy the Brother update utility, I think people would simply give up and not update the printer firmware.

And don’t think you can cheat and tell Brother you’re running Linux – the downloads for Linux don’t include a method to update the firmware.

De-bloating the Dell Server Update Utility – Continued

Dell has released the 2014.12 SUU, and it continues the tradition of expanding:

12/18/2014 07:13 AM 10,589,175,808 SUU_14.12.200.69.ISO

It is no longer sufficient to simply delete all the .exe files in the \repository directory if you still want it to fit on a single-layer DVD. You should delete all of the files in \bin\Windows and \java\Windows as well. This will leave you with 4,467,253,896 bytes, which is small enough to fit on a single layer DVD.

At some point in the future, unless Dell deals with the SUU bloat by splitting the Windows and Linux discs, you will need to use a double layer DVD, even with the Windows executables removed.

Dell OptiPlex 9020 mini-review

I previously reviewed the OptiPlex 755 here, along with entries about upgrading them and installing Windows 7. Click here for those entries.

Since then, I upgraded to OptiPlex 960 systems, but I didn’t feel that it would be fair to review them since I had built them from spare parts (starting with “barebones” chassis from eBay, which are scratch & dent discards from Dell Manufacturing, and adding the necessary parts) and this wouldn’t paint a true picture of the 960. I will say that the OptiPlex 960 is the first Dell business tower system that I would consider truly attractive – they obviously spent a lot of time on the case aesthetics.

My 960’s are getting a bit long in the tooth, since I have been using them since late 2010. Windows has been getting twitchier over time, with things like Virtual PC not wanting to start, “Internet Explorer has crashed”, and so on. This is a still a record for a Windows install, since earlier versions tended to die from “bit rot” and need a wipe and reinstall every few years.

I decided to try the newest OptiPlex model, the 9020 Mini Tower, as on paper its specs looked quite good. It eliminated the floppy disk / multi-card reader bay (which I don’t use, anyway) and was rearranged internally to provide a more useful layout and selection of expansion card slots. I had hoped that this would avoid some of the hassles I’d had in the past with getting a video card to fit into the system. With two PCI Express x16 slots (one of which is only wired x4), I hoped I would be able to experiment with my Intel X540 10 Gigabit Ethernet cards.

Unfortunately, when I went to the Dell business site to configure and purchase a 9020, it seems that they only have pre-configured models available. You can’t specify which processor you want, or even if you want extra memory installed! None of the pre-configured systems were available with a set of options I felt comfortable starting with, so I ordered the 730-8285 configuration from an authorized reseller. This system’s specs are:

  • OptiPlex 9020 Mini Tower
  • Intel® Core™ i7-4770 Processor (Quad Core, 3.4GHz Turbo, 8MB, w/ HD Graphics 4600)
  • Operating System: Windows 7 Pro 64-bit (includes Windows 8.1 Pro License and Installation Disk)
  • Graphics Card: AMD Radeon HD 8570, 1GB DDR3, 1DP 1DVI
  • 8 GB 1600MHz DDR3 Memory (2 x 4 GB)
  • Keyboard: Dell KB212-B QuietKey
  • Mouse: Dell USB Optical Mouse MS111
  • Hard Drive: 1 Terabyte 7,200 RPM
  • Internal Audio Speaker
  • Intel vPro Technology Enabled
  • Resource DVD contains Diagnostics and Drivers
  • 16X DVD+/-RW Drive
  • Chassis Intrusion Switch
  • Dell 3-Year NBD Warranty

At the same time, I ordered a Samsung 840 EVO SSD (250GB) and a pair of 4GB memory modules (to upgrade the system to 16GB total). I planned on using a BDR-206BK Blu-ray burner and HIS R9 270 video card, along with an ASUS Xonar D2X (for digital audio output) from inventory to round out the system.

Upon opening the chassis, I discovered that Dell is using a new, 12V-only power supply. This is based on a concept by Fujitsu (PDF whitepaper here). Unfortunately, despite that paper ending with “The 12 V Only System is not an industry standard yet but a proprietary solution, which is currently implemented by all Tier 1 Systembuilders like e.g. Dell, HP, Fujitsu!”, each of those manufacturers seems to use a slightly different implementation. Thus, there don’t seem to be any 3rd-party manufacturers building compatible power supplies. A search for supplies only turned up people complaining about the problem, not any replacements.

The theory behind the 12V-only power supply is that most of the power requirements on the motherboard are for 12V (newer systems have had 12V rails dedicated to the processor for some time), and the remaining voltages can be more efficiently generated on the motherboard.

So, I’m stuck with the somewhat-anemic stock 290W power supply, and don’t have a good way to power the HIS video card I was planning on using. It might be possible to do something using a reverse SATA power adapter to convert the 2nd HDD SATA power connector into a pair of Molex 4-pin connectors, then use a PCI Express power adapter to convert the HIS card’s connector to a pair of mating Molex 4-pin connectors. However, this may lead to overloading something, as the HDD power connectors are supplied via a single pin from the motherboard. It doesn’t seem to be worth risking damage to the motherboard to try to make this work.

So I am now looking for an attractive case (attractive in the sense of the OptiPlex 960, not in the “Fast and the Furious” sense with neon lighting, see-through panels, etc.) and will buy a generic motherboard (probably from SuperMicro) and components to build a system from scratch. At least I won’t be limited to half-length single-slot video cards.

In summary, I would classify the OptiPlex 9020 as “Not Recommended” due to the inability to configure the system as needed. The power supply issue is probably not relevant for most business users (the primary target market for OptiPlex users). Dell originally designed the OptiPlex line “for customers who are traumatized by change” (actual quote from a Dell Marketing VP many years ago), with a guarantee that the same system configuration would continue to be orderable for a year. The limited number of packaged configurations available means that the customer may wind up with multiple versions of the 9020 if ordered in separate batches.

Bear in mind that this reviewer represents the “traditionalist” view. Articles in the trade press keep telling us that “the next generation of things (be they desktops, notebooks, or tablets) will be the last big update” because the world will have moved on to something else by the time they are due for replacement. If you take that viewpoint, the smaller form-factor OptiPlex 9020 models (which can be treated as non-upgradable) may be an appropriate fit for the business environment. But selling a “classic” mini tower form factor system with limited options and where add-in cards are limited by lack of power just doesn’t make a lot of sense.

De-bloating the Dell Server Update Utility (SUU) DVD Image

Dell issues a quarterly Server Update Utility (SUU) image which is used to update most firmware on PowerEdge servers (and some other Dell devices). As I use FreeBSD on my servers (which is not supported by Dell) I have to boot the Dell CDU CD to get a standalone Linux system suitable for launching SUU. Unfortunately, the SUU ISO image has become increasingly bloated over time, and is now too big to either burn to a double-layer DVD or upload to the 8GB vFlash card in the iDRAC. I suppose there’s some method for dealing with this if you’re running a Dell-supported operating system, but us FreeBSD users are left out. Here is a list of the last 4 quarters of SUU images, showing their sizes:

01/03/2014 08:08 AM 7,986,208,768 SUU_740_Q42013_A00.ISO
04/13/2014 08:00 AM 8,434,493,440 SUU_14.03.00_A00.iso
07/26/2014 06:36 AM 9,057,501,184 SUU_14.07.00_A00.ISO
10/21/2014 03:23 AM 9,922,859,008 SUU_14.10.200.117.iso

The main part of the bloat is that the disc contains two versions of every update utility, one for Linux systems and one for Windows systems. Since the CDU provides a Linux system, we can delete all of the Windows files with no impact. I found it easiest to copy the entire SUU DVD to a scratch directory and then delete all the .exe files from the \repository directory. There’s quite a few of them:

F:\repository>dir *.exe
Volume in drive F is SUU743_117
Volume Serial Number is 442E-5D5D

Directory of F:\repository

[snip]

400 File(s) 5,490,684,272 bytes
0 Dir(s) 0 bytes free

Once I deleted these un-necessary files, I burned the remaining files (preserving the directory structure) to a DVD (a single layer DVD is now sufficient) with ImgBurn. There are more Windows files in other directories (for example, a Java runtime) but it isn’t necessary to delete those to get the size below the limit of a single layer DVD. Booting CDU and then switching to my modified SUU disc worked fine, and installed the few updates I was missing on my PowerEdge R710.

I don’t know why Dell doesn’t create separate SUU ISO images for Windows and Linux – it would cut people’s download times in half. Until they decide to do something, the above method should give you a usable SUU DVD.

Troubleshooting Catalyst 4948-10GE red status LED and no console output

This is not intended as a complete DIY. It requires equipment most of my readers won’t have, such as a hot air PCB rework station with magnifier. I am posting it to give you an idea of what is involved in the repair of these devices, and to provide info to any readers who do have the necessary equipment and just need to know the repair procedure.

I have been encountering more and more dead Catalyst 4948-10GE switches lately. These usually have a solid red Status LED and do not display any messages on the console when power is applied. This means that the switch did not make any progress at all in booting (one of the first steps in the boot process is to change the Status LED from red to orange). Catalyst 4948-10GE switches with this type of fault are frequently listed on eBay in the $250-$350 price range (usually marked “For parts or not working”). When troubleshooting these, the problem is often defective memory. Unfortunately, this memory is soldered to the circuit board in the switch, so it isn’t simply a matter of removing a faulty memory module and replacing it with a known good one. The old memory needs to be de-soldered and new memory soldered in, and you need to have a source for the obsolete memory chips needed for replacements.

These switches have 256MB of ECC memory, implemented via 5 32MB x 16-bit memory chips such as the Micron MT46V32M16-6T F. Three of the chips are located on the top side of the motherboard next to the power supply, and another two are located on the underside of the board (all images in this post can be clicked for a larger version):

Top side of board

Top side of board

Bottom side of board

Bottom side of board

In each of the boards I have repaired, the fault has always been in one of the bottom two chips. This makes sense as there is no airflow across the bottom of the board, so those chips are more likely to overheat than the ones on the top of the board. Cisco has announced an issue with an unspecified memory supplier (often rumored to be Micron), and the Catalyst 49xx family is on that list. However, the switches that I am seeing failures on are not on a Cisco support contract, and I haven’t read anything about Cisco fixing equipment not on a support contract for free, so I’ve been repairing them myself.

The first step is to remove the two existing memory chips from the underside of the board and clean and prepare the board for the new chips:

Memory removed

The next step is to solder the replacement chips into place:

New memory installed

Of course, you need to ensure that the chips are installed in the correct orientation (of course!) and that all pins are soldered to their respective pads (66 pins per chip) and that there are no shorts between pins. You also need to avoid damaging any of the neighboring components or the circuit board itself while doing this.

If all goes well, when you reinstall the board in the chassis and apply power, you will be greeted with the appropriate console messages and the switch will boot up normally. If not, remove the board and examine the area around the replaced chips under a magnifier to double check for bad connections or solder bridges.

2.5″ enterprise hard drives and power savings

I admit it – I used to have an unwarranted prejudice against 2.5″ enterprise hard drives, considering them “toy” drives, or at best suited for notebook use, or non-critical use in enterprise systems. I used WD Velociraptor drives on my Dell desktops (before I upgraded to SSDs), but that particular model was discontinued, and the WD web site has this discouraging note about the current models: “Models WD1000CHTZ, WD5000BHTZ and WD2500BHTZ are available on a build to order basis, contact your WD Sales representative for more information.” which I interpreted as “people aren’t buying these, but if you want a bazillion of ’em, we’ll restart the production line”. I also used WD 2.5″ drives as the operating system volume on my RAIDzilla II file servers, but the actual data volumes were built with 16 x 2TB 3.5″ drives.

However, in an attempt to reduce power consumption here, I decided to test 2.5″ enterprise drives as a replacement for identical-capacity 3.5″ drives, and the results were surprising (to me, at least). I upgraded one of my Dell PowerEdge R710 systems (gate.glaver.org, the system that is serving this web page that you’re reading) from 6 x 146GB 15K RPM 3.5″ SAS drives (ST3300657SS-H*) to 6 x 146GB 15K RPM 2.5″ SAS drives (ST9146852SS). All other components remained the same*. The drives are in a 5-drive RAID5 array controlled by a Dell PERC H700 controller, with the 6th drive being a dedicated hot spare.

Power consumption on this busy system dropped from 237W to 204W and became much more even (apparently, seeking on the 3.5″ drives consumes much more power than on the 2.5″ drives):

gate.glaver.org power consumption

Click the picture for a larger view

The PowerEdge R710 is already a pretty efficient system – this particular box has 2 x X5680 6-core Xeon CPUs, 48GB of registered ECC RAM, hardware RAID controller, etc.

Even more surprising was the discovery that disk I/O was still very fast, at well over 600MByte/sec:

(0:1) gate:~terry# dd if=/dev/mfid0 of=/dev/null bs=1m count=102400
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 171.422439 secs (626371804 bytes/sec)

Based on this, I will certainly give serious consideration to using 2.5″ drives in future builds.

Seagate has announced 2.5″ enterprise drives with up to 2TB capacity (in both SAS and SATA variants). While that is lagging behind the announced capacity for 3.5″ drives (8TB at this time), you can fit a lot more 2.5″ drives in a given chassis. I expect to use one or two additional drive generations in my existing RAIDzilla II chassis (upgrading to 4TB drives at some point, and then in the future to 8TB or 10TB drives). After that, it will be time to design the RAIDzilla III.

* Yes, I know this is normally a 300GB drive. Seagate didn’t make a native 146GB drive in the Cheetah 15K.7 family, and the -H suffix indicates a half-capacity drive for OEMs who needed to match existing drive capacities.

* This is not a particularly easy conversion, as the Dell chassis for the R710 is not modular. However, various sellers on eBay are selling new or used 2.5″ chassis (part number 33P6Y). You can move just about all of the old components from the 3.5″ chassis over – the only item you will need (other than the actual 2.5″ drives and trays) are the appropriate cables from the drive backplane to the RAID controller. For a PERC H700, that is 2 x R145M mini-SAS cables.

A few more words of advice for used equipment sellers

Today I’m going to expand on the advice I provided in my earlier post, “A few words of advice for used equipment sellers“. Today I’m going to address the issues with “As-Is / Not Working / For Parts Only” listing types. These are terms used by eBay, but this advice also applies to anyone else selling equipment in this category.

In general, this type of item is offered by sellers at a lower price in the hope of recovering some money from a piece of equipment that is either not operating properly or is not able to be tested by the seller. Some sellers are very scrupulous about describing the equipment, providing lots of pictures and as much information as they know about the item. At the other end are sellers who use a stock photograph and product description, perhaps with some words like “Couldn’t power on – didn’t test.”

Any buyer who purchases items in this category is hoping to find a bargain by ending up with a piece of working equipment after performing minimal repairs. [There are probably people who buy this material for other purposes, such as scrap metal recovery, components for artwork, and so on, but I’ll leave those out of this discussion.] As such, you (as the seller) want to provide as much information as possible to potential buyers so you both end up with a good experience.

There are quite a few categories of “untested / not working”, and I’ll go through these from best to worst:

  • Unable to test / Not tested – this means that the seller lacks the ability to test the item, either because it is a sub-component of a larger device the seller does not have, lack of necessary cabling to connect it, or due to it requiring specialized test / calibration equipment. Items in this category are truly untested and may or may not work. This category should NOT be used for items that the seller did test, but were found to be non-operational. It should also NOT be used for equipment with obvious physical defects which would make the unit not fit for use.
  • Tested to power on only – this means the seller was able to apply power to the unit and it did something. Perhaps the seller lacked cabling or test equipment to perform further tests. Any observed behavior (patterns and colors of indicator lights, fans turning / not turning, unusual beeps or other noises, etc.) should be described in detail. Like the above category, it should not have any of the defects noted by NOT.
  • Tested, found defective – this means that the seller was able to perform further testing and determined that there was indeed a problem with the unit. The seller should clearly state the nature of the defect (to whatever extent they investigated), such as “no console output”, “Status light solid red”, “displays fatal error message”, and so forth. Again, any physical defects would bump this to a lower category.
  • Tested, found defective, investigated in depth – in this category, the seller has somewhat more knowledge of the device and has done further investigation. There might be concealed damage or the seller might have disassembled the unit to investigate further. Essential components may have been found to be missing. Any results of the investigation should be included in the listing, and the seller should return the unit to the condition as found (re-installing all components, including case screws, etc.) or note in the listing why this was not done.
  • Physical damage, repairable – the device has some sort of physical damage which renders it partially or completely unusable, such as damaged connectors, bent or broken components, etc. The damage should be described as completely as possible, preferably with good quality photographs of the damaged areas. Buyers should evaluate the usability of the device without using the damaged areas or their ability to repair the damage. Note that modern electronic equipment often uses surface-mount components on multi-layer circuit boards, meaning that the skills and equipment needed to perform the repairs are beyond the reach of most users.
  • Physical damage, non-repairable – the device has obvious physical damage which would prevent it from being repaired or being usable as a complete unit. Sometimes it may be possible to salvage components from the device (power supplies, faceplate, memory, etc.). The damage should be described as completely as possible, preferably with good quality photographs of the damaged and un-damaged components.

Now, I’d like to provide a few examples of actual listings that I’ve purchased, and what I’ve found. I am not naming any sellers here, since it is possible that they received the item from somewhere “up the food chain” and did not investigate it completely.

  1. Catalyst WS-C4948-10GE switch – Listing simply said “Being sold AS IS for Parts or Not working. Power on but no console. No return, No refunds. AS IS!!!“. The listing also included pictures of the device, including one which showed the status LED being red.

    When I received this unit, the first thing I did was open it up to make sure there were no loose parts inside. During this inspection I discovered that 12 of the 14 screws that hold the cover on were missing and that the memory battery backup battery had been ripped off the main board (and was nowhere to be found inside the chassis). I also found that all of the screws holding the main board to the chassis were loose (but at least they were all present). Based on this, I determined that someone had been inside the unit already and had diagnosed it at least as far as removing the main board.

    I contacted the seller and they said they received it that way from the company that was using it, and the company ripped the battery off to erase the config because they were “security conscious”.

    Soldering in a new battery was not sufficient to get the switch working. I suspected the problem might be due to defective memory components soldered onto the main board, as described in this Cisco Field Notice. I ordered a tray of memory chips from a specialist in obsolete components (they are long-discontinued DDR333 parts) and replaced the two chips on the underside of the board. Since the ones on the bottom were made by Micron and the 3 on the top were from Samsung, I guessed (correctly, as it turned out) that the fault was in the Micron ones.

    After reinstalling the main board in the chassis and powering the switch up, I was greeted with the normal startup messages on the console*. After enabling priv mode in ROMMON, I tested the memory for an hour or so and it passed without errors. I then updated the ROMMON and IOS to the latest versions and gave the switch a 72-hour burn-in test, which it passed. Not bad for $255 plus another $10 in replacement memory chips and an hour or so’s work.

    * To my amusement, it appears that the battery on this switch is only used to maintain the date/time, not power the configuration memory. When the switch booted up after I repaired it, it put up a full-page banner with dire warnings about accessing the network without authorization, part of the saved config file that it had retained the whole while.

  2. More items to be added as I purchase them.

IPv4Scan.com – scan or scam?

One of my occasional consulting customers called me in a panic because all of their HP printers printed out the same page at the same time:

GET http://ipv4scan.com/hello/check.txt HTTP/1.1
Host: ipv4scan.com
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: IPv4Scan (+http://ipv4scan.com)

Now, I have nothing against most network measurement bots. Most are useful, and the rest are usually well-intentioned, even if they are counterproductive. The one thing these have in common is that they have a page that tells you what they’re doing, why they’re doing it, and who to contact if you have further questions.

The http://IPv4Scan.com page does none of those:

Screen capture

There is no contact information provided on the page, there is no statement of how the data is being used (other than that it is “not for sale, rental or release”). The web page source does not contain any useful contact information, either. So they’re collecting this data for their own, unspecified, purposes.

Ok, maybe it is legit, just with a spectacularly bad public relations campaign. Let’s look and see who is behind this:

(0:115) host:~terry# jwhois ipv4scan.com
[whois.internet.bs]
Domain Name: IPV4SCAN.COM
Registry Domain ID: 1824307886_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.internet.bs
Registrar URL: http://www.internetbs.net
Updated Date: 2013-08-30T10:37:11Z
Creation Date: 2013-08-30T10:21:44Z
Registrar Registration Expiration Date: 2014-08-30T10:21:44Z
Registrar: Internet.bs Corp.
Registrar IANA ID: 814
Registrar Abuse Contact Email: abuse@internet.bs
Registrar Abuse Contact Phone:
Reseller:
Domain Status: clientTransferProhibited
Registry Registrant ID:
Registrant Name: Domain Administrator
Registrant Organization: Fundacion Private Whois
Registrant Street: Attn: ipv4scan.com, Aptds. 0850-00056
Registrant City: Panama
Registrant State/Province:
Registrant Postal Code: Zona 15
Registrant Country: PA
Registrant Phone: +507.65967959
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email: 5230a6158jiing35@5225b4d0pi3627q9.privatewhois.net
Registry Admin ID:
Admin Name: Domain Administrator
Admin Organization: Fundacion Private Whois
Admin Street: Attn: ipv4scan.com, Aptds. 0850-00056
Admin City: Panama
Admin State/Province:
Admin Postal Code: Zona 15
Admin Country: PA
Admin Phone: +507.65967959
Admin Phone Ext:
Admin Fax:
Admin Fax Ext:
Admin Email: 5230a6157t3qutyb@5225b4d0pi3627q9.privatewhois.net
Registry Tech ID:
Tech Name: Domain Administrator
Tech Organization: Fundacion Private Whois
Tech Street: Attn: ipv4scan.com, Aptds. 0850-00056
Tech City: Panama
Tech State/Province:
Tech Postal Code: Zona 15
Tech Country: PA
Tech Phone: +507.65967959
Tech Phone Ext:
Tech Fax:
Tech Fax Ext:
Tech Email: 5230a615n285uy95@5225b4d0pi3627q9.privatewhois.net
Name Server: ns-canada.topdns.com
Name Server: ns-usa.topdns.com
Name Server: ns-uk.topdns.com
DNSSEC: unsigned
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2014-04-29T05:00:41Z <<<

Ok, so they're hiding behind a privacy service, but seem to be located in Panama. Let's see if the IP address they're using matches:

(0:116) host:~terry# host ipv4scan.com
ipv4scan.com has address 93.174.93.51
ipv4scan.com mail is handled by 5 smtp09.topdns.com.
ipv4scan.com mail is handled by 5 smtp01.topdns.com.
(0:117) host:~terry# jwhois 93.174.93.51
[whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.

% Information related to '93.174.93.0 - 93.174.93.255'

% Abuse contact for '93.174.93.0 - 93.174.93.255' is 'admin@ecatel.net'

inetnum: 93.174.93.0 - 93.174.93.255
netname: NL-ECATEL
descr: ECATEL LTD
descr: Dedicated servers
descr: http://www.ecatel.net/
country: NL
admin-c: EL25-RIPE
tech-c: EL25-RIPE
status: ASSIGNED PA
mnt-by: ECATEL-MNT
mnt-lower: ECATEL-MNT
mnt-routes: ECATEL-MNT
source: RIPE # Filtered

role: Ecatel LTD
address: P.O.Box 19533
address: 2521 CA The Hague
address: Netherlands
abuse-mailbox: abuse@ecatel.info
remarks: ----------------------------------------------------
remarks: ECATEL LTD
remarks: Dedicated and Co-location hosting services
remarks: ----------------------------------------------------
remarks: for abuse complaints : abuse@ecatel.info
remarks: for any other questions : info@ecatel.info
remarks: ----------------------------------------------------
admin-c: EL25-RIPE
tech-c: EL25-RIPE
nic-hdl: EL25-RIPE
mnt-by: ECATEL-MNT
source: RIPE # Filtered

% Information related to '93.174.88.0/21AS29073'

route: 93.174.88.0/21
descr: AS29073, Route object
origin: AS29073
mnt-by: ECATEL-MNT
source: RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.72 (DBC-WHOIS3)

So, they're using an IP address allocated to Ecatel in the Netherlands. Not exactly close to Panama, is it? Let's see if that address is actually in the Netherlands:

(0:118) host:~terry# traceroute ipv4scan.com
traceroute to ipv4scan.com (93.174.93.51), 64 hops max, 52 byte packets
[snip]
8 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 20.530 ms
be2097.ccr22.bos01.atlas.cogentco.com (154.54.30.118) 19.664 ms
be2095.ccr21.bos01.atlas.cogentco.com (154.54.30.38) 20.657 ms
9 be2387.ccr22.lpl01.atlas.cogentco.com (154.54.44.166) 85.582 ms 85.667 ms
be2386.ccr21.lpl01.atlas.cogentco.com (154.54.44.162) 85.388 ms
10 be2183.ccr42.ams03.atlas.cogentco.com (154.54.58.70) 95.882 ms
be2182.ccr41.ams03.atlas.cogentco.com (154.54.77.245) 95.035 ms
be2183.ccr42.ams03.atlas.cogentco.com (154.54.58.70) 97.517 ms
11 be2311.ccr21.ams04.atlas.cogentco.com (154.54.74.90) 130.510 ms
be2312.ccr21.ams04.atlas.cogentco.com (154.54.74.94) 94.574 ms
be2311.ccr21.ams04.atlas.cogentco.com (154.54.74.90) 101.849 ms
12 149.11.38.179 (149.11.38.179) 101.548 ms 118.302 ms 102.141 ms
13 server.anonymous-hosting-service.com (93.174.93.51) 98.234 ms 97.335 ms 96.958 ms

Ok, the server is in Amsterdam, Netherlands. But hiding behind anonymous-hosting-service.com seems suspicious. Let's see where they are:

(0:119) host:~terry# jwhois anonymous-hosting-service.com
[Querying whois.verisign-grs.com]
[Redirected to whois.onlinenic.com]
[Querying whois.onlinenic.com]
[whois.onlinenic.com]

Domain Name: anonymous-hosting-service.com
Registry Domain ID:
Registrar WHOIS Server: whois.onlinenic.com
Registrar URL: http://www.onlinenic.com
Updated Date: 2014-04-06 03:14:38
Creation Date: 2009-09-08
Registrar Registration Expiration Date: 2015-09-08
Registrar: Onlinenic Inc
Registrar IANA ID: 82
Registrar Abuse Contact Email: onlinenic-enduser@onlinenic.com
Registrar Abuse Contact Phone: +1.5107698492
Domain Status: clientTransferProhibited
Registry Registrant ID:
Registrant Name: Laura Yun
Registrant Organization: Vindo International Ltd.
Registrant Street: Oliaji TradeCenter - 1st floor
Registrant City: Victoria
Registrant State/Province: Mahe
Registrant Postal Code: 5567
Registrant Country: SC
Registrant Phone: +248.6629012
Registrant Phone Ext:
Registrant Fax: +248.24822575500
Registrant Fax Ext:
Registrant Email: anonymous.client@vindohosting.com
Registry Admin ID:
Admin Name: Laura Yun
Admin Organization: Vindo International Ltd.
Admin Street: Oliaji TradeCenter - 1st floor
Admin City: Victoria
Admin State/Province: Mahe
Admin Postal Code: 5567
Admin Country: SC
Admin Phone: +248.6629012
Admin Phone Ext:
Admin Fax: +248.24822575500
Admin Fax Ext:
Admin Email: anonymous.client@vindohosting.com
Registry Tech ID:
Tech Name: Laura Yun
Tech Organization: Vindo International Ltd.
Tech Street: Oliaji TradeCenter - 1st floor
Tech City: Victoria
Tech State/Province: Mahe
Tech Postal Code: 5567
Tech Country: SC
Tech Phone: +248.6629012
Tech Phone Ext:
Tech Fax: +248.24822575500
Tech Fax Ext:
Tech Email: anonymous.client@vindohosting.com
Name Server: ns1.anonymous-hosting-service.com
Name Server: ns2.anonymous-hosting-service.com
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2014-04-06 03:14:38 <<<

Well, this is definitely fishy. No legitimate survey would be hiding behind so many levels of indirection.

I used the site's form to "opt out" 0.0.0.0/1 with an email address requesting they contact me about their project. I've also sent email to the abuse contacts shown above, pointing them to this blog entry, in the hope that they can get some sort of explanation from their customer.

In the meantime, you may want to fine-tune your firewall rules to prevent this type of probe. That would (at a minimum) include blocking all outside connection attempts on ports 80 (http) and 443 (https) to anything on your network that is not intended to be a public web server. I cannot recommend using their opt-out form as there is no indication of what they do with the information. For all I know, it has the same effect as sending "unsubscribe" in response to a spam email - it just targets you for more spam.

If I receive any information from my inquiries, I'll update this blog entry accordingly.