Troubleshooting Catalyst 4948-10GE red status LED and no console output

This is not intended as a complete DIY. It requires equipment most of my readers won’t have, such as a hot air PCB rework station with magnifier. I am posting it to give you an idea of what is involved in the repair of these devices, and to provide info to any readers who do have the necessary equipment and just need to know the repair procedure.

I have been encountering more and more dead Catalyst 4948-10GE switches lately. These usually have a solid red Status LED and do not display any messages on the console when power is applied. This means that the switch did not make any progress at all in booting (one of the first steps in the boot process is to change the Status LED from red to orange). Catalyst 4948-10GE switches with this type of fault are frequently listed on eBay in the $250-$350 price range (usually marked “For parts or not working”). When troubleshooting these, the problem is often defective memory. Unfortunately, this memory is soldered to the circuit board in the switch, so it isn’t simply a matter of removing a faulty memory module and replacing it with a known good one. The old memory needs to be de-soldered and new memory soldered in, and you need to have a source for the obsolete memory chips needed for replacements.

These switches have 256MB of ECC memory, implemented via 5 32MB x 16-bit memory chips such as the Micron MT46V32M16-6T F. Three of the chips are located on the top side of the motherboard next to the power supply, and another two are located on the underside of the board (all images in this post can be clicked for a larger version):

Top side of board

Top side of board

Bottom side of board

Bottom side of board

In each of the boards I have repaired, the fault has always been in one of the bottom two chips. This makes sense as there is no airflow across the bottom of the board, so those chips are more likely to overheat than the ones on the top of the board. Cisco has announced an issue with an unspecified memory supplier (often rumored to be Micron), and the Catalyst 49xx family is on that list. However, the switches that I am seeing failures on are not on a Cisco support contract, and I haven’t read anything about Cisco fixing equipment not on a support contract for free, so I’ve been repairing them myself.

The first step is to remove the two existing memory chips from the underside of the board and clean and prepare the board for the new chips:

Memory removed

The next step is to solder the replacement chips into place:

New memory installed

Of course, you need to ensure that the chips are installed in the correct orientation (of course!) and that all pins are soldered to their respective pads (66 pins per chip) and that there are no shorts between pins. You also need to avoid damaging any of the neighboring components or the circuit board itself while doing this.

If all goes well, when you reinstall the board in the chassis and apply power, you will be greeted with the appropriate console messages and the switch will boot up normally. If not, remove the board and examine the area around the replaced chips under a magnifier to double check for bad connections or solder bridges.

2.5″ enterprise hard drives and power savings

I admit it – I used to have an unwarranted prejudice against 2.5″ enterprise hard drives, considering them “toy” drives, or at best suited for notebook use, or non-critical use in enterprise systems. I used WD Velociraptor drives on my Dell desktops (before I upgraded to SSDs), but that particular model was discontinued, and the WD web site has this discouraging note about the current models: “Models WD1000CHTZ, WD5000BHTZ and WD2500BHTZ are available on a build to order basis, contact your WD Sales representative for more information.” which I interpreted as “people aren’t buying these, but if you want a bazillion of ’em, we’ll restart the production line”. I also used WD 2.5″ drives as the operating system volume on my RAIDzilla II file servers, but the actual data volumes were built with 16 x 2TB 3.5″ drives.

However, in an attempt to reduce power consumption here, I decided to test 2.5″ enterprise drives as a replacement for identical-capacity 3.5″ drives, and the results were surprising (to me, at least). I upgraded one of my Dell PowerEdge R710 systems (gate.glaver.org, the system that is serving this web page that you’re reading) from 6 x 146GB 15K RPM 3.5″ SAS drives (ST3300657SS-H*) to 6 x 146GB 15K RPM 2.5″ SAS drives (ST9146852SS). All other components remained the same*. The drives are in a 5-drive RAID5 array controlled by a Dell PERC H700 controller, with the 6th drive being a dedicated hot spare.

Power consumption on this busy system dropped from 237W to 204W and became much more even (apparently, seeking on the 3.5″ drives consumes much more power than on the 2.5″ drives):

gate.glaver.org power consumption

Click the picture for a larger view

The PowerEdge R710 is already a pretty efficient system – this particular box has 2 x X5680 6-core Xeon CPUs, 48GB of registered ECC RAM, hardware RAID controller, etc.

Even more surprising was the discovery that disk I/O was still very fast, at well over 600MByte/sec:

(0:1) gate:~terry# dd if=/dev/mfid0 of=/dev/null bs=1m count=102400
102400+0 records in
102400+0 records out
107374182400 bytes transferred in 171.422439 secs (626371804 bytes/sec)

Based on this, I will certainly give serious consideration to using 2.5″ drives in future builds.

Seagate has announced 2.5″ enterprise drives with up to 2TB capacity (in both SAS and SATA variants). While that is lagging behind the announced capacity for 3.5″ drives (8TB at this time), you can fit a lot more 2.5″ drives in a given chassis. I expect to use one or two additional drive generations in my existing RAIDzilla II chassis (upgrading to 4TB drives at some point, and then in the future to 8TB or 10TB drives). After that, it will be time to design the RAIDzilla III.

* Yes, I know this is normally a 300GB drive. Seagate didn’t make a native 146GB drive in the Cheetah 15K.7 family, and the -H suffix indicates a half-capacity drive for OEMs who needed to match existing drive capacities.

* This is not a particularly easy conversion, as the Dell chassis for the R710 is not modular. However, various sellers on eBay are selling new or used 2.5″ chassis (part number 33P6Y). You can move just about all of the old components from the 3.5″ chassis over – the only item you will need (other than the actual 2.5″ drives and trays) are the appropriate cables from the drive backplane to the RAID controller. For a PERC H700, that is 2 x R145M mini-SAS cables.

A few more words of advice for used equipment sellers

Today I’m going to expand on the advice I provided in my earlier post, “A few words of advice for used equipment sellers“. Today I’m going to address the issues with “As-Is / Not Working / For Parts Only” listing types. These are terms used by eBay, but this advice also applies to anyone else selling equipment in this category.

In general, this type of item is offered by sellers at a lower price in the hope of recovering some money from a piece of equipment that is either not operating properly or is not able to be tested by the seller. Some sellers are very scrupulous about describing the equipment, providing lots of pictures and as much information as they know about the item. At the other end are sellers who use a stock photograph and product description, perhaps with some words like “Couldn’t power on – didn’t test.”

Any buyer who purchases items in this category is hoping to find a bargain by ending up with a piece of working equipment after performing minimal repairs. [There are probably people who buy this material for other purposes, such as scrap metal recovery, components for artwork, and so on, but I’ll leave those out of this discussion.] As such, you (as the seller) want to provide as much information as possible to potential buyers so you both end up with a good experience.

There are quite a few categories of “untested / not working”, and I’ll go through these from best to worst:

  • Unable to test / Not tested – this means that the seller lacks the ability to test the item, either because it is a sub-component of a larger device the seller does not have, lack of necessary cabling to connect it, or due to it requiring specialized test / calibration equipment. Items in this category are truly untested and may or may not work. This category should NOT be used for items that the seller did test, but were found to be non-operational. It should also NOT be used for equipment with obvious physical defects which would make the unit not fit for use.
  • Tested to power on only – this means the seller was able to apply power to the unit and it did something. Perhaps the seller lacked cabling or test equipment to perform further tests. Any observed behavior (patterns and colors of indicator lights, fans turning / not turning, unusual beeps or other noises, etc.) should be described in detail. Like the above category, it should not have any of the defects noted by NOT.
  • Tested, found defective – this means that the seller was able to perform further testing and determined that there was indeed a problem with the unit. The seller should clearly state the nature of the defect (to whatever extent they investigated), such as “no console output”, “Status light solid red”, “displays fatal error message”, and so forth. Again, any physical defects would bump this to a lower category.
  • Tested, found defective, investigated in depth – in this category, the seller has somewhat more knowledge of the device and has done further investigation. There might be concealed damage or the seller might have disassembled the unit to investigate further. Essential components may have been found to be missing. Any results of the investigation should be included in the listing, and the seller should return the unit to the condition as found (re-installing all components, including case screws, etc.) or note in the listing why this was not done.
  • Physical damage, repairable – the device has some sort of physical damage which renders it partially or completely unusable, such as damaged connectors, bent or broken components, etc. The damage should be described as completely as possible, preferably with good quality photographs of the damaged areas. Buyers should evaluate the usability of the device without using the damaged areas or their ability to repair the damage. Note that modern electronic equipment often uses surface-mount components on multi-layer circuit boards, meaning that the skills and equipment needed to perform the repairs are beyond the reach of most users.
  • Physical damage, non-repairable – the device has obvious physical damage which would prevent it from being repaired or being usable as a complete unit. Sometimes it may be possible to salvage components from the device (power supplies, faceplate, memory, etc.). The damage should be described as completely as possible, preferably with good quality photographs of the damaged and un-damaged components.

Now, I’d like to provide a few examples of actual listings that I’ve purchased, and what I’ve found. I am not naming any sellers here, since it is possible that they received the item from somewhere “up the food chain” and did not investigate it completely.

  1. Catalyst WS-C4948-10GE switch – Listing simply said “Being sold AS IS for Parts or Not working. Power on but no console. No return, No refunds. AS IS!!!“. The listing also included pictures of the device, including one which showed the status LED being red.

    When I received this unit, the first thing I did was open it up to make sure there were no loose parts inside. During this inspection I discovered that 12 of the 14 screws that hold the cover on were missing and that the memory battery backup battery had been ripped off the main board (and was nowhere to be found inside the chassis). I also found that all of the screws holding the main board to the chassis were loose (but at least they were all present). Based on this, I determined that someone had been inside the unit already and had diagnosed it at least as far as removing the main board.

    I contacted the seller and they said they received it that way from the company that was using it, and the company ripped the battery off to erase the config because they were “security conscious”.

    Soldering in a new battery was not sufficient to get the switch working. I suspected the problem might be due to defective memory components soldered onto the main board, as described in this Cisco Field Notice. I ordered a tray of memory chips from a specialist in obsolete components (they are long-discontinued DDR333 parts) and replaced the two chips on the underside of the board. Since the ones on the bottom were made by Micron and the 3 on the top were from Samsung, I guessed (correctly, as it turned out) that the fault was in the Micron ones.

    After reinstalling the main board in the chassis and powering the switch up, I was greeted with the normal startup messages on the console*. After enabling priv mode in ROMMON, I tested the memory for an hour or so and it passed without errors. I then updated the ROMMON and IOS to the latest versions and gave the switch a 72-hour burn-in test, which it passed. Not bad for $255 plus another $10 in replacement memory chips and an hour or so’s work.

    * To my amusement, it appears that the battery on this switch is only used to maintain the date/time, not power the configuration memory. When the switch booted up after I repaired it, it put up a full-page banner with dire warnings about accessing the network without authorization, part of the saved config file that it had retained the whole while.

  2. More items to be added as I purchase them.

Ghost Like Sun – The IUMA Demos (CD)

Update: The second Ghost Like Sun album, Human Satellite, is now available right here, on this blog.

A long time ago, at the very dawn of the web, the Internet Underground Music Archive (IUMA) was born. It was designed to allow unsigned artists to post their music for listening, downloading, and comments. Rather like soundcloud.com is today. As time went on, IUMA was purchased by a succession of companies, eventually vanishing in 2006.

I happened to click on a random artist while browsing that early web on a VAXstation 3100 using NCSA Mosaic and happened to like what I heard. That band was Ghost Like Sun (Internet Archive copy). I ended up having an extended correspondence with the guitarist, Leigh Newsome, and bought their first (and only) album, Loud as Light. There was going to be a second album, but as far as I know, it was never released. Eventually the Ghost Like Sun web site followed IUMA into the digital dustbin. The domain is now parked in Japan.

When listiening to Loud as Light in the car this past weekend, I said to myself “I wonder what ever happened to the IUMA demos I downloaded all those years ago”. Well, this is what happened to them:

Z:\Terry\Music>dir *.mp2
Volume in drive Z is data
Volume Serial Number is 10B3-C748

Directory of Z:\Terry\Music

10/30/1997 06:13 AM 7,105,100 victoria.mp2
05/03/1997 12:42 AM 5,229,924 Sign_Of_One.mp2
05/03/1997 12:53 AM 3,353,952 Sorrow.mp2
05/03/1997 12:50 AM 9,309,920 The_Wheel.mp2
4 File(s) 24,998,896 bytes
0 Dir(s) 6,679,371,872,256 bytes free

They’re in MP2 format, which is as ancient as the web itself. I visited the Internet Archive’s IUMA collection, but the songs had been converted to low-resolution MP3 files and “Sorrow” was missing completely. I decided to convert my copies into a modern format, and via a number of conversion programs they were re-sampled, gain-adjusted, and burned to an audio CD, which I am making available for your listening pleasure here (150MB Zip file containing .bin/.cue suitable for use with burning utilities like ImgBurn).

Of course, if you don’t care about having these on high-quality CD, you can get 3 of the 4 songs from the Internet Archive (link above).

Note: As these are simply converted versions of material freely available at the Internet Archive, I don’t see any issues with making them available here. If anyone from Ghost Like Sun objects, simply drop me a line (see the “Contact Info” in the “LINKS” section to the right of this post). Of course, that means I’m going to ask you where the second album went, etc.

IPv4Scan.com – scan or scam?

One of my occasional consulting customers called me in a panic because all of their HP printers printed out the same page at the same time:

GET http://ipv4scan.com/hello/check.txt HTTP/1.1
Host: ipv4scan.com
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: IPv4Scan (+http://ipv4scan.com)

Now, I have nothing against most network measurement bots. Most are useful, and the rest are usually well-intentioned, even if they are counterproductive. The one thing these have in common is that they have a page that tells you what they’re doing, why they’re doing it, and who to contact if you have further questions.

The http://IPv4Scan.com page does none of those:

Screen capture

There is no contact information provided on the page, there is no statement of how the data is being used (other than that it is “not for sale, rental or release”). The web page source does not contain any useful contact information, either. So they’re collecting this data for their own, unspecified, purposes.

Ok, maybe it is legit, just with a spectacularly bad public relations campaign. Let’s look and see who is behind this:

(0:115) host:~terry# jwhois ipv4scan.com
[whois.internet.bs]
Domain Name: IPV4SCAN.COM
Registry Domain ID: 1824307886_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.internet.bs
Registrar URL: http://www.internetbs.net
Updated Date: 2013-08-30T10:37:11Z
Creation Date: 2013-08-30T10:21:44Z
Registrar Registration Expiration Date: 2014-08-30T10:21:44Z
Registrar: Internet.bs Corp.
Registrar IANA ID: 814
Registrar Abuse Contact Email: abuse@internet.bs
Registrar Abuse Contact Phone:
Reseller:
Domain Status: clientTransferProhibited
Registry Registrant ID:
Registrant Name: Domain Administrator
Registrant Organization: Fundacion Private Whois
Registrant Street: Attn: ipv4scan.com, Aptds. 0850-00056
Registrant City: Panama
Registrant State/Province:
Registrant Postal Code: Zona 15
Registrant Country: PA
Registrant Phone: +507.65967959
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email: 5230a6158jiing35@5225b4d0pi3627q9.privatewhois.net
Registry Admin ID:
Admin Name: Domain Administrator
Admin Organization: Fundacion Private Whois
Admin Street: Attn: ipv4scan.com, Aptds. 0850-00056
Admin City: Panama
Admin State/Province:
Admin Postal Code: Zona 15
Admin Country: PA
Admin Phone: +507.65967959
Admin Phone Ext:
Admin Fax:
Admin Fax Ext:
Admin Email: 5230a6157t3qutyb@5225b4d0pi3627q9.privatewhois.net
Registry Tech ID:
Tech Name: Domain Administrator
Tech Organization: Fundacion Private Whois
Tech Street: Attn: ipv4scan.com, Aptds. 0850-00056
Tech City: Panama
Tech State/Province:
Tech Postal Code: Zona 15
Tech Country: PA
Tech Phone: +507.65967959
Tech Phone Ext:
Tech Fax:
Tech Fax Ext:
Tech Email: 5230a615n285uy95@5225b4d0pi3627q9.privatewhois.net
Name Server: ns-canada.topdns.com
Name Server: ns-usa.topdns.com
Name Server: ns-uk.topdns.com
DNSSEC: unsigned
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2014-04-29T05:00:41Z <<<

Ok, so they're hiding behind a privacy service, but seem to be located in Panama. Let's see if the IP address they're using matches:

(0:116) host:~terry# host ipv4scan.com
ipv4scan.com has address 93.174.93.51
ipv4scan.com mail is handled by 5 smtp09.topdns.com.
ipv4scan.com mail is handled by 5 smtp01.topdns.com.
(0:117) host:~terry# jwhois 93.174.93.51
[whois.ripe.net]
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.

% Information related to '93.174.93.0 - 93.174.93.255'

% Abuse contact for '93.174.93.0 - 93.174.93.255' is 'admin@ecatel.net'

inetnum: 93.174.93.0 - 93.174.93.255
netname: NL-ECATEL
descr: ECATEL LTD
descr: Dedicated servers
descr: http://www.ecatel.net/
country: NL
admin-c: EL25-RIPE
tech-c: EL25-RIPE
status: ASSIGNED PA
mnt-by: ECATEL-MNT
mnt-lower: ECATEL-MNT
mnt-routes: ECATEL-MNT
source: RIPE # Filtered

role: Ecatel LTD
address: P.O.Box 19533
address: 2521 CA The Hague
address: Netherlands
abuse-mailbox: abuse@ecatel.info
remarks: ----------------------------------------------------
remarks: ECATEL LTD
remarks: Dedicated and Co-location hosting services
remarks: ----------------------------------------------------
remarks: for abuse complaints : abuse@ecatel.info
remarks: for any other questions : info@ecatel.info
remarks: ----------------------------------------------------
admin-c: EL25-RIPE
tech-c: EL25-RIPE
nic-hdl: EL25-RIPE
mnt-by: ECATEL-MNT
source: RIPE # Filtered

% Information related to '93.174.88.0/21AS29073'

route: 93.174.88.0/21
descr: AS29073, Route object
origin: AS29073
mnt-by: ECATEL-MNT
source: RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.72 (DBC-WHOIS3)

So, they're using an IP address allocated to Ecatel in the Netherlands. Not exactly close to Panama, is it? Let's see if that address is actually in the Netherlands:

(0:118) host:~terry# traceroute ipv4scan.com
traceroute to ipv4scan.com (93.174.93.51), 64 hops max, 52 byte packets
[snip]
8 be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14) 20.530 ms
be2097.ccr22.bos01.atlas.cogentco.com (154.54.30.118) 19.664 ms
be2095.ccr21.bos01.atlas.cogentco.com (154.54.30.38) 20.657 ms
9 be2387.ccr22.lpl01.atlas.cogentco.com (154.54.44.166) 85.582 ms 85.667 ms
be2386.ccr21.lpl01.atlas.cogentco.com (154.54.44.162) 85.388 ms
10 be2183.ccr42.ams03.atlas.cogentco.com (154.54.58.70) 95.882 ms
be2182.ccr41.ams03.atlas.cogentco.com (154.54.77.245) 95.035 ms
be2183.ccr42.ams03.atlas.cogentco.com (154.54.58.70) 97.517 ms
11 be2311.ccr21.ams04.atlas.cogentco.com (154.54.74.90) 130.510 ms
be2312.ccr21.ams04.atlas.cogentco.com (154.54.74.94) 94.574 ms
be2311.ccr21.ams04.atlas.cogentco.com (154.54.74.90) 101.849 ms
12 149.11.38.179 (149.11.38.179) 101.548 ms 118.302 ms 102.141 ms
13 server.anonymous-hosting-service.com (93.174.93.51) 98.234 ms 97.335 ms 96.958 ms

Ok, the server is in Amsterdam, Netherlands. But hiding behind anonymous-hosting-service.com seems suspicious. Let's see where they are:

(0:119) host:~terry# jwhois anonymous-hosting-service.com
[Querying whois.verisign-grs.com]
[Redirected to whois.onlinenic.com]
[Querying whois.onlinenic.com]
[whois.onlinenic.com]

Domain Name: anonymous-hosting-service.com
Registry Domain ID:
Registrar WHOIS Server: whois.onlinenic.com
Registrar URL: http://www.onlinenic.com
Updated Date: 2014-04-06 03:14:38
Creation Date: 2009-09-08
Registrar Registration Expiration Date: 2015-09-08
Registrar: Onlinenic Inc
Registrar IANA ID: 82
Registrar Abuse Contact Email: onlinenic-enduser@onlinenic.com
Registrar Abuse Contact Phone: +1.5107698492
Domain Status: clientTransferProhibited
Registry Registrant ID:
Registrant Name: Laura Yun
Registrant Organization: Vindo International Ltd.
Registrant Street: Oliaji TradeCenter - 1st floor
Registrant City: Victoria
Registrant State/Province: Mahe
Registrant Postal Code: 5567
Registrant Country: SC
Registrant Phone: +248.6629012
Registrant Phone Ext:
Registrant Fax: +248.24822575500
Registrant Fax Ext:
Registrant Email: anonymous.client@vindohosting.com
Registry Admin ID:
Admin Name: Laura Yun
Admin Organization: Vindo International Ltd.
Admin Street: Oliaji TradeCenter - 1st floor
Admin City: Victoria
Admin State/Province: Mahe
Admin Postal Code: 5567
Admin Country: SC
Admin Phone: +248.6629012
Admin Phone Ext:
Admin Fax: +248.24822575500
Admin Fax Ext:
Admin Email: anonymous.client@vindohosting.com
Registry Tech ID:
Tech Name: Laura Yun
Tech Organization: Vindo International Ltd.
Tech Street: Oliaji TradeCenter - 1st floor
Tech City: Victoria
Tech State/Province: Mahe
Tech Postal Code: 5567
Tech Country: SC
Tech Phone: +248.6629012
Tech Phone Ext:
Tech Fax: +248.24822575500
Tech Fax Ext:
Tech Email: anonymous.client@vindohosting.com
Name Server: ns1.anonymous-hosting-service.com
Name Server: ns2.anonymous-hosting-service.com
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2014-04-06 03:14:38 <<<

Well, this is definitely fishy. No legitimate survey would be hiding behind so many levels of indirection.

I used the site's form to "opt out" 0.0.0.0/1 with an email address requesting they contact me about their project. I've also sent email to the abuse contacts shown above, pointing them to this blog entry, in the hope that they can get some sort of explanation from their customer.

In the meantime, you may want to fine-tune your firewall rules to prevent this type of probe. That would (at a minimum) include blocking all outside connection attempts on ports 80 (http) and 443 (https) to anything on your network that is not intended to be a public web server. I cannot recommend using their opt-out form as there is no indication of what they do with the information. For all I know, it has the same effect as sending "unsubscribe" in response to a spam email - it just targets you for more spam.

If I receive any information from my inquiries, I'll update this blog entry accordingly.

Does your bank care about online security? Mine (Citibank) doesn’t…

Updated March 4th, 2021 to change some long-dead links to copies at the Internet Archive
Updated July 16th, 2013 to document further idiocy – see the bottom of this post.

Today provided yet another indication that Citibank (and by extension, MasterCard) have absolutely no clue about online security, and past events have shown that they simply don’t care.

As background, I’m sure you remember all the warnings your bank / credit card company gave you about never giving out information to unknown entities, to always make sure that the name of the bank / credit card company is in the URL, and so forth. It sure would be nice if they’d take their own advice…

Today’s experience was triggered by an order on newegg.com. After clicking on the “confirm order” button, I was told that I might be redirected to my bank’s web site to confirm the order. So far so good – I’ve had experiences in the past where every single Newegg order caused my card to be flagged for fraud. But then I was greeted with a web page claiming to be “MasterCard SecureCode”, but with a URL showing “securesuite.net”, which demanded a bunch of sensitive info, including the last 4 digits of my SSN and my billing Zip Code. What the heck? Looks like an obvious phishing site. I let the page sit there while I contacted Citibank MasterCard. The agent said that it was obviously a fake and that I should never enter any info into an online form like that (a statement I strongly agree with). I clicked the “cancel” button and figured that I’d just place my order somewhere else. However, Newegg told me my order had been placed successfully and subsequently sent me an email letting me know that my credit card had been charged.

I then decided to investigate what this “securesuite.net” site was. There aren’t many useful search engine hits, but there is history going back at least seven years, all of which points out the confusing nature of that site. For example:

For an actual scholarly paper about this problem, I suggest reading “Verified by Visa and MasterCard SecureCode: or, How Not to Design Authentication“.

If you browse to https://www.securesuite.net, you get (as of this writing) a blank page – it doesn’t even return any HTML headers. If by some chance you happen to find https://www.securesuite.net/csi/docs/contact_support.jsp, you’ll find a singularly uninformative page which contains such gems as “Call us at your Financial Institution’s support phone.” To be fair, that may just be a generic template page not intended to be shown to users.

The main point is that after telling us to never trust unknown web sites, the banks and credit card companies are sending people to just those sorts of sites. Talk about mixed messages!

Compounding this, if you do get a call from the Citibank Fraud Department, it will show up as “Unavailable” or “Private” in Caller ID. While it’s true that Caller ID is easily faked, I’d be more inclined to answer the phone if it didn’t look like a random telemarketing call. For added security, that automated call could simply say “This is a fraud warning about your Citi MasterCard ending in 1234. Please call the number on the back of your card immediately.”

This is not a new problem – I’ve been reporting Citibank’s own email to their anti-phishing department becase my mail server (correctly) flags it as fraudulent due to forged headers. In particular, they like to send out email with the subject “Important information regarding your statement”. It is actually just a canned solicitation to switch to online billing, not “Important information”. But Citibank doesn’t send it themselves – instead, they use companies called bigfootinteractive.com and epsiloninteractive.com. As I said in my unacknowledged complaints to Citibank, “Imagine you got an email claiming to be from the IRS entitled “Important information about your tax return”, where the email was sent from a Yahoo account through a GMail account to you. Wouldn’t you be suspicious? You’re doing the exact same thing with the mail you send out.”

These companies should require the use of their own domains and SSL certificates rather than apparently-unassociated third parties, or at least correct information when users call them and ask if the third-party site is legitimate.

It’s a sad day when I have to admit that PayPal does a much better job with this sort of thing than Citibank does.

This total disregard for security isn’t just in their online communications, either. Citibank started sending me unsolicited “balance transfer” checks in the mail again, despite my having gotten them to stop some years ago. I had to call yet again and have my account flagged to not receive them. I said to the phone rep “Who in this day and age thinks sending blank checks in the mail is a good idea?” and she agreed with me. She apparently gets lots of calls about this.

Update as of July 16th:

As I wrote yesterday, I canceled the “MasterCard SecureCode” window and Newegg apparently processed my order, notifying me that they’d received the order and later that it had been successfully charged to my credit card. That’s where things were at the time I wrote the above post.

Last night I received email from Newegg telling me that my order had shipped and tracking information was available, and that I could expect to receive the order on the 17th. That’s excellent service, considering that I had used the “free 4-5 day shipping” option. I figured everything was all set. Little did I know…

Today at 6:37 PM (note that this is at least 12 hours after my Newegg order shipped – talk about “locking the barn door…”) I get the usual “Unavailable” Caller ID phone call from the Citibank Fraud “Early” Warning Department, telling me that my card has been frozen and asking me to confirm that my Newegg purchase was legitimate (oddly, they had no problem with my Amazon purchase later that same day). I told the agent it was, and explained that I’d received the phony-looking SecureCode page and after contacting the same department she was calling me from, who told me it was bogus and to never provide information on that sort of suspicious page, I clicked “cancel”.

The agent proceeded to tell me how important the SecureCode was. She was unable or unwilling (perhaps due to the “script” they’re required to work from) to understand that her department was the one who told me to never provide that information. We went around in circles for about 10 minutes as I tried to get her to understand that, and also to get the point across that they are the ones who say to never provide information to an untrusted 3rd party.

It’s easy enough to dismiss this as “somebody else’s problem”, but the banks, card companies and merchants are covering the losses they incur due to their own stupidity by charging everybody a little more. So it’s everybody’s problem – I just wish the bank could see that it is a problem entirely of their own making.

The GEN IIv7 MOD-6_7971

Carl and Michael recently released the latest version of their MOD-6_7971 Nixie clock. This is pretty much the same hardware as before, but uses a higher-powered radio transmitter in the remote GPS receiver so it can be located further from the clock. There have been a huge number of software changes, however. To give you an idea, the version that shipped with the last batch of clocks was V07-09. The version on this batch is V07-53!

A lot of that was me pestering Carl for changes, but there’s also a lot of other neat new stuff in there.

One other new thing is that the clock now has a 67 page user manual written by me. If I may be permitted to brag a little bit (Ok, a lot!) I think this sets a new standard for Nixie clock documentation. You can read it (and the accompanying updating instructions – you can update any of the older clocks to this software) here:

User Manual
Updating Instructions

For more info or to express interest in ordering a kit or assembled clock from a future batch, visit the MOD-6 page at BadNixie.com.

MOD-6 Nixie clock

President Obama, you make me ashamed to be an American…

As my long-time readers know, I don’t make my views on politics, religion, etc. known. That’s because I’ve seen too many message boards (and friendships) torn apart by disagreements between people with different points of view. However, recent events compel me to speak up. As long as the comments remain civil, I’ll leave comments enabled on this post. If things get out of control, I may lock this post temporarily or permanently, or delete the whole thing. With that out of the way…

“They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” – Benjamin Franklin

I’m stunned by the revelation that the NSA is continuing to monitor all phone calls, collecting and storing for an indefinite length of time the phone numbers of both parties, the locations of both parties, and the starting time and length of each phone call. [Reference].

Even more troubling is President Obama’s statement that “In the abstract, you can complain about Big Brother or how this is a potential program run amok, but when you actually look at the details, then I think we’ve struck the right balance.” [Reference]

Apparently Congress was informed of this and yet not a single member appears to have raised any objection.

On the heels of that revelation, further information was published showing that a different data collection operation is in progress, sending everything from email to photos to VoIP phone calls, etc. to the NSA. [Reference]

Next, it was revealed that the NSA is also receiving details of credit card transactions. [Reference]

President Obama, I voted for you (twice!) because you campaigned on a platform saying you were different from President Bush and would repeal the “surveillance state” legislation passed as a knee-jerk reaction to 9/11. Not to mention your promise to “Close Guantanamo Bay”. Instead, this is what we get? You make me ashamed to be an American.

“People should not be afraid of their governments. Governments should be afraid of their people.” – V, “V for Vendetta”

A few words of advice for used equipment sellers

I purchase a good deal of used electronic equipment for both work and personal use. Some of that equipment comes from eBay, some is purchased from companies who sell used equipment for a living. The two aren’t mutually exclusive, of course – there are a number of commercial vendors who sell through eBay as well as their own site.

Used equipment can represent a sizable savings over new, particularly when a manufacturer only has a “list price” and doesn’t offer discounts to any but their largest customers. Of course, you need to consider the cost of any required re-licensing (for example, on Cisco gear) when comparing the used price with new. But a large number of manufacturers make updates available for free to all, and in that cases you can often save a great deal of money. Most used equipment will come with at least a one-week warranty against being defective, but some sellers will offer a longer warranty – up to 1 year is common.

One of the best times for great deals is just as a device is no longer being sold as new by the manufacturer. There’s a further drop once the manufacturer no longer supports it with software updates, spare parts, and so on – but you probably don’t want to buy something that far along unless you plan to use it for spare parts yourself.

That’s the benefit to the buyer. But I’d now like to give some advice to sellers, both to ensure the largest market for their items and to avoid potential problems.

Getting the item ready for sale

  1. If the device has any configuration data, erase it before listing the device for sale.
    • Some devices have no way of resetting them to the default state unless the existing password is given, which means that if the seller doesn’t erase it before selling, the only way a buyer will be able to use it is if the seller is willing to tell them what the password is (not practical if it is the same password the seller is using on equipment they’re still using, or if they don’t know it). Otherwise, the device has to go back to the seller and the transaction voided.
    • Some other devices have a “reset the password only” option, or (insecurely) a “backdoor” password that works on all units. If the user does that, they will have access to the entire configuration of the device as the seller last used it – at a minimum, things like IP addresses, SNMP communities, and so on. Potentially even more sensitive information like access lists can be disclosed. Additionally, at least two major brands of devices have the (undocumented, but widely known) ability to read or decrypt the original password cleartext once a password recovery procedure is performed.
    • This is particularly important for disk drives and other storage media. Even if the drives were part of a RAID set, it might still be possible to recover chunks of data from individual drives. You can use a utility such as DBAN to erase drives that are still in the system. It offers a variety of erasure options, from a simple “write zeros to the whole drive” to multiple erase passes with random data. Note that even with this type of erasure, it may still be possible to recover data from certain areas of the disk (replaced defective sectors, for example). If you (or your company) doesn’t want to take the risk, you can remove the drives – but read on for a suggestion about disk trays and mounting hardware.
  2. If you’re selling something like a server and your company policy requires removal of the drives before the sale, put the empty hot-swap drive trays back in the server instead of trashing them with the drives. If the trays require oddball hardware to hold the drives in, put the screws in a small plastic bag and tape them securely to the disk tray(s). The buyer will thank you as they won’t have to scavenge for drive trays to get the server running with new drives.
  3. Unless you’re explicitly selling the item “as-is” or “non-working”, please test it before listing it. Having a 14-day (or longer) “no questions asked” return policy is nice, but neither the buyer nor you want to deal with shipping defective items back and forth. For some items, this can simply be installing (or leaving) them in a system and seeing if they work. Mechanical items like disk drives need some additional testing. Modern drives (anything in the last decade or so) have S.M.A.R.T. testing built in, so it is a simple matter to use something like smartmontools to test the drive and see if it has any problems before listing it. Just today I received a pair of SAS drives, each with less than 30 power-on hours on them, which had over 50 media errors each and had been logging S.M.A.R.T. errors since new (the first failure was logged at 0 power-on hours).
  4. Along with the above, it would be helpful to update the device to the latest available firmware “while you’re in there”, if that is something the manufacturer allows. I’ve received devices that were so old that several intermediate firmware updates were needed to get them to the current revision. In a number of those cases, the intermediate updates were themselves so old that the manufacturer had removed them from their web site as obsolete. That requires the user to go on a “scavenger hunt” through potentially untrustworthy sites to try to find firmware. Another reason to update before selling is that in some cases, the update procedure will only work in the specific brand of equipment the device came from. An example is Dell network cards – the Dell Server Update Utility only runs on Dell-branded servers. Dell network cards are mostly-generic Broadcom, Intel, etc. cards but often have Dell listed in the PCI Vendor ID on the card. This means that generic firmware updates from the manufacturer may fail to recognize the card. To continue my example, even if the user is putting the card in a Dell server, unless Dell offered the specific option card for the user’s server, the appropriate Server Update Utility may not detect / update it.

Listing the item for sale

  1. Be as descriptive as possible when listing the item. To give a specific example of why this is a problem, look for “PowerEdge R300” on eBay. That model was available with or without hot-swap drives and with or without redundant power supplies. It is not possible to convert a chassis from any of those configurations to another. Many times a seller will just say something like “PowerEdge R300 Quad-core 2.33GHz 4GB 2x 146GB HDD”. That doesn’t convey much useful information – in addition to the chassis type, it would be useful to know the exact CPU model, whether the disks are SATA or SAS and if there’s an add-on disk controller in the system, and whether or not there’s a remote access card. This is made even worse by the sellers that say “Stock photo” or “Photo may not represent actual item”. To add insult to injury, some of those same sellers will say “if it isn’t in the picture, it isn’t included” in the body of the listing. Dell’s web site is pretty good – if you know the “service tag” of a system, Dell’s site will show you the configuration as it shipped from Dell. Of course, the seller or a previous owner may have added, removed, or modified components, so don’t take the Dell list as the last word. As the seller, you can go to the Dell site and copy/paste the configuration into your sale listing once you verify that it’s accurate.
  2. If you’re selling something that isn’t an add-on component (like a network card or a disk drive), but can function as a standalone device (like a server, Ethernet switch or network-controlled outlet strip), provide all of the necessary accessories with it or explain clearly that they’re missing. This definitely includes rack mount ears/rails (if the device is rack mountable) and console cables (no two vendors do exactly the same thing once you get to anything newer than 9-pin serial connectors). If the device has cable-management hardware (bracket, etc.) and you have it, include that with the item. Likewise for the faceplate. It is also thoughtful to include the required power cord, at least if the seller and the buyer use the same type of electrical outlets. This isn’t vital, as there are a small number of possible mating power cords for modern equipment. But the buyer will usually appreciate your thoughfulness, particularly if it is an unsual cord like an IEC C20 and they have to order one once they receive your shipment.

Shipping the item to the buyer

Pack the item well, preferably using the original manufacturer packaging (if still available). You’d be amazed at the way some stuff arrives here. I’ve received memory DIMMs ratlling around loose inside a cardboard box. I’ve received servers where parts of the chassis were dented or damaged (usually parts that protrude beyond the basic rectangular shape, but sometimes the main chassis itself). I’ve received devices with glass faceplates that were smashed. I’ve received boxes where the cardboard was too thin for the weight of the item and has ripped during normal handling, with accessories falling out of the box and being lost in transit.

I’d like to be able to say “just take the item to your nearest parcel store and have them pack and ship it”, but that’s generally not a good idea. It seems that their solution for shipping anything is a thin-wall cardboard box and packing peanuts. Those peanuts are not acceptable for anything that might shift around or settle in the box. With enough practice, it is possible to ship fragile items using common materials – I have purchased many items from ex-Soviet countries where the contents were packaged entirely (but carefully) in newspaper and placed in a cardboard box and which arrived here in perfect condition despite their international travel and the rough handling of various foreign postal services.

Large items are generally either heavy or are light enough that they get charged “dimensional weight”, where the shipping company charges the package as if it weighs a certain amount per cubic inch. In general, the cost of reasonable insurance (value up to some hundreds of dollars) will be a small part of the total shipping cost, so it makes sense to insure the package. If you have to file a claim, be aware that you will often be asked to provide proof of adequate packaging before the shipping company will process the claim. I know of one company that took pictures of each box while it was being packaged and retained those pictures, both to deal with shipping damage claims and to prove that a certain item was in the box when it was shipped.

Conclusion

If, as a seller, you follow these steps I think you will find that your items will sell faster and your customers will be happier. And if I’m the customer, I’ll definitely be happier.

Dell PowerEdge R300 ESM / BMC firmware updates on non-supported operating systems

Dell has generally been quite good about making firmware updates available in a variety of formats. In addition to the normal Windows and Linux versions, most patches are also available as a floppy / USB image or an ISO image (depending on size). Those of us who don’t run one of the operating systems Dell provides support for appreciate them going through the trouble.

However, newer updates for older systems and updates for newer systems seem to no longer provide standalone installers. In theory, Dell provides a quarterly packaged roll-up of all available updates on a pair of DVD images (CDU and SUU). Booting these and wasting about 10 minutes switching discs should get your system updated to the latest versions of all firmware without any additional steps.

Unfortunately, the firmware for the R300’s ESM / BMC has not been on any SUU discs I’ve looked at, and the update is listed as “Critical Security Update” on Dell’s site (look under ESM on the R300’s downloads and drivers page). The only two formats it is available in are “Windows Update Package” and “Linux Update Package”. I figure that’s not a problem, as I can boot a Windows 7 recovery disk and then run the ESM update from a USB drive. Unfortunately, that doesn’t work. You get an error about “unsupported operating system”.

Next, I boot the CDU DVD and select F3 for Advanced Options. This eventually gets me to a Linux shell prompt (CDU/SUU operates under Linux). I mount the USB drive and execute the Linux version of the ESM update. That errors out with “Not compatible with your system configuration” for some unknown reason. Time to investigate further…

Clicking on “Previous Versions” on the Dell page shows the previous version as 2.46 from 2009. Looking at the available formats, one is listed as “Hard-Drive”. Depending on the mood Dell is in when they create the kit, this could be anything from a freestanding binary that writes a floppy image to a drive, to creating an ISO file, or something that just unpacks into a bunch of loose files somewhere, perhaps then trying to run them (incorrectly) on the local system.

I downloaded that file (link here) and discovered it created 3 useful files when it was executed:

  • bmcfl16d.exe – a DOS-based flash utility
  • bmccfg.def – some sort of configuration file
  • bmcflsh.dat – the actual firmware to be flashed

Now all I needed to do was to find newer versions of the last 2 files inside either the Linux or Windows installer. The Linux installer was a pain, and I quickly gave up on it. I had much better luck with the Windows version (link here). Despite being an EXE file, I was able to use WinZip 16.5 to open the file (browse to the directory where you downloaded the Dell update, then make sure you’ve selected “All files (*.*)” in WinZip’s Open Archive dialog). There’s a whole load of un-needed stuff in there (which doesn’t completely explain how a 655KB update turns into a 4800KB Windows binary). Find the bmccfg.def and bmcflsh.dat files and extract them on top of (replacing) the ones from unpacking the older download.

I copied the 3 files onto a bootable USB stick and then used that to boot the R300 to be updated. Here are some screnshots of the various stages of the procedure (it’s very simple – just answer Y or N when asked if you want to perform the update):



If the firmware is already at the latest revision, the utility will tell you that and exit. This can also be used to double-check that the update was successful:

That’s all there is to it. If you want a pre-built .ZIP file with the flash utility and the 2.50 image, I have placed one here for your convenience.

Advanced topics

The bmcfl16d.exe utility has a number of documented and undocumented additional features. You can use the -help option to get a list of the documented features. Before using one of these features when updating a system, be sure you know what you’re doing and have a fallback plan in case the update fails and you’re left with a non-operable system.

There is also an undocumented -advhelp (advanced help) option, which shows the additional undocumented options:

The above caution about knowing what you’re doing and having a fallback plan is doubly important if you try using any of the advanced options.