NAME
hippi-part-1 – HIPPI on the Onyx2, part 1
1. Why the cluster gear from the nineties
I've written before about how I ended up here: a Personal Iris under my father's desk, SGI and Cray posters on my bedroom wall, and a career as a chip designer posing as a networking guy. That last part is the real reason I like the cluster hardware from the nineties and early 2000s.
In that era, building one fast computer out of several smaller ones was a hardware problem first. Ethernet was 10 or 100 Mbit/s for most of the decade, while the machines worth connecting moved hundreds of megabytes per second internally. There was no commodity answer, so the vendors built their own: crossbar switches, network cards with their own processors, connection-oriented links, and "OS bypass" so an MPI message never had to touch the kernel. Every one of those ideas lives on in today's data centres, but in 1998 you could still see them, on a board you can hold in one hand.
The other reason is scale. An Origin 2000 cabinet was never going to fit in my house (the Onyx2 story covers how close I came). The building blocks of those systems do fit: an Onyx2, a couple of Octane2s, an Origin 350 cluster, a HIPPI link and a Myrinet switch. A supercomputer you can understand end to end, in a home office. As a kid I read about this stuff. Now I get to run it.
2. HIPPI and Myrinet
HIPPI, the High Performance Parallel Interface, was developed in the late eighties to move data between supercomputers and their storage and graphics. The basic unit is a simplex channel of 100 MB/s (800 Mbit/s); you use two of them for a duplex link. The original version ran over fat 50-pair copper cables of up to 25 metres. Serial HIPPI, which is what I have, puts the same thing on a pair of fibres: multimode for a few hundred metres, single mode for up to 10 km.
It is connection-oriented, which is what makes it feel so different from Ethernet. A source opens a connection to a destination by presenting a 32-bit I-field, crossbar switches route the connection on that I-field, the data goes across in bursts, and the connection closes. Without a switch, two ports simply cable to each other. IP over HIPPI was standardised in RFC 1374 and later RFC 2067, with its own ARP flavour (HARP) in RFC 2834.
SGI supported HIPPI throughout: copper HIO boards for Challenge and Onyx, then the XIO HIPPI-Serial board for the Origin 2000, the Onyx2 and the Origin 3000. The last IRIS HIPPI software release, 4.0.1, dates from 2003. On top of plain IP, SGI's MPI had a HIPPI "bypass" that moved MPI messages between two SGI boards' firmware without a system call per message.
There was also a third-party scene. Essential Communications in Albuquerque made HIPPI cards for SGI's own workstations, GIO for the Indigo2 and PCI for the O2, Octane and Origin 200, built around their RoadRunner chip (the same one Linux's rrunner driver talks to). Look closely at SGI's XIO board and you find two RoadRunners on it, alongside two SGI LINC ASICs and two IDT RV4640 MIPS processors. Same silicon at both ends of my link.
Myrinet came from Myricom a few years later and took a different route: a switched, source-routed network where every card has its own LANai processor running a control program, and a messaging layer (GM) that lets applications talk to the card directly. Myricom's IRIX support was always a source-only beta, with 1.4.1 the last build they released. I've been porting GM 2.0.8 to IRIX. GM itself now builds and works, while MPI over GM and IP over GM are still in progress. The old 1.6.4 build already managed 13.7 microseconds half round trip and 87 MB/s between lucy and aurora. The goal is three or four nodes on one Myrinet switch, 28 CPUs in total.
3. A board from eBay
The XIO board came first, before the Onyx2 even arrived: an untested XIO HIPPI-Serial board on eBay, from the same seller as the Shoehorn that later turned out to be dead on arrival. I also had two Essential PCI HIPPI cards for the Octane2s.
The Essential cards were their own adventure. They need Essential's driver, not SGI's, and they don't show up in hinv at all. With help from a few fantastic people on the IRIX forums I found the 2.0.3 driver kit for Octane and Origin, and I scanned Essential's installation guide and put it on archive.org so the next person doesn't have to hunt for it.
The XIO board showed up in hinv straight away, and IRIX created hippi0 and the bypass device nodes. That is as far as it got. As far as I could tell it received but would not transmit. I pulled it and parked it.
4. A board from Ian
The second board came from Ian at SGIdepot. Into XIO slot 5 it went, cabled to dave, the test-bed Octane2 with an Essential card. Same symptom.
arthur# hipcntl status Origin HIPPI Serial XIO Adapter FLAGS: ACCEPTING DST.FSYNC DST.SIG_DET SRC connections : 0 SRC glink reset : 33779 SRC glink lost : 96
The destination side was happy: signal detected, frame sync, dave's packets arriving. The source side never came up. On dave, the Essential driver counted 62,840 G-link errors and 1.4 million alternating flag errors: light from arthur was arriving, but it could not be decoded.
Then the elimination round:
- dave's own fibre loopback: link up instantly, so dave's optics are fine.
- the cable: multimode, matching the 770-860 nm label on the board, so not a single-mode mix-up.
- a phone camera on arthur's TX port: the laser is lit.
- arthur's fibre loopback: frame sync drops on its own signal. Arthur's receiver reads dave perfectly but can't read itself, so the problem is arthur's transmit side.
hipcntl loopback, the board's electrical loopback before the optics: clean. The board logic, serialiser and clock are all fine.
That leaves the optical transmit path: the laser module or what's in front of it. One lesson on the way: SRC glink reset is not an error counter. It ticks at a steady 50 or so per second whenever the source isn't happy, and stops when it is. It's a pass/fail light, not a rate.
A proper fibre cleaning pen moved the numbers, which was the first sign of life:
| dave, per second | before | clean 1 | clean 2 | 4 min later |
|---|---|---|---|---|
| G-link errors | ~60 | ~20,000 | ~11,400 | ~6,000 |
| alternating flag errors | ~10,000 | ~4,800 | ~4,300 | ~3,600 |
Link-ready started flickering on, so it was getting closer. But not there.
5. The module fix
The laser lives in a Methode MDX-19-5-1-S, an 850 nm transceiver from 1997 in the industry-standard 1x9 package. That's good news: 1x9 modules were made by the million for Fibre Channel and Gigabit Ethernet, and Methode listed HIPPI-Serial among the protocols for this family.
What has to match: 5V supply, an 850 nm multimode laser good for around 1.2 Gbit/s, and the right kind of signal detect output. With the board unpowered, the square pad marks pin 1 and the TD and RD probe points confirm which end is transmit. Pin 4, signal detect, measured about 240 ohms to ground against about 113 on the supply pins: its own net with a pull-down, which means PECL. That gives a shopping list of 1x9, SC duplex, 850 nm, 1.0625 to 1.25 Gbit/s, 5V, PECL signal detect. Avago's HFBR-53D5 is the textbook part, and Rochester Electronics, who specialise in obsolete parts, still had genuine HFBR-53D5EMs on the shelf. The catch was a $250 minimum order, so I bought seven at $39.38 each. That covers this board, with spares left over for the eBay one.
Getting the old module out was the hard part. My gear is modest: a soldering iron and a desoldering pen from AliExpress, both somewhere between five and ten years old. It took about twenty rounds to get the small holes for the module's pins open again. The board is dense, with SMD parts on both sides, and I kept my fingers crossed the whole time that I wasn't damaging anything critical along the way.
New module in, board back in slot 5, power on:
arthur# hipcntl status Origin HIPPI Serial XIO Adapter FLAGS: ACCEPTING DST.LNK_RDY DST.FSYNC DST.OH8SYNC DST.SIG_DET SRC glink reset : 0 SRC glink lost : 0
Link ready, frame sync, and the reset counter sitting at zero instead of ticking away. On dave, esshippi status said LINK_ON. A link, at last.
6. End to end with blast
With the link up, I wanted to know what the link itself can do, before TCP/IP gets involved. Essential ships blast and sink, which push raw HIPPI-FP packets through the card's character driver. They only run on the Essential side, but SGI's hiptest can play the other end. Both sides just have to agree on the upper-layer protocol number.
arthur# hiptest -r -f -u 0x82 65536 10000 dave# ./blast -l 65528 -u 0x82 -n 10000 blast(ess0): 10000 packets, length 65528, I=0x00000000, ulp=0x82 blast(ess0): 624.92 Meg, 11.45 s (54.58 MB/s)
hiptest reported all 10,000 packets received. For the other direction, sink receives on dave and hiptest sends from arthur, timed with timex. All figures are in blast's units (2^20 bytes):
| direction | packet size | options | MB/s |
|---|---|---|---|
| dave to arthur | 64 KB | 54.6 | |
| dave to arthur | 32 KB | 44.2 | |
| dave to arthur | 64 KB | -C, one long connection | 54.8 |
| dave to arthur | 64 KB | -M, buffer pinned in memory | 54.8 |
| dave to arthur | 64 KB | -2, two writers | 57.6 |
| arthur to dave | 64 KB | 46.3 | |
| arthur to dave | 32 KB | 33.2 |
Two packet sizes let you split each packet's time into a fixed cost and a streaming rate. Going from 32 KB to 64 KB costs the same extra 0.41 to 0.44 ms in both directions, so the link streams at 71 to 77 MB/s, most of Serial HIPPI's ceiling. The difference is the fixed cost per packet: 0.27 ms when dave sends, 0.53 ms when arthur sends. At 64 KB that overhead takes roughly a quarter to 40 percent of the time.
A single long connection (-C) changed nothing, so it isn't HIPPI connection setup. Pinning the buffer (-M) changed nothing, so it isn't page locking. Two writers (-2) helped a little. What's left is the system call, DMA setup and completion interrupt for every packet, with only one packet in flight at a time. That's a limit of how blast and hiptest work, not of the link. The IP drivers keep many packets queued, which is exactly what should hide that cost.
A footnote for the next person: blast and sink also have a -k option for pre-mapped kernel buffers, which Essential says nearly doubles throughput. On IRIX 6.5 it fails with "couldn't open special hippi device". The code opens the unit name, ess0, as a file in the current directory, a leftover from drivers that had a /dev/ess0. I left that alone: I couldn't tell what the mapping exposes, and writing a test pattern into the card's own memory is a good way to hang it.
7. Talking IP
The raw link is nice, but real work needs IP. Without a HIPPI switch there is no address resolution worth having, so each side gets a static entry for the other. On arthur, SGI's map file:
# /usr/etc/hippi.imap IP, I-field, ULA 10.42.1.15 0x070000dc 0:a0:88:1:82:dc # dave-hippi
On dave, in Essential's /etc/init.d/Essnic.post, with dnd ("do not disturb") so dave stops sending ARP traffic to a peer that doesn't answer it:
/usr/etc/ifconfig ess0 up /usr/Essential/hippi/bin/essarp -s 10.42.1.17 8:0:69:4:6a:c6 011 dnd
The Essential card ships set to switched mode. For a back-to-back link, store direct mode in its EEPROM once (esshippi on short fp network direct, then esshippi off and esstune -l -e), or it comes up after every reboot probing for a switch that isn't there.
dave# ping arthur-hippi 64 bytes from 10.42.1.17: icmp_seq=0 ttl=255 time=0.671 ms 64 bytes from 10.42.1.17: icmp_seq=1 ttl=255 time=0.552 ms
Both directions, by name, across reboots. Both drivers also speak HARP, HIPPI's ARP, so arthur could act as the server and dave would register itself. That's an experiment for another day.
8. TCP with ttcp
By now lucy, my other Octane2, had joined in with the same Essential card. It got the same setup as dave: direct mode in the EEPROM and static entries for both peers, after which moving one fibre pair was enough to switch lucy between dave and arthur. One trap on the way: lucy still had an older static entry for arthur, with a ULA that isn't the one on arthur's current board. Two entries for one address is a quiet way to lose packets, so run essarp -a after swapping a board.
ttcp is the classic tool for this: one side listens, the other writes 64 KB buffers as fast as TCP takes them. The -b option asks for 512 KB socket buffers.
arthur# ttcp -r -s -l 65536 -b 524288 lucy# ttcp -t -s -l 65536 -n 20000 -b 524288 10.42.1.17 ttcp-t: 1310720000 bytes in 17.52 real seconds = 73055.77 KB/sec +++
Each HIPPI run moves 1.3 GB, the 100 Mbit run 128 MB. Same units as blast, 2^20 bytes:
| path | link | MB/s |
|---|---|---|
| arthur to lucy | HIPPI | 82.1 |
| lucy to arthur | HIPPI | 71.3 |
| lucy to dave | HIPPI | 71.2 |
| lucy to dave | HIPPI, default socket buffers | 35.2 |
| lucy to dave | tg2, gigabit Ethernet | 59.4 |
| arthur to lucy | ef0, 100 Mbit Ethernet | 11.3 |
TCP beats blast by a wide margin, which settles the question from section 6. With the IP driver keeping packets queued, the next packet's DMA can overlap the current one on the fibre, and the fixed cost per packet disappears. What's left is each card's streaming rate. lucy sends at 71 MB/s to either peer, so that's the Essential card's own limit as a sender. arthur, the slow sender under hiptest, is the fastest here at 82.1 MB/s: 86 million bytes per second, or 86 percent of what Serial HIPPI can carry. Its 0.53 ms per packet was the whole story.
Socket buffers matter. With IRIX's defaults, ttcp's time per 64 KB write doubled from 0.90 to 1.82 ms and the rate halved: one packet in flight, then a wait for its acknowledgement. On HIPPI, raise them.
Against Ethernet, it depends on which Ethernet. The Octane2s' gigabit tg2 cards manage 59.4 MB/s, so HIPPI's lead there is about 20 percent. arthur's 100 Mbit ef0 tops out at 11.3 MB/s, and HIPPI moves more than seven times as much.
Latency is a different story. A small ping over HIPPI takes about 0.44 ms round trip, no better than Ethernet, and even a 32 KB ping is slower: 2.90 ms against 2.43 ms over gigabit. A single HIPPI packet goes to the card, across the fibre and off the other card one step after the other, while Ethernet's 1,500-byte fragments overlap those steps. HIPPI is a bandwidth pipe, not a low-latency link.
9. Part 2
That's the link done. Part 2 puts real work on it: MPI workloads with sgi-tess, lucy and arthur over HIPPI against lucy, aurora and arthur over GM on the Myrinet switch, once MPI over GM runs. On paper Myrinet wins on latency by a wide margin and roughly matches HIPPI on bandwidth, so the question is what sgi-tess actually needs.
I also want a proper look at CPU cost. ttcp's own system time was higher over HIPPI, because the CPU checksums HIPPI traffic while the gigabit cards do it in hardware. But ttcp doesn't see interrupt handling, and that is where Ethernet's roughly 40 times more packets per megabyte should cost.
SOURCES
- HIPPI NIC SGI PCI and GIO Installation Guide 1.5, Essential Communications, 1999 -- my scan
- HIPPI board drivers for Essential Communications, irixnet -- where the drivers turned up
- RFC 2067, IP over HIPPI and RFC 2834, ARP and IP Broadcast over HIPPI-800
- IRIS HIPPI Administrator's Guide (007-2229-006) and the IRIS HIPPI 4.0.1 release notes and man pages
- Rochester Electronics HFBR53D5EM replacement part
SEE ALSO
systems/arthur, systems/dave, systems/lucy, articles/onyx2-from-kentucky