"IBM said LinuxONE Emperor can scale up to 8,000 virtual machines or thousands of containers, which would be the most for any single Linux system."
I wish article (or IBM) provided some specs. Thousands of containers is just too broad of a statement- what are these containers doing? Running just a bash script hello world program or minting bitcoins?
In an article which states "..IBM's z13 mainframe computer, which had been designed for high-volume mobile transactions.", all bets are off with respect to reality. I mean, what does 'designed for high-volume mobile transactions' even mean in the context of having Linux running on the hardware?
Not sure if this is still correct for the z13 but in general: "System z servers offload such functions as I/O processing, cryptography, memory control, and various service functions (such as hardware configuration management and error logging) to dedicated processors."
It's called a zIIP engine (http://www-03.ibm.com/systems/z/hardware/features/ziip/) and it utilizes the IFL also (http://www-03.ibm.com/systems/z/os/linux/solutions/ifl.html). We use the zIIP on our z13 basically to offload any work that takes a ton of I/O (DB2 data serves, dataset scans for performance monitoring, RMF and CICS reporting). It ends up saving a lot of money monthly because we are charged by maximum number of MSU's during any four-hour window of the month. The zIIP reduces the number of MSU's needed at any given time. In our case, 5-10.
Back in 2005, I installed SuSe on an s/390 box. If you could throw a lot of memory at it or MIPs, then it would run pretty well, and could be used as a file or email server. But anything that crunched data was horribly slow. Can't say that it's any better now, but if it's utilizing the zIIP, I'm sure it's miles above where it was 10 years ago.
Still, pretty high cost of ownership on the mainframe to get this. I can't imagine many people buying a million dollar mainframe just to run Linux. Seems like those who have a mainframe already would have been like us: let's partition out a Linux instance and try it out.
> I can't imagine many people buying a million dollar mainframe just to run Linux.
The Techcrunch version of this story [1] has an interesting bit on the pricing model, suggesting that the up-front cost will be much lower than usual (but no specific numbers).
By offering an elastic, cloud-like pricing model, [IBM] is hoping to land more customers who might have been scared away previously by the up-front cost of investing in a mainframe. The metered mainframe will still sit inside the customer’s on-premises data center, but billing will be based on how much the customer uses the system, much like a cloud model, Mauri explained.
Of course it depends a lot on the specifics, since IBM does already bill in part based on usage. So this could be either a significant departure in pricing approach, or just marketing a tweak on it.
To clarify, zIIP processing is only relevant under z/OS. Linux processing runs under IFLs. But that's not to say you can't get some offloading in Linux. By design, mainframes can hardware accelerate some processing. The crytpocards [0] are an example of this.
But you are right. A lot of shops go through a lot of effort to maintain their zIIP window in their z/OS processing. And a big selling point for many mainframe software vendors is that their product's processing is "zIIP-enabled".
Yes, you are right there, and thank you for clarifying that. I should have been more clear that the zIIP is useful to Linux through accessing data via z/VM. You can't offload all of the I/O, but you can offload some of it if accessing the mainframe for data. Or so I remember at this point. It's been a few years since leaving my mainframe systems programmer position.
I am complete naive in this subject area. What would you say to a clueless peer who said, "my laptop has those same features, except built into the CPU (like the AES instructions found in i7 cores) so they have less latency than a mainframe would"?
I cut my teeth on an Amiga and understand the appeal of external processors, but it seems like bringing those features on-die would be better yet. Mainframes seem to be going the other way, though. What am I missing out on?
All of that is standard for a mainframe. Nothing on that list distinguishes this machine for handling mobile transactions as opposed to non mobile ones. IBM hasn't innovated here, they've merely iterated and added a buzzword.
It shows how outmoded mainframes are that they're now selling one that only supports being a fleet of Linux machines. Mainframe customers are no longer scaling their legacy zOS systems, meaning the big mainframe users must be processing their transactions in the Linux cloud now.
I mean, what does 'designed for high-volume mobile transactions' even mean in the context of having Linux running on the hardware?
Excellent question. Statements like this, grandiose but nonspecific, are intended for managers and those who are looking for a justification to spend money.
Do you have a source on if the LinuxONE Emprorer is actually a z13 underneath? It wouldn't be surprised if the Emperor and Rockhopper are just re-branded z13s and z114s.
but surely those are super-expensive and I doubt all that effective when it comes to performance per watt. I'm actually a bit more excited about Power8 servers, but are there any non-IBM available yet?
I can fit 4 C7000 chassis in a single rack, which top out at an aggregate 32 TB of memory and over 2000 cores. 5 GHz cores running s390x microcode are not, FYI, twice as fast as Xeons.
It would be great if people who weren't familiar with the state of the art in x86 kit didn't blindly assume IBM et al's advertising was reality.
I'm no expert, but my understanding is that the IBM "z" platform has a number of interesting features perhaps not found in more typical server platforms/configurations, beyond just scaling out to loads of cores and RAM.
As with all IBM mainframes going right back to the 60s, the "z" systems are designed for continuous uptime, on the order of decades, and this is evident in many of the design decisions. For example, various subsystems and components have hot-spares available, so that even outright component failures will not cause downtime. Many components are then hot-swappable, even those that might not ordinarily be in other architectures, such as processors and main memory. No interruption to OS or application-level services is expected by hot-swapping such hardware. Across the useful life of the mainframe, most repairs and maintenance would be carried out without ever shutting it down.
I understand the platform also has extensive internal integrity checking built in, a potentially important factor for various types of jobs. Its auditing service is capable of detecting unusual conditions in various subsystems or jobs ("I've just picked up a fault in the AE-35 unit"), automatically retrying instructions on the processor if they executed anomalously. If the fault continues, the suspect processor is routed-around with no interruption to OS or applications, the job is resumed from last checkpoint on another processor, and the system phones home to IBM to log a service call. This monitoring is not being performed by processes running in userland or by the kernel, but is in fact baked into the hardware/firmware platform.
Furthermore, the systems can be configured with a variety of specialty offload processors or subsystems for tasks like encryption, key management, compression, and even logging -- which again might not be so commonly found on-board of some commodity servers.
(And, of course, even if you could put together an analogous solution with commodity kit, it's IBM! For the sorts of companies looking at a mainframe in 2015, having the IBM name on the SLA has got to be a pretty big part of the equation, right?)
Moreover, if you proposed to build and manage these sorts of capabilities from commodity x86 kit, I imagine IBM would claim that they'd have the lower TCO.
Whereas I have hands on experience with this stuff. So I guess I should thank you for giving me first-hand experience of being on the receiving end of that mansplaining thing people complain about.
> I imagine IBM would claim that they'd have the lower TCO.
Yes, they will. The funny thing is, the man from HP was in just last month explaining to me that Xeon Superdomes have a lower TCO, too, and last year the lady from Oracle was telling me how I shouldn't balk at the headline cost of ExaData and ExaLogic because, from a TCO perspective, they'd save me money.
People fall into the mistake of just comparing cores and memory specs between x86_64 and s390x. The hardware redundancy benefits are pretty huge. You really need a full proof-of-concept to get any picture of how your workload might run on System z.
NB your AE-35 comment, despite loud protestations there's a very good case that 2001 was written and directed as a sharp critique on IBM. Including IBM logos in numerous places.
No. Which workloads do you have that need more than 40 cores per image, are happy with a maximum of a hundred on, and won't run on clusters? (I hope it's not one that's going to be crippled by zVM's slow scan of large memory areas suspending guest execution).
> I/O offloaded
If you've actually worked with zLinux (I have) you'll know there's little effective offload, and that zVM overhead increases as the virtual IO ramps up.
> and five 9's
Is zVM offering sysplex? No. So you're relying on your single LPAR to be five nines? Never upgrading zVM? Never updating PR/SM?
Is the datacentre five 9s? The power? The network? Really?
> most rack cannot fit 10 TB of RAM and 140x5GHz cores
Any rack can fit 10TB of RAM and the equivalent of 140x5GHz POWER cores.
I can easily get ~1TB RAM and 64 2GHz+ x86 cores in 1U. A standard rack is at least 42U. In terms of density it's nothing special.
What makes these different is that you get it in a box that someone else will be maintaining, and where all the dirty work of designing and building a high availability redundant system has been done for you. Most people never build anything remotely as redundant as these things tend to be.
I'm guessing part of the appeal will also be super-wide bandwidth between the cores. Interconnect throughput and/or shared memory could make this significantly more interesting than a modular rack-based system for some workloads.
> I'm guessing part of the appeal will also be super-wide bandwidth between the cores.
If it's anything like previous-generations, that will be infiniband between books. The benefits on Z-class systems tend to be very big per-core, per-socket caches and, as the person you're replying to said, a lot of internal redundancy and automated failover (multiple backplanes with failover, spare memory and cores with failover, and so on and so forth).
The high-end POWER 7/8 hardware has incredible single core performance, beating the pants off Xeon. It uses huge amounts of power to do that, so it's not appropriate for all roles. Low end POWER is pretty niche. Freescale uses the architecture for telecoms applications.
In general Linux upstream these days "just works" on ppc64 & ppc64le. There's RHEL for POWER already, and IBM have loaned hardware to the CentOS project so we'll get CentOS on POWER pretty soon.
The licensing of (Open-)POWER is more open than x86 (but not as open as stuff like RISC-V), and there are several second sources for chips, in situations where that matters.
You get a pretty decent multiple of performance on power vs. intel. Probably something like 4-7x for common use cases.
Generally speaking, you buy the hardware because your workload needs the single core performance, or you're arbitraging vendor licensing costs for software.
Also, IBM's bread and butter is "peaky" financial services and gov't business, so they have business models that makes it work from a $ pov. You can buy a box with 100 cores, pay for 20, and lease 30 more for a few days to meet your peak demands for tax/christmas/billing season.
It may be one basket, but IBM high end kit is the nuclear fallout shelter of baskets.
E.g. we used to have an IBM Enterprise Storage System (aka the Shark) back in the day (around 2000), and it came in American fridge size, full of drawers of drives. You could just yank any drawer safe in the knowledge that all the raid volumes were distributed over multiple drawers. If a SCSI controller failed, you could yank a drawer of SCSI controllers and hotswap them safe in the knowledge they were fully redundant.
The "brains" of the thing consisted of a fully redundant pair of two AIX RS/6000 servers, and you could yank either one of them without losing data (all writes were committed to at least non-volatile memory on both servers before being acknowledged). Either server also had at least hot-swap RAM (raid-like memory controller) and may have had hot swap CPU's (tell the OS to move all threads off a CPU, swap, switch back).
On top of that, it had a phone connection and would dial out to report any early warnings of problems directly to IBM who'd send out a technician before anything even failed as long as you kept paying your support plan.
So yes, you can get that density with standard kit easily, and probably much cheaper too. Assuming you have enough skilled staff to manage it. The reason IBM still manages to sell this kind of kit, on the other hand, is because what they are really selling is peace of mind that most issues are Someone Elses Problem. For some people it makes sense to pay a lot for that.
Back in the late 1990s I was involved in provisioning a large Sun e15k. Not indestructable but nearly.
It broke. You know what happened? The factory roof leaked and poured water onto the DC sub-building which the roof then collapsed onto the e15k which promptly blew up and caused a spectacular fire, halon dump and about a month of work arguing with insurance companies and guys with shovels.
In that circumstance, it doesn't matter what promises the vendor make. That's still all your eggs in one basket.
Buy two and keep one somewhere else didn't help either as the network termination, switching and routing layers were down and all the people using it were about 300 miles away from the backup location anyway. So some poor fucker had to dismantle the backup e15k and disk arrays, bring them in a large truck[1] to the original location and erect a temp DC in a portakabin outside the building.
Edit: We would have been better served with two smaller DCs with off the shelf kit on the same site but different buildings running a mirrored arrangement. All for pocket change compared to a zSeries...
That's what the company I work for now do. We have off the shelf kit,SAN replication, ESX, redundant routing and multiple peers in different locations.
That's why you never deploy to just one location no matter how reliable the actual kit is.
You'd be in exactly the same situation if you had off the shelf "normal" servers in a rack. The point is one IBM mainframe is generally going to be more reliable than the vast majority of "homegrown" setups in a single location.
If you're comparing against a setup in multiple locations, then you should compare against two or more of these.
And there too these kind of solutions are far more reliable if you are willing to pay the money. E.g. IBM provides a range of options up to full synchronous mirroring of mainframe setups over up to about 200km where both systems can be administered as one identical unit (distance is down to latency). They also provide a range of other options for various performance vs. amount of data you can potentially lose vs. cost tradeoffs.
> Buy two and keep one somewhere else didn't help either as the network termination, switching and routing layers were down and all the people using it were about 300 miles away from the backup location anyway.
And this wouldn't have been any better if you had two racks of kit instead of two mainframes.
> All for pocket change compared to a zSeries...
There we agree. I'll likely never buy or recommend one of these, for the reason that I tend to work on cost sensitive projects.
Except that you're going to pay a lot more than you would for those off the shelf "normal" servers in a rack. Probably enough that you can afford doubly-redundant normal servers for the cost of a non-redundant IBM mainframe, with quite a bit of cash left over.
Yes but when that system falls over, your boss is yelling at you, and you're on the hook. With IBM, you can all yell at IBM. And that's why big enterprise companies buy IBM.
This story is obviously a bit ridiculous nowadays, since no one that can afford an ESS is buying a single site. In fact, most can't due to legal regulations about having data redundancy. These regulations typically lead to having a secondary site across town and a tertiary site across the country.
We have an AS/400 (excuse me, iSeries) and the damn thing is rock solid. It also alerts us and IBM when it needs maintenance. Its basically a tank with a logistics chain.
Designed for business apps, future-proofing, integrated database, largely self-managing, capability-security, continuing on solid (POWER) hardware... did about everything right. That's why we regularly fix crashed Windows and 'NIX machines but my company's AS/400 has been running for around 10 years.
I've always wanted a modern, clean-slated version of the System/38 w/out relics from that time and with any tricks we've learned since. Throw in hardware acceleration for garbage collection and some NonStop-style tricks for fault-tolerance to have a beast of a machine.
A similarly amazing machine that IBM's System/38 learned from a little bit. Somebody posted a link to an emulator but honestly I don't want to dredge through that. Like you said, a modern system that reimplemented its best attributes without the limitations or baggage would be nice.
Mainframes are complex enough that there's rarely projects to implement them but there's lots of work on safer CPU's. See crash-safe.org's early publications for a CPU that combined Burrough's-style checks, Alpha ISA, and functional programming at system level. Given stack preference, you might like these:
I started out my career in IT as an AS/400 operator / Netware 3.12 admin, and while AS/400 / iSeries aren't "en vogue" these days, I have a lot of respect for those machines. As you say, they are rock solid. One of the places I worked for had an even older machine, an IBM S/36 (predecessor to the AS/400) and while ancient, it just kept plugging away, day after day after day after day...
OTOH, you couldn't pay me to program in RPG/400 using SEU. Building menus, or playing around with a little CL on on the '400 is one thing. But RPG programming sucks. Well, it did anyway. Maybe things have gotten better. I understand the ILE stuff made RPG less column oriented and closer to a free-form language, but I had never had a chance to use that.
I remember original RPG as being the electronic descendant of the old IBM unit record machines, with their plug boards and mechanical processing cycles. That heritage likely predates even COBOL. IBM added many extensions over the years, and at one of my mainframe workplaces we even did online CICS programming with RPG (not fun at all!).
Who are you and how have you stolen my[1] early career history?!
Hahaha... well, it's a long story, regarding the early part of my career. Especially the whole bit about exactly how I got involved with AS/400's in the first place.
Let me guess you started in the early 90's, right?
Almost. I graduated H.S. in '91, started programming in '92 or so, but didn't start my first IT job until 1997.
Except that the OS is still provided by a different company to your actual hardware, so there's plenty of room for blame-passing, and most of the OS development is being done on x86 machines by people with no access to IBM Power hardware of any kind.
If I never hear "one throat to choke" again, I'll die a happy man.
This only works well if you can negotiate an acceptable SLA, your main vendor doesn't balk when integrating with subcontractors or other vendors and if you have a rock-solid vendor manager on your side enforcing the SLA.
Oh, that works great when you lose power to the rack. Or the datacentre. Or the SAN fails. Or the core routers. Or any of the many other SPOFs that can and do occur in a datacentre.
All of which are accounted for by having two or more of these, combined with the feature they call (I'm not kidding) Geographically Dispersed Parallel Sysplex (GDPS).
You can hook up multiple IBM mainframes remotely and set them up to automatically ensure consistent replication of machine state to various extents depending on your reliability vs. performance tradeoffs and replication distance (latency being the issue), all the way up to active-active operation across systems.
So in other words: It works far better than the failover options most people deploy on their off the shelf servers in their self-wired racks (and yes, I run my own setup across off the shelf servers; and no, they're not nearly as redundant as a pair of IBM mainframes).
Problem is we kitted out two 42U racks in two DCs with HP and EMC kit on VMware and got four humans for five years for less than the comparable quote from IBM. And we've tested replication and failover to the same extent and didnt have to rewrite the 2 million lines or so of code we have...
> All of which are accounted for by having two or more of these, combined with the feature they call (I'm not kidding) Geographically Dispersed Parallel Sysplex (GDPS).
And it is an awesome thing - although I didn't realise it supported zVM these days, rather than just zOS.
In any case, you've still got two baskets, which was my point.
I wish article (or IBM) provided some specs. Thousands of containers is just too broad of a statement- what are these containers doing? Running just a bash script hello world program or minting bitcoins?