Ksplice: kernel patches without reboots
| This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible. |
The kernel developers are generally quite good about responding to security problems. Once a vulnerability in the kernel has been found, a patch comes out in short order; system administrators can then apply the patch (or get a patched kernel from their distributor), reboot the system, and get on with life knowing that the vulnerability has been fixed. It is a system which works pretty well.
One little problem remains, though: rebooting the system is a pain. At a minimum, it requires a few minutes of down time. In many situations, that down time cannot be tolerated. Reboots also disrupt any ongoing work, break existing network connections, and can cause the loss of results from long-running processes. And, most importantly of all, reboots prove traumatic for a certain subset of Linux administrators who prize a long uptime above almost all other things. Administrators currently have to choose between multi-year uptimes and security fixes; anything which frees them from a dilemma of this magnitude can only be welcome.
That "anything" might just be a recently-announced project called ksplice. With ksplice, system administrators can have the best of both worlds: security fixes without unsightly reboots.
An in-depth explanation of how ksplice works can be found in this document [PDF]. In short, ksplice requires as input the source tree for the running kernel and the security patch. It will then build two kernels, one with the patch and one without; the kernels are built with a special set of options which makes it easy to figure out which functions change as a result of the patch. The two kernels will be compared, with the purpose of finding those functions. Changes can propagate further than one might expect, especially if, for example, an inline function is modified.
Once a list of changed functions has been made, the updated code for those functions is packaged into a kernel module and loaded into the system. Then comes the tricky part: getting the running kernel to start using the new code. That requires patching the running code, which is a risky thing to do. Ksplice starts with a call to stop_machine_run(), which dumps a high-priority thread onto each processor, thus taking control of all processors in the system. It then examines all threads in the system to ensure that none of them are running in the functions to be replaced; if so, trampoline jumps are patched into the beginning of each replaced function (they "bounce" the call to the old code into the replacement code) and life continues. Otherwise ksplice will back off and try again later.
This method imposes a number of limitations. One is that only code changes can be patched in with ksplice; patches which make changes to data structures cannot be accommodated. Another comes from the retry-based approach to ensuring that no threads are running in the patched functions; what happens if one of those functions is never free? Kernel functions like schedule(), sys_poll(), or sys_waitid() are likely to always have processes running within them. In cases like this, ksplice will eventually give up and inform the user that the patch cannot be done; it is simply not possible to make changes to those particular functions.
These limitations mean that, out of 50 security patches examined by the ksplice developers, eight could not be applied with ksplice. So multi-year uptimes are probably still incompatible with the application of all security patches. Even so, ksplice certainly has the potential to reduce patch-related downtime considerably. Chances are good that there will be a fair amount of interest in ksplice in sites running high-uptime, mission-critical systems.
There are few things in the way of an immediate merge of this code into the mainline. One is a matter of coding quality and can be fixed. Then, there is the matter of the lead developer being unconvinced that merging this code makes sense since it is, essentially, a standalone feature. Andi Kleen's response made the (usual) reasons for merging the code clear:
So, presumably, the code will eventually be proposed for a mainline merge. But there is one other little difficulty pointed out by Tomasz Chmielewski: Microsoft holds a patent described this way:
Microsoft came up with this novel new technique in the distant past: 2002. The posting immediately brought out a crowd of surprised graybeards who distinctly remember using such techniques on their PDP-11 systems some decades before Microsoft "invented" hot-patching. The basic claim of the patent would thus appear to be invalidated by some decades' worth of prior art, but some of the dependent claims include features (such as capturing all other processors on the system) which were unlikely to be useful on PDP-11s.
Given that the kernel developers are now well aware of this
patent, they must take it into account when deciding whether to accept this
code into the mainline. It would not be surprising if they chose to avoid
baiting the Microsoft FUD machine in this way, even if they all agreed that
the patent lacked validity. So a promising technology risks being left out
of the kernel as the result of a software patent which was filed at least
30 years too late.
| Index entries for this article | |
|---|---|
| Kernel | Ksplice |
| Kernel | Live patching |
| Kernel | Security/Security technologies |
(Log in to post comments)
Ksplice: kernel patches without reboots
Posted Apr 29, 2008 18:45 UTC (Tue) by xav (guest, #18536) [Link]
Time for a config option like CONFIG_COUNTRY_WITH_SANE_PATENT_LAWS on which that feature could depend ...
Ksplice: kernel patches without reboots
Posted May 8, 2008 20:14 UTC (Thu) by FredMobach (guest, #1755) [Link]
Why not invalidate the patent because of prior art ? In the eighties we had a sysadmin utility Helga on the Siemens BS2000 mainframe systems which offered the functionality of code reps in run time. Like a process I always ran at boot time : /EXEC $HELGA CHANGE X'4700',PBINIIH+BE END (A patch not supported by Siemens, but it worked for me.)
Ksplice: kernel patches without reboots
Posted Apr 29, 2008 20:47 UTC (Tue) by aleXXX (subscriber, #2742) [Link]
To me this sounds more like "weird hack" than "promising technology". I mean, is it really worth that much to save the reboot compared to having a strangely binary patched kernel ? Alex
Ksplice: kernel patches without reboots
Posted Apr 29, 2008 21:11 UTC (Tue) by proski (subscriber, #104) [Link]
Many promising technologies started as weird hacks.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 2:44 UTC (Wed) by clugstj (subscriber, #4020) [Link]
But many more weird hacks started as weird hacks.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 12:47 UTC (Wed) by hmh (subscriber, #3838) [Link]
And remain as weird hacks, causing trouble and maintenance nightmares. Which in fact IS a big issue.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 4:32 UTC (Wed) by jamesh (guest, #1159) [Link]
If there is a critical security vulnerability that must be fixed as soon as possible, but you have jobs running that you'd prefer not to kill, then it might be worth using such a tool (assuming it doesn't risk causing corruption). In the best case, things run smoothly and the vulnerability is patched. You can then perform the reboot at your leisure. If things don't work out, you can reboot into the new kernel as you would have done previously.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 8:40 UTC (Wed) by tzafrir (subscriber, #11501) [Link]
Such hack seems to be applicable to distro kernels: exactly the same kernel on many machines. Hence this work only needs to be done once.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 12:58 UTC (Wed) by miahfost (guest, #51602) [Link]
Think of a small switch somewhere running linux. An embedded version of linux perhaps. Now imagine it routes telephone calls, and is placed in a desert, 100 kilometers from anywhere, 10,000 kilometers from where it was developed. Now imagine it needs a new patch _without_ taking down 10,000 subscribers' service. Would be nice to have a thing like Ksplice, no?
Ksplice: kernel patches without reboots
Posted May 1, 2008 4:09 UTC (Thu) by brouhaha (subscriber, #1698) [Link]
Which is why phone switches like the AT&T 5ESS were designed such that the control processor (running Unix) could be rebooted without disrupting phone calls. At worst it introduced a slight delay in a subscriber getting a dial tone if they picked up just as the system was rebooting. No clever patch mechanism necessary. For something more complex than a phone switch, in-service-upgrade is more difficult, so this patch mechanism may be very useful.
Ksplice: kernel patches without reboots
Posted Jun 11, 2008 23:03 UTC (Wed) by telcoman (guest, #52500) [Link]
HW failure will tear down the calls anyway, if you rely purely on Ksplice. For telco systems you need HW redundancy, too. Especially if you are running a commercial network, you had better not loosing charging data due to HW issues. Nevertheless, Ksplice alone could help stateless services, like Google's search, where you can allow data loss due to HW issues. And it can help small systems, too, where HW failure is rare and less disturbing. Using Ksplice along with clustering solutions could improve the situation. Keep in mind, though, that you are just patching the kernel, but the applications remain unfixed. So you did not end up with a real solution.
Ksplice: kernel patches without reboots
Posted May 9, 2008 6:35 UTC (Fri) by xamal (guest, #51976) [Link]
I need to take the time to respond to this. This sort of thing is NOT new and has been done for a long long long time.
NASA does this as part of normal practice. Basically diff(objectold, objectnew) ... then carefully smush the diff in place. In an old Dr. Dobbs (of which I have a hard copy) the following was said:
DDJ: And when you had to hot-patch in flight?
GR: That's standard procedure. You always build in the ability to change it. ( Read ~ http://www.ddj.com/184411097 )
As further evidence that this is done on Spacecraft please read: http://www.flightsoftware.org/files/FSW07_Calder.ppt
If you really want to get into this read http://en.wikipedia.org/wiki/Binary_File_Descriptor_library
As I said before this stuff has been done since time immemorial (or at least when the first computers showed up) ... also read up self-modifying-code.
Ksplice: kernel patches without reboots
Posted Apr 29, 2008 21:00 UTC (Tue) by danielpf (guest, #4723) [Link]
For those wanting 100% uptime the dual (or more) computer "high availability" (HA) configuration already exists. This technique covers much more possible problems than kernel patches. So one can wonder if the ksplice feature is really useful. It could perhaps change the uptime from 99.995% to 99.996%, but those wanting 100% would anyway need a HA configuration.
Ksplice: kernel patches without reboots
Posted Apr 29, 2008 22:16 UTC (Tue) by gravious (guest, #7662) [Link]
You are quite right. As has been pointed out here recently, the system as a whole may achieve 100% availability by using pervasive redundancy but the individual machines in the system won't. The associated PDF suggests that there may be long lived processes that would be intolerant to any outages due to their inability to save state properly or would exhibit a certain unhappiness with network connections dying. One would suggest that a system administrator designing and implementing a HA environment would make sure that each and every piece of software on each machine could handle a reboot if necessary. While I love the idea of hot updating and while I think this implementation is a fine fine hack worthy of much admiration I would welcome a concrete use case other than, "look I have 5 years, 2 months, 4 hours and 16 minutes uptime".
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 2:52 UTC (Wed) by gdt (subscriber, #6284) [Link]
There are many cases where the redundancy path to high availability is too expensive. Linux running on an closet ethernet switch would be one example. Are you really going to provision a second set of switches and cabling for the average corporate desktop computer?
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 2:57 UTC (Wed) by dlang (guest, #313) [Link]
there is not much overlap between the set of uses where cost prohibits having redundant hardware and the set of uses where a system cannot be down for a reboot. Many upgrades to Cisco equipment requires a reboot, and they are used in many places that are extremely sensitive to outages
Ksplice: kernel patches without reboots
Posted May 1, 2008 2:33 UTC (Thu) by a9db0 (subscriber, #2181) [Link]
Here's one: telecom Big phone switches aren't usually redundant, and are frequently utilized 24x7x365(6). The rise in VoIP has brought this into higher relief, and I'd expect to see some of the telecom folks looking very hard at this. My uptime is never that good - the power around here is way too flaky. Even for my oversized ups.
Ksplice: kernel patches without reboots
Posted May 1, 2008 16:42 UTC (Thu) by piggy (guest, #18693) [Link]
The reason big phone switches have traditionally been non-redundant is that they were staggeringly expensive when first created. The telecom industry is still absorbing the consequences of a 1000X improvement in both price and performance. Reliability of individual components has also dropped by a couple orders of magnitude, so redundancy is becoming the solution of choice. I agree with earlier assertions that the disjunction between businesses who want long uptimes and and those willing to put in redundant equipment is vanishingly small. Perhaps individuals after long uptime for geek-cred are a large enough population to sustain ksplice.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 18:14 UTC (Wed) by droundy (subscriber, #4559) [Link]
Indeed. Another example would be that of scientific computing. If I've got a job that has been running for a couple of weeks, and will finish in just a couple weeks more, I'd rather not reboot the system. Redundancy gains me nothing (since I'm utilizing all my computing resources already). The code could be trained to checkpoint (and some of my code is so enabled), but that generally has a high cost (in terms of bandwidth and disk use), so you don't want to checkpoint very often, if at all.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 20:53 UTC (Wed) by dlang (guest, #313) [Link]
any system like this should be isolated anyway, so delaying the security update for a week or a month to let your job finish should not be a big problem. remember that if a box is not exposed it doesn't need a security update.
Ksplice: kernel patches without reboots
Posted May 1, 2008 0:22 UTC (Thu) by gdt (subscriber, #6284) [Link]
any system like this should be isolated anyway
In practice that's increasingly difficult. Datasets are growing so large that the last thing you want is two copies of them, so you end up with the input data being remotely hosted and pulled across the Internet on demand. It's this sort of use that the academic community created the Internet for.
The other problem with scientific computing is simply that I might not want to reboot the system at this moment. Imagine that I've concurrently booked four radiotelescopes, which is about a six-month wait. I've got them streaming into my processing cluster. A security patch arrives. If I apply the patch and reboot then I lose resolution, and thus my experiment may be inconclusive. If I don't apply the patch and the machine is subverted then there are data integrity issues and again the experiment is inconclusive. In both cases I wait another six months and try again. My favoured choice would be to apply the patch whilst still running the telescope correlation.
I'm not saying the ksplice is the best thing since sliced bread. But it does have some use, particularly outside of the typical server application that Linux is generally used for.
Ksplice: kernel patches without reboots
Posted May 1, 2008 0:36 UTC (Thu) by dlang (guest, #313) [Link]
if you don't have more then one copy of your data you run the serious risk of loosing it. even for huge datasets, it's cheaper to keep an extra copy then to recreate the data. I'm not saying that ksplice is worthless, I'm disagreeing with the idea that was posted that it's required for these situations.
Ksplice: kernel patches without reboots
Posted May 1, 2008 12:57 UTC (Thu) by nix (subscriber, #2304) [Link]
Yeah, but using an extra copy for failover requires that it be online *now*. Using an extra copy for redundancy only does not require that (and is much cheaper: how will you keep an extra online copy of the ATLAS detector's collected data? It's far too large to keep even *one* copy at any one site: keeping an extra online copy means doubling the size of an already large collaboration...)
Ksplice: kernel patches without reboots
Posted May 8, 2008 11:40 UTC (Thu) by anandsr21 (guest, #28562) [Link]
Do you know how much data Google keeps. And they keep three copies not too. And in Geographically separated locations. So the solution is essentially to make multiple copies. Actually as Google has shown even two copies are not enough.
Ksplice: kernel patches without reboots
Posted May 1, 2008 13:04 UTC (Thu) by richardr (guest, #14799) [Link]
But the point about academic workloads is that often we use every desktop in the department as a distributed supercomputer, so the nodes are both exposed to every possible attack because people want their desktops accessible from outside (at least via ssh) and want to be able to surf the web, and may be running background jobs for weeks at a time belonging to other people who don't want them to be restarted. The conflict between these two factors is where this kind of technology becomes important.
Ksplice: kernel patches without reboots
Posted May 1, 2008 21:20 UTC (Thu) by dlang (guest, #313) [Link]
if you are running on random desktops that are used for other things, your software had better be able to handle reboots/crashes/power outages anyway as those events will happen. while I see some use for live patching, I really don't see where it becomes a killer feature
Orthogonal elements
Posted Apr 29, 2008 22:26 UTC (Tue) by tialaramex (subscriber, #21167) [Link]
These seem like orthogonal elements of a HA systems approach. You seem to equate "there are two boxes and some HA software" with 100% availability, but obviously it doesn't really deliver that, otherwise the phrase "belt, braces and skyhook" wouldn't exist. Suppose you have a two box HA system, and one day you take down box A to upgrade the kernel. But just short while into this procedure a tiny capacitor blows up inside box B, and the smoke leaks out. Now you have zero boxes, and your HA solution has failed. If, in contrast, you had moved the workload from box A to box B not so that you could reboot box A into a new kernel, but just so that it could be patched using ksplice while quiescent, then you'd survive this disaster with only mild inconvenience (box B having failed, box A would suddenly become very busy, most likely causing your ksplice to fail) Availability is a trade, usually against some combination of hardware investment, administrative overheads and system performance. For some people ksplice could let them spend a bit less money and get the same availability.
Orthogonal elements
Posted Apr 30, 2008 6:52 UTC (Wed) by NAR (subscriber, #1313) [Link]
Obviously you can't ever achieve 100% availability - what if the Sun turns into supernova and burns the whole Earth? Of course, this has very low probability, but I believe that hardware failure on the live machine while the standby is being upgraded is also a pretty low probability event. The other important thing: 8 fixes out of 50 couldn't be applied this way - in other words: the user will have to reboot anyway sooner or later. So why not design the system in a way that it could deal with reboots without service outage, if it's really important?
Orthogonal elements
Posted Apr 30, 2008 12:20 UTC (Wed) by nix (subscriber, #2304) [Link]
Because it may be really *expensive*. Fundamentally, if ever a system is taken out of service, whatever redundancy mechanisms may be in place may happen to fail then. Thus it is wise to reduce the time for which a machine is out of service. ksplice can, for a large proportion of smaller patches, reduce this period to a fraction of a second, with no loss of non-persistent state. And that seems worthwhile to me.
Orthogonal elements
Posted Apr 30, 2008 12:48 UTC (Wed) by NAR (subscriber, #1313) [Link]
Because it may be really *expensive*.I believe in this case there's no free lunch. I think if a user can't accept a couple of minutes downtime for a reboot, but isn't willing to invest in spare hardware won't really get high availability, only a sense of high availabilty.
I've checked CVE and found 10 linux kernel problems for this year. At this rate there could be 30 such problems in a year - if all problems need a reboot, than that's about 60-90 minutes of downtime. I guess a hardware failure would lead to a longer downtime (it takes time to get the new hardware) and a hardware failure isn't schedulable to off-business or off-peak hours (unlike the kernel change).
To me, this solution looks to be more like a toy (or a weird hack, as it was put earlier in this thread). It's nice, but I wouldn't use it for real.
Orthogonal elements
Posted May 1, 2008 3:56 UTC (Thu) by himi (subscriber, #340) [Link]
60-90 minutes of downtime for thirty reboots? Hell, I couldn't guarantee that with the tiny little Dell PE860s I'm working with at the moment - it'd be more like 300 minutes for some of the larger and more interesting hardware configurations we use in my organisation. Or far more, when one of those reboots includes an fsck of a multi-terabyte filesystem. ksplice sounds like a weird kludge at first but I can actually see it being very useful, particularly for systems running unmodified vendor kernels. It might fail occasionally (and a 1 in 5 failure rate is pretty good for something like this) but that still means 80% fewer reboots. himi
Orthogonal elements
Posted Apr 30, 2008 12:26 UTC (Wed) by pboddie (subscriber, #50784) [Link]
Obviously you can't ever achieve 100% availability - what if the Sun turns into supernova and burns the whole Earth?
You'll be glad you had that redundant Sun and redundant Earth floating around! ;-)
Orthogonal elements
Posted Apr 30, 2008 23:16 UTC (Wed) by nix (subscriber, #2304) [Link]
Hence the term `hot spare'...
Orthogonal elements
Posted May 1, 2008 4:18 UTC (Thu) by lysse (guest, #3190) [Link]
> Obviously you can't ever achieve 100% availability - what if the Sun turns into supernova and burns the whole Earth? Of course, this has very low probability ...not to mention that if your clients' main concern in such an eventuality is the discontinuation of their service, there are a few thousand UFOlogists out there who are going to want to examine your contract very closely...
Orthogonal elements
Posted Apr 30, 2008 14:36 UTC (Wed) by Los__D (guest, #15263) [Link]
Dammit, we lost the magic smoke again.
Determining applicability is hard
Posted Apr 29, 2008 23:36 UTC (Tue) by jreiser (subscriber, #11027) [Link]
"A person is expected to confirm that the target security patch does not make any semantic changes to data structures. Performing this check requires only seconds or a few minutes for most security patches." (Section 1)Changing a CRC algorithm is a semantic change to any piece of data that is derived from any output of that CRC algorithm. Thus patching a CRC algorithm must not be allowed as long as there are any outputs "in view", whether static globals, static locals, the stack of any thread, or dynamically allocated data. Now go back and reread this paragraph, deleting all instances of "CRC". Therefore, almost anything that is to be patched must be memoryless, either forever, or at the moment. That's hard.
Determining applicability is hard
Posted Apr 30, 2008 2:20 UTC (Wed) by dvdeug (subscriber, #10998) [Link]
True, if you change a CRC algorithm, you'll have to fix the data in memory. But there's a lot of changes out there that can be done without changing the data in the memory. If your CRC algorithm crashes on certain inputs, that can be changed without changing the data in the memory, for example. Anytime the bug is "this algorithm does the wrong thing in these circumstances that mean a hacker is knocking on your door", it can be changed to do the right thing in just those circumstances without affecting data in memory at all.
Determining applicability is hard
Posted Apr 30, 2008 3:45 UTC (Wed) by jreiser (subscriber, #11027) [Link]
"But there's a lot of changes out there that can be done without changing the data in the memory."The implication is: for any subroutine that is to be patched, along the dynamic execution path from the entry point to any instruction that is effectively modifed by the patch, then there have been no stores to memory (ever in the total execution history since booting the operating system, particularly including past intrusion attempts) except to on-stack local variables of the present subroutine. Within any patched subroutine, all prefixes of execution history up to any point of change must be (and have been) read-only except for automatic locals. That is more than a little restrictive.
Determining applicability is hard
Posted Apr 30, 2008 11:00 UTC (Wed) by dvdeug (subscriber, #10998) [Link]
Why? That's not what "semantic changes to data structures" means; it means that you can't change the meaning or layout of data structures. What implies that it can't use stores to memory?
Determining applicability is hard
Posted Apr 30, 2008 14:54 UTC (Wed) by jreiser (subscriber, #11027) [Link]
If any of the past outputs are "in view" (are stored in memory, or were used as inputs to compute data that is stored in memory at the time of the patch), then changing an algorithm such that its output changes, is a semantic change to the past outputs. Running the patched algorithm on the original inputs won't give the same outputs as the original algorithm, and many things rely on getting the same output.
Determining applicability is hard
Posted May 1, 2008 0:59 UTC (Thu) by jzbiciak (guest, #5246) [Link]
I'm not sure I follow. Let's look at a patch, shall we?
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -983,6 +983,8 @@ int get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
int i;
unsigned int vm_flags;
+ if (len <= 0)
+ return 0;
/*
* Require read or write permissions.
* If 'force' is set, we only require the "MAY" flags.
(If our dear editor is reading, he might get a chuckle out of my selection.)
I would argue that "get_user_pages" has plenty of side effects in memory on nearly all interesting execution paths. And yet, I imagine this splicing trick works just fine with this patch.
We've changed the algorithm for "get_user_pages" so it doesn't do strange and mysterious things (occasionally leading to local-root exploits) when the length is negative. Since we didn't free all other user pages ahead of time, previous outputs are "in view."
What I *will* buy is if someone exploited this hole on a running kernel, patching the bug after the fact won't cause the bogus allocation to disappear. But, in that case, you're b0rked anyway.
Your argument applies only when valid inputs give materially different outputs. Note that I say materially different. If I change Initial Sequence Number generation in my TCP/IP stack (a noticeable change!), I don't have to close all other TCP connections that are in the process of being established. I just get better behavior going forward.
Determining applicability is hard
Posted May 9, 2008 6:40 UTC (Fri) by xamal (guest, #51976) [Link]
Read my comments above where I post references to how and why this is done. Some systems do not have the luxury of someone physically going to install new software and flicking a switch. When applying a HOT-PATCH you do want to ensure that the patched area is not cached or some intensive application is going on. The ideal way is to send the computer to some idle state (some sort of idle loop), do the patch and transition from the idle mode.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 3:11 UTC (Wed) by joey (guest, #328) [Link]
One use case I can see is large xen hosting providers. Upgrading the domU kernels is a real pain for such providers, because just rebooting all the user's machines won't work; a certian number will fail to come up, users will hate you. Instead, a good hosting provider will let the user choose when to reboot into a fixed kernel. But then some percentage of users never will, and instead get cracked. So I can see hotpatching the kernel being really appealing in that situation, even if it's annoying to manage. (Personally, I'd rather my xen hosting provider just let me build and run my *own* kernel. But that's just me.)
Foolish, not good
Posted Apr 30, 2008 4:33 UTC (Wed) by khim (subscriber, #9252) [Link]
Instead, a good hosting provider will let the user choose when to reboot into a fixed kernel.
Why? Xen long had a way to solve this problem: live migration. When and how user updates kernel in VPS installation is another question...
Foolish, not good
Posted Apr 30, 2008 6:51 UTC (Wed) by joey (guest, #328) [Link]
Live migration to a different version of the kernel? I thought that xen's migration was of a whole domU, including kernel.
Foolish, not good
Posted May 1, 2008 13:20 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]
You are correct. khim is probably thinking of what you do when dom0's kernel needs to be patched.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 6:41 UTC (Wed) by NAR (subscriber, #1313) [Link]
Well, erlang has this feature since the middle of 90s, so that's a prior art too...
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 7:40 UTC (Wed) by jamesh (guest, #1159) [Link]
Python's reload() builtin might also be covered by the patent too. If that is considered too simple, some of the "super reload" modules that try harder to replace references to the old code might be covered.
Ksplice: kernel patches without reboots
Posted May 1, 2008 0:48 UTC (Thu) by flewellyn (subscriber, #5047) [Link]
And various Lisp systems had it in the 1970s, so there's a great deal more prior art there.
Ksplice: kernel patches without reboots
Posted May 8, 2008 14:09 UTC (Thu) by forthy (guest, #1525) [Link]
Interactive programming languages typically have this feature, i.e. Lisp, Smalltalk, and to some extent Forth (Forth usually does not change old functions when you redefine one, but it's often possible to patch whatever old function to point to a newer one if you like to). You just reload your source.
This is only so tricky, because Linux is written in C, a language that has no intention to generate, replace, or modify code at run-time (it's a classical batch-oriented edit-compile-link-go language). Therefore, the programmers have no idea how to do that properly - it's fairly trivial if done right, but needs the support of the language. Changing e.g. GCC to add (on x86) a simple 5-byte noop in front of each function to be patched with a jmp to the replacement function would allow seamless patching of the kernel - you don't have to wait for the previous call to this function to complete, since the replacement is somewhere else, and all further callers of this function will call the replacement instead.
Ksplice: kernel patches without reboots
Posted May 8, 2008 19:55 UTC (Thu) by nix (subscriber, #2304) [Link]
How do you ensure that concurrent callers on other CPUs aren't located inside those five bytes?
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 8:14 UTC (Wed) by Jel (guest, #22988) [Link]
This sounds surprisingly like the Amiga's old SetFunction() call, which was used for various things: live updates to the ROM kernel from more recent disks (very similar indeed), nifty tricks by applications (such as installing hooks around clock and GUI functions)... but also for installing viruses. The thing is... I always thought that unix's equivalent of SetFunction was LD_PRELOAD. It's interesting to see that the Amiga way is gaining traction.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 8:49 UTC (Wed) by tialaramex (subscriber, #21167) [Link]
What you're talking about is called "hooking" and is much older than the Amiga. It's also the technique used to create background programs in MS DOS, and to live patch some operating systems a decade or so before the Amiga. There have been tools that hooked the Linux kernel for a long time, but generally they're frowned on. For example, there's an old program written by Russinovich IIRC that hooks the file-related syscalls so that it can tell you which programs are opening and closing specific files. The /right/ way to do this is to instrument the functions rather than trying to hook them in this way and so in a modern Linux kernel that's what's provided. Ksplice isn't hooking, it's a little more subtle, the objective is to actually alter the code in place, rather than just patching the address of a function. This means that the result, when it works, should be more robust. LD_PRELOAD affects the userspace environment by configuring the runtime linker, the Amiga doesn't have anything analogous to that, but then it forces everything into a single shared address space, so allowing LD_PRELOAD type tricks would cause all sorts of problems on an Amiga.
Ksplice: kernel patches without reboots
Posted May 1, 2008 4:25 UTC (Thu) by lysse (guest, #3190) [Link]
> What you're talking about is called "hooking" and is much older than the Amiga. It's also the technique used to create background programs in MS DOS Indeed, DOS itself does it to the BIOS. However, as the allusion to seizing all processors in a system illustrates, that bit probably isn't the interesting bit of the patent; on the other hand, whilst SMP may not have been a big concern for the PDP-11, DEC does hold quite a lot of prior art around the whole subject of SMP, thanks to the PDP-10 series; if a technique was used on the '11, chances are it was worked out on the '10. And of course, the developer of NT would know about that, given his history (even if the '10 team did disdain him for his "small system thinking").
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 10:00 UTC (Wed) by dambacher (subscriber, #1710) [Link]
what if someone provides a patch to silently _include_ a security hole? How can a sysadmin make sure his kernel won't be patched that way?
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 11:16 UTC (Wed) by nowster (subscriber, #67) [Link]
The black hats have already done this. Modules that patch the kernel have been part of rootkits for a while.
Ksplice: kernel patches without reboots
Posted Apr 30, 2008 12:24 UTC (Wed) by nix (subscriber, #2304) [Link]
Don't allow module loading and remove CAP_SYS_RAWIO from the capability bounding set so that use of /dev/mem, /dev/kmem et al is barred. (Of course this stops you using ksplice, systemtap et al as well.)
The MS Patent
Posted Apr 30, 2008 12:50 UTC (Wed) by davecb (subscriber, #1574) [Link]
Was investigated on LKML and found to have been rejected. MS can still appeal, of course. --dave
The MS Patent
Posted May 1, 2008 4:51 UTC (Thu) by roelofs (guest, #2599) [Link]
Sigh... People really need to learn the difference between a patent application and an issued patent. Any idiot with a big enough pile of cash (on the order of $10K+, IIRC) can apply for a patent...Greg
Ksplice: kernel patches without reboots
Posted May 1, 2008 7:50 UTC (Thu) by stewart (subscriber, #50665) [Link]
There's a lot of prior art for this. Various telco switches had this capability... Ericsson AXE switches did at least... how much this was used or ever worked though, I'm not sure.
Ksplice: kernel patches without reboots
Posted May 1, 2008 15:00 UTC (Thu) by Lennie (subscriber, #49641) [Link]
Just out of curiosity... the comment above said: 'Well, erlang has this feature since the middle of 90s, so that's a prior art too...' And I know Erlang was created by Ericsson. Would it be wrong to conclude the switches might have been running Erlang ?
Ksplice: kernel patches without reboots
Posted May 1, 2008 23:24 UTC (Thu) by stewart (subscriber, #50665) [Link]
they weren't... Ericsson is pretty big, so lots of things happen and there's certainly been more than one programming language invented there. I was thinking of PLEX, used in AXE switches.
Ksplice: kernel patches without reboots
Posted Jun 11, 2008 22:13 UTC (Wed) by telcoman (guest, #52500) [Link]
Yes, Ericsson, Siemens and possibly many other telco manufacturers use the same solution for decades. So MS's patent is likely to be invalid. Patching is a very good thing, as long as you apply the right patches. Just remember the Ericsson scandal: http://www.spectrum.ieee.org/print/5280 So being able to patch a running kernel is a double edged sword. If someone gains root rights he can apply a nasty patch secretly and leave immediately. In most HA systems there is an alternative of patching. On resilient systems you have quite enough hardware, so all you need to do is update the code (even the most frequently used ones) on a spare computer, move all the data (incl. data from RAM) there and continue the execution using the new code (or new hardware, if you had HW issues instead of SW ones). That is how Nokia DX200 switches work. The same restrictions apply for data structure changes as for patching, though.
Ksplice: kernel patches without reboots
Posted May 2, 2008 8:17 UTC (Fri) by krkumar2 (guest, #34243) [Link]
Can this be achieved by virtualization instead? Create a new partition with the freshly built kernel, move all processes to this partition, and delete the old partition and move all resources to the new partition?