summaryrefslogtreecommitdiff
path: root/arch/x86/mm/tlb.c
diff options
context:
space:
mode:
authorGreg Kroah-Hartman <gregkh@google.com>2019-05-16 20:09:36 +0200
committerGreg Kroah-Hartman <gregkh@google.com>2019-05-16 20:09:36 +0200
commit71cb827c0249e87c62139bd4d5897cf59f51cb55 (patch)
tree4fefc02eec875a9874d8776cca490ee0885998b7 /arch/x86/mm/tlb.c
parentce469a70f3bc41a1df2efc5c7f53502f0d88d2ec (diff)
parent0f654c12cd720e65f1fb3174a7ee468f1daa09e7 (diff)
Merge 4.4.180 into android-4.4
Changes in 4.4.180 kbuild: simplify ld-option implementation KVM: fail KVM_SET_VCPU_EVENTS with invalid exception number cifs: do not attempt cifs operation on smb2+ rename error MIPS: scall64-o32: Fix indirect syscall number load trace: Fix preempt_enable_no_resched() abuse sched/numa: Fix a possible divide-by-zero ceph: ensure d_name stability in ceph_dentry_hash() ceph: fix ci->i_head_snapc leak nfsd: Don't release the callback slot unless it was actually held sunrpc: don't mark uninitialised items as VALID. USB: Add new USB LPM helpers USB: Consolidate LPM checks to avoid enabling LPM twice powerpc/xmon: Add RFI flush related fields to paca dump powerpc/64s: Improve RFI L1-D cache flush fallback powerpc/pseries: Support firmware disable of RFI flush powerpc/powernv: Support firmware disable of RFI flush powerpc/rfi-flush: Move the logic to avoid a redo into the debugfs code powerpc/rfi-flush: Make it possible to call setup_rfi_flush() again powerpc/rfi-flush: Always enable fallback flush on pseries powerpc/rfi-flush: Differentiate enabled and patched flush types powerpc/pseries: Add new H_GET_CPU_CHARACTERISTICS flags powerpc/rfi-flush: Call setup_rfi_flush() after LPM migration powerpc: Add security feature flags for Spectre/Meltdown powerpc/pseries: Set or clear security feature flags powerpc/powernv: Set or clear security feature flags powerpc/64s: Move cpu_show_meltdown() powerpc/64s: Enhance the information in cpu_show_meltdown() powerpc/powernv: Use the security flags in pnv_setup_rfi_flush() powerpc/pseries: Use the security flags in pseries_setup_rfi_flush() powerpc/64s: Wire up cpu_show_spectre_v1() powerpc/64s: Wire up cpu_show_spectre_v2() powerpc/pseries: Fix clearing of security feature flags powerpc: Move default security feature flags powerpc/pseries: Restore default security feature flags on setup powerpc/64s: Fix section mismatch warnings from setup_rfi_flush() powerpc/64s: Add support for a store forwarding barrier at kernel entry/exit powerpc/64s: Add barrier_nospec powerpc/64s: Add support for ori barrier_nospec patching powerpc/64s: Patch barrier_nospec in modules powerpc/64s: Enable barrier_nospec based on firmware settings powerpc/64: Use barrier_nospec in syscall entry powerpc: Use barrier_nospec in copy_from_user() powerpc/64s: Enhance the information in cpu_show_spectre_v1() powerpc64s: Show ori31 availability in spectre_v1 sysfs file not v2 powerpc/64: Disable the speculation barrier from the command line powerpc/64: Make stf barrier PPC_BOOK3S_64 specific. powerpc/64: Add CONFIG_PPC_BARRIER_NOSPEC powerpc/64: Call setup_barrier_nospec() from setup_arch() powerpc/64: Make meltdown reporting Book3S 64 specific powerpc/fsl: Add barrier_nospec implementation for NXP PowerPC Book3E powerpc/asm: Add a patch_site macro & helpers for patching instructions powerpc/64s: Add new security feature flags for count cache flush powerpc/64s: Add support for software count cache flush powerpc/pseries: Query hypervisor for count cache flush settings powerpc/powernv: Query firmware for count cache flush settings powerpc: Avoid code patching freed init sections powerpc/fsl: Add infrastructure to fixup branch predictor flush powerpc/fsl: Add macro to flush the branch predictor powerpc/fsl: Fix spectre_v2 mitigations reporting powerpc/fsl: Add nospectre_v2 command line argument powerpc/fsl: Flush the branch predictor at each kernel entry (64bit) powerpc/fsl: Update Spectre v2 reporting powerpc/security: Fix spectre_v2 reporting powerpc/fsl: Fix the flush of branch predictor. tipc: handle the err returned from cmd header function slip: make slhc_free() silently accept an error pointer intel_th: gth: Fix an off-by-one in output unassigning fs/proc/proc_sysctl.c: Fix a NULL pointer dereference NFS: Forbid setting AF_INET6 to "struct sockaddr_in"->sin_family. netfilter: ebtables: CONFIG_COMPAT: drop a bogus WARN_ON tipc: check bearer name with right length in tipc_nl_compat_bearer_enable tipc: check link name with right length in tipc_nl_compat_link_set bpf: reject wrong sized filters earlier Revert "block/loop: Use global lock for ioctl() operation." ipv4: add sanity checks in ipv4_link_failure() team: fix possible recursive locking when add slaves net: stmmac: move stmmac_check_ether_addr() to driver probe ipv4: set the tcp_min_rtt_wlen range from 0 to one day powerpc/fsl: Enable runtime patching if nospectre_v2 boot arg is used powerpc/fsl: Flush branch predictor when entering KVM powerpc/fsl: Emulate SPRN_BUCSR register powerpc/fsl: Flush the branch predictor at each kernel entry (32 bit) powerpc/fsl: Sanitize the syscall table for NXP PowerPC 32 bit platforms powerpc/fsl: Fixed warning: orphan section `__btb_flush_fixup' powerpc/fsl: Add FSL_PPC_BOOK3E as supported arch for nospectre_v2 boot arg Documentation: Add nospectre_v1 parameter usbnet: ipheth: prevent TX queue timeouts when device not ready usbnet: ipheth: fix potential null pointer dereference in ipheth_carrier_set qlcnic: Avoid potential NULL pointer dereference netfilter: bridge: set skb transport_header before entering NF_INET_PRE_ROUTING sc16is7xx: missing unregister/delete driver on error in sc16is7xx_init() usb: gadget: net2280: Fix overrun of OUT messages usb: gadget: net2280: Fix net2280_dequeue() usb: gadget: net2272: Fix net2272_dequeue() ARM: dts: pfla02: increase phy reset duration net: ks8851: Dequeue RX packets explicitly net: ks8851: Reassert reset pin if chip ID check fails net: ks8851: Delay requesting IRQ until opened net: ks8851: Set initial carrier state to down net: xilinx: fix possible object reference leak net: ibm: fix possible object reference leak net: ethernet: ti: fix possible object reference leak scsi: qla4xxx: fix a potential NULL pointer dereference usb: u132-hcd: fix resource leak ceph: fix use-after-free on symlink traversal scsi: zfcp: reduce flood of fcrscn1 trace records on multi-element RSCN libata: fix using DMA buffers on stack kconfig/[mn]conf: handle backspace (^H) key vfio/type1: Limit DMA mappings per container ALSA: line6: use dynamic buffers ipv4: ip_do_fragment: Preserve skb_iif during fragmentation ipv6/flowlabel: wait rcu grace period before put_pid() ipv6: invert flowlabel sharing check in process and user mode bnxt_en: Improve multicast address setup logic. packet: validate msg_namelen in send directly USB: yurex: Fix protection fault after device removal USB: w1 ds2490: Fix bug caused by improper use of altsetting array USB: core: Fix unterminated string returned by usb_string() USB: core: Fix bug caused by duplicate interface PM usage counter HID: debug: fix race condition with between rdesc_show() and device removal rtc: sh: Fix invalid alarm warning for non-enabled alarm igb: Fix WARN_ONCE on runtime suspend bonding: show full hw address in sysfs for slave entries jffs2: fix use-after-free on symlink traversal debugfs: fix use-after-free on symlink traversal rtc: da9063: set uie_unsupported when relevant vfio/pci: use correct format characters scsi: storvsc: Fix calculation of sub-channel count net: hns: Use NAPI_POLL_WEIGHT for hns driver net: hns: Fix WARNING when remove HNS driver with SMMU enabled hugetlbfs: fix memory leak for resv_map xsysace: Fix error handling in ace_setup ARM: orion: don't use using 64-bit DMA masks ARM: iop: don't use using 64-bit DMA masks usb: usbip: fix isoc packet num validation in get_pipe staging: iio: adt7316: allow adt751x to use internal vref for all dacs staging: iio: adt7316: fix the dac read calculation staging: iio: adt7316: fix the dac write calculation Input: snvs_pwrkey - initialize necessary driver data before enabling IRQ selinux: never allow relabeling on context mounts x86/mce: Improve error message when kernel cannot recover, p2 media: v4l2: i2c: ov7670: Fix PLL bypass register values scsi: libsas: fix a race condition when smp task timeout ASoC:soc-pcm:fix a codec fixup issue in TDM case ASoC: cs4270: Set auto-increment bit for register writes ASoC: tlv320aic32x4: Fix Common Pins perf/x86/intel: Fix handling of wakeup_events for multi-entry PEBS scsi: csiostor: fix missing data copy in csio_scsi_err_handler() iommu/amd: Set exclusion range correctly genirq: Prevent use-after-free and work list corruption usb: dwc3: Fix default lpm_nyet_threshold value scsi: qla2xxx: Fix incorrect region-size setting in optrom SYSFS routines Bluetooth: hidp: fix buffer overflow Bluetooth: Align minimum encryption key size for LE and BR/EDR connections UAS: fix alignment of scatter/gather segments ipv6: fix a potential deadlock in do_ipv6_setsockopt() ASoC: Intel: avoid Oops if DMA setup fails timer/debug: Change /proc/timer_stats from 0644 to 0600 netfilter: compat: initialize all fields in xt_init platform/x86: sony-laptop: Fix unintentional fall-through iio: adc: xilinx: fix potential use-after-free on remove HID: input: add mapping for Expose/Overview key HID: input: add mapping for keyboard Brightness Up/Down/Toggle keys libnvdimm/btt: Fix a kmemdup failure check s390/dasd: Fix capacity calculation for large volumes s390/3270: fix lockdep false positive on view->lock KVM: x86: avoid misreporting level-triggered irqs as edge-triggered in tracing tools lib traceevent: Fix missing equality check for strcmp init: initialize jump labels before command line option parsing ipvs: do not schedule icmp errors from tunnels s390: ctcm: fix ctcm_new_device error return code selftests/net: correct the return value for run_netsocktests gpu: ipu-v3: dp: fix CSC handling cw1200: fix missing unlock on error in cw1200_hw_scan() x86/vdso: Pass --eh-frame-hdr to the linker Don't jump to compute_result state from check_result state locking/static_keys: Provide DECLARE and well as DEFINE macros x86/microcode/intel: Add a helper which gives the microcode revision x86: stop exporting msr-index.h to userland bitops: avoid integer overflow in GENMASK(_ULL) x86/microcode/intel: Check microcode revision before updating sibling threads x86/MCE: Save microcode revision in machine check records x86/cpufeatures: Hide AMD-specific speculation flags x86/speculation: Support Enhanced IBRS on future CPUs x86/speculation: Simplify the CPU bug detection logic x86/bugs: Add AMD's variant of SSB_NO x86/bugs: Add AMD's SPEC_CTRL MSR usage x86/bugs: Switch the selection of mitigation from CPU vendor to CPU features locking/atomics, asm-generic: Move some macros from <linux/bitops.h> to a new <linux/bits.h> file x86/bugs: Fix the AMD SSBD usage of the SPEC_CTRL MSR x86/speculation: Remove SPECTRE_V2_IBRS in enum spectre_v2_mitigation x86/microcode: Make sure boot_cpu_data.microcode is up-to-date x86/microcode: Update the new microcode revision unconditionally x86/cpu: Sanitize FAM6_ATOM naming KVM: x86: SVM: Call x86_spec_ctrl_set_guest/host() with interrupts disabled x86/mm: Use WRITE_ONCE() when setting PTEs x86/speculation: Apply IBPB more strictly to avoid cross-process data leak x86/speculation: Enable cross-hyperthread spectre v2 STIBP mitigation x86/speculation: Propagate information about RSB filling mitigation to sysfs x86/speculation: Update the TIF_SSBD comment x86/speculation: Clean up spectre_v2_parse_cmdline() x86/speculation: Remove unnecessary ret variable in cpu_show_common() x86/speculation: Move STIPB/IBPB string conditionals out of cpu_show_common() x86/speculation: Disable STIBP when enhanced IBRS is in use x86/speculation: Rename SSBD update functions x86/speculation: Reorganize speculation control MSRs update x86/Kconfig: Select SCHED_SMT if SMP enabled sched: Add sched_smt_active() x86/speculation: Rework SMT state change x86/speculation: Reorder the spec_v2 code x86/speculation: Mark string arrays const correctly x86/speculataion: Mark command line parser data __initdata x86/speculation: Unify conditional spectre v2 print functions x86/speculation: Add command line control for indirect branch speculation x86/speculation: Prepare for per task indirect branch speculation control x86/process: Consolidate and simplify switch_to_xtra() code x86/speculation: Avoid __switch_to_xtra() calls x86/speculation: Prepare for conditional IBPB in switch_mm() x86/speculation: Split out TIF update x86/speculation: Prepare arch_smt_update() for PRCTL mode x86/speculation: Prevent stale SPEC_CTRL msr content x86/speculation: Add prctl() control for indirect branch speculation x86/speculation: Enable prctl mode for spectre_v2_user x86/speculation: Add seccomp Spectre v2 user space protection mode x86/speculation: Provide IBPB always command line options kvm: x86: Report STIBP on GET_SUPPORTED_CPUID x86/msr-index: Cleanup bit defines x86/speculation: Consolidate CPU whitelists x86/speculation/mds: Add basic bug infrastructure for MDS x86/speculation/mds: Add BUG_MSBDS_ONLY x86/kvm: Expose X86_FEATURE_MD_CLEAR to guests x86/speculation/mds: Add mds_clear_cpu_buffers() x86/speculation/mds: Clear CPU buffers on exit to user x86/speculation/mds: Conditionally clear CPU buffers on idle entry x86/speculation/mds: Add mitigation control for MDS x86/speculation/l1tf: Document l1tf in sysfs x86/speculation/mds: Add sysfs reporting for MDS x86/speculation/mds: Add mitigation mode VMWERV Documentation: Move L1TF to separate directory Documentation: Add MDS vulnerability documentation x86/cpu/bugs: Use __initconst for 'const' init data x86/speculation: Move arch_smt_update() call to after mitigation decisions x86/speculation/mds: Add SMT warning message x86/speculation/mds: Fix comment x86/speculation/mds: Print SMT vulnerable on MSBDS with mitigations off cpu/speculation: Add 'mitigations=' cmdline option x86/speculation: Support 'mitigations=' cmdline option x86/speculation/mds: Add 'mitigations=' support for MDS x86/mds: Add MDSUM variant to the MDS documentation Documentation: Correct the possible MDS sysfs values x86/speculation/mds: Fix documentation typo x86/bugs: Change L1TF mitigation string to match upstream USB: serial: use variable for status USB: serial: fix unthrottle races powerpc/64s: Include cpu header bridge: Fix error path for kobject_init_and_add() net: ucc_geth - fix Oops when changing number of buffers in the ring packet: Fix error path in packet_init vlan: disable SIOCSHWTSTAMP in container vrf: sit mtu should not be updated when vrf netdev is the link ipv4: Fix raw socket lookup for local traffic bonding: fix arp_validate toggling in active-backup mode drivers/virt/fsl_hypervisor.c: dereferencing error pointers in ioctl drivers/virt/fsl_hypervisor.c: prevent integer overflow in ioctl powerpc/booke64: set RI in default MSR powerpc/lib: fix book3s/32 boot failure due to code patching Linux 4.4.180 Change-Id: If2d2fdd451b55c002666b32022b269cec9545607 Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Diffstat (limited to 'arch/x86/mm/tlb.c')
-rw-r--r--arch/x86/mm/tlb.c114
1 files changed, 90 insertions, 24 deletions
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 6d683bbb3502..f3237e4cb18f 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -30,6 +30,12 @@
* Implement flush IPI by CALL_FUNCTION_VECTOR, Alex Shi
*/
+/*
+ * Use bit 0 to mangle the TIF_SPEC_IB state into the mm pointer which is
+ * stored in cpu_tlb_state.last_user_mm_ibpb.
+ */
+#define LAST_USER_MM_IBPB 0x1UL
+
atomic64_t last_mm_ctx_id = ATOMIC64_INIT(1);
struct flush_tlb_info {
@@ -101,41 +107,101 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,
local_irq_restore(flags);
}
-void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
- struct task_struct *tsk)
+static inline unsigned long mm_mangle_tif_spec_ib(struct task_struct *next)
{
- unsigned cpu = smp_processor_id();
+ unsigned long next_tif = task_thread_info(next)->flags;
+ unsigned long ibpb = (next_tif >> TIF_SPEC_IB) & LAST_USER_MM_IBPB;
- if (likely(prev != next)) {
- u64 last_ctx_id = this_cpu_read(cpu_tlbstate.last_ctx_id);
+ return (unsigned long)next->mm | ibpb;
+}
+
+static void cond_ibpb(struct task_struct *next)
+{
+ if (!next || !next->mm)
+ return;
+
+ /*
+ * Both, the conditional and the always IBPB mode use the mm
+ * pointer to avoid the IBPB when switching between tasks of the
+ * same process. Using the mm pointer instead of mm->context.ctx_id
+ * opens a hypothetical hole vs. mm_struct reuse, which is more or
+ * less impossible to control by an attacker. Aside of that it
+ * would only affect the first schedule so the theoretically
+ * exposed data is not really interesting.
+ */
+ if (static_branch_likely(&switch_mm_cond_ibpb)) {
+ unsigned long prev_mm, next_mm;
/*
- * Avoid user/user BTB poisoning by flushing the branch
- * predictor when switching between processes. This stops
- * one process from doing Spectre-v2 attacks on another.
+ * This is a bit more complex than the always mode because
+ * it has to handle two cases:
+ *
+ * 1) Switch from a user space task (potential attacker)
+ * which has TIF_SPEC_IB set to a user space task
+ * (potential victim) which has TIF_SPEC_IB not set.
+ *
+ * 2) Switch from a user space task (potential attacker)
+ * which has TIF_SPEC_IB not set to a user space task
+ * (potential victim) which has TIF_SPEC_IB set.
+ *
+ * This could be done by unconditionally issuing IBPB when
+ * a task which has TIF_SPEC_IB set is either scheduled in
+ * or out. Though that results in two flushes when:
+ *
+ * - the same user space task is scheduled out and later
+ * scheduled in again and only a kernel thread ran in
+ * between.
+ *
+ * - a user space task belonging to the same process is
+ * scheduled in after a kernel thread ran in between
*
- * As an optimization, flush indirect branches only when
- * switching into processes that disable dumping. This
- * protects high value processes like gpg, without having
- * too high performance overhead. IBPB is *expensive*!
+ * - a user space task belonging to the same process is
+ * scheduled in immediately.
*
- * This will not flush branches when switching into kernel
- * threads. It will also not flush if we switch to idle
- * thread and back to the same process. It will flush if we
- * switch to a different non-dumpable process.
+ * Optimize this with reasonably small overhead for the
+ * above cases. Mangle the TIF_SPEC_IB bit into the mm
+ * pointer of the incoming task which is stored in
+ * cpu_tlbstate.last_user_mm_ibpb for comparison.
*/
- if (tsk && tsk->mm &&
- tsk->mm->context.ctx_id != last_ctx_id &&
- get_dumpable(tsk->mm) != SUID_DUMP_USER)
+ next_mm = mm_mangle_tif_spec_ib(next);
+ prev_mm = this_cpu_read(cpu_tlbstate.last_user_mm_ibpb);
+
+ /*
+ * Issue IBPB only if the mm's are different and one or
+ * both have the IBPB bit set.
+ */
+ if (next_mm != prev_mm &&
+ (next_mm | prev_mm) & LAST_USER_MM_IBPB)
indirect_branch_prediction_barrier();
+ this_cpu_write(cpu_tlbstate.last_user_mm_ibpb, next_mm);
+ }
+
+ if (static_branch_unlikely(&switch_mm_always_ibpb)) {
/*
- * Record last user mm's context id, so we can avoid
- * flushing branch buffer with IBPB if we switch back
- * to the same user.
+ * Only flush when switching to a user space task with a
+ * different context than the user space task which ran
+ * last on this CPU.
+ */
+ if (this_cpu_read(cpu_tlbstate.last_user_mm) != next->mm) {
+ indirect_branch_prediction_barrier();
+ this_cpu_write(cpu_tlbstate.last_user_mm, next->mm);
+ }
+ }
+}
+
+void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
+ struct task_struct *tsk)
+{
+ unsigned cpu = smp_processor_id();
+
+ if (likely(prev != next)) {
+ /*
+ * Avoid user/user BTB poisoning by flushing the branch
+ * predictor when switching between processes. This stops
+ * one process from doing Spectre-v2 attacks on another.
*/
- if (next != &init_mm)
- this_cpu_write(cpu_tlbstate.last_ctx_id, next->context.ctx_id);
+ cond_ibpb(tsk);
this_cpu_write(cpu_tlbstate.state, TLBSTATE_OK);
this_cpu_write(cpu_tlbstate.active_mm, next);