Coverage Report

Created: 2019-07-24 05:18

/Users/buildslave/jenkins/workspace/clang-stage2-coverage-R/llvm/lib/Target/AArch64/AArch64SpeculationHardening.cpp
Line
Count
Source (jump to first uncovered line)
1
//===- AArch64SpeculationHardening.cpp - Harden Against Missspeculation  --===//
2
//
3
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4
// See https://llvm.org/LICENSE.txt for license information.
5
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6
//
7
//===----------------------------------------------------------------------===//
8
//
9
// This file contains a pass to insert code to mitigate against side channel
10
// vulnerabilities that may happen under control flow miss-speculation.
11
//
12
// The pass implements tracking of control flow miss-speculation into a "taint"
13
// register. That taint register can then be used to mask off registers with
14
// sensitive data when executing under miss-speculation, a.k.a. "transient
15
// execution".
16
// This pass is aimed at mitigating against SpectreV1-style vulnarabilities.
17
//
18
// It also implements speculative load hardening, i.e. using the taint register
19
// to automatically mask off loaded data.
20
//
21
// As a possible follow-on improvement, also an intrinsics-based approach as
22
// explained at https://lwn.net/Articles/759423/ could be implemented on top of
23
// the current design.
24
//
25
// For AArch64, the following implementation choices are made to implement the
26
// tracking of control flow miss-speculation into a taint register:
27
// Some of these are different than the implementation choices made in
28
// the similar pass implemented in X86SpeculativeLoadHardening.cpp, as
29
// the instruction set characteristics result in different trade-offs.
30
// - The speculation hardening is done after register allocation. With a
31
//   relative abundance of registers, one register is reserved (X16) to be
32
//   the taint register. X16 is expected to not clash with other register
33
//   reservation mechanisms with very high probability because:
34
//   . The AArch64 ABI doesn't guarantee X16 to be retained across any call.
35
//   . The only way to request X16 to be used as a programmer is through
36
//     inline assembly. In the rare case a function explicitly demands to
37
//     use X16/W16, this pass falls back to hardening against speculation
38
//     by inserting a DSB SYS/ISB barrier pair which will prevent control
39
//     flow speculation.
40
// - It is easy to insert mask operations at this late stage as we have
41
//   mask operations available that don't set flags.
42
// - The taint variable contains all-ones when no miss-speculation is detected,
43
//   and contains all-zeros when miss-speculation is detected. Therefore, when
44
//   masking, an AND instruction (which only changes the register to be masked,
45
//   no other side effects) can easily be inserted anywhere that's needed.
46
// - The tracking of miss-speculation is done by using a data-flow conditional
47
//   select instruction (CSEL) to evaluate the flags that were also used to
48
//   make conditional branch direction decisions. Speculation of the CSEL
49
//   instruction can be limited with a CSDB instruction - so the combination of
50
//   CSEL + a later CSDB gives the guarantee that the flags as used in the CSEL
51
//   aren't speculated. When conditional branch direction gets miss-speculated,
52
//   the semantics of the inserted CSEL instruction is such that the taint
53
//   register will contain all zero bits.
54
//   One key requirement for this to work is that the conditional branch is
55
//   followed by an execution of the CSEL instruction, where the CSEL
56
//   instruction needs to use the same flags status as the conditional branch.
57
//   This means that the conditional branches must not be implemented as one
58
//   of the AArch64 conditional branches that do not use the flags as input
59
//   (CB(N)Z and TB(N)Z). This is implemented by ensuring in the instruction
60
//   selectors to not produce these instructions when speculation hardening
61
//   is enabled. This pass will assert if it does encounter such an instruction.
62
// - On function call boundaries, the miss-speculation state is transferred from
63
//   the taint register X16 to be encoded in the SP register as value 0.
64
//
65
// For the aspect of automatically hardening loads, using the taint register,
66
// (a.k.a. speculative load hardening, see
67
//  https://llvm.org/docs/SpeculativeLoadHardening.html), the following
68
// implementation choices are made for AArch64:
69
//   - Many of the optimizations described at
70
//     https://llvm.org/docs/SpeculativeLoadHardening.html to harden fewer
71
//     loads haven't been implemented yet - but for some of them there are
72
//     FIXMEs in the code.
73
//   - loads that load into general purpose (X or W) registers get hardened by
74
//     masking the loaded data. For loads that load into other registers, the
75
//     address loaded from gets hardened. It is expected that hardening the
76
//     loaded data may be more efficient; but masking data in registers other
77
//     than X or W is not easy and may result in being slower than just
78
//     hardening the X address register loaded from.
79
//   - On AArch64, CSDB instructions are inserted between the masking of the
80
//     register and its first use, to ensure there's no non-control-flow
81
//     speculation that might undermine the hardening mechanism.
82
//
83
// Future extensions/improvements could be:
84
// - Implement this functionality using full speculation barriers, akin to the
85
//   x86-slh-lfence option. This may be more useful for the intrinsics-based
86
//   approach than for the SLH approach to masking.
87
//   Note that this pass already inserts the full speculation barriers if the
88
//   function for some niche reason makes use of X16/W16.
89
// - no indirect branch misprediction gets protected/instrumented; but this
90
//   could be done for some indirect branches, such as switch jump tables.
91
//===----------------------------------------------------------------------===//
92
93
#include "AArch64InstrInfo.h"
94
#include "AArch64Subtarget.h"
95
#include "Utils/AArch64BaseInfo.h"
96
#include "llvm/ADT/BitVector.h"
97
#include "llvm/ADT/SmallVector.h"
98
#include "llvm/CodeGen/MachineBasicBlock.h"
99
#include "llvm/CodeGen/MachineFunction.h"
100
#include "llvm/CodeGen/MachineFunctionPass.h"
101
#include "llvm/CodeGen/MachineInstr.h"
102
#include "llvm/CodeGen/MachineInstrBuilder.h"
103
#include "llvm/CodeGen/MachineOperand.h"
104
#include "llvm/CodeGen/MachineRegisterInfo.h"
105
#include "llvm/CodeGen/RegisterScavenging.h"
106
#include "llvm/IR/DebugLoc.h"
107
#include "llvm/Pass.h"
108
#include "llvm/Support/CodeGen.h"
109
#include "llvm/Target/TargetMachine.h"
110
#include <cassert>
111
112
using namespace llvm;
113
114
#define DEBUG_TYPE "aarch64-speculation-hardening"
115
116
266k
#define AARCH64_SPECULATION_HARDENING_NAME "AArch64 speculation hardening pass"
117
118
cl::opt<bool> HardenLoads("aarch64-slh-loads", cl::Hidden,
119
                          cl::desc("Sanitize loads from memory."),
120
                          cl::init(true));
121
122
namespace {
123
124
class AArch64SpeculationHardening : public MachineFunctionPass {
125
public:
126
  const TargetInstrInfo *TII;
127
  const TargetRegisterInfo *TRI;
128
129
  static char ID;
130
131
8.78k
  AArch64SpeculationHardening() : MachineFunctionPass(ID) {
132
8.78k
    initializeAArch64SpeculationHardeningPass(*PassRegistry::getPassRegistry());
133
8.78k
  }
134
135
  bool runOnMachineFunction(MachineFunction &Fn) override;
136
137
266k
  StringRef getPassName() const override {
138
266k
    return AARCH64_SPECULATION_HARDENING_NAME;
139
266k
  }
140
141
private:
142
  unsigned MisspeculatingTaintReg;
143
  unsigned MisspeculatingTaintReg32Bit;
144
  bool UseControlFlowSpeculationBarrier;
145
  BitVector RegsNeedingCSDBBeforeUse;
146
  BitVector RegsAlreadyMasked;
147
148
  bool functionUsesHardeningRegister(MachineFunction &MF) const;
149
  bool instrumentControlFlow(MachineBasicBlock &MBB,
150
                             bool &UsesFullSpeculationBarrier);
151
  bool endsWithCondControlFlow(MachineBasicBlock &MBB, MachineBasicBlock *&TBB,
152
                               MachineBasicBlock *&FBB,
153
                               AArch64CC::CondCode &CondCode) const;
154
  void insertTrackingCode(MachineBasicBlock &SplitEdgeBB,
155
                          AArch64CC::CondCode &CondCode, DebugLoc DL) const;
156
  void insertSPToRegTaintPropagation(MachineBasicBlock &MBB,
157
                                     MachineBasicBlock::iterator MBBI) const;
158
  void insertRegToSPTaintPropagation(MachineBasicBlock &MBB,
159
                                     MachineBasicBlock::iterator MBBI,
160
                                     unsigned TmpReg) const;
161
  void insertFullSpeculationBarrier(MachineBasicBlock &MBB,
162
                                    MachineBasicBlock::iterator MBBI,
163
                                    DebugLoc DL) const;
164
165
  bool slhLoads(MachineBasicBlock &MBB);
166
  bool makeGPRSpeculationSafe(MachineBasicBlock &MBB,
167
                              MachineBasicBlock::iterator MBBI,
168
                              MachineInstr &MI, unsigned Reg);
169
  bool lowerSpeculationSafeValuePseudos(MachineBasicBlock &MBB,
170
                                        bool UsesFullSpeculationBarrier);
171
  bool expandSpeculationSafeValue(MachineBasicBlock &MBB,
172
                                  MachineBasicBlock::iterator MBBI,
173
                                  bool UsesFullSpeculationBarrier);
174
  bool insertCSDB(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
175
                  DebugLoc DL);
176
};
177
178
} // end anonymous namespace
179
180
char AArch64SpeculationHardening::ID = 0;
181
182
INITIALIZE_PASS(AArch64SpeculationHardening, "aarch64-speculation-hardening",
183
                AARCH64_SPECULATION_HARDENING_NAME, false, false)
184
185
bool AArch64SpeculationHardening::endsWithCondControlFlow(
186
    MachineBasicBlock &MBB, MachineBasicBlock *&TBB, MachineBasicBlock *&FBB,
187
114
    AArch64CC::CondCode &CondCode) const {
188
114
  SmallVector<MachineOperand, 1> analyzeBranchCondCode;
189
114
  if (TII->analyzeBranch(MBB, TBB, FBB, analyzeBranchCondCode, false))
190
44
    return false;
191
70
192
70
  // Ignore if the BB ends in an unconditional branch/fall-through.
193
70
  if (analyzeBranchCondCode.empty())
194
54
    return false;
195
16
196
16
  // If the BB ends with a single conditional branch, FBB will be set to
197
16
  // nullptr (see API docs for TII->analyzeBranch). For the rest of the
198
16
  // analysis we want the FBB block to be set always.
199
16
  assert(TBB != nullptr);
200
16
  if (FBB == nullptr)
201
15
    FBB = MBB.getFallThrough();
202
16
203
16
  // If both the true and the false condition jump to the same basic block,
204
16
  // there isn't need for any protection - whether the branch is speculated
205
16
  // correctly or not, we end up executing the architecturally correct code.
206
16
  if (TBB == FBB)
207
0
    return false;
208
16
209
16
  assert(MBB.succ_size() == 2);
210
16
  // translate analyzeBranchCondCode to CondCode.
211
16
  assert(analyzeBranchCondCode.size() == 1 && "unknown Cond array format");
212
16
  CondCode = AArch64CC::CondCode(analyzeBranchCondCode[0].getImm());
213
16
  return true;
214
16
}
215
216
void AArch64SpeculationHardening::insertFullSpeculationBarrier(
217
    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
218
9
    DebugLoc DL) const {
219
9
  // A full control flow speculation barrier consists of (DSB SYS + ISB)
220
9
  BuildMI(MBB, MBBI, DL, TII->get(AArch64::DSB)).addImm(0xf);
221
9
  BuildMI(MBB, MBBI, DL, TII->get(AArch64::ISB)).addImm(0xf);
222
9
}
223
224
void AArch64SpeculationHardening::insertTrackingCode(
225
    MachineBasicBlock &SplitEdgeBB, AArch64CC::CondCode &CondCode,
226
32
    DebugLoc DL) const {
227
32
  if (UseControlFlowSpeculationBarrier) {
228
4
    insertFullSpeculationBarrier(SplitEdgeBB, SplitEdgeBB.begin(), DL);
229
28
  } else {
230
28
    BuildMI(SplitEdgeBB, SplitEdgeBB.begin(), DL, TII->get(AArch64::CSELXr))
231
28
        .addDef(MisspeculatingTaintReg)
232
28
        .addUse(MisspeculatingTaintReg)
233
28
        .addUse(AArch64::XZR)
234
28
        .addImm(CondCode);
235
28
    SplitEdgeBB.addLiveIn(AArch64::NZCV);
236
28
  }
237
32
}
238
239
bool AArch64SpeculationHardening::instrumentControlFlow(
240
114
    MachineBasicBlock &MBB, bool &UsesFullSpeculationBarrier) {
241
114
  LLVM_DEBUG(dbgs() << "Instrument control flow tracking on MBB: " << MBB);
242
114
243
114
  bool Modified = false;
244
114
  MachineBasicBlock *TBB = nullptr;
245
114
  MachineBasicBlock *FBB = nullptr;
246
114
  AArch64CC::CondCode CondCode;
247
114
248
114
  if (!endsWithCondControlFlow(MBB, TBB, FBB, CondCode)) {
249
98
    LLVM_DEBUG(dbgs() << "... doesn't end with CondControlFlow\n");
250
98
  } else {
251
16
    // Now insert:
252
16
    // "CSEL MisSpeculatingR, MisSpeculatingR, XZR, cond" on the True edge and
253
16
    // "CSEL MisSpeculatingR, MisSpeculatingR, XZR, Invertcond" on the False
254
16
    // edge.
255
16
    AArch64CC::CondCode InvCondCode = AArch64CC::getInvertedCondCode(CondCode);
256
16
257
16
    MachineBasicBlock *SplitEdgeTBB = MBB.SplitCriticalEdge(TBB, *this);
258
16
    MachineBasicBlock *SplitEdgeFBB = MBB.SplitCriticalEdge(FBB, *this);
259
16
260
16
    assert(SplitEdgeTBB != nullptr);
261
16
    assert(SplitEdgeFBB != nullptr);
262
16
263
16
    DebugLoc DL;
264
16
    if (MBB.instr_end() != MBB.instr_begin())
265
16
      DL = (--MBB.instr_end())->getDebugLoc();
266
16
267
16
    insertTrackingCode(*SplitEdgeTBB, CondCode, DL);
268
16
    insertTrackingCode(*SplitEdgeFBB, InvCondCode, DL);
269
16
270
16
    LLVM_DEBUG(dbgs() << "SplitEdgeTBB: " << *SplitEdgeTBB << "\n");
271
16
    LLVM_DEBUG(dbgs() << "SplitEdgeFBB: " << *SplitEdgeFBB << "\n");
272
16
    Modified = true;
273
16
  }
274
114
275
114
  // Perform correct code generation around function calls and before returns.
276
114
  // The below variables record the return/terminator instructions and the call
277
114
  // instructions respectively; including which register is available as a
278
114
  // temporary register just before the recorded instructions.
279
114
  SmallVector<std::pair<MachineInstr *, unsigned>, 4> ReturnInstructions;
280
114
  SmallVector<std::pair<MachineInstr *, unsigned>, 4> CallInstructions;
281
114
  // if a temporary register is not available for at least one of the
282
114
  // instructions for which we need to transfer taint to the stack pointer, we
283
114
  // need to insert a full speculation barrier.
284
114
  // TmpRegisterNotAvailableEverywhere tracks that condition.
285
114
  bool TmpRegisterNotAvailableEverywhere = false;
286
114
287
114
  RegScavenger RS;
288
114
  RS.enterBasicBlock(MBB);
289
114
290
572
  for (MachineBasicBlock::iterator I = MBB.begin(); I != MBB.end(); 
I++458
) {
291
458
    MachineInstr &MI = *I;
292
458
    if (!MI.isReturn() && 
!MI.isCall()415
)
293
399
      continue;
294
59
295
59
    // The RegScavenger represents registers available *after* the MI
296
59
    // instruction pointed to by RS.getCurrentPosition().
297
59
    // We need to have a register that is available *before* the MI is executed.
298
59
    if (I != MBB.begin())
299
48
      RS.forward(std::prev(I));
300
59
    // FIXME: The below just finds *a* unused register. Maybe code could be
301
59
    // optimized more if this looks for the register that isn't used for the
302
59
    // longest time around this place, to enable more scheduling freedom. Not
303
59
    // sure if that would actually result in a big performance difference
304
59
    // though. Maybe RegisterScavenger::findSurvivorBackwards has some logic
305
59
    // already to do this - but it's unclear if that could easily be used here.
306
59
    unsigned TmpReg = RS.FindUnusedReg(&AArch64::GPR64commonRegClass);
307
59
    LLVM_DEBUG(dbgs() << "RS finds "
308
59
                      << ((TmpReg == 0) ? "no register " : "register ");
309
59
               if (TmpReg != 0) dbgs() << printReg(TmpReg, TRI) << " ";
310
59
               dbgs() << "to be available at MI " << MI);
311
59
    if (TmpReg == 0)
312
1
      TmpRegisterNotAvailableEverywhere = true;
313
59
    if (MI.isReturn())
314
43
      ReturnInstructions.push_back({&MI, TmpReg});
315
16
    else if (MI.isCall())
316
16
      CallInstructions.push_back({&MI, TmpReg});
317
59
  }
318
114
319
114
  if (TmpRegisterNotAvailableEverywhere) {
320
1
    // When a temporary register is not available everywhere in this basic
321
1
    // basic block where a propagate-taint-to-sp operation is needed, just
322
1
    // emit a full speculation barrier at the start of this basic block, which
323
1
    // renders the taint/speculation tracking in this basic block unnecessary.
324
1
    insertFullSpeculationBarrier(MBB, MBB.begin(),
325
1
                                 (MBB.begin())->getDebugLoc());
326
1
    UsesFullSpeculationBarrier = true;
327
1
    Modified = true;
328
113
  } else {
329
113
    for (auto MI_Reg : ReturnInstructions) {
330
42
      assert(MI_Reg.second != 0);
331
42
      LLVM_DEBUG(
332
42
          dbgs()
333
42
          << " About to insert Reg to SP taint propagation with temp register "
334
42
          << printReg(MI_Reg.second, TRI)
335
42
          << " on instruction: " << *MI_Reg.first);
336
42
      insertRegToSPTaintPropagation(MBB, MI_Reg.first, MI_Reg.second);
337
42
      Modified = true;
338
42
    }
339
113
340
113
    for (auto MI_Reg : CallInstructions) {
341
16
      assert(MI_Reg.second != 0);
342
16
      LLVM_DEBUG(dbgs() << " About to insert Reg to SP and back taint "
343
16
                           "propagation with temp register "
344
16
                        << printReg(MI_Reg.second, TRI)
345
16
                        << " around instruction: " << *MI_Reg.first);
346
16
      // Just after the call:
347
16
      insertSPToRegTaintPropagation(
348
16
          MBB, std::next((MachineBasicBlock::iterator)MI_Reg.first));
349
16
      // Just before the call:
350
16
      insertRegToSPTaintPropagation(MBB, MI_Reg.first, MI_Reg.second);
351
16
      Modified = true;
352
16
    }
353
113
  }
354
114
  return Modified;
355
114
}
356
357
void AArch64SpeculationHardening::insertSPToRegTaintPropagation(
358
53
    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI) const {
359
53
  // If full control flow speculation barriers are used, emit a control flow
360
53
  // barrier to block potential miss-speculation in flight coming in to this
361
53
  // function.
362
53
  if (UseControlFlowSpeculationBarrier) {
363
4
    insertFullSpeculationBarrier(MBB, MBBI, DebugLoc());
364
4
    return;
365
4
  }
366
49
367
49
  // CMP   SP, #0   === SUBS   xzr, SP, #0
368
49
  BuildMI(MBB, MBBI, DebugLoc(), TII->get(AArch64::SUBSXri))
369
49
      .addDef(AArch64::XZR)
370
49
      .addUse(AArch64::SP)
371
49
      .addImm(0)
372
49
      .addImm(0); // no shift
373
49
  // CSETM x16, NE  === CSINV  x16, xzr, xzr, EQ
374
49
  BuildMI(MBB, MBBI, DebugLoc(), TII->get(AArch64::CSINVXr))
375
49
      .addDef(MisspeculatingTaintReg)
376
49
      .addUse(AArch64::XZR)
377
49
      .addUse(AArch64::XZR)
378
49
      .addImm(AArch64CC::EQ);
379
49
}
380
381
void AArch64SpeculationHardening::insertRegToSPTaintPropagation(
382
    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
383
58
    unsigned TmpReg) const {
384
58
  // If full control flow speculation barriers are used, there will not be
385
58
  // miss-speculation when returning from this function, and therefore, also
386
58
  // no need to encode potential miss-speculation into the stack pointer.
387
58
  if (UseControlFlowSpeculationBarrier)
388
4
    return;
389
54
390
54
  // mov   Xtmp, SP  === ADD  Xtmp, SP, #0
391
54
  BuildMI(MBB, MBBI, DebugLoc(), TII->get(AArch64::ADDXri))
392
54
      .addDef(TmpReg)
393
54
      .addUse(AArch64::SP)
394
54
      .addImm(0)
395
54
      .addImm(0); // no shift
396
54
  // and   Xtmp, Xtmp, TaintReg === AND Xtmp, Xtmp, TaintReg, #0
397
54
  BuildMI(MBB, MBBI, DebugLoc(), TII->get(AArch64::ANDXrs))
398
54
      .addDef(TmpReg, RegState::Renamable)
399
54
      .addUse(TmpReg, RegState::Kill | RegState::Renamable)
400
54
      .addUse(MisspeculatingTaintReg, RegState::Kill)
401
54
      .addImm(0);
402
54
  // mov   SP, Xtmp === ADD SP, Xtmp, #0
403
54
  BuildMI(MBB, MBBI, DebugLoc(), TII->get(AArch64::ADDXri))
404
54
      .addDef(AArch64::SP)
405
54
      .addUse(TmpReg, RegState::Kill)
406
54
      .addImm(0)
407
54
      .addImm(0); // no shift
408
54
}
409
410
bool AArch64SpeculationHardening::functionUsesHardeningRegister(
411
34
    MachineFunction &MF) const {
412
80
  for (MachineBasicBlock &MBB : MF) {
413
264
    for (MachineInstr &MI : MBB) {
414
264
      // treat function calls specially, as the hardening register does not
415
264
      // need to remain live across function calls.
416
264
      if (MI.isCall())
417
18
        continue;
418
246
      if (MI.readsRegister(MisspeculatingTaintReg, TRI) ||
419
246
          MI.modifiesRegister(MisspeculatingTaintReg, TRI))
420
3
        return true;
421
246
    }
422
80
  }
423
34
  
return false31
;
424
34
}
425
426
// Make GPR register Reg speculation-safe by putting it through the
427
// SpeculationSafeValue pseudo instruction, if we can't prove that
428
// the value in the register has already been hardened.
429
bool AArch64SpeculationHardening::makeGPRSpeculationSafe(
430
    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, MachineInstr &MI,
431
47
    unsigned Reg) {
432
47
  assert(AArch64::GPR32allRegClass.contains(Reg) ||
433
47
         AArch64::GPR64allRegClass.contains(Reg));
434
47
435
47
  // Loads cannot directly load a value into the SP (nor WSP).
436
47
  // Therefore, if Reg is SP or WSP, it is because the instruction loads from
437
47
  // the stack through the stack pointer.
438
47
  //
439
47
  // Since the stack pointer is never dynamically controllable, don't harden it.
440
47
  if (Reg == AArch64::SP || 
Reg == AArch64::WSP39
)
441
8
    return false;
442
39
443
39
  // Do not harden the register again if already hardened before.
444
39
  if (RegsAlreadyMasked[Reg])
445
0
    return false;
446
39
447
39
  const bool Is64Bit = AArch64::GPR64allRegClass.contains(Reg);
448
39
  LLVM_DEBUG(dbgs() << "About to harden register : " << Reg << "\n");
449
39
  BuildMI(MBB, MBBI, MI.getDebugLoc(),
450
39
          TII->get(Is64Bit ? 
AArch64::SpeculationSafeValueX35
451
39
                           : 
AArch64::SpeculationSafeValueW4
))
452
39
      .addDef(Reg)
453
39
      .addUse(Reg);
454
39
  RegsAlreadyMasked.set(Reg);
455
39
  return true;
456
39
}
457
458
82
bool AArch64SpeculationHardening::slhLoads(MachineBasicBlock &MBB) {
459
82
  bool Modified = false;
460
82
461
82
  LLVM_DEBUG(dbgs() << "slhLoads running on MBB: " << MBB);
462
82
463
82
  RegsAlreadyMasked.reset();
464
82
465
82
  MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
466
82
  MachineBasicBlock::iterator NextMBBI;
467
360
  for (; MBBI != E; 
MBBI = NextMBBI278
) {
468
278
    MachineInstr &MI = *MBBI;
469
278
    NextMBBI = std::next(MBBI);
470
278
    // Only harden loaded values or addresses used in loads.
471
278
    if (!MI.mayLoad())
472
250
      continue;
473
28
474
28
    LLVM_DEBUG(dbgs() << "About to harden: " << MI);
475
28
476
28
    // For general purpose register loads, harden the registers loaded into.
477
28
    // For other loads, harden the address loaded from.
478
28
    // Masking the loaded value is expected to result in less performance
479
28
    // overhead, as the load can still execute speculatively in comparison to
480
28
    // when the address loaded from gets masked. However, masking is only
481
28
    // easy to do efficiently on GPR registers, so for loads into non-GPR
482
28
    // registers (e.g. floating point loads), mask the address loaded from.
483
48
    bool AllDefsAreGPR = llvm::all_of(MI.defs(), [&](MachineOperand &Op) {
484
48
      return Op.isReg() && (AArch64::GPR32allRegClass.contains(Op.getReg()) ||
485
48
                            
AArch64::GPR64allRegClass.contains(Op.getReg())43
);
486
48
    });
487
28
    // FIXME: it might be a worthwhile optimization to not mask loaded
488
28
    // values if all the registers involved in address calculation are already
489
28
    // hardened, leading to this load not able to execute on a miss-speculated
490
28
    // path.
491
28
    bool HardenLoadedData = AllDefsAreGPR;
492
28
    bool HardenAddressLoadedFrom = !HardenLoadedData;
493
28
494
28
    // First remove registers from AlreadyMaskedRegisters if their value is
495
28
    // updated by this instruction - it makes them contain a new value that is
496
28
    // not guaranteed to already have been masked.
497
28
    for (MachineOperand Op : MI.defs())
498
311
      
for (MCRegAliasIterator AI(Op.getReg(), TRI, true); 48
AI.isValid();
++AI263
)
499
263
        RegsAlreadyMasked.reset(*AI);
500
28
501
28
    // FIXME: loads from the stack with an immediate offset from the stack
502
28
    // pointer probably shouldn't be hardened, which could result in a
503
28
    // significant optimization. See section "Don’t check loads from
504
28
    // compile-time constant stack offsets", in
505
28
    // https://llvm.org/docs/SpeculativeLoadHardening.html
506
28
507
28
    if (HardenLoadedData)
508
45
      
for (auto Def : MI.defs())25
{
509
45
        if (Def.isDead())
510
1
          // Do not mask a register that is not used further.
511
1
          continue;
512
44
        // FIXME: For pre/post-increment addressing modes, the base register
513
44
        // used in address calculation is also defined by this instruction.
514
44
        // It might be a worthwhile optimization to not harden that
515
44
        // base register increment/decrement when the increment/decrement is
516
44
        // an immediate.
517
44
        Modified |= makeGPRSpeculationSafe(MBB, NextMBBI, MI, Def.getReg());
518
44
      }
519
28
520
28
    if (HardenAddressLoadedFrom)
521
8
      
for (auto Use : MI.uses())3
{
522
8
        if (!Use.isReg())
523
3
          continue;
524
5
        unsigned Reg = Use.getReg();
525
5
        // Some loads of floating point data have implicit defs/uses on a
526
5
        // super register of that floating point data. Some examples:
527
5
        // $s0 = LDRSui $sp, 22, implicit-def $q0
528
5
        // $q0 = LD1i64 $q0, 1, renamable $x0
529
5
        // We need to filter out these uses for non-GPR register which occur
530
5
        // because the load partially fills a non-GPR register with the loaded
531
5
        // data. Just skipping all non-GPR registers is safe (for now) as all
532
5
        // AArch64 load instructions only use GPR registers to perform the
533
5
        // address calculation. FIXME: However that might change once we can
534
5
        // produce SVE gather instructions.
535
5
        if (!(AArch64::GPR32allRegClass.contains(Reg) ||
536
5
              AArch64::GPR64allRegClass.contains(Reg)))
537
2
          continue;
538
3
        Modified |= makeGPRSpeculationSafe(MBB, MBBI, MI, Reg);
539
3
      }
540
28
  }
541
82
  return Modified;
542
82
}
543
544
/// \brief If MBBI references a pseudo instruction that should be expanded
545
/// here, do the expansion and return true. Otherwise return false.
546
bool AArch64SpeculationHardening::expandSpeculationSafeValue(
547
    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
548
654
    bool UsesFullSpeculationBarrier) {
549
654
  MachineInstr &MI = *MBBI;
550
654
  unsigned Opcode = MI.getOpcode();
551
654
  bool Is64Bit = true;
552
654
553
654
  switch (Opcode) {
554
654
  default:
555
615
    break;
556
654
  case AArch64::SpeculationSafeValueW:
557
4
    Is64Bit = false;
558
4
    LLVM_FALLTHROUGH;
559
39
  case AArch64::SpeculationSafeValueX:
560
39
    // Just remove the SpeculationSafe pseudo's if control flow
561
39
    // miss-speculation isn't happening because we're already inserting barriers
562
39
    // to guarantee that.
563
39
    if (!UseControlFlowSpeculationBarrier && 
!UsesFullSpeculationBarrier36
) {
564
35
      unsigned DstReg = MI.getOperand(0).getReg();
565
35
      unsigned SrcReg = MI.getOperand(1).getReg();
566
35
      // Mark this register and all its aliasing registers as needing to be
567
35
      // value speculation hardened before its next use, by using a CSDB
568
35
      // barrier instruction.
569
35
      for (MachineOperand Op : MI.defs())
570
175
        
for (MCRegAliasIterator AI(Op.getReg(), TRI, true); 35
AI.isValid();
++AI140
)
571
140
          RegsNeedingCSDBBeforeUse.set(*AI);
572
35
573
35
      // Mask off with taint state.
574
35
      BuildMI(MBB, MBBI, MI.getDebugLoc(),
575
35
              Is64Bit ? 
TII->get(AArch64::ANDXrs)31
:
TII->get(AArch64::ANDWrs)4
)
576
35
          .addDef(DstReg)
577
35
          .addUse(SrcReg, RegState::Kill)
578
35
          .addUse(Is64Bit ? 
MisspeculatingTaintReg31
579
35
                          : 
MisspeculatingTaintReg32Bit4
)
580
35
          .addImm(0);
581
35
    }
582
39
    MI.eraseFromParent();
583
39
    return true;
584
615
  }
585
615
  return false;
586
615
}
587
588
bool AArch64SpeculationHardening::insertCSDB(MachineBasicBlock &MBB,
589
                                             MachineBasicBlock::iterator MBBI,
590
17
                                             DebugLoc DL) {
591
17
  assert(!UseControlFlowSpeculationBarrier && "No need to insert CSDBs when "
592
17
                                              "control flow miss-speculation "
593
17
                                              "is already blocked");
594
17
  // insert data value speculation barrier (CSDB)
595
17
  BuildMI(MBB, MBBI, DL, TII->get(AArch64::HINT)).addImm(0x14);
596
17
  RegsNeedingCSDBBeforeUse.reset();
597
17
  return true;
598
17
}
599
600
bool AArch64SpeculationHardening::lowerSpeculationSafeValuePseudos(
601
114
    MachineBasicBlock &MBB, bool UsesFullSpeculationBarrier) {
602
114
  bool Modified = false;
603
114
604
114
  RegsNeedingCSDBBeforeUse.reset();
605
114
606
114
  // The following loop iterates over all instructions in the basic block,
607
114
  // and performs 2 operations:
608
114
  // 1. Insert a CSDB at this location if needed.
609
114
  // 2. Expand the SpeculationSafeValuePseudo if the current instruction is
610
114
  // one.
611
114
  //
612
114
  // The insertion of the CSDB is done as late as possible (i.e. just before
613
114
  // the use of a masked register), in the hope that that will reduce the
614
114
  // total number of CSDBs in a block when there are multiple masked registers
615
114
  // in the block.
616
114
  MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
617
114
  DebugLoc DL;
618
768
  while (MBBI != E) {
619
654
    MachineInstr &MI = *MBBI;
620
654
    DL = MI.getDebugLoc();
621
654
    MachineBasicBlock::iterator NMBBI = std::next(MBBI);
622
654
623
654
    // First check if a CSDB needs to be inserted due to earlier registers
624
654
    // that were masked and that are used by the next instruction.
625
654
    // Also emit the barrier on any potential control flow changes.
626
654
    bool NeedToEmitBarrier = false;
627
654
    if (RegsNeedingCSDBBeforeUse.any() && 
(63
MI.isCall()63
||
MI.isTerminator()62
))
628
8
      NeedToEmitBarrier = true;
629
654
    if (!NeedToEmitBarrier)
630
646
      for (MachineOperand Op : MI.uses())
631
1.79k
        if (Op.isReg() && 
RegsNeedingCSDBBeforeUse[Op.getReg()]1.02k
) {
632
6
          NeedToEmitBarrier = true;
633
6
          break;
634
6
        }
635
654
636
654
    if (NeedToEmitBarrier && 
!UsesFullSpeculationBarrier14
)
637
14
      Modified |= insertCSDB(MBB, MBBI, DL);
638
654
639
654
    Modified |=
640
654
        expandSpeculationSafeValue(MBB, MBBI, UsesFullSpeculationBarrier);
641
654
642
654
    MBBI = NMBBI;
643
654
  }
644
114
645
114
  if (RegsNeedingCSDBBeforeUse.any() && 
!UsesFullSpeculationBarrier3
)
646
3
    Modified |= insertCSDB(MBB, MBBI, DL);
647
114
648
114
  return Modified;
649
114
}
650
651
257k
bool AArch64SpeculationHardening::runOnMachineFunction(MachineFunction &MF) {
652
257k
  if (!MF.getFunction().hasFnAttribute(Attribute::SpeculativeLoadHardening))
653
257k
    return false;
654
34
655
34
  MisspeculatingTaintReg = AArch64::X16;
656
34
  MisspeculatingTaintReg32Bit = AArch64::W16;
657
34
  TII = MF.getSubtarget().getInstrInfo();
658
34
  TRI = MF.getSubtarget().getRegisterInfo();
659
34
  RegsNeedingCSDBBeforeUse.resize(TRI->getNumRegs());
660
34
  RegsAlreadyMasked.resize(TRI->getNumRegs());
661
34
  UseControlFlowSpeculationBarrier = functionUsesHardeningRegister(MF);
662
34
663
34
  bool Modified = false;
664
34
665
34
  // Step 1: Enable automatic insertion of SpeculationSafeValue.
666
34
  if (HardenLoads) {
667
34
    LLVM_DEBUG(
668
34
        dbgs() << "***** AArch64SpeculationHardening - automatic insertion of "
669
34
                  "SpeculationSafeValue intrinsics *****\n");
670
34
    for (auto &MBB : MF)
671
82
      Modified |= slhLoads(MBB);
672
34
  }
673
34
674
34
  // 2. Add instrumentation code to function entry and exits.
675
34
  LLVM_DEBUG(
676
34
      dbgs()
677
34
      << "***** AArch64SpeculationHardening - track control flow *****\n");
678
34
679
34
  SmallVector<MachineBasicBlock *, 2> EntryBlocks;
680
34
  EntryBlocks.push_back(&MF.front());
681
34
  for (const LandingPadInfo &LPI : MF.getLandingPads())
682
3
    EntryBlocks.push_back(LPI.LandingPadBlock);
683
34
  for (auto Entry : EntryBlocks)
684
37
    insertSPToRegTaintPropagation(
685
37
        *Entry, Entry->SkipPHIsLabelsAndDebug(Entry->begin()));
686
34
687
34
  // 3. Add instrumentation code to every basic block.
688
114
  for (auto &MBB : MF) {
689
114
    bool UsesFullSpeculationBarrier = false;
690
114
    Modified |= instrumentControlFlow(MBB, UsesFullSpeculationBarrier);
691
114
    Modified |=
692
114
        lowerSpeculationSafeValuePseudos(MBB, UsesFullSpeculationBarrier);
693
114
  }
694
34
695
34
  return Modified;
696
34
}
697
698
/// \brief Returns an instance of the pseudo instruction expansion pass.
699
8.78k
FunctionPass *llvm::createAArch64SpeculationHardeningPass() {
700
8.78k
  return new AArch64SpeculationHardening();
701
8.78k
}