Coverage Report

Created: 2019-07-24 05:18

/Users/buildslave/jenkins/workspace/clang-stage2-coverage-R/llvm/lib/Transforms/Instrumentation/MemorySanitizer.cpp
Line
Count
Source (jump to first uncovered line)
1
//===- MemorySanitizer.cpp - detector of uninitialized reads --------------===//
2
//
3
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
4
// See https://llvm.org/LICENSE.txt for license information.
5
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
6
//
7
//===----------------------------------------------------------------------===//
8
//
9
/// \file
10
/// This file is a part of MemorySanitizer, a detector of uninitialized
11
/// reads.
12
///
13
/// The algorithm of the tool is similar to Memcheck
14
/// (http://goo.gl/QKbem). We associate a few shadow bits with every
15
/// byte of the application memory, poison the shadow of the malloc-ed
16
/// or alloca-ed memory, load the shadow bits on every memory read,
17
/// propagate the shadow bits through some of the arithmetic
18
/// instruction (including MOV), store the shadow bits on every memory
19
/// write, report a bug on some other instructions (e.g. JMP) if the
20
/// associated shadow is poisoned.
21
///
22
/// But there are differences too. The first and the major one:
23
/// compiler instrumentation instead of binary instrumentation. This
24
/// gives us much better register allocation, possible compiler
25
/// optimizations and a fast start-up. But this brings the major issue
26
/// as well: msan needs to see all program events, including system
27
/// calls and reads/writes in system libraries, so we either need to
28
/// compile *everything* with msan or use a binary translation
29
/// component (e.g. DynamoRIO) to instrument pre-built libraries.
30
/// Another difference from Memcheck is that we use 8 shadow bits per
31
/// byte of application memory and use a direct shadow mapping. This
32
/// greatly simplifies the instrumentation code and avoids races on
33
/// shadow updates (Memcheck is single-threaded so races are not a
34
/// concern there. Memcheck uses 2 shadow bits per byte with a slow
35
/// path storage that uses 8 bits per byte).
36
///
37
/// The default value of shadow is 0, which means "clean" (not poisoned).
38
///
39
/// Every module initializer should call __msan_init to ensure that the
40
/// shadow memory is ready. On error, __msan_warning is called. Since
41
/// parameters and return values may be passed via registers, we have a
42
/// specialized thread-local shadow for return values
43
/// (__msan_retval_tls) and parameters (__msan_param_tls).
44
///
45
///                           Origin tracking.
46
///
47
/// MemorySanitizer can track origins (allocation points) of all uninitialized
48
/// values. This behavior is controlled with a flag (msan-track-origins) and is
49
/// disabled by default.
50
///
51
/// Origins are 4-byte values created and interpreted by the runtime library.
52
/// They are stored in a second shadow mapping, one 4-byte value for 4 bytes
53
/// of application memory. Propagation of origins is basically a bunch of
54
/// "select" instructions that pick the origin of a dirty argument, if an
55
/// instruction has one.
56
///
57
/// Every 4 aligned, consecutive bytes of application memory have one origin
58
/// value associated with them. If these bytes contain uninitialized data
59
/// coming from 2 different allocations, the last store wins. Because of this,
60
/// MemorySanitizer reports can show unrelated origins, but this is unlikely in
61
/// practice.
62
///
63
/// Origins are meaningless for fully initialized values, so MemorySanitizer
64
/// avoids storing origin to memory when a fully initialized value is stored.
65
/// This way it avoids needless overwritting origin of the 4-byte region on
66
/// a short (i.e. 1 byte) clean store, and it is also good for performance.
67
///
68
///                            Atomic handling.
69
///
70
/// Ideally, every atomic store of application value should update the
71
/// corresponding shadow location in an atomic way. Unfortunately, atomic store
72
/// of two disjoint locations can not be done without severe slowdown.
73
///
74
/// Therefore, we implement an approximation that may err on the safe side.
75
/// In this implementation, every atomically accessed location in the program
76
/// may only change from (partially) uninitialized to fully initialized, but
77
/// not the other way around. We load the shadow _after_ the application load,
78
/// and we store the shadow _before_ the app store. Also, we always store clean
79
/// shadow (if the application store is atomic). This way, if the store-load
80
/// pair constitutes a happens-before arc, shadow store and load are correctly
81
/// ordered such that the load will get either the value that was stored, or
82
/// some later value (which is always clean).
83
///
84
/// This does not work very well with Compare-And-Swap (CAS) and
85
/// Read-Modify-Write (RMW) operations. To follow the above logic, CAS and RMW
86
/// must store the new shadow before the app operation, and load the shadow
87
/// after the app operation. Computers don't work this way. Current
88
/// implementation ignores the load aspect of CAS/RMW, always returning a clean
89
/// value. It implements the store part as a simple atomic store by storing a
90
/// clean shadow.
91
///
92
///                      Instrumenting inline assembly.
93
///
94
/// For inline assembly code LLVM has little idea about which memory locations
95
/// become initialized depending on the arguments. It can be possible to figure
96
/// out which arguments are meant to point to inputs and outputs, but the
97
/// actual semantics can be only visible at runtime. In the Linux kernel it's
98
/// also possible that the arguments only indicate the offset for a base taken
99
/// from a segment register, so it's dangerous to treat any asm() arguments as
100
/// pointers. We take a conservative approach generating calls to
101
///   __msan_instrument_asm_store(ptr, size)
102
/// , which defer the memory unpoisoning to the runtime library.
103
/// The latter can perform more complex address checks to figure out whether
104
/// it's safe to touch the shadow memory.
105
/// Like with atomic operations, we call __msan_instrument_asm_store() before
106
/// the assembly call, so that changes to the shadow memory will be seen by
107
/// other threads together with main memory initialization.
108
///
109
///                  KernelMemorySanitizer (KMSAN) implementation.
110
///
111
/// The major differences between KMSAN and MSan instrumentation are:
112
///  - KMSAN always tracks the origins and implies msan-keep-going=true;
113
///  - KMSAN allocates shadow and origin memory for each page separately, so
114
///    there are no explicit accesses to shadow and origin in the
115
///    instrumentation.
116
///    Shadow and origin values for a particular X-byte memory location
117
///    (X=1,2,4,8) are accessed through pointers obtained via the
118
///      __msan_metadata_ptr_for_load_X(ptr)
119
///      __msan_metadata_ptr_for_store_X(ptr)
120
///    functions. The corresponding functions check that the X-byte accesses
121
///    are possible and returns the pointers to shadow and origin memory.
122
///    Arbitrary sized accesses are handled with:
123
///      __msan_metadata_ptr_for_load_n(ptr, size)
124
///      __msan_metadata_ptr_for_store_n(ptr, size);
125
///  - TLS variables are stored in a single per-task struct. A call to a
126
///    function __msan_get_context_state() returning a pointer to that struct
127
///    is inserted into every instrumented function before the entry block;
128
///  - __msan_warning() takes a 32-bit origin parameter;
129
///  - local variables are poisoned with __msan_poison_alloca() upon function
130
///    entry and unpoisoned with __msan_unpoison_alloca() before leaving the
131
///    function;
132
///  - the pass doesn't declare any global variables or add global constructors
133
///    to the translation unit.
134
///
135
/// Also, KMSAN currently ignores uninitialized memory passed into inline asm
136
/// calls, making sure we're on the safe side wrt. possible false positives.
137
///
138
///  KernelMemorySanitizer only supports X86_64 at the moment.
139
///
140
//===----------------------------------------------------------------------===//
141
142
#include "llvm/Transforms/Instrumentation/MemorySanitizer.h"
143
#include "llvm/ADT/APInt.h"
144
#include "llvm/ADT/ArrayRef.h"
145
#include "llvm/ADT/DepthFirstIterator.h"
146
#include "llvm/ADT/SmallSet.h"
147
#include "llvm/ADT/SmallString.h"
148
#include "llvm/ADT/SmallVector.h"
149
#include "llvm/ADT/StringExtras.h"
150
#include "llvm/ADT/StringRef.h"
151
#include "llvm/ADT/Triple.h"
152
#include "llvm/Analysis/TargetLibraryInfo.h"
153
#include "llvm/IR/Argument.h"
154
#include "llvm/IR/Attributes.h"
155
#include "llvm/IR/BasicBlock.h"
156
#include "llvm/IR/CallSite.h"
157
#include "llvm/IR/CallingConv.h"
158
#include "llvm/IR/Constant.h"
159
#include "llvm/IR/Constants.h"
160
#include "llvm/IR/DataLayout.h"
161
#include "llvm/IR/DerivedTypes.h"
162
#include "llvm/IR/Function.h"
163
#include "llvm/IR/GlobalValue.h"
164
#include "llvm/IR/GlobalVariable.h"
165
#include "llvm/IR/IRBuilder.h"
166
#include "llvm/IR/InlineAsm.h"
167
#include "llvm/IR/InstVisitor.h"
168
#include "llvm/IR/InstrTypes.h"
169
#include "llvm/IR/Instruction.h"
170
#include "llvm/IR/Instructions.h"
171
#include "llvm/IR/IntrinsicInst.h"
172
#include "llvm/IR/Intrinsics.h"
173
#include "llvm/IR/LLVMContext.h"
174
#include "llvm/IR/MDBuilder.h"
175
#include "llvm/IR/Module.h"
176
#include "llvm/IR/Type.h"
177
#include "llvm/IR/Value.h"
178
#include "llvm/IR/ValueMap.h"
179
#include "llvm/Pass.h"
180
#include "llvm/Support/AtomicOrdering.h"
181
#include "llvm/Support/Casting.h"
182
#include "llvm/Support/CommandLine.h"
183
#include "llvm/Support/Compiler.h"
184
#include "llvm/Support/Debug.h"
185
#include "llvm/Support/ErrorHandling.h"
186
#include "llvm/Support/MathExtras.h"
187
#include "llvm/Support/raw_ostream.h"
188
#include "llvm/Transforms/Instrumentation.h"
189
#include "llvm/Transforms/Utils/BasicBlockUtils.h"
190
#include "llvm/Transforms/Utils/Local.h"
191
#include "llvm/Transforms/Utils/ModuleUtils.h"
192
#include <algorithm>
193
#include <cassert>
194
#include <cstddef>
195
#include <cstdint>
196
#include <memory>
197
#include <string>
198
#include <tuple>
199
200
using namespace llvm;
201
202
#define DEBUG_TYPE "msan"
203
204
static const unsigned kOriginSize = 4;
205
static const unsigned kMinOriginAlignment = 4;
206
static const unsigned kShadowTLSAlignment = 8;
207
208
// These constants must be kept in sync with the ones in msan.h.
209
static const unsigned kParamTLSSize = 800;
210
static const unsigned kRetvalTLSSize = 800;
211
212
// Accesses sizes are powers of two: 1, 2, 4, 8.
213
static const size_t kNumberOfAccessSizes = 4;
214
215
/// Track origins of uninitialized values.
216
///
217
/// Adds a section to MemorySanitizer report that points to the allocation
218
/// (stack or heap) the uninitialized bits came from originally.
219
static cl::opt<int> ClTrackOrigins("msan-track-origins",
220
       cl::desc("Track origins (allocation sites) of poisoned memory"),
221
       cl::Hidden, cl::init(0));
222
223
static cl::opt<bool> ClKeepGoing("msan-keep-going",
224
       cl::desc("keep going after reporting a UMR"),
225
       cl::Hidden, cl::init(false));
226
227
static cl::opt<bool> ClPoisonStack("msan-poison-stack",
228
       cl::desc("poison uninitialized stack variables"),
229
       cl::Hidden, cl::init(true));
230
231
static cl::opt<bool> ClPoisonStackWithCall("msan-poison-stack-with-call",
232
       cl::desc("poison uninitialized stack variables with a call"),
233
       cl::Hidden, cl::init(false));
234
235
static cl::opt<int> ClPoisonStackPattern("msan-poison-stack-pattern",
236
       cl::desc("poison uninitialized stack variables with the given pattern"),
237
       cl::Hidden, cl::init(0xff));
238
239
static cl::opt<bool> ClPoisonUndef("msan-poison-undef",
240
       cl::desc("poison undef temps"),
241
       cl::Hidden, cl::init(true));
242
243
static cl::opt<bool> ClHandleICmp("msan-handle-icmp",
244
       cl::desc("propagate shadow through ICmpEQ and ICmpNE"),
245
       cl::Hidden, cl::init(true));
246
247
static cl::opt<bool> ClHandleICmpExact("msan-handle-icmp-exact",
248
       cl::desc("exact handling of relational integer ICmp"),
249
       cl::Hidden, cl::init(false));
250
251
static cl::opt<bool> ClHandleLifetimeIntrinsics(
252
    "msan-handle-lifetime-intrinsics",
253
    cl::desc(
254
        "when possible, poison scoped variables at the beginning of the scope "
255
        "(slower, but more precise)"),
256
    cl::Hidden, cl::init(true));
257
258
// When compiling the Linux kernel, we sometimes see false positives related to
259
// MSan being unable to understand that inline assembly calls may initialize
260
// local variables.
261
// This flag makes the compiler conservatively unpoison every memory location
262
// passed into an assembly call. Note that this may cause false positives.
263
// Because it's impossible to figure out the array sizes, we can only unpoison
264
// the first sizeof(type) bytes for each type* pointer.
265
// The instrumentation is only enabled in KMSAN builds, and only if
266
// -msan-handle-asm-conservative is on. This is done because we may want to
267
// quickly disable assembly instrumentation when it breaks.
268
static cl::opt<bool> ClHandleAsmConservative(
269
    "msan-handle-asm-conservative",
270
    cl::desc("conservative handling of inline assembly"), cl::Hidden,
271
    cl::init(true));
272
273
// This flag controls whether we check the shadow of the address
274
// operand of load or store. Such bugs are very rare, since load from
275
// a garbage address typically results in SEGV, but still happen
276
// (e.g. only lower bits of address are garbage, or the access happens
277
// early at program startup where malloc-ed memory is more likely to
278
// be zeroed. As of 2012-08-28 this flag adds 20% slowdown.
279
static cl::opt<bool> ClCheckAccessAddress("msan-check-access-address",
280
       cl::desc("report accesses through a pointer which has poisoned shadow"),
281
       cl::Hidden, cl::init(true));
282
283
static cl::opt<bool> ClDumpStrictInstructions("msan-dump-strict-instructions",
284
       cl::desc("print out instructions with default strict semantics"),
285
       cl::Hidden, cl::init(false));
286
287
static cl::opt<int> ClInstrumentationWithCallThreshold(
288
    "msan-instrumentation-with-call-threshold",
289
    cl::desc(
290
        "If the function being instrumented requires more than "
291
        "this number of checks and origin stores, use callbacks instead of "
292
        "inline checks (-1 means never use callbacks)."),
293
    cl::Hidden, cl::init(3500));
294
295
static cl::opt<bool>
296
    ClEnableKmsan("msan-kernel",
297
                  cl::desc("Enable KernelMemorySanitizer instrumentation"),
298
                  cl::Hidden, cl::init(false));
299
300
// This is an experiment to enable handling of cases where shadow is a non-zero
301
// compile-time constant. For some unexplainable reason they were silently
302
// ignored in the instrumentation.
303
static cl::opt<bool> ClCheckConstantShadow("msan-check-constant-shadow",
304
       cl::desc("Insert checks for constant shadow values"),
305
       cl::Hidden, cl::init(false));
306
307
// This is off by default because of a bug in gold:
308
// https://sourceware.org/bugzilla/show_bug.cgi?id=19002
309
static cl::opt<bool> ClWithComdat("msan-with-comdat",
310
       cl::desc("Place MSan constructors in comdat sections"),
311
       cl::Hidden, cl::init(false));
312
313
// These options allow to specify custom memory map parameters
314
// See MemoryMapParams for details.
315
static cl::opt<uint64_t> ClAndMask("msan-and-mask",
316
                                   cl::desc("Define custom MSan AndMask"),
317
                                   cl::Hidden, cl::init(0));
318
319
static cl::opt<uint64_t> ClXorMask("msan-xor-mask",
320
                                   cl::desc("Define custom MSan XorMask"),
321
                                   cl::Hidden, cl::init(0));
322
323
static cl::opt<uint64_t> ClShadowBase("msan-shadow-base",
324
                                      cl::desc("Define custom MSan ShadowBase"),
325
                                      cl::Hidden, cl::init(0));
326
327
static cl::opt<uint64_t> ClOriginBase("msan-origin-base",
328
                                      cl::desc("Define custom MSan OriginBase"),
329
                                      cl::Hidden, cl::init(0));
330
331
static const char *const kMsanModuleCtorName = "msan.module_ctor";
332
static const char *const kMsanInitName = "__msan_init";
333
334
namespace {
335
336
// Memory map parameters used in application-to-shadow address calculation.
337
// Offset = (Addr & ~AndMask) ^ XorMask
338
// Shadow = ShadowBase + Offset
339
// Origin = OriginBase + Offset
340
struct MemoryMapParams {
341
  uint64_t AndMask;
342
  uint64_t XorMask;
343
  uint64_t ShadowBase;
344
  uint64_t OriginBase;
345
};
346
347
struct PlatformMemoryMapParams {
348
  const MemoryMapParams *bits32;
349
  const MemoryMapParams *bits64;
350
};
351
352
} // end anonymous namespace
353
354
// i386 Linux
355
static const MemoryMapParams Linux_I386_MemoryMapParams = {
356
  0x000080000000,  // AndMask
357
  0,               // XorMask (not used)
358
  0,               // ShadowBase (not used)
359
  0x000040000000,  // OriginBase
360
};
361
362
// x86_64 Linux
363
static const MemoryMapParams Linux_X86_64_MemoryMapParams = {
364
#ifdef MSAN_LINUX_X86_64_OLD_MAPPING
365
  0x400000000000,  // AndMask
366
  0,               // XorMask (not used)
367
  0,               // ShadowBase (not used)
368
  0x200000000000,  // OriginBase
369
#else
370
  0,               // AndMask (not used)
371
  0x500000000000,  // XorMask
372
  0,               // ShadowBase (not used)
373
  0x100000000000,  // OriginBase
374
#endif
375
};
376
377
// mips64 Linux
378
static const MemoryMapParams Linux_MIPS64_MemoryMapParams = {
379
  0,               // AndMask (not used)
380
  0x008000000000,  // XorMask
381
  0,               // ShadowBase (not used)
382
  0x002000000000,  // OriginBase
383
};
384
385
// ppc64 Linux
386
static const MemoryMapParams Linux_PowerPC64_MemoryMapParams = {
387
  0xE00000000000,  // AndMask
388
  0x100000000000,  // XorMask
389
  0x080000000000,  // ShadowBase
390
  0x1C0000000000,  // OriginBase
391
};
392
393
// aarch64 Linux
394
static const MemoryMapParams Linux_AArch64_MemoryMapParams = {
395
  0,               // AndMask (not used)
396
  0x06000000000,   // XorMask
397
  0,               // ShadowBase (not used)
398
  0x01000000000,   // OriginBase
399
};
400
401
// i386 FreeBSD
402
static const MemoryMapParams FreeBSD_I386_MemoryMapParams = {
403
  0x000180000000,  // AndMask
404
  0x000040000000,  // XorMask
405
  0x000020000000,  // ShadowBase
406
  0x000700000000,  // OriginBase
407
};
408
409
// x86_64 FreeBSD
410
static const MemoryMapParams FreeBSD_X86_64_MemoryMapParams = {
411
  0xc00000000000,  // AndMask
412
  0x200000000000,  // XorMask
413
  0x100000000000,  // ShadowBase
414
  0x380000000000,  // OriginBase
415
};
416
417
// x86_64 NetBSD
418
static const MemoryMapParams NetBSD_X86_64_MemoryMapParams = {
419
  0,               // AndMask
420
  0x500000000000,  // XorMask
421
  0,               // ShadowBase
422
  0x100000000000,  // OriginBase
423
};
424
425
static const PlatformMemoryMapParams Linux_X86_MemoryMapParams = {
426
  &Linux_I386_MemoryMapParams,
427
  &Linux_X86_64_MemoryMapParams,
428
};
429
430
static const PlatformMemoryMapParams Linux_MIPS_MemoryMapParams = {
431
  nullptr,
432
  &Linux_MIPS64_MemoryMapParams,
433
};
434
435
static const PlatformMemoryMapParams Linux_PowerPC_MemoryMapParams = {
436
  nullptr,
437
  &Linux_PowerPC64_MemoryMapParams,
438
};
439
440
static const PlatformMemoryMapParams Linux_ARM_MemoryMapParams = {
441
  nullptr,
442
  &Linux_AArch64_MemoryMapParams,
443
};
444
445
static const PlatformMemoryMapParams FreeBSD_X86_MemoryMapParams = {
446
  &FreeBSD_I386_MemoryMapParams,
447
  &FreeBSD_X86_64_MemoryMapParams,
448
};
449
450
static const PlatformMemoryMapParams NetBSD_X86_MemoryMapParams = {
451
  nullptr,
452
  &NetBSD_X86_64_MemoryMapParams,
453
};
454
455
namespace {
456
457
/// Instrument functions of a module to detect uninitialized reads.
458
///
459
/// Instantiating MemorySanitizer inserts the msan runtime library API function
460
/// declarations into the module if they don't exist already. Instantiating
461
/// ensures the __msan_init function is in the list of global constructors for
462
/// the module.
463
class MemorySanitizer {
464
public:
465
574
  MemorySanitizer(Module &M, MemorySanitizerOptions Options) {
466
574
    this->CompileKernel =
467
574
        ClEnableKmsan.getNumOccurrences() > 0 ? 
ClEnableKmsan55
:
Options.Kernel519
;
468
574
    if (ClTrackOrigins.getNumOccurrences() > 0)
469
190
      this->TrackOrigins = ClTrackOrigins;
470
384
    else
471
384
      this->TrackOrigins = this->CompileKernel ? 
268
:
Options.TrackOrigins316
;
472
574
    this->Recover = ClKeepGoing.getNumOccurrences() > 0
473
574
                        ? 
ClKeepGoing0
474
574
                        : (this->CompileKernel | Options.Recover);
475
574
    initializeModule(M);
476
574
  }
477
478
  // MSan cannot be moved or copied because of MapParams.
479
  MemorySanitizer(MemorySanitizer &&) = delete;
480
  MemorySanitizer &operator=(MemorySanitizer &&) = delete;
481
  MemorySanitizer(const MemorySanitizer &) = delete;
482
  MemorySanitizer &operator=(const MemorySanitizer &) = delete;
483
484
  bool sanitizeFunction(Function &F, TargetLibraryInfo &TLI);
485
486
private:
487
  friend struct MemorySanitizerVisitor;
488
  friend struct VarArgAMD64Helper;
489
  friend struct VarArgMIPS64Helper;
490
  friend struct VarArgAArch64Helper;
491
  friend struct VarArgPowerPC64Helper;
492
493
  void initializeModule(Module &M);
494
  void initializeCallbacks(Module &M);
495
  void createKernelApi(Module &M);
496
  void createUserspaceApi(Module &M);
497
498
  /// True if we're compiling the Linux kernel.
499
  bool CompileKernel;
500
  /// Track origins (allocation points) of uninitialized values.
501
  int TrackOrigins;
502
  bool Recover;
503
504
  LLVMContext *C;
505
  Type *IntptrTy;
506
  Type *OriginTy;
507
508
  // XxxTLS variables represent the per-thread state in MSan and per-task state
509
  // in KMSAN.
510
  // For the userspace these point to thread-local globals. In the kernel land
511
  // they point to the members of a per-task struct obtained via a call to
512
  // __msan_get_context_state().
513
514
  /// Thread-local shadow storage for function parameters.
515
  Value *ParamTLS;
516
517
  /// Thread-local origin storage for function parameters.
518
  Value *ParamOriginTLS;
519
520
  /// Thread-local shadow storage for function return value.
521
  Value *RetvalTLS;
522
523
  /// Thread-local origin storage for function return value.
524
  Value *RetvalOriginTLS;
525
526
  /// Thread-local shadow storage for in-register va_arg function
527
  /// parameters (x86_64-specific).
528
  Value *VAArgTLS;
529
530
  /// Thread-local shadow storage for in-register va_arg function
531
  /// parameters (x86_64-specific).
532
  Value *VAArgOriginTLS;
533
534
  /// Thread-local shadow storage for va_arg overflow area
535
  /// (x86_64-specific).
536
  Value *VAArgOverflowSizeTLS;
537
538
  /// Thread-local space used to pass origin value to the UMR reporting
539
  /// function.
540
  Value *OriginTLS;
541
542
  /// Are the instrumentation callbacks set up?
543
  bool CallbacksInitialized = false;
544
545
  /// The run-time callback to print a warning.
546
  FunctionCallee WarningFn;
547
548
  // These arrays are indexed by log2(AccessSize).
549
  FunctionCallee MaybeWarningFn[kNumberOfAccessSizes];
550
  FunctionCallee MaybeStoreOriginFn[kNumberOfAccessSizes];
551
552
  /// Run-time helper that generates a new origin value for a stack
553
  /// allocation.
554
  FunctionCallee MsanSetAllocaOrigin4Fn;
555
556
  /// Run-time helper that poisons stack on function entry.
557
  FunctionCallee MsanPoisonStackFn;
558
559
  /// Run-time helper that records a store (or any event) of an
560
  /// uninitialized value and returns an updated origin id encoding this info.
561
  FunctionCallee MsanChainOriginFn;
562
563
  /// MSan runtime replacements for memmove, memcpy and memset.
564
  FunctionCallee MemmoveFn, MemcpyFn, MemsetFn;
565
566
  /// KMSAN callback for task-local function argument shadow.
567
  StructType *MsanContextStateTy;
568
  FunctionCallee MsanGetContextStateFn;
569
570
  /// Functions for poisoning/unpoisoning local variables
571
  FunctionCallee MsanPoisonAllocaFn, MsanUnpoisonAllocaFn;
572
573
  /// Each of the MsanMetadataPtrXxx functions returns a pair of shadow/origin
574
  /// pointers.
575
  FunctionCallee MsanMetadataPtrForLoadN, MsanMetadataPtrForStoreN;
576
  FunctionCallee MsanMetadataPtrForLoad_1_8[4];
577
  FunctionCallee MsanMetadataPtrForStore_1_8[4];
578
  FunctionCallee MsanInstrumentAsmStoreFn;
579
580
  /// Helper to choose between different MsanMetadataPtrXxx().
581
  FunctionCallee getKmsanShadowOriginAccessFn(bool isStore, int size);
582
583
  /// Memory map parameters used in application-to-shadow calculation.
584
  const MemoryMapParams *MapParams;
585
586
  /// Custom memory map parameters used when -msan-shadow-base or
587
  // -msan-origin-base is provided.
588
  MemoryMapParams CustomMapParams;
589
590
  MDNode *ColdCallWeights;
591
592
  /// Branch weights for origin store.
593
  MDNode *OriginStoreWeights;
594
595
  /// An empty volatile inline asm that prevents callback merge.
596
  InlineAsm *EmptyAsm;
597
598
  Function *MsanCtorFunction;
599
};
600
601
/// A legacy function pass for msan instrumentation.
602
///
603
/// Instruments functions to detect unitialized reads.
604
struct MemorySanitizerLegacyPass : public FunctionPass {
605
  // Pass identification, replacement for typeid.
606
  static char ID;
607
608
  MemorySanitizerLegacyPass(MemorySanitizerOptions Options = {})
609
91
      : FunctionPass(ID), Options(Options) {}
610
570
  StringRef getPassName() const override { return "MemorySanitizerLegacyPass"; }
611
612
91
  void getAnalysisUsage(AnalysisUsage &AU) const override {
613
91
    AU.addRequired<TargetLibraryInfoWrapperPass>();
614
91
  }
615
616
570
  bool runOnFunction(Function &F) override {
617
570
    return MSan->sanitizeFunction(
618
570
        F, getAnalysis<TargetLibraryInfoWrapperPass>().getTLI());
619
570
  }
620
  bool doInitialization(Module &M) override;
621
622
  Optional<MemorySanitizer> MSan;
623
  MemorySanitizerOptions Options;
624
};
625
626
} // end anonymous namespace
627
628
PreservedAnalyses MemorySanitizerPass::run(Function &F,
629
483
                                           FunctionAnalysisManager &FAM) {
630
483
  MemorySanitizer Msan(*F.getParent(), Options);
631
483
  if (Msan.sanitizeFunction(F, FAM.getResult<TargetLibraryAnalysis>(F)))
632
416
    return PreservedAnalyses::none();
633
67
  return PreservedAnalyses::all();
634
67
}
635
636
char MemorySanitizerLegacyPass::ID = 0;
637
638
11.0k
INITIALIZE_PASS_BEGIN(MemorySanitizerLegacyPass, "msan",
639
11.0k
                      "MemorySanitizer: detects uninitialized reads.", false,
640
11.0k
                      false)
641
11.0k
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
642
11.0k
INITIALIZE_PASS_END(MemorySanitizerLegacyPass, "msan",
643
                    "MemorySanitizer: detects uninitialized reads.", false,
644
                    false)
645
646
FunctionPass *
647
21
llvm::createMemorySanitizerLegacyPassPass(MemorySanitizerOptions Options) {
648
21
  return new MemorySanitizerLegacyPass(Options);
649
21
}
650
651
/// Create a non-const global initialized with the given string.
652
///
653
/// Creates a writable global for Str so that we can pass it to the
654
/// run-time lib. Runtime uses first 4 bytes of the string to store the
655
/// frame ID, so the string needs to be mutable.
656
static GlobalVariable *createPrivateNonConstGlobalForString(Module &M,
657
110
                                                            StringRef Str) {
658
110
  Constant *StrConst = ConstantDataArray::getString(M.getContext(), Str);
659
110
  return new GlobalVariable(M, StrConst->getType(), /*isConstant=*/false,
660
110
                            GlobalValue::PrivateLinkage, StrConst, "");
661
110
}
662
663
/// Create KMSAN API callbacks.
664
68
void MemorySanitizer::createKernelApi(Module &M) {
665
68
  IRBuilder<> IRB(*C);
666
68
667
68
  // These will be initialized in insertKmsanPrologue().
668
68
  RetvalTLS = nullptr;
669
68
  RetvalOriginTLS = nullptr;
670
68
  ParamTLS = nullptr;
671
68
  ParamOriginTLS = nullptr;
672
68
  VAArgTLS = nullptr;
673
68
  VAArgOriginTLS = nullptr;
674
68
  VAArgOverflowSizeTLS = nullptr;
675
68
  // OriginTLS is unused in the kernel.
676
68
  OriginTLS = nullptr;
677
68
678
68
  // __msan_warning() in the kernel takes an origin.
679
68
  WarningFn = M.getOrInsertFunction("__msan_warning", IRB.getVoidTy(),
680
68
                                    IRB.getInt32Ty());
681
68
  // Requests the per-task context state (kmsan_context_state*) from the
682
68
  // runtime library.
683
68
  MsanContextStateTy = StructType::get(
684
68
      ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
685
68
      ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8),
686
68
      ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8),
687
68
      ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8), /* va_arg_origin */
688
68
      IRB.getInt64Ty(), ArrayType::get(OriginTy, kParamTLSSize / 4), OriginTy,
689
68
      OriginTy);
690
68
  MsanGetContextStateFn = M.getOrInsertFunction(
691
68
      "__msan_get_context_state", PointerType::get(MsanContextStateTy, 0));
692
68
693
68
  Type *RetTy = StructType::get(PointerType::get(IRB.getInt8Ty(), 0),
694
68
                                PointerType::get(IRB.getInt32Ty(), 0));
695
68
696
340
  for (int ind = 0, size = 1; ind < 4; 
ind++, size <<= 1272
) {
697
272
    std::string name_load =
698
272
        "__msan_metadata_ptr_for_load_" + std::to_string(size);
699
272
    std::string name_store =
700
272
        "__msan_metadata_ptr_for_store_" + std::to_string(size);
701
272
    MsanMetadataPtrForLoad_1_8[ind] = M.getOrInsertFunction(
702
272
        name_load, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
703
272
    MsanMetadataPtrForStore_1_8[ind] = M.getOrInsertFunction(
704
272
        name_store, RetTy, PointerType::get(IRB.getInt8Ty(), 0));
705
272
  }
706
68
707
68
  MsanMetadataPtrForLoadN = M.getOrInsertFunction(
708
68
      "__msan_metadata_ptr_for_load_n", RetTy,
709
68
      PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
710
68
  MsanMetadataPtrForStoreN = M.getOrInsertFunction(
711
68
      "__msan_metadata_ptr_for_store_n", RetTy,
712
68
      PointerType::get(IRB.getInt8Ty(), 0), IRB.getInt64Ty());
713
68
714
68
  // Functions for poisoning and unpoisoning memory.
715
68
  MsanPoisonAllocaFn =
716
68
      M.getOrInsertFunction("__msan_poison_alloca", IRB.getVoidTy(),
717
68
                            IRB.getInt8PtrTy(), IntptrTy, IRB.getInt8PtrTy());
718
68
  MsanUnpoisonAllocaFn = M.getOrInsertFunction(
719
68
      "__msan_unpoison_alloca", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy);
720
68
}
721
722
3.51k
static Constant *getOrInsertGlobal(Module &M, StringRef Name, Type *Ty) {
723
3.51k
  return M.getOrInsertGlobal(Name, Ty, [&] {
724
1.16k
    return new GlobalVariable(M, Ty, false, GlobalVariable::ExternalLinkage,
725
1.16k
                              nullptr, Name, nullptr,
726
1.16k
                              GlobalVariable::InitialExecTLSModel);
727
1.16k
  });
728
3.51k
}
729
730
/// Insert declarations for userspace-specific functions and globals.
731
439
void MemorySanitizer::createUserspaceApi(Module &M) {
732
439
  IRBuilder<> IRB(*C);
733
439
  // Create the callback.
734
439
  // FIXME: this function should have "Cold" calling conv,
735
439
  // which is not yet implemented.
736
439
  StringRef WarningFnName = Recover ? 
"__msan_warning"0
737
439
                                    : "__msan_warning_noreturn";
738
439
  WarningFn = M.getOrInsertFunction(WarningFnName, IRB.getVoidTy());
739
439
740
439
  // Create the global TLS variables.
741
439
  RetvalTLS =
742
439
      getOrInsertGlobal(M, "__msan_retval_tls",
743
439
                        ArrayType::get(IRB.getInt64Ty(), kRetvalTLSSize / 8));
744
439
745
439
  RetvalOriginTLS = getOrInsertGlobal(M, "__msan_retval_origin_tls", OriginTy);
746
439
747
439
  ParamTLS =
748
439
      getOrInsertGlobal(M, "__msan_param_tls",
749
439
                        ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
750
439
751
439
  ParamOriginTLS =
752
439
      getOrInsertGlobal(M, "__msan_param_origin_tls",
753
439
                        ArrayType::get(OriginTy, kParamTLSSize / 4));
754
439
755
439
  VAArgTLS =
756
439
      getOrInsertGlobal(M, "__msan_va_arg_tls",
757
439
                        ArrayType::get(IRB.getInt64Ty(), kParamTLSSize / 8));
758
439
759
439
  VAArgOriginTLS =
760
439
      getOrInsertGlobal(M, "__msan_va_arg_origin_tls",
761
439
                        ArrayType::get(OriginTy, kParamTLSSize / 4));
762
439
763
439
  VAArgOverflowSizeTLS =
764
439
      getOrInsertGlobal(M, "__msan_va_arg_overflow_size_tls", IRB.getInt64Ty());
765
439
  OriginTLS = getOrInsertGlobal(M, "__msan_origin_tls", IRB.getInt32Ty());
766
439
767
2.19k
  for (size_t AccessSizeIndex = 0; AccessSizeIndex < kNumberOfAccessSizes;
768
1.75k
       AccessSizeIndex++) {
769
1.75k
    unsigned AccessSize = 1 << AccessSizeIndex;
770
1.75k
    std::string FunctionName = "__msan_maybe_warning_" + itostr(AccessSize);
771
1.75k
    MaybeWarningFn[AccessSizeIndex] = M.getOrInsertFunction(
772
1.75k
        FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8),
773
1.75k
        IRB.getInt32Ty());
774
1.75k
775
1.75k
    FunctionName = "__msan_maybe_store_origin_" + itostr(AccessSize);
776
1.75k
    MaybeStoreOriginFn[AccessSizeIndex] = M.getOrInsertFunction(
777
1.75k
        FunctionName, IRB.getVoidTy(), IRB.getIntNTy(AccessSize * 8),
778
1.75k
        IRB.getInt8PtrTy(), IRB.getInt32Ty());
779
1.75k
  }
780
439
781
439
  MsanSetAllocaOrigin4Fn = M.getOrInsertFunction(
782
439
    "__msan_set_alloca_origin4", IRB.getVoidTy(), IRB.getInt8PtrTy(), IntptrTy,
783
439
    IRB.getInt8PtrTy(), IntptrTy);
784
439
  MsanPoisonStackFn =
785
439
      M.getOrInsertFunction("__msan_poison_stack", IRB.getVoidTy(),
786
439
                            IRB.getInt8PtrTy(), IntptrTy);
787
439
}
788
789
/// Insert extern declaration of runtime-provided functions and globals.
790
908
void MemorySanitizer::initializeCallbacks(Module &M) {
791
908
  // Only do this once.
792
908
  if (CallbacksInitialized)
793
401
    return;
794
507
795
507
  IRBuilder<> IRB(*C);
796
507
  // Initialize callbacks that are common for kernel and userspace
797
507
  // instrumentation.
798
507
  MsanChainOriginFn = M.getOrInsertFunction(
799
507
    "__msan_chain_origin", IRB.getInt32Ty(), IRB.getInt32Ty());
800
507
  MemmoveFn = M.getOrInsertFunction(
801
507
    "__msan_memmove", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
802
507
    IRB.getInt8PtrTy(), IntptrTy);
803
507
  MemcpyFn = M.getOrInsertFunction(
804
507
    "__msan_memcpy", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt8PtrTy(),
805
507
    IntptrTy);
806
507
  MemsetFn = M.getOrInsertFunction(
807
507
    "__msan_memset", IRB.getInt8PtrTy(), IRB.getInt8PtrTy(), IRB.getInt32Ty(),
808
507
    IntptrTy);
809
507
  // We insert an empty inline asm after __msan_report* to avoid callback merge.
810
507
  EmptyAsm = InlineAsm::get(FunctionType::get(IRB.getVoidTy(), false),
811
507
                            StringRef(""), StringRef(""),
812
507
                            /*hasSideEffects=*/true);
813
507
814
507
  MsanInstrumentAsmStoreFn =
815
507
      M.getOrInsertFunction("__msan_instrument_asm_store", IRB.getVoidTy(),
816
507
                            PointerType::get(IRB.getInt8Ty(), 0), IntptrTy);
817
507
818
507
  if (CompileKernel) {
819
68
    createKernelApi(M);
820
439
  } else {
821
439
    createUserspaceApi(M);
822
439
  }
823
507
  CallbacksInitialized = true;
824
507
}
825
826
FunctionCallee MemorySanitizer::getKmsanShadowOriginAccessFn(bool isStore,
827
185
                                                             int size) {
828
185
  FunctionCallee *Fns =
829
185
      isStore ? 
MsanMetadataPtrForStore_1_8112
:
MsanMetadataPtrForLoad_1_873
;
830
185
  switch (size) {
831
185
  case 1:
832
22
    return Fns[0];
833
185
  case 2:
834
4
    return Fns[1];
835
185
  case 4:
836
119
    return Fns[2];
837
185
  case 8:
838
36
    return Fns[3];
839
185
  default:
840
4
    return nullptr;
841
185
  }
842
185
}
843
844
/// Module-level initialization.
845
///
846
/// inserts a call to __msan_init to the module's constructor list.
847
574
void MemorySanitizer::initializeModule(Module &M) {
848
574
  auto &DL = M.getDataLayout();
849
574
850
574
  bool ShadowPassed = ClShadowBase.getNumOccurrences() > 0;
851
574
  bool OriginPassed = ClOriginBase.getNumOccurrences() > 0;
852
574
  // Check the overrides first
853
574
  if (ShadowPassed || 
OriginPassed562
) {
854
15
    CustomMapParams.AndMask = ClAndMask;
855
15
    CustomMapParams.XorMask = ClXorMask;
856
15
    CustomMapParams.ShadowBase = ClShadowBase;
857
15
    CustomMapParams.OriginBase = ClOriginBase;
858
15
    MapParams = &CustomMapParams;
859
559
  } else {
860
559
    Triple TargetTriple(M.getTargetTriple());
861
559
    switch (TargetTriple.getOS()) {
862
559
      case Triple::FreeBSD:
863
0
        switch (TargetTriple.getArch()) {
864
0
          case Triple::x86_64:
865
0
            MapParams = FreeBSD_X86_MemoryMapParams.bits64;
866
0
            break;
867
0
          case Triple::x86:
868
0
            MapParams = FreeBSD_X86_MemoryMapParams.bits32;
869
0
            break;
870
0
          default:
871
0
            report_fatal_error("unsupported architecture");
872
0
        }
873
0
        break;
874
0
      case Triple::NetBSD:
875
0
        switch (TargetTriple.getArch()) {
876
0
          case Triple::x86_64:
877
0
            MapParams = NetBSD_X86_MemoryMapParams.bits64;
878
0
            break;
879
0
          default:
880
0
            report_fatal_error("unsupported architecture");
881
0
        }
882
0
        break;
883
559
      case Triple::Linux:
884
559
        switch (TargetTriple.getArch()) {
885
559
          case Triple::x86_64:
886
511
            MapParams = Linux_X86_MemoryMapParams.bits64;
887
511
            break;
888
559
          case Triple::x86:
889
3
            MapParams = Linux_X86_MemoryMapParams.bits32;
890
3
            break;
891
559
          case Triple::mips64:
892
14
          case Triple::mips64el:
893
14
            MapParams = Linux_MIPS_MemoryMapParams.bits64;
894
14
            break;
895
23
          case Triple::ppc64:
896
23
          case Triple::ppc64le:
897
23
            MapParams = Linux_PowerPC_MemoryMapParams.bits64;
898
23
            break;
899
23
          case Triple::aarch64:
900
8
          case Triple::aarch64_be:
901
8
            MapParams = Linux_ARM_MemoryMapParams.bits64;
902
8
            break;
903
8
          default:
904
0
            report_fatal_error("unsupported architecture");
905
559
        }
906
559
        break;
907
559
      default:
908
0
        report_fatal_error("unsupported operating system");
909
574
    }
910
574
  }
911
574
912
574
  C = &(M.getContext());
913
574
  IRBuilder<> IRB(*C);
914
574
  IntptrTy = IRB.getIntPtrTy(DL);
915
574
  OriginTy = IRB.getInt32Ty();
916
574
917
574
  ColdCallWeights = MDBuilder(*C).createBranchWeights(1, 1000);
918
574
  OriginStoreWeights = MDBuilder(*C).createBranchWeights(1, 1000);
919
574
920
574
  if (!CompileKernel) {
921
506
    std::tie(MsanCtorFunction, std::ignore) =
922
506
        getOrCreateSanitizerCtorAndInitFunctions(
923
506
            M, kMsanModuleCtorName, kMsanInitName,
924
506
            /*InitArgTypes=*/{},
925
506
            /*InitArgs=*/{},
926
506
            // This callback is invoked when the functions are created the first
927
506
            // time. Hook them into the global ctors list in that case:
928
506
            [&](Function *Ctor, FunctionCallee) {
929
145
              if (!ClWithComdat) {
930
145
                appendToGlobalCtors(M, Ctor, 0);
931
145
                return;
932
145
              }
933
0
              Comdat *MsanCtorComdat = M.getOrInsertComdat(kMsanModuleCtorName);
934
0
              Ctor->setComdat(MsanCtorComdat);
935
0
              appendToGlobalCtors(M, Ctor, 0, Ctor);
936
0
            });
937
506
938
506
    if (TrackOrigins)
939
194
      M.getOrInsertGlobal("__msan_track_origins", IRB.getInt32Ty(), [&] {
940
45
        return new GlobalVariable(
941
45
            M, IRB.getInt32Ty(), true, GlobalValue::WeakODRLinkage,
942
45
            IRB.getInt32(TrackOrigins), "__msan_track_origins");
943
45
      });
944
506
945
506
    if (Recover)
946
0
      M.getOrInsertGlobal("__msan_keep_going", IRB.getInt32Ty(), [&] {
947
0
        return new GlobalVariable(M, IRB.getInt32Ty(), true,
948
0
                                  GlobalValue::WeakODRLinkage,
949
0
                                  IRB.getInt32(Recover), "__msan_keep_going");
950
0
      });
951
506
}
952
574
}
953
954
91
bool MemorySanitizerLegacyPass::doInitialization(Module &M) {
955
91
  MSan.emplace(M, Options);
956
91
  return true;
957
91
}
958
959
namespace {
960
961
/// A helper class that handles instrumentation of VarArg
962
/// functions on a particular platform.
963
///
964
/// Implementations are expected to insert the instrumentation
965
/// necessary to propagate argument shadow through VarArg function
966
/// calls. Visit* methods are called during an InstVisitor pass over
967
/// the function, and should avoid creating new basic blocks. A new
968
/// instance of this class is created for each instrumented function.
969
struct VarArgHelper {
970
908
  virtual ~VarArgHelper() = default;
971
972
  /// Visit a CallSite.
973
  virtual void visitCallSite(CallSite &CS, IRBuilder<> &IRB) = 0;
974
975
  /// Visit a va_start call.
976
  virtual void visitVAStartInst(VAStartInst &I) = 0;
977
978
  /// Visit a va_copy call.
979
  virtual void visitVACopyInst(VACopyInst &I) = 0;
980
981
  /// Finalize function instrumentation.
982
  ///
983
  /// This method is called after visiting all interesting (see above)
984
  /// instructions in a function.
985
  virtual void finalizeInstrumentation() = 0;
986
};
987
988
struct MemorySanitizerVisitor;
989
990
} // end anonymous namespace
991
992
static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
993
                                        MemorySanitizerVisitor &Visitor);
994
995
308
static unsigned TypeSizeToSizeIndex(unsigned TypeSize) {
996
308
  if (TypeSize <= 8) 
return 050
;
997
258
  return Log2_32_Ceil((TypeSize + 7) / 8);
998
258
}
999
1000
namespace {
1001
1002
/// This class does all the work for a given function. Store and Load
1003
/// instructions store and load corresponding shadow and origin
1004
/// values. Most instructions propagate shadow from arguments to their
1005
/// return values. Certain instructions (most importantly, BranchInst)
1006
/// test their argument shadow and print reports (with a runtime call) if it's
1007
/// non-zero.
1008
struct MemorySanitizerVisitor : public InstVisitor<MemorySanitizerVisitor> {
1009
  Function &F;
1010
  MemorySanitizer &MS;
1011
  SmallVector<PHINode *, 16> ShadowPHINodes, OriginPHINodes;
1012
  ValueMap<Value*, Value*> ShadowMap, OriginMap;
1013
  std::unique_ptr<VarArgHelper> VAHelper;
1014
  const TargetLibraryInfo *TLI;
1015
  BasicBlock *ActualFnStart;
1016
1017
  // The following flags disable parts of MSan instrumentation based on
1018
  // blacklist contents and command-line options.
1019
  bool InsertChecks;
1020
  bool PropagateShadow;
1021
  bool PoisonStack;
1022
  bool PoisonUndef;
1023
  bool CheckReturnValue;
1024
1025
  struct ShadowOriginAndInsertPoint {
1026
    Value *Shadow;
1027
    Value *Origin;
1028
    Instruction *OrigIns;
1029
1030
    ShadowOriginAndInsertPoint(Value *S, Value *O, Instruction *I)
1031
249
      : Shadow(S), Origin(O), OrigIns(I) {}
1032
  };
1033
  SmallVector<ShadowOriginAndInsertPoint, 16> InstrumentationList;
1034
  bool InstrumentLifetimeStart = ClHandleLifetimeIntrinsics;
1035
  SmallSet<AllocaInst *, 16> AllocaSet;
1036
  SmallVector<std::pair<IntrinsicInst *, AllocaInst *>, 16> LifetimeStartList;
1037
  SmallVector<StoreInst *, 16> StoreList;
1038
1039
  MemorySanitizerVisitor(Function &F, MemorySanitizer &MS,
1040
                         const TargetLibraryInfo &TLI)
1041
908
      : F(F), MS(MS), VAHelper(CreateVarArgHelper(F, MS, *this)), TLI(&TLI) {
1042
908
    bool SanitizeFunction = F.hasFnAttribute(Attribute::SanitizeMemory);
1043
908
    InsertChecks = SanitizeFunction;
1044
908
    PropagateShadow = SanitizeFunction;
1045
908
    PoisonStack = SanitizeFunction && 
ClPoisonStack762
;
1046
908
    PoisonUndef = SanitizeFunction && 
ClPoisonUndef762
;
1047
908
    // FIXME: Consider using SpecialCaseList to specify a list of functions that
1048
908
    // must always return fully initialized values. For now, we hardcode "main".
1049
908
    CheckReturnValue = SanitizeFunction && 
(F.getName() == "main")762
;
1050
908
1051
908
    MS.initializeCallbacks(*F.getParent());
1052
908
    if (MS.CompileKernel)
1053
107
      ActualFnStart = insertKmsanPrologue(F);
1054
801
    else
1055
801
      ActualFnStart = &F.getEntryBlock();
1056
908
1057
908
    LLVM_DEBUG(if (!InsertChecks) dbgs()
1058
908
               << "MemorySanitizer is not inserting checks into '"
1059
908
               << F.getName() << "'\n");
1060
908
  }
1061
1062
67
  Value *updateOrigin(Value *V, IRBuilder<> &IRB) {
1063
67
    if (MS.TrackOrigins <= 1) 
return V42
;
1064
25
    return IRB.CreateCall(MS.MsanChainOriginFn, V);
1065
25
  }
1066
1067
63
  Value *originToIntptr(IRBuilder<> &IRB, Value *Origin) {
1068
63
    const DataLayout &DL = F.getParent()->getDataLayout();
1069
63
    unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1070
63
    if (IntptrSize == kOriginSize) 
return Origin0
;
1071
63
    assert(IntptrSize == kOriginSize * 2);
1072
63
    Origin = IRB.CreateIntCast(Origin, MS.IntptrTy, /* isSigned */ false);
1073
63
    return IRB.CreateOr(Origin, IRB.CreateShl(Origin, kOriginSize * 8));
1074
63
  }
1075
1076
  /// Fill memory range with the given origin value.
1077
  void paintOrigin(IRBuilder<> &IRB, Value *Origin, Value *OriginPtr,
1078
119
                   unsigned Size, unsigned Alignment) {
1079
119
    const DataLayout &DL = F.getParent()->getDataLayout();
1080
119
    unsigned IntptrAlignment = DL.getABITypeAlignment(MS.IntptrTy);
1081
119
    unsigned IntptrSize = DL.getTypeStoreSize(MS.IntptrTy);
1082
119
    assert(IntptrAlignment >= kMinOriginAlignment);
1083
119
    assert(IntptrSize >= kOriginSize);
1084
119
1085
119
    unsigned Ofs = 0;
1086
119
    unsigned CurrentAlignment = Alignment;
1087
119
    if (Alignment >= IntptrAlignment && 
IntptrSize > kOriginSize63
) {
1088
63
      Value *IntptrOrigin = originToIntptr(IRB, Origin);
1089
63
      Value *IntptrOriginPtr =
1090
63
          IRB.CreatePointerCast(OriginPtr, PointerType::get(MS.IntptrTy, 0));
1091
109
      for (unsigned i = 0; i < Size / IntptrSize; 
++i46
) {
1092
46
        Value *Ptr = i ? 
IRB.CreateConstGEP1_32(MS.IntptrTy, IntptrOriginPtr, i)6
1093
46
                       : 
IntptrOriginPtr40
;
1094
46
        IRB.CreateAlignedStore(IntptrOrigin, Ptr, CurrentAlignment);
1095
46
        Ofs += IntptrSize / kOriginSize;
1096
46
        CurrentAlignment = IntptrAlignment;
1097
46
      }
1098
63
    }
1099
119
1100
263
    for (unsigned i = Ofs; i < (Size + kOriginSize - 1) / kOriginSize; 
++i144
) {
1101
144
      Value *GEP =
1102
144
          i ? 
IRB.CreateConstGEP1_32(MS.OriginTy, OriginPtr, i)65
:
OriginPtr79
;
1103
144
      IRB.CreateAlignedStore(Origin, GEP, CurrentAlignment);
1104
144
      CurrentAlignment = kMinOriginAlignment;
1105
144
    }
1106
119
  }
1107
1108
  void storeOrigin(IRBuilder<> &IRB, Value *Addr, Value *Shadow, Value *Origin,
1109
196
                   Value *OriginPtr, unsigned Alignment, bool AsCall) {
1110
196
    const DataLayout &DL = F.getParent()->getDataLayout();
1111
196
    unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1112
196
    unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
1113
196
    if (Shadow->getType()->isAggregateType()) {
1114
8
      paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1115
8
                  OriginAlignment);
1116
188
    } else {
1117
188
      Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB);
1118
188
      Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow);
1119
188
      if (ConstantShadow) {
1120
127
        if (ClCheckConstantShadow && 
!ConstantShadow->isZeroValue()4
)
1121
2
          paintOrigin(IRB, updateOrigin(Origin, IRB), OriginPtr, StoreSize,
1122
2
                      OriginAlignment);
1123
127
        return;
1124
127
      }
1125
61
1126
61
      unsigned TypeSizeInBits =
1127
61
          DL.getTypeSizeInBits(ConvertedShadow->getType());
1128
61
      unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1129
61
      if (AsCall && 
SizeIndex < kNumberOfAccessSizes4
&&
!MS.CompileKernel4
) {
1130
4
        FunctionCallee Fn = MS.MaybeStoreOriginFn[SizeIndex];
1131
4
        Value *ConvertedShadow2 = IRB.CreateZExt(
1132
4
            ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1133
4
        IRB.CreateCall(Fn, {ConvertedShadow2,
1134
4
                            IRB.CreatePointerCast(Addr, IRB.getInt8PtrTy()),
1135
4
                            Origin});
1136
57
      } else {
1137
57
        Value *Cmp = IRB.CreateICmpNE(
1138
57
            ConvertedShadow, getCleanShadow(ConvertedShadow), "_mscmp");
1139
57
        Instruction *CheckTerm = SplitBlockAndInsertIfThen(
1140
57
            Cmp, &*IRB.GetInsertPoint(), false, MS.OriginStoreWeights);
1141
57
        IRBuilder<> IRBNew(CheckTerm);
1142
57
        paintOrigin(IRBNew, updateOrigin(Origin, IRBNew), OriginPtr, StoreSize,
1143
57
                    OriginAlignment);
1144
57
      }
1145
61
    }
1146
196
  }
1147
1148
908
  void materializeStores(bool InstrumentWithCalls) {
1149
908
    for (StoreInst *SI : StoreList) {
1150
342
      IRBuilder<> IRB(SI);
1151
342
      Value *Val = SI->getValueOperand();
1152
342
      Value *Addr = SI->getPointerOperand();
1153
342
      Value *Shadow = SI->isAtomic() ? 
getCleanShadow(Val)24
:
getShadow(Val)318
;
1154
342
      Value *ShadowPtr, *OriginPtr;
1155
342
      Type *ShadowTy = Shadow->getType();
1156
342
      unsigned Alignment = SI->getAlignment();
1157
342
      unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1158
342
      std::tie(ShadowPtr, OriginPtr) =
1159
342
          getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ true);
1160
342
1161
342
      StoreInst *NewSI = IRB.CreateAlignedStore(Shadow, ShadowPtr, Alignment);
1162
342
      LLVM_DEBUG(dbgs() << "  STORE: " << *NewSI << "\n");
1163
342
      (void)NewSI;
1164
342
1165
342
      if (SI->isAtomic())
1166
24
        SI->setOrdering(addReleaseOrdering(SI->getOrdering()));
1167
342
1168
342
      if (MS.TrackOrigins && 
!SI->isAtomic()212
)
1169
196
        storeOrigin(IRB, Addr, Shadow, getOrigin(Val), OriginPtr,
1170
196
                    OriginAlignment, InstrumentWithCalls);
1171
342
    }
1172
908
  }
1173
1174
  /// Helper function to insert a warning at IRB's current insert point.
1175
227
  void insertWarningFn(IRBuilder<> &IRB, Value *Origin) {
1176
227
    if (!Origin)
1177
106
      Origin = (Value *)IRB.getInt32(0);
1178
227
    if (MS.CompileKernel) {
1179
91
      IRB.CreateCall(MS.WarningFn, Origin);
1180
136
    } else {
1181
136
      if (MS.TrackOrigins) {
1182
26
        IRB.CreateStore(Origin, MS.OriginTLS);
1183
26
      }
1184
136
      IRB.CreateCall(MS.WarningFn, {});
1185
136
    }
1186
227
    IRB.CreateCall(MS.EmptyAsm, {});
1187
227
    // FIXME: Insert UnreachableInst if !MS.Recover?
1188
227
    // This may invalidate some of the following checks and needs to be done
1189
227
    // at the very end.
1190
227
  }
1191
1192
  void materializeOneCheck(Instruction *OrigIns, Value *Shadow, Value *Origin,
1193
249
                           bool AsCall) {
1194
249
    IRBuilder<> IRB(OrigIns);
1195
249
    LLVM_DEBUG(dbgs() << "  SHAD0 : " << *Shadow << "\n");
1196
249
    Value *ConvertedShadow = convertToShadowTyNoVec(Shadow, IRB);
1197
249
    LLVM_DEBUG(dbgs() << "  SHAD1 : " << *ConvertedShadow << "\n");
1198
249
1199
249
    Constant *ConstantShadow = dyn_cast_or_null<Constant>(ConvertedShadow);
1200
249
    if (ConstantShadow) {
1201
2
      if (ClCheckConstantShadow && !ConstantShadow->isZeroValue()) {
1202
2
        insertWarningFn(IRB, Origin);
1203
2
      }
1204
2
      return;
1205
2
    }
1206
247
1207
247
    const DataLayout &DL = OrigIns->getModule()->getDataLayout();
1208
247
1209
247
    unsigned TypeSizeInBits = DL.getTypeSizeInBits(ConvertedShadow->getType());
1210
247
    unsigned SizeIndex = TypeSizeToSizeIndex(TypeSizeInBits);
1211
247
    if (AsCall && 
SizeIndex < kNumberOfAccessSizes24
&&
!MS.CompileKernel22
) {
1212
22
      FunctionCallee Fn = MS.MaybeWarningFn[SizeIndex];
1213
22
      Value *ConvertedShadow2 =
1214
22
          IRB.CreateZExt(ConvertedShadow, IRB.getIntNTy(8 * (1 << SizeIndex)));
1215
22
      IRB.CreateCall(Fn, {ConvertedShadow2, MS.TrackOrigins && 
Origin4
1216
22
                                                ? 
Origin4
1217
22
                                                : 
(Value *)IRB.getInt32(0)18
});
1218
225
    } else {
1219
225
      Value *Cmp = IRB.CreateICmpNE(ConvertedShadow,
1220
225
                                    getCleanShadow(ConvertedShadow), "_mscmp");
1221
225
      Instruction *CheckTerm = SplitBlockAndInsertIfThen(
1222
225
          Cmp, OrigIns,
1223
225
          /* Unreachable */ !MS.Recover, MS.ColdCallWeights);
1224
225
1225
225
      IRB.SetInsertPoint(CheckTerm);
1226
225
      insertWarningFn(IRB, Origin);
1227
225
      LLVM_DEBUG(dbgs() << "  CHECK: " << *Cmp << "\n");
1228
225
    }
1229
247
  }
1230
1231
908
  void materializeChecks(bool InstrumentWithCalls) {
1232
908
    for (const auto &ShadowData : InstrumentationList) {
1233
249
      Instruction *OrigIns = ShadowData.OrigIns;
1234
249
      Value *Shadow = ShadowData.Shadow;
1235
249
      Value *Origin = ShadowData.Origin;
1236
249
      materializeOneCheck(OrigIns, Shadow, Origin, InstrumentWithCalls);
1237
249
    }
1238
908
    LLVM_DEBUG(dbgs() << "DONE:\n" << F);
1239
908
  }
1240
1241
107
  BasicBlock *insertKmsanPrologue(Function &F) {
1242
107
    BasicBlock *ret =
1243
107
        SplitBlock(&F.getEntryBlock(), F.getEntryBlock().getFirstNonPHI());
1244
107
    IRBuilder<> IRB(F.getEntryBlock().getFirstNonPHI());
1245
107
    Value *ContextState = IRB.CreateCall(MS.MsanGetContextStateFn, {});
1246
107
    Constant *Zero = IRB.getInt32(0);
1247
107
    MS.ParamTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1248
107
                                {Zero, IRB.getInt32(0)}, "param_shadow");
1249
107
    MS.RetvalTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1250
107
                                 {Zero, IRB.getInt32(1)}, "retval_shadow");
1251
107
    MS.VAArgTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1252
107
                                {Zero, IRB.getInt32(2)}, "va_arg_shadow");
1253
107
    MS.VAArgOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1254
107
                                      {Zero, IRB.getInt32(3)}, "va_arg_origin");
1255
107
    MS.VAArgOverflowSizeTLS =
1256
107
        IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1257
107
                      {Zero, IRB.getInt32(4)}, "va_arg_overflow_size");
1258
107
    MS.ParamOriginTLS = IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1259
107
                                      {Zero, IRB.getInt32(5)}, "param_origin");
1260
107
    MS.RetvalOriginTLS =
1261
107
        IRB.CreateGEP(MS.MsanContextStateTy, ContextState,
1262
107
                      {Zero, IRB.getInt32(6)}, "retval_origin");
1263
107
    return ret;
1264
107
  }
1265
1266
  /// Add MemorySanitizer instrumentation to a function.
1267
908
  bool runOnFunction() {
1268
908
    // In the presence of unreachable blocks, we may see Phi nodes with
1269
908
    // incoming nodes from such blocks. Since InstVisitor skips unreachable
1270
908
    // blocks, such nodes will not have any shadow value associated with them.
1271
908
    // It's easier to remove unreachable blocks than deal with missing shadow.
1272
908
    removeUnreachableBlocks(F);
1273
908
1274
908
    // Iterate all BBs in depth-first order and create shadow instructions
1275
908
    // for all instructions (where applicable).
1276
908
    // For PHI nodes we create dummy shadow PHIs which will be finalized later.
1277
908
    for (BasicBlock *BB : depth_first(ActualFnStart))
1278
1.05k
      visit(*BB);
1279
908
1280
908
    // Finalize PHI nodes.
1281
908
    for (PHINode *PN : ShadowPHINodes) {
1282
8
      PHINode *PNS = cast<PHINode>(getShadow(PN));
1283
8
      PHINode *PNO = MS.TrackOrigins ? 
cast<PHINode>(getOrigin(PN))6
:
nullptr2
;
1284
8
      size_t NumValues = PN->getNumIncomingValues();
1285
24
      for (size_t v = 0; v < NumValues; 
v++16
) {
1286
16
        PNS->addIncoming(getShadow(PN, v), PN->getIncomingBlock(v));
1287
16
        if (PNO) 
PNO->addIncoming(getOrigin(PN, v), PN->getIncomingBlock(v))12
;
1288
16
      }
1289
8
    }
1290
908
1291
908
    VAHelper->finalizeInstrumentation();
1292
908
1293
908
    // Poison llvm.lifetime.start intrinsics, if we haven't fallen back to
1294
908
    // instrumenting only allocas.
1295
908
    if (InstrumentLifetimeStart) {
1296
897
      for (auto Item : LifetimeStartList) {
1297
35
        instrumentAlloca(*Item.second, Item.first);
1298
35
        AllocaSet.erase(Item.second);
1299
35
      }
1300
897
    }
1301
908
    // Poison the allocas for which we didn't instrument the corresponding
1302
908
    // lifetime intrinsics.
1303
908
    for (AllocaInst *AI : AllocaSet)
1304
204
      instrumentAlloca(*AI);
1305
908
1306
908
    bool InstrumentWithCalls = ClInstrumentationWithCallThreshold >= 0 &&
1307
908
                               InstrumentationList.size() + StoreList.size() >
1308
908
                                   (unsigned)ClInstrumentationWithCallThreshold;
1309
908
1310
908
    // Insert shadow value checks.
1311
908
    materializeChecks(InstrumentWithCalls);
1312
908
1313
908
    // Delayed instrumentation of StoreInst.
1314
908
    // This may not add new address checks.
1315
908
    materializeStores(InstrumentWithCalls);
1316
908
1317
908
    return true;
1318
908
  }
1319
1320
  /// Compute the shadow type that corresponds to a given Value.
1321
4.65k
  Type *getShadowTy(Value *V) {
1322
4.65k
    return getShadowTy(V->getType());
1323
4.65k
  }
1324
1325
  /// Compute the shadow type that corresponds to a given Type.
1326
11.9k
  Type *getShadowTy(Type *OrigTy) {
1327
11.9k
    if (!OrigTy->isSized()) {
1328
397
      return nullptr;
1329
397
    }
1330
11.5k
    // For integer type, shadow is the same as the original type.
1331
11.5k
    // This may return weird-sized types like i1.
1332
11.5k
    if (IntegerType *IT = dyn_cast<IntegerType>(OrigTy))
1333
8.75k
      return IT;
1334
2.81k
    const DataLayout &DL = F.getParent()->getDataLayout();
1335
2.81k
    if (VectorType *VT = dyn_cast<VectorType>(OrigTy)) {
1336
606
      uint32_t EltSize = DL.getTypeSizeInBits(VT->getElementType());
1337
606
      return VectorType::get(IntegerType::get(*MS.C, EltSize),
1338
606
                             VT->getNumElements());
1339
606
    }
1340
2.20k
    if (ArrayType *AT = dyn_cast<ArrayType>(OrigTy)) {
1341
96
      return ArrayType::get(getShadowTy(AT->getElementType()),
1342
96
                            AT->getNumElements());
1343
96
    }
1344
2.11k
    if (StructType *ST = dyn_cast<StructType>(OrigTy)) {
1345
142
      SmallVector<Type*, 4> Elements;
1346
466
      for (unsigned i = 0, n = ST->getNumElements(); i < n; 
i++324
)
1347
324
        Elements.push_back(getShadowTy(ST->getElementType(i)));
1348
142
      StructType *Res = StructType::get(*MS.C, Elements, ST->isPacked());
1349
142
      LLVM_DEBUG(dbgs() << "getShadowTy: " << *ST << " ===> " << *Res << "\n");
1350
142
      return Res;
1351
142
    }
1352
1.96k
    uint32_t TypeSize = DL.getTypeSizeInBits(OrigTy);
1353
1.96k
    return IntegerType::get(*MS.C, TypeSize);
1354
1.96k
  }
1355
1356
  /// Flatten a vector type.
1357
463
  Type *getShadowTyNoVec(Type *ty) {
1358
463
    if (VectorType *vt = dyn_cast<VectorType>(ty))
1359
12
      return IntegerType::get(*MS.C, vt->getBitWidth());
1360
451
    return ty;
1361
451
  }
1362
1363
  /// Convert a shadow value to it's flattened variant.
1364
461
  Value *convertToShadowTyNoVec(Value *V, IRBuilder<> &IRB) {
1365
461
    Type *Ty = V->getType();
1366
461
    Type *NoVecTy = getShadowTyNoVec(Ty);
1367
461
    if (Ty == NoVecTy) 
return V451
;
1368
10
    return IRB.CreateBitCast(V, NoVecTy);
1369
10
  }
1370
1371
  /// Compute the integer shadow offset that corresponds to a given
1372
  /// application address.
1373
  ///
1374
  /// Offset = (Addr & ~AndMask) ^ XorMask
1375
715
  Value *getShadowPtrOffset(Value *Addr, IRBuilder<> &IRB) {
1376
715
    Value *OffsetLong = IRB.CreatePointerCast(Addr, MS.IntptrTy);
1377
715
1378
715
    uint64_t AndMask = MS.MapParams->AndMask;
1379
715
    if (AndMask)
1380
42
      OffsetLong =
1381
42
          IRB.CreateAnd(OffsetLong, ConstantInt::get(MS.IntptrTy, ~AndMask));
1382
715
1383
715
    uint64_t XorMask = MS.MapParams->XorMask;
1384
715
    if (XorMask)
1385
707
      OffsetLong =
1386
707
          IRB.CreateXor(OffsetLong, ConstantInt::get(MS.IntptrTy, XorMask));
1387
715
    return OffsetLong;
1388
715
  }
1389
1390
  /// Compute the shadow and origin addresses corresponding to a given
1391
  /// application address.
1392
  ///
1393
  /// Shadow = ShadowBase + Offset
1394
  /// Origin = (OriginBase + Offset) & ~3ULL
1395
  std::pair<Value *, Value *> getShadowOriginPtrUserspace(Value *Addr,
1396
                                                          IRBuilder<> &IRB,
1397
                                                          Type *ShadowTy,
1398
715
                                                          unsigned Alignment) {
1399
715
    Value *ShadowOffset = getShadowPtrOffset(Addr, IRB);
1400
715
    Value *ShadowLong = ShadowOffset;
1401
715
    uint64_t ShadowBase = MS.MapParams->ShadowBase;
1402
715
    if (ShadowBase != 0) {
1403
44
      ShadowLong =
1404
44
        IRB.CreateAdd(ShadowLong,
1405
44
                      ConstantInt::get(MS.IntptrTy, ShadowBase));
1406
44
    }
1407
715
    Value *ShadowPtr =
1408
715
        IRB.CreateIntToPtr(ShadowLong, PointerType::get(ShadowTy, 0));
1409
715
    Value *OriginPtr = nullptr;
1410
715
    if (MS.TrackOrigins) {
1411
272
      Value *OriginLong = ShadowOffset;
1412
272
      uint64_t OriginBase = MS.MapParams->OriginBase;
1413
272
      if (OriginBase != 0)
1414
272
        OriginLong = IRB.CreateAdd(OriginLong,
1415
272
                                   ConstantInt::get(MS.IntptrTy, OriginBase));
1416
272
      if (Alignment < kMinOriginAlignment) {
1417
123
        uint64_t Mask = kMinOriginAlignment - 1;
1418
123
        OriginLong =
1419
123
            IRB.CreateAnd(OriginLong, ConstantInt::get(MS.IntptrTy, ~Mask));
1420
123
      }
1421
272
      OriginPtr =
1422
272
          IRB.CreateIntToPtr(OriginLong, PointerType::get(MS.OriginTy, 0));
1423
272
    }
1424
715
    return std::make_pair(ShadowPtr, OriginPtr);
1425
715
  }
1426
1427
  std::pair<Value *, Value *>
1428
  getShadowOriginPtrKernel(Value *Addr, IRBuilder<> &IRB, Type *ShadowTy,
1429
185
                           unsigned Alignment, bool isStore) {
1430
185
    Value *ShadowOriginPtrs;
1431
185
    const DataLayout &DL = F.getParent()->getDataLayout();
1432
185
    int Size = DL.getTypeStoreSize(ShadowTy);
1433
185
1434
185
    FunctionCallee Getter = MS.getKmsanShadowOriginAccessFn(isStore, Size);
1435
185
    Value *AddrCast =
1436
185
        IRB.CreatePointerCast(Addr, PointerType::get(IRB.getInt8Ty(), 0));
1437
185
    if (Getter) {
1438
181
      ShadowOriginPtrs = IRB.CreateCall(Getter, AddrCast);
1439
181
    } else {
1440
4
      Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
1441
4
      ShadowOriginPtrs = IRB.CreateCall(isStore ? 
MS.MsanMetadataPtrForStoreN2
1442
4
                                                : 
MS.MsanMetadataPtrForLoadN2
,
1443
4
                                        {AddrCast, SizeVal});
1444
4
    }
1445
185
    Value *ShadowPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 0);
1446
185
    ShadowPtr = IRB.CreatePointerCast(ShadowPtr, PointerType::get(ShadowTy, 0));
1447
185
    Value *OriginPtr = IRB.CreateExtractValue(ShadowOriginPtrs, 1);
1448
185
1449
185
    return std::make_pair(ShadowPtr, OriginPtr);
1450
185
  }
1451
1452
  std::pair<Value *, Value *> getShadowOriginPtr(Value *Addr, IRBuilder<> &IRB,
1453
                                                 Type *ShadowTy,
1454
                                                 unsigned Alignment,
1455
900
                                                 bool isStore) {
1456
900
    std::pair<Value *, Value *> ret;
1457
900
    if (MS.CompileKernel)
1458
185
      ret = getShadowOriginPtrKernel(Addr, IRB, ShadowTy, Alignment, isStore);
1459
715
    else
1460
715
      ret = getShadowOriginPtrUserspace(Addr, IRB, ShadowTy, Alignment);
1461
900
    return ret;
1462
900
  }
1463
1464
  /// Compute the shadow address for a given function argument.
1465
  ///
1466
  /// Shadow = ParamTLS+ArgOffset.
1467
  Value *getShadowPtrForArgument(Value *A, IRBuilder<> &IRB,
1468
2.28k
                                 int ArgOffset) {
1469
2.28k
    Value *Base = IRB.CreatePointerCast(MS.ParamTLS, MS.IntptrTy);
1470
2.28k
    if (ArgOffset)
1471
1.67k
      Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1472
2.28k
    return IRB.CreateIntToPtr(Base, PointerType::get(getShadowTy(A), 0),
1473
2.28k
                              "_msarg");
1474
2.28k
  }
1475
1476
  /// Compute the origin address for a given function argument.
1477
  Value *getOriginPtrForArgument(Value *A, IRBuilder<> &IRB,
1478
271
                                 int ArgOffset) {
1479
271
    if (!MS.TrackOrigins)
1480
0
      return nullptr;
1481
271
    Value *Base = IRB.CreatePointerCast(MS.ParamOriginTLS, MS.IntptrTy);
1482
271
    if (ArgOffset)
1483
118
      Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
1484
271
    return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
1485
271
                              "_msarg_o");
1486
271
  }
1487
1488
  /// Compute the shadow address for a retval.
1489
710
  Value *getShadowPtrForRetval(Value *A, IRBuilder<> &IRB) {
1490
710
    return IRB.CreatePointerCast(MS.RetvalTLS,
1491
710
                                 PointerType::get(getShadowTy(A), 0),
1492
710
                                 "_msret");
1493
710
  }
1494
1495
  /// Compute the origin address for a retval.
1496
192
  Value *getOriginPtrForRetval(IRBuilder<> &IRB) {
1497
192
    // We keep a single origin for the entire retval. Might be too optimistic.
1498
192
    return MS.RetvalOriginTLS;
1499
192
  }
1500
1501
  /// Set SV to be the shadow value for V.
1502
1.81k
  void setShadow(Value *V, Value *SV) {
1503
1.81k
    assert(!ShadowMap.count(V) && "Values may only have one shadow");
1504
1.81k
    ShadowMap[V] = PropagateShadow ? 
SV1.38k
:
getCleanShadow(V)425
;
1505
1.81k
  }
1506
1507
  /// Set Origin to be the origin value for V.
1508
2.02k
  void setOrigin(Value *V, Value *Origin) {
1509
2.02k
    if (!MS.TrackOrigins) 
return909
;
1510
1.11k
    assert(!OriginMap.count(V) && "Values may only have one origin");
1511
1.11k
    LLVM_DEBUG(dbgs() << "ORIGIN: " << *V << "  ==> " << *Origin << "\n");
1512
1.11k
    OriginMap[V] = Origin;
1513
1.11k
  }
1514
1515
5.55k
  Constant *getCleanShadow(Type *OrigTy) {
1516
5.55k
    Type *ShadowTy = getShadowTy(OrigTy);
1517
5.55k
    if (!ShadowTy)
1518
397
      return nullptr;
1519
5.16k
    return Constant::getNullValue(ShadowTy);
1520
5.16k
  }
1521
1522
  /// Create a clean shadow value for a given value.
1523
  ///
1524
  /// Clean shadow (all zeroes) means all bits of the value are defined
1525
  /// (initialized).
1526
5.53k
  Constant *getCleanShadow(Value *V) {
1527
5.53k
    return getCleanShadow(V->getType());
1528
5.53k
  }
1529
1530
  /// Create a dirty shadow of a given shadow type.
1531
82
  Constant *getPoisonedShadow(Type *ShadowTy) {
1532
82
    assert(ShadowTy);
1533
82
    if (isa<IntegerType>(ShadowTy) || 
isa<VectorType>(ShadowTy)30
)
1534
56
      return Constant::getAllOnesValue(ShadowTy);
1535
26
    if (ArrayType *AT = dyn_cast<ArrayType>(ShadowTy)) {
1536
12
      SmallVector<Constant *, 4> Vals(AT->getNumElements(),
1537
12
                                      getPoisonedShadow(AT->getElementType()));
1538
12
      return ConstantArray::get(AT, Vals);
1539
12
    }
1540
14
    if (StructType *ST = dyn_cast<StructType>(ShadowTy)) {
1541
14
      SmallVector<Constant *, 4> Vals;
1542
42
      for (unsigned i = 0, n = ST->getNumElements(); i < n; 
i++28
)
1543
28
        Vals.push_back(getPoisonedShadow(ST->getElementType(i)));
1544
14
      return ConstantStruct::get(ST, Vals);
1545
14
    }
1546
0
    llvm_unreachable("Unexpected shadow type");
1547
0
  }
1548
1549
  /// Create a dirty shadow for a given value.
1550
34
  Constant *getPoisonedShadow(Value *V) {
1551
34
    Type *ShadowTy = getShadowTy(V);
1552
34
    if (!ShadowTy)
1553
0
      return nullptr;
1554
34
    return getPoisonedShadow(ShadowTy);
1555
34
  }
1556
1557
  /// Create a clean (zero) origin.
1558
1.46k
  Value *getCleanOrigin() {
1559
1.46k
    return Constant::getNullValue(MS.OriginTy);
1560
1.46k
  }
1561
1562
  /// Get the shadow value for a given Value.
1563
  ///
1564
  /// This function either returns the value set earlier with setShadow,
1565
  /// or extracts if from ParamTLS (for function arguments).
1566
6.06k
  Value *getShadow(Value *V) {
1567
6.06k
    if (!PropagateShadow) 
return getCleanShadow(V)3.24k
;
1568
2.82k
    if (Instruction *I = dyn_cast<Instruction>(V)) {
1569
1.35k
      if (I->getMetadata("nosanitize"))
1570
12
        return getCleanShadow(V);
1571
1.33k
      // For instructions the shadow is already stored in the map.
1572
1.33k
      Value *Shadow = ShadowMap[V];
1573
1.33k
      if (!Shadow) {
1574
0
        LLVM_DEBUG(dbgs() << "No shadow: " << *V << "\n" << *(I->getParent()));
1575
0
        (void)I;
1576
0
        assert(Shadow && "No shadow for a value");
1577
0
      }
1578
1.33k
      return Shadow;
1579
1.33k
    }
1580
1.47k
    if (UndefValue *U = dyn_cast<UndefValue>(V)) {
1581
34
      Value *AllOnes = PoisonUndef ? getPoisonedShadow(V) : 
getCleanShadow(V)0
;
1582
34
      LLVM_DEBUG(dbgs() << "Undef: " << *U << " ==> " << *AllOnes << "\n");
1583
34
      (void)U;
1584
34
      return AllOnes;
1585
34
    }
1586
1.43k
    if (Argument *A = dyn_cast<Argument>(V)) {
1587
804
      // For arguments we compute the shadow on demand and store it in the map.
1588
804
      Value **ShadowPtr = &ShadowMap[V];
1589
804
      if (*ShadowPtr)
1590
114
        return *ShadowPtr;
1591
690
      Function *F = A->getParent();
1592
690
      IRBuilder<> EntryIRB(ActualFnStart->getFirstNonPHI());
1593
690
      unsigned ArgOffset = 0;
1594
690
      const DataLayout &DL = F->getParent()->getDataLayout();
1595
1.28k
      for (auto &FArg : F->args()) {
1596
1.28k
        if (!FArg.getType()->isSized()) {
1597
0
          LLVM_DEBUG(dbgs() << "Arg is not sized\n");
1598
0
          continue;
1599
0
        }
1600
1.28k
        unsigned Size =
1601
1.28k
            FArg.hasByValAttr()
1602
1.28k
                ? 
DL.getTypeAllocSize(FArg.getType()->getPointerElementType())4
1603
1.28k
                : 
DL.getTypeAllocSize(FArg.getType())1.28k
;
1604
1.28k
        if (A == &FArg) {
1605
690
          bool Overflow = ArgOffset + Size > kParamTLSSize;
1606
690
          Value *Base = getShadowPtrForArgument(&FArg, EntryIRB, ArgOffset);
1607
690
          if (FArg.hasByValAttr()) {
1608
4
            // ByVal pointer itself has clean shadow. We copy the actual
1609
4
            // argument shadow to the underlying memory.
1610
4
            // Figure out maximal valid memcpy alignment.
1611
4
            unsigned ArgAlign = FArg.getParamAlignment();
1612
4
            if (ArgAlign == 0) {
1613
4
              Type *EltType = A->getType()->getPointerElementType();
1614
4
              ArgAlign = DL.getABITypeAlignment(EltType);
1615
4
            }
1616
4
            Value *CpShadowPtr =
1617
4
                getShadowOriginPtr(V, EntryIRB, EntryIRB.getInt8Ty(), ArgAlign,
1618
4
                                   /*isStore*/ true)
1619
4
                    .first;
1620
4
            // TODO(glider): need to copy origins.
1621
4
            if (Overflow) {
1622
0
              // ParamTLS overflow.
1623
0
              EntryIRB.CreateMemSet(
1624
0
                  CpShadowPtr, Constant::getNullValue(EntryIRB.getInt8Ty()),
1625
0
                  Size, ArgAlign);
1626
4
            } else {
1627
4
              unsigned CopyAlign = std::min(ArgAlign, kShadowTLSAlignment);
1628
4
              Value *Cpy = EntryIRB.CreateMemCpy(CpShadowPtr, CopyAlign, Base,
1629
4
                                                 CopyAlign, Size);
1630
4
              LLVM_DEBUG(dbgs() << "  ByValCpy: " << *Cpy << "\n");
1631
4
              (void)Cpy;
1632
4
            }
1633
4
            *ShadowPtr = getCleanShadow(V);
1634
686
          } else {
1635
686
            if (Overflow) {
1636
0
              // ParamTLS overflow.
1637
0
              *ShadowPtr = getCleanShadow(V);
1638
686
            } else {
1639
686
              *ShadowPtr = EntryIRB.CreateAlignedLoad(getShadowTy(&FArg), Base,
1640
686
                                                      kShadowTLSAlignment);
1641
686
            }
1642
686
          }
1643
690
          LLVM_DEBUG(dbgs()
1644
690
                     << "  ARG:    " << FArg << " ==> " << **ShadowPtr << "\n");
1645
690
          if (MS.TrackOrigins && 
!Overflow239
) {
1646
239
            Value *OriginPtr =
1647
239
                getOriginPtrForArgument(&FArg, EntryIRB, ArgOffset);
1648
239
            setOrigin(A, EntryIRB.CreateLoad(MS.OriginTy, OriginPtr));
1649
451
          } else {
1650
451
            setOrigin(A, getCleanOrigin());
1651
451
          }
1652
690
        }
1653
1.28k
        ArgOffset += alignTo(Size, kShadowTLSAlignment);
1654
1.28k
      }
1655
690
      assert(*ShadowPtr && "Could not find shadow for an argument");
1656
690
      return *ShadowPtr;
1657
690
    }
1658
634
    // For everything else the shadow is zero.
1659
634
    return getCleanShadow(V);
1660
634
  }
1661
1662
  /// Get the shadow for i-th argument of the instruction I.
1663
458
  Value *getShadow(Instruction *I, int i) {
1664
458
    return getShadow(I->getOperand(i));
1665
458
  }
1666
1667
  /// Get the origin for a value.
1668
1.45k
  Value *getOrigin(Value *V) {
1669
1.45k
    if (!MS.TrackOrigins) 
return nullptr332
;
1670
1.12k
    if (!PropagateShadow) 
return getCleanOrigin()232
;
1671
889
    if (isa<Constant>(V)) 
return getCleanOrigin()121
;
1672
768
    assert((isa<Instruction>(V) || isa<Argument>(V)) &&
1673
768
           "Unexpected value type in getOrigin()");
1674
768
    if (Instruction *I = dyn_cast<Instruction>(V)) {
1675
519
      if (I->getMetadata("nosanitize"))
1676
6
        return getCleanOrigin();
1677
762
    }
1678
762
    Value *Origin = OriginMap[V];
1679
762
    assert(Origin && "Missing origin");
1680
762
    return Origin;
1681
762
  }
1682
1683
  /// Get the origin for i-th argument of the instruction I.
1684
270
  Value *getOrigin(Instruction *I, int i) {
1685
270
    return getOrigin(I->getOperand(i));
1686
270
  }
1687
1688
  /// Remember the place where a shadow check should be inserted.
1689
  ///
1690
  /// This location will be later instrumented with a check that will print a
1691
  /// UMR warning in runtime if the shadow value is not 0.
1692
249
  void insertShadowCheck(Value *Shadow, Value *Origin, Instruction *OrigIns) {
1693
249
    assert(Shadow);
1694
249
    if (!InsertChecks) 
return0
;
1695
#ifndef NDEBUG
1696
    Type *ShadowTy = Shadow->getType();
1697
    assert((isa<IntegerType>(ShadowTy) || isa<VectorType>(ShadowTy)) &&
1698
           "Can only insert checks for integer and vector shadow types");
1699
#endif
1700
249
    InstrumentationList.push_back(
1701
249
        ShadowOriginAndInsertPoint(Shadow, Origin, OrigIns));
1702
249
  }
1703
1704
  /// Remember the place where a shadow check should be inserted.
1705
  ///
1706
  /// This location will be later instrumented with a check that will print a
1707
  /// UMR warning in runtime if the value is not fully defined.
1708
869
  void insertShadowCheck(Value *Val, Instruction *OrigIns) {
1709
869
    assert(Val);
1710
869
    Value *Shadow, *Origin;
1711
869
    if (ClCheckConstantShadow) {
1712
2
      Shadow = getShadow(Val);
1713
2
      if (!Shadow) 
return0
;
1714
2
      Origin = getOrigin(Val);
1715
867
    } else {
1716
867
      Shadow = dyn_cast_or_null<Instruction>(getShadow(Val));
1717
867
      if (!Shadow) 
return628
;
1718
239
      Origin = dyn_cast_or_null<Instruction>(getOrigin(Val));
1719
239
    }
1720
869
    insertShadowCheck(Shadow, Origin, OrigIns);
1721
241
  }
1722
1723
48
  AtomicOrdering addReleaseOrdering(AtomicOrdering a) {
1724
48
    switch (a) {
1725
48
      case AtomicOrdering::NotAtomic:
1726
0
        return AtomicOrdering::NotAtomic;
1727
48
      case AtomicOrdering::Unordered:
1728
24
      case AtomicOrdering::Monotonic:
1729
24
      case AtomicOrdering::Release:
1730
24
        return AtomicOrdering::Release;
1731
24
      case AtomicOrdering::Acquire:
1732
0
      case AtomicOrdering::AcquireRelease:
1733
0
        return AtomicOrdering::AcquireRelease;
1734
24
      case AtomicOrdering::SequentiallyConsistent:
1735
24
        return AtomicOrdering::SequentiallyConsistent;
1736
0
    }
1737
0
    llvm_unreachable("Unknown ordering");
1738
0
  }
1739
1740
24
  AtomicOrdering addAcquireOrdering(AtomicOrdering a) {
1741
24
    switch (a) {
1742
24
      case AtomicOrdering::NotAtomic:
1743
0
        return AtomicOrdering::NotAtomic;
1744
24
      case AtomicOrdering::Unordered:
1745
18
      case AtomicOrdering::Monotonic:
1746
18
      case AtomicOrdering::Acquire:
1747
18
        return AtomicOrdering::Acquire;
1748
18
      case AtomicOrdering::Release:
1749
0
      case AtomicOrdering::AcquireRelease:
1750
0
        return AtomicOrdering::AcquireRelease;
1751
6
      case AtomicOrdering::SequentiallyConsistent:
1752
6
        return AtomicOrdering::SequentiallyConsistent;
1753
0
    }
1754
0
    llvm_unreachable("Unknown ordering");
1755
0
  }
1756
1757
  // ------------------- Visitors.
1758
  using InstVisitor<MemorySanitizerVisitor>::visit;
1759
3.39k
  void visit(Instruction &I) {
1760
3.39k
    if (!I.getMetadata("nosanitize"))
1761
3.32k
      InstVisitor<MemorySanitizerVisitor>::visit(I);
1762
3.39k
  }
1763
1764
  /// Instrument LoadInst
1765
  ///
1766
  /// Loads the corresponding shadow and (optionally) origin.
1767
  /// Optionally, checks that the load address is fully defined.
1768
272
  void visitLoadInst(LoadInst &I) {
1769
272
    assert(I.getType()->isSized() && "Load type must have size");
1770
272
    assert(!I.getMetadata("nosanitize"));
1771
272
    IRBuilder<> IRB(I.getNextNode());
1772
272
    Type *ShadowTy = getShadowTy(&I);
1773
272
    Value *Addr = I.getPointerOperand();
1774
272
    Value *ShadowPtr, *OriginPtr;
1775
272
    unsigned Alignment = I.getAlignment();
1776
272
    if (PropagateShadow) {
1777
238
      std::tie(ShadowPtr, OriginPtr) =
1778
238
          getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
1779
238
      setShadow(&I,
1780
238
                IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
1781
238
    } else {
1782
34
      setShadow(&I, getCleanShadow(&I));
1783
34
    }
1784
272
1785
272
    if (ClCheckAccessAddress)
1786
102
      insertShadowCheck(I.getPointerOperand(), &I);
1787
272
1788
272
    if (I.isAtomic())
1789
24
      I.setOrdering(addAcquireOrdering(I.getOrdering()));
1790
272
1791
272
    if (MS.TrackOrigins) {
1792
143
      if (PropagateShadow) {
1793
121
        unsigned OriginAlignment = std::max(kMinOriginAlignment, Alignment);
1794
121
        setOrigin(
1795
121
            &I, IRB.CreateAlignedLoad(MS.OriginTy, OriginPtr, OriginAlignment));
1796
121
      } else {
1797
22
        setOrigin(&I, getCleanOrigin());
1798
22
      }
1799
143
    }
1800
272
  }
1801
1802
  /// Instrument StoreInst
1803
  ///
1804
  /// Stores the corresponding shadow and (optionally) origin.
1805
  /// Optionally, checks that the store address is fully defined.
1806
342
  void visitStoreInst(StoreInst &I) {
1807
342
    StoreList.push_back(&I);
1808
342
    if (ClCheckAccessAddress)
1809
104
      insertShadowCheck(I.getPointerOperand(), &I);
1810
342
  }
1811
1812
24
  void handleCASOrRMW(Instruction &I) {
1813
24
    assert(isa<AtomicRMWInst>(I) || isa<AtomicCmpXchgInst>(I));
1814
24
1815
24
    IRBuilder<> IRB(&I);
1816
24
    Value *Addr = I.getOperand(0);
1817
24
    Value *ShadowPtr = getShadowOriginPtr(Addr, IRB, I.getType(),
1818
24
                                          /*Alignment*/ 1, /*isStore*/ true)
1819
24
                           .first;
1820
24
1821
24
    if (ClCheckAccessAddress)
1822
0
      insertShadowCheck(Addr, &I);
1823
24
1824
24
    // Only test the conditional argument of cmpxchg instruction.
1825
24
    // The other argument can potentially be uninitialized, but we can not
1826
24
    // detect this situation reliably without possible false positives.
1827
24
    if (isa<AtomicCmpXchgInst>(I))
1828
12
      insertShadowCheck(I.getOperand(1), &I);
1829
24
1830
24
    IRB.CreateStore(getCleanShadow(&I), ShadowPtr);
1831
24
1832
24
    setShadow(&I, getCleanShadow(&I));
1833
24
    setOrigin(&I, getCleanOrigin());
1834
24
  }
1835
1836
12
  void visitAtomicRMWInst(AtomicRMWInst &I) {
1837
12
    handleCASOrRMW(I);
1838
12
    I.setOrdering(addReleaseOrdering(I.getOrdering()));
1839
12
  }
1840
1841
12
  void visitAtomicCmpXchgInst(AtomicCmpXchgInst &I) {
1842
12
    handleCASOrRMW(I);
1843
12
    I.setSuccessOrdering(addReleaseOrdering(I.getSuccessOrdering()));
1844
12
  }
1845
1846
  // Vector manipulation.
1847
6
  void visitExtractElementInst(ExtractElementInst &I) {
1848
6
    insertShadowCheck(I.getOperand(1), &I);
1849
6
    IRBuilder<> IRB(&I);
1850
6
    setShadow(&I, IRB.CreateExtractElement(getShadow(&I, 0), I.getOperand(1),
1851
6
              "_msprop"));
1852
6
    setOrigin(&I, getOrigin(&I, 0));
1853
6
  }
1854
1855
22
  void visitInsertElementInst(InsertElementInst &I) {
1856
22
    insertShadowCheck(I.getOperand(2), &I);
1857
22
    IRBuilder<> IRB(&I);
1858
22
    setShadow(&I, IRB.CreateInsertElement(getShadow(&I, 0), getShadow(&I, 1),
1859
22
              I.getOperand(2), "_msprop"));
1860
22
    setOriginForNaryOp(I);
1861
22
  }
1862
1863
6
  void visitShuffleVectorInst(ShuffleVectorInst &I) {
1864
6
    insertShadowCheck(I.getOperand(2), &I);
1865
6
    IRBuilder<> IRB(&I);
1866
6
    setShadow(&I, IRB.CreateShuffleVector(getShadow(&I, 0), getShadow(&I, 1),
1867
6
              I.getOperand(2), "_msprop"));
1868
6
    setOriginForNaryOp(I);
1869
6
  }
1870
1871
  // Casts.
1872
11
  void visitSExtInst(SExtInst &I) {
1873
11
    IRBuilder<> IRB(&I);
1874
11
    setShadow(&I, IRB.CreateSExt(getShadow(&I, 0), I.getType(), "_msprop"));
1875
11
    setOrigin(&I, getOrigin(&I, 0));
1876
11
  }
1877
1878
4
  void visitZExtInst(ZExtInst &I) {
1879
4
    IRBuilder<> IRB(&I);
1880
4
    setShadow(&I, IRB.CreateZExt(getShadow(&I, 0), I.getType(), "_msprop"));
1881
4
    setOrigin(&I, getOrigin(&I, 0));
1882
4
  }
1883
1884
4
  void visitTruncInst(TruncInst &I) {
1885
4
    IRBuilder<> IRB(&I);
1886
4
    setShadow(&I, IRB.CreateTrunc(getShadow(&I, 0), I.getType(), "_msprop"));
1887
4
    setOrigin(&I, getOrigin(&I, 0));
1888
4
  }
1889
1890
224
  void visitBitCastInst(BitCastInst &I) {
1891
224
    // Special case: if this is the bitcast (there is exactly 1 allowed) between
1892
224
    // a musttail call and a ret, don't instrument. New instructions are not
1893
224
    // allowed after a musttail call.
1894
224
    if (auto *CI = dyn_cast<CallInst>(I.getOperand(0)))
1895
7
      if (CI->isMustTailCall())
1896
4
        return;
1897
220
    IRBuilder<> IRB(&I);
1898
220
    setShadow(&I, IRB.CreateBitCast(getShadow(&I, 0), getShadowTy(&I)));
1899
220
    setOrigin(&I, getOrigin(&I, 0));
1900
220
  }
1901
1902
0
  void visitPtrToIntInst(PtrToIntInst &I) {
1903
0
    IRBuilder<> IRB(&I);
1904
0
    setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
1905
0
             "_msprop_ptrtoint"));
1906
0
    setOrigin(&I, getOrigin(&I, 0));
1907
0
  }
1908
1909
9
  void visitIntToPtrInst(IntToPtrInst &I) {
1910
9
    IRBuilder<> IRB(&I);
1911
9
    setShadow(&I, IRB.CreateIntCast(getShadow(&I, 0), getShadowTy(&I), false,
1912
9
             "_msprop_inttoptr"));
1913
9
    setOrigin(&I, getOrigin(&I, 0));
1914
9
  }
1915
1916
0
  void visitFPToSIInst(CastInst& I) { handleShadowOr(I); }
1917
0
  void visitFPToUIInst(CastInst& I) { handleShadowOr(I); }
1918
0
  void visitSIToFPInst(CastInst& I) { handleShadowOr(I); }
1919
0
  void visitUIToFPInst(CastInst& I) { handleShadowOr(I); }
1920
0
  void visitFPExtInst(CastInst& I) { handleShadowOr(I); }
1921
0
  void visitFPTruncInst(CastInst& I) { handleShadowOr(I); }
1922
1923
  /// Propagate shadow for bitwise AND.
1924
  ///
1925
  /// This code is exact, i.e. if, for example, a bit in the left argument
1926
  /// is defined and 0, then neither the value not definedness of the
1927
  /// corresponding bit in B don't affect the resulting shadow.
1928
0
  void visitAnd(BinaryOperator &I) {
1929
0
    IRBuilder<> IRB(&I);
1930
0
    //  "And" of 0 and a poisoned value results in unpoisoned value.
1931
0
    //  1&1 => 1;     0&1 => 0;     p&1 => p;
1932
0
    //  1&0 => 0;     0&0 => 0;     p&0 => 0;
1933
0
    //  1&p => p;     0&p => 0;     p&p => p;
1934
0
    //  S = (S1 & S2) | (V1 & S2) | (S1 & V2)
1935
0
    Value *S1 = getShadow(&I, 0);
1936
0
    Value *S2 = getShadow(&I, 1);
1937
0
    Value *V1 = I.getOperand(0);
1938
0
    Value *V2 = I.getOperand(1);
1939
0
    if (V1->getType() != S1->getType()) {
1940
0
      V1 = IRB.CreateIntCast(V1, S1->getType(), false);
1941
0
      V2 = IRB.CreateIntCast(V2, S2->getType(), false);
1942
0
    }
1943
0
    Value *S1S2 = IRB.CreateAnd(S1, S2);
1944
0
    Value *V1S2 = IRB.CreateAnd(V1, S2);
1945
0
    Value *S1V2 = IRB.CreateAnd(S1, V2);
1946
0
    setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
1947
0
    setOriginForNaryOp(I);
1948
0
  }
1949
1950
0
  void visitOr(BinaryOperator &I) {
1951
0
    IRBuilder<> IRB(&I);
1952
0
    //  "Or" of 1 and a poisoned value results in unpoisoned value.
1953
0
    //  1|1 => 1;     0|1 => 1;     p|1 => 1;
1954
0
    //  1|0 => 1;     0|0 => 0;     p|0 => p;
1955
0
    //  1|p => 1;     0|p => p;     p|p => p;
1956
0
    //  S = (S1 & S2) | (~V1 & S2) | (S1 & ~V2)
1957
0
    Value *S1 = getShadow(&I, 0);
1958
0
    Value *S2 = getShadow(&I, 1);
1959
0
    Value *V1 = IRB.CreateNot(I.getOperand(0));
1960
0
    Value *V2 = IRB.CreateNot(I.getOperand(1));
1961
0
    if (V1->getType() != S1->getType()) {
1962
0
      V1 = IRB.CreateIntCast(V1, S1->getType(), false);
1963
0
      V2 = IRB.CreateIntCast(V2, S2->getType(), false);
1964
0
    }
1965
0
    Value *S1S2 = IRB.CreateAnd(S1, S2);
1966
0
    Value *V1S2 = IRB.CreateAnd(V1, S2);
1967
0
    Value *S1V2 = IRB.CreateAnd(S1, V2);
1968
0
    setShadow(&I, IRB.CreateOr({S1S2, V1S2, S1V2}));
1969
0
    setOriginForNaryOp(I);
1970
0
  }
1971
1972
  /// Default propagation of shadow and/or origin.
1973
  ///
1974
  /// This class implements the general case of shadow propagation, used in all
1975
  /// cases where we don't know and/or don't care about what the operation
1976
  /// actually does. It converts all input shadow values to a common type
1977
  /// (extending or truncating as necessary), and bitwise OR's them.
1978
  ///
1979
  /// This is much cheaper than inserting checks (i.e. requiring inputs to be
1980
  /// fully initialized), and less prone to false positives.
1981
  ///
1982
  /// This class also implements the general case of origin propagation. For a
1983
  /// Nary operation, result origin is set to the origin of an argument that is
1984
  /// not entirely initialized. If there is more than one such arguments, the
1985
  /// rightmost of them is picked. It does not matter which one is picked if all
1986
  /// arguments are initialized.
1987
  template <bool CombineShadow>
1988
  class Combiner {
1989
    Value *Shadow = nullptr;
1990
    Value *Origin = nullptr;
1991
    IRBuilder<> &IRB;
1992
    MemorySanitizerVisitor *MSV;
1993
1994
  public:
1995
    Combiner(MemorySanitizerVisitor *MSV, IRBuilder<> &IRB)
1996
290
        : IRB(IRB), MSV(MSV) {}
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<true>::Combiner((anonymous namespace)::MemorySanitizerVisitor*, llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>&)
Line
Count
Source
1996
169
        : IRB(IRB), MSV(MSV) {}
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<false>::Combiner((anonymous namespace)::MemorySanitizerVisitor*, llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>&)
Line
Count
Source
1996
121
        : IRB(IRB), MSV(MSV) {}
1997
1998
    /// Add a pair of shadow and origin values to the mix.
1999
616
    Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2000
616
      if (CombineShadow) {
2001
436
        assert(OpShadow);
2002
436
        if (!Shadow)
2003
169
          Shadow = OpShadow;
2004
267
        else {
2005
267
          OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2006
267
          Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2007
267
        }
2008
436
      }
2009
616
2010
616
      if (MSV->MS.TrackOrigins) {
2011
342
        assert(OpOrigin);
2012
342
        if (!Origin) {
2013
184
          Origin = OpOrigin;
2014
184
        } else {
2015
158
          Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2016
158
          // No point in adding something that might result in 0 origin value.
2017
158
          if (!ConstOrigin || 
!ConstOrigin->isNullValue()134
) {
2018
24
            Value *FlatShadow = MSV->convertToShadowTyNoVec(OpShadow, IRB);
2019
24
            Value *Cond =
2020
24
                IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow));
2021
24
            Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2022
24
          }
2023
158
        }
2024
342
      }
2025
616
      return *this;
2026
616
    }
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<true>::Add(llvm::Value*, llvm::Value*)
Line
Count
Source
1999
436
    Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2000
436
      if (CombineShadow) {
2001
436
        assert(OpShadow);
2002
436
        if (!Shadow)
2003
169
          Shadow = OpShadow;
2004
267
        else {
2005
267
          OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2006
267
          Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2007
267
        }
2008
436
      }
2009
436
2010
436
      if (MSV->MS.TrackOrigins) {
2011
162
        assert(OpOrigin);
2012
162
        if (!Origin) {
2013
63
          Origin = OpOrigin;
2014
99
        } else {
2015
99
          Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2016
99
          // No point in adding something that might result in 0 origin value.
2017
99
          if (!ConstOrigin || 
!ConstOrigin->isNullValue()95
) {
2018
4
            Value *FlatShadow = MSV->convertToShadowTyNoVec(OpShadow, IRB);
2019
4
            Value *Cond =
2020
4
                IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow));
2021
4
            Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2022
4
          }
2023
99
        }
2024
162
      }
2025
436
      return *this;
2026
436
    }
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<false>::Add(llvm::Value*, llvm::Value*)
Line
Count
Source
1999
180
    Combiner &Add(Value *OpShadow, Value *OpOrigin) {
2000
180
      if (CombineShadow) {
2001
0
        assert(OpShadow);
2002
0
        if (!Shadow)
2003
0
          Shadow = OpShadow;
2004
0
        else {
2005
0
          OpShadow = MSV->CreateShadowCast(IRB, OpShadow, Shadow->getType());
2006
0
          Shadow = IRB.CreateOr(Shadow, OpShadow, "_msprop");
2007
0
        }
2008
0
      }
2009
180
2010
180
      if (MSV->MS.TrackOrigins) {
2011
180
        assert(OpOrigin);
2012
180
        if (!Origin) {
2013
121
          Origin = OpOrigin;
2014
121
        } else {
2015
59
          Constant *ConstOrigin = dyn_cast<Constant>(OpOrigin);
2016
59
          // No point in adding something that might result in 0 origin value.
2017
59
          if (!ConstOrigin || 
!ConstOrigin->isNullValue()39
) {
2018
20
            Value *FlatShadow = MSV->convertToShadowTyNoVec(OpShadow, IRB);
2019
20
            Value *Cond =
2020
20
                IRB.CreateICmpNE(FlatShadow, MSV->getCleanShadow(FlatShadow));
2021
20
            Origin = IRB.CreateSelect(Cond, OpOrigin, Origin);
2022
20
          }
2023
59
        }
2024
180
      }
2025
180
      return *this;
2026
180
    }
2027
2028
    /// Add an application value to the mix.
2029
616
    Combiner &Add(Value *V) {
2030
616
      Value *OpShadow = MSV->getShadow(V);
2031
616
      Value *OpOrigin = MSV->MS.TrackOrigins ? 
MSV->getOrigin(V)342
:
nullptr274
;
2032
616
      return Add(OpShadow, OpOrigin);
2033
616
    }
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<true>::Add(llvm::Value*)
Line
Count
Source
2029
436
    Combiner &Add(Value *V) {
2030
436
      Value *OpShadow = MSV->getShadow(V);
2031
436
      Value *OpOrigin = MSV->MS.TrackOrigins ? 
MSV->getOrigin(V)162
:
nullptr274
;
2032
436
      return Add(OpShadow, OpOrigin);
2033
436
    }
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<false>::Add(llvm::Value*)
Line
Count
Source
2029
180
    Combiner &Add(Value *V) {
2030
180
      Value *OpShadow = MSV->getShadow(V);
2031
180
      Value *OpOrigin = MSV->MS.TrackOrigins ? MSV->getOrigin(V) : 
nullptr0
;
2032
180
      return Add(OpShadow, OpOrigin);
2033
180
    }
2034
2035
    /// Set the current combined values as the given instruction's shadow
2036
    /// and origin.
2037
290
    void Done(Instruction *I) {
2038
290
      if (CombineShadow) {
2039
169
        assert(Shadow);
2040
169
        Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2041
169
        MSV->setShadow(I, Shadow);
2042
169
      }
2043
290
      if (MSV->MS.TrackOrigins) {
2044
184
        assert(Origin);
2045
184
        MSV->setOrigin(I, Origin);
2046
184
      }
2047
290
    }
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<true>::Done(llvm::Instruction*)
Line
Count
Source
2037
169
    void Done(Instruction *I) {
2038
169
      if (CombineShadow) {
2039
169
        assert(Shadow);
2040
169
        Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2041
169
        MSV->setShadow(I, Shadow);
2042
169
      }
2043
169
      if (MSV->MS.TrackOrigins) {
2044
63
        assert(Origin);
2045
63
        MSV->setOrigin(I, Origin);
2046
63
      }
2047
169
    }
MemorySanitizer.cpp:(anonymous namespace)::MemorySanitizerVisitor::Combiner<false>::Done(llvm::Instruction*)
Line
Count
Source
2037
121
    void Done(Instruction *I) {
2038
121
      if (CombineShadow) {
2039
0
        assert(Shadow);
2040
0
        Shadow = MSV->CreateShadowCast(IRB, Shadow, MSV->getShadowTy(I));
2041
0
        MSV->setShadow(I, Shadow);
2042
0
      }
2043
121
      if (MSV->MS.TrackOrigins) {
2044
121
        assert(Origin);
2045
121
        MSV->setOrigin(I, Origin);
2046
121
      }
2047
121
    }
2048
  };
2049
2050
  using ShadowAndOriginCombiner = Combiner<true>;
2051
  using OriginCombiner = Combiner<false>;
2052
2053
  /// Propagate origin for arbitrary operation.
2054
247
  void setOriginForNaryOp(Instruction &I) {
2055
247
    if (!MS.TrackOrigins) 
return126
;
2056
121
    IRBuilder<> IRB(&I);
2057
121
    OriginCombiner OC(this, IRB);
2058
301
    for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); 
++OI180
)
2059
180
      OC.Add(OI->get());
2060
121
    OC.Done(&I);
2061
121
  }
2062
2063
916
  size_t VectorOrPrimitiveTypeSizeInBits(Type *Ty) {
2064
916
    assert(!(Ty->isVectorTy() && Ty->getScalarType()->isPointerTy()) &&
2065
916
           "Vector of pointers is not a valid shadow type");
2066
916
    return Ty->isVectorTy() ?
2067
32
      Ty->getVectorNumElements() * Ty->getScalarSizeInBits() :
2068
916
      
Ty->getPrimitiveSizeInBits()884
;
2069
916
  }
2070
2071
  /// Cast between two shadow types, extending or truncating as
2072
  /// necessary.
2073
  Value *CreateShadowCast(IRBuilder<> &IRB, Value *V, Type *dstTy,
2074
458
                          bool Signed = false) {
2075
458
    Type *srcTy = V->getType();
2076
458
    size_t srcSizeInBits = VectorOrPrimitiveTypeSizeInBits(srcTy);
2077
458
    size_t dstSizeInBits = VectorOrPrimitiveTypeSizeInBits(dstTy);
2078
458
    if (srcSizeInBits > 1 && 
dstSizeInBits == 1440
)
2079
9
      return IRB.CreateICmpNE(V, getCleanShadow(V));
2080
449
2081
449
    if (dstTy->isIntegerTy() && 
srcTy->isIntegerTy()429
)
2082
425
      return IRB.CreateIntCast(V, dstTy, Signed);
2083
24
    if (dstTy->isVectorTy() && 
srcTy->isVectorTy()20
&&
2084
24
        
dstTy->getVectorNumElements() == srcTy->getVectorNumElements()8
)
2085
8
      return IRB.CreateIntCast(V, dstTy, Signed);
2086
16
    Value *V1 = IRB.CreateBitCast(V, Type::getIntNTy(*MS.C, srcSizeInBits));
2087
16
    Value *V2 =
2088
16
      IRB.CreateIntCast(V1, Type::getIntNTy(*MS.C, dstSizeInBits), Signed);
2089
16
    return IRB.CreateBitCast(V2, dstTy);
2090
16
    // TODO: handle struct types.
2091
16
  }
2092
2093
  /// Cast an application value to the type of its own shadow.
2094
72
  Value *CreateAppToShadowCast(IRBuilder<> &IRB, Value *V) {
2095
72
    Type *ShadowTy = getShadowTy(V);
2096
72
    if (V->getType() == ShadowTy)
2097
24
      return V;
2098
48
    if (V->getType()->isPtrOrPtrVectorTy())
2099
44
      return IRB.CreatePtrToInt(V, ShadowTy);
2100
4
    else
2101
4
      return IRB.CreateBitCast(V, ShadowTy);
2102
48
  }
2103
2104
  /// Propagate shadow for arbitrary operation.
2105
165
  void handleShadowOr(Instruction &I) {
2106
165
    IRBuilder<> IRB(&I);
2107
165
    ShadowAndOriginCombiner SC(this, IRB);
2108
593
    for (Instruction::op_iterator OI = I.op_begin(); OI != I.op_end(); 
++OI428
)
2109
428
      SC.Add(OI->get());
2110
165
    SC.Done(&I);
2111
165
  }
2112
2113
4
  void visitFNeg(UnaryOperator &I) { handleShadowOr(I); }
2114
2115
  // Handle multiplication by constant.
2116
  //
2117
  // Handle a special case of multiplication by constant that may have one or
2118
  // more zeros in the lower bits. This makes corresponding number of lower bits
2119
  // of the result zero as well. We model it by shifting the other operand
2120
  // shadow left by the required number of bits. Effectively, we transform
2121
  // (X * (A * 2**B)) to ((X << B) * A) and instrument (X << B) as (Sx << B).
2122
  // We use multiplication by 2**N instead of shift to cover the case of
2123
  // multiplication by 0, which may occur in some elements of a vector operand.
2124
  void handleMulByConstant(BinaryOperator &I, Constant *ConstArg,
2125
18
                           Value *OtherArg) {
2126
18
    Constant *ShadowMul;
2127
18
    Type *Ty = ConstArg->getType();
2128
18
    if (Ty->isVectorTy()) {
2129
4
      unsigned NumElements = Ty->getVectorNumElements();
2130
4
      Type *EltTy = Ty->getSequentialElementType();
2131
4
      SmallVector<Constant *, 16> Elements;
2132
16
      for (unsigned Idx = 0; Idx < NumElements; 
++Idx12
) {
2133
12
        if (ConstantInt *Elt =
2134
10
                dyn_cast<ConstantInt>(ConstArg->getAggregateElement(Idx))) {
2135
10
          const APInt &V = Elt->getValue();
2136
10
          APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
2137
10
          Elements.push_back(ConstantInt::get(EltTy, V2));
2138
10
        } else {
2139
2
          Elements.push_back(ConstantInt::get(EltTy, 1));
2140
2
        }
2141
12
      }
2142
4
      ShadowMul = ConstantVector::get(Elements);
2143
14
    } else {
2144
14
      if (ConstantInt *Elt = dyn_cast<ConstantInt>(ConstArg)) {
2145
12
        const APInt &V = Elt->getValue();
2146
12
        APInt V2 = APInt(V.getBitWidth(), 1) << V.countTrailingZeros();
2147
12
        ShadowMul = ConstantInt::get(Ty, V2);
2148
12
      } else {
2149
2
        ShadowMul = ConstantInt::get(Ty, 1);
2150
2
      }
2151
14
    }
2152
18
2153
18
    IRBuilder<> IRB(&I);
2154
18
    setShadow(&I,
2155
18
              IRB.CreateMul(getShadow(OtherArg), ShadowMul, "msprop_mul_cst"));
2156
18
    setOrigin(&I, getOrigin(OtherArg));
2157
18
  }
2158
2159
18
  void visitMul(BinaryOperator &I) {
2160
18
    Constant *constOp0 = dyn_cast<Constant>(I.getOperand(0));
2161
18
    Constant *constOp1 = dyn_cast<Constant>(I.getOperand(1));
2162
18
    if (constOp0 && 
!constOp10
)
2163
0
      handleMulByConstant(I, constOp0, I.getOperand(1));
2164
18
    else if (constOp1 && !constOp0)
2165
18
      handleMulByConstant(I, constOp1, I.getOperand(0));
2166
0
    else
2167
0
      handleShadowOr(I);
2168
18
  }
2169
2170
0
  void visitFAdd(BinaryOperator &I) { handleShadowOr(I); }
2171
0
  void visitFSub(BinaryOperator &I) { handleShadowOr(I); }
2172
0
  void visitFMul(BinaryOperator &I) { handleShadowOr(I); }
2173
31
  void visitAdd(BinaryOperator &I) { handleShadowOr(I); }
2174
0
  void visitSub(BinaryOperator &I) { handleShadowOr(I); }
2175
0
  void visitXor(BinaryOperator &I) { handleShadowOr(I); }
2176
2177
4
  void handleIntegerDiv(Instruction &I) {
2178
4
    IRBuilder<> IRB(&I);
2179
4
    // Strict on the second argument.
2180
4
    insertShadowCheck(I.getOperand(1), &I);
2181
4
    setShadow(&I, getShadow(&I, 0));
2182
4
    setOrigin(&I, getOrigin(&I, 0));
2183
4
  }
2184
2185
4
  void visitUDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2186
0
  void visitSDiv(BinaryOperator &I) { handleIntegerDiv(I); }
2187
0
  void visitURem(BinaryOperator &I) { handleIntegerDiv(I); }
2188
0
  void visitSRem(BinaryOperator &I) { handleIntegerDiv(I); }
2189
2190
  // Floating point division is side-effect free. We can not require that the
2191
  // divisor is fully initialized and must propagate shadow. See PR37523.
2192
4
  void visitFDiv(BinaryOperator &I) { handleShadowOr(I); }
2193
0
  void visitFRem(BinaryOperator &I) { handleShadowOr(I); }
2194
2195
  /// Instrument == and != comparisons.
2196
  ///
2197
  /// Sometimes the comparison result is known even if some of the bits of the
2198
  /// arguments are not.
2199
40
  void handleEqualityComparison(ICmpInst &I) {
2200
40
    IRBuilder<> IRB(&I);
2201
40
    Value *A = I.getOperand(0);
2202
40
    Value *B = I.getOperand(1);
2203
40
    Value *Sa = getShadow(A);
2204
40
    Value *Sb = getShadow(B);
2205
40
2206
40
    // Get rid of pointers and vectors of pointers.
2207
40
    // For ints (and vectors of ints), types of A and Sa match,
2208
40
    // and this is a no-op.
2209
40
    A = IRB.CreatePointerCast(A, Sa->getType());
2210
40
    B = IRB.CreatePointerCast(B, Sb->getType());
2211
40
2212
40
    // A == B  <==>  (C = A^B) == 0
2213
40
    // A != B  <==>  (C = A^B) != 0
2214
40
    // Sc = Sa | Sb
2215
40
    Value *C = IRB.CreateXor(A, B);
2216
40
    Value *Sc = IRB.CreateOr(Sa, Sb);
2217
40
    // Now dealing with i = (C == 0) comparison (or C != 0, does not matter now)
2218
40
    // Result is defined if one of the following is true
2219
40
    // * there is a defined 1 bit in C
2220
40
    // * C is fully defined
2221
40
    // Si = !(C & ~Sc) && Sc
2222
40
    Value *Zero = Constant::getNullValue(Sc->getType());
2223
40
    Value *MinusOne = Constant::getAllOnesValue(Sc->getType());
2224
40
    Value *Si =
2225
40
      IRB.CreateAnd(IRB.CreateICmpNE(Sc, Zero),
2226
40
                    IRB.CreateICmpEQ(
2227
40
                      IRB.CreateAnd(IRB.CreateXor(Sc, MinusOne), C), Zero));
2228
40
    Si->setName("_msprop_icmp");
2229
40
    setShadow(&I, Si);
2230
40
    setOriginForNaryOp(I);
2231
40
  }
2232
2233
  /// Build the lowest possible value of V, taking into account V's
2234
  ///        uninitialized bits.
2235
  Value *getLowestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2236
22
                                bool isSigned) {
2237
22
    if (isSigned) {
2238
0
      // Split shadow into sign bit and other bits.
2239
0
      Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2240
0
      Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2241
0
      // Maximise the undefined shadow bit, minimize other undefined bits.
2242
0
      return
2243
0
        IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaOtherBits)), SaSignBit);
2244
22
    } else {
2245
22
      // Minimize undefined bits.
2246
22
      return IRB.CreateAnd(A, IRB.CreateNot(Sa));
2247
22
    }
2248
22
  }
2249
2250
  /// Build the highest possible value of V, taking into account V's
2251
  ///        uninitialized bits.
2252
  Value *getHighestPossibleValue(IRBuilder<> &IRB, Value *A, Value *Sa,
2253
22
                                bool isSigned) {
2254
22
    if (isSigned) {
2255
0
      // Split shadow into sign bit and other bits.
2256
0
      Value *SaOtherBits = IRB.CreateLShr(IRB.CreateShl(Sa, 1), 1);
2257
0
      Value *SaSignBit = IRB.CreateXor(Sa, SaOtherBits);
2258
0
      // Minimise the undefined shadow bit, maximise other undefined bits.
2259
0
      return
2260
0
        IRB.CreateOr(IRB.CreateAnd(A, IRB.CreateNot(SaSignBit)), SaOtherBits);
2261
22
    } else {
2262
22
      // Maximize undefined bits.
2263
22
      return IRB.CreateOr(A, Sa);
2264
22
    }
2265
22
  }
2266
2267
  /// Instrument relational comparisons.
2268
  ///
2269
  /// This function does exact shadow propagation for all relational
2270
  /// comparisons of integers, pointers and vectors of those.
2271
  /// FIXME: output seems suboptimal when one of the operands is a constant
2272
11
  void handleRelationalComparisonExact(ICmpInst &I) {
2273
11
    IRBuilder<> IRB(&I);
2274
11
    Value *A = I.getOperand(0);
2275
11
    Value *B = I.getOperand(1);
2276
11
    Value *Sa = getShadow(A);
2277
11
    Value *Sb = getShadow(B);
2278
11
2279
11
    // Get rid of pointers and vectors of pointers.
2280
11
    // For ints (and vectors of ints), types of A and Sa match,
2281
11
    // and this is a no-op.
2282
11
    A = IRB.CreatePointerCast(A, Sa->getType());
2283
11
    B = IRB.CreatePointerCast(B, Sb->getType());
2284
11
2285
11
    // Let [a0, a1] be the interval of possible values of A, taking into account
2286
11
    // its undefined bits. Let [b0, b1] be the interval of possible values of B.
2287
11
    // Then (A cmp B) is defined iff (a0 cmp b1) == (a1 cmp b0).
2288
11
    bool IsSigned = I.isSigned();
2289
11
    Value *S1 = IRB.CreateICmp(I.getPredicate(),
2290
11
                               getLowestPossibleValue(IRB, A, Sa, IsSigned),
2291
11
                               getHighestPossibleValue(IRB, B, Sb, IsSigned));
2292
11
    Value *S2 = IRB.CreateICmp(I.getPredicate(),
2293
11
                               getHighestPossibleValue(IRB, A, Sa, IsSigned),
2294
11
                               getLowestPossibleValue(IRB, B, Sb, IsSigned));
2295
11
    Value *Si = IRB.CreateXor(S1, S2);
2296
11
    setShadow(&I, Si);
2297
11
    setOriginForNaryOp(I);
2298
11
  }
2299
2300
  /// Instrument signed relational comparisons.
2301
  ///
2302
  /// Handle sign bit tests: x<0, x>=0, x<=-1, x>-1 by propagating the highest
2303
  /// bit of the shadow. Everything else is delegated to handleShadowOr().
2304
49
  void handleSignedRelationalComparison(ICmpInst &I) {
2305
49
    Constant *constOp;
2306
49
    Value *op = nullptr;
2307
49
    CmpInst::Predicate pre;
2308
49
    if ((constOp = dyn_cast<Constant>(I.getOperand(1)))) {
2309
27
      op = I.getOperand(0);
2310
27
      pre = I.getPredicate();
2311
27
    } else 
if (22
(constOp = dyn_cast<Constant>(I.getOperand(0)))22
) {
2312
20
      op = I.getOperand(1);
2313
20
      pre = I.getSwappedPredicate();
2314
20
    } else {
2315
2
      handleShadowOr(I);
2316
2
      return;
2317
2
    }
2318
47
2319
47
    if ((constOp->isNullValue() &&
2320
47
         
(27
pre == CmpInst::ICMP_SLT27
||
pre == CmpInst::ICMP_SGE15
)) ||
2321
47
        
(27
constOp->isAllOnesValue()27
&&
2322
40
         
(20
pre == CmpInst::ICMP_SGT20
||
pre == CmpInst::ICMP_SLE8
))) {
2323
40
      IRBuilder<> IRB(&I);
2324
40
      Value *Shadow = IRB.CreateICmpSLT(getShadow(op), getCleanShadow(op),
2325
40
                                        "_msprop_icmp_s");
2326
40
      setShadow(&I, Shadow);
2327
40
      setOrigin(&I, getOrigin(op));
2328
40
    } else {
2329
7
      handleShadowOr(I);
2330
7
    }
2331
47
  }
2332
2333
100
  void visitICmpInst(ICmpInst &I) {
2334
100
    if (!ClHandleICmp) {
2335
0
      handleShadowOr(I);
2336
0
      return;
2337
0
    }
2338
100
    if (I.isEquality()) {
2339
40
      handleEqualityComparison(I);
2340
40
      return;
2341
40
    }
2342
60
2343
60
    assert(I.isRelational());
2344
60
    if (ClHandleICmpExact) {
2345
0
      handleRelationalComparisonExact(I);
2346
0
      return;
2347
0
    }
2348
60
    if (I.isSigned()) {
2349
49
      handleSignedRelationalComparison(I);
2350
49
      return;
2351
49
    }
2352
11
2353
11
    assert(I.isUnsigned());
2354
11
    if ((isa<Constant>(I.getOperand(0)) || isa<Constant>(I.getOperand(1)))) {
2355
11
      handleRelationalComparisonExact(I);
2356
11
      return;
2357
11
    }
2358
0
2359
0
    handleShadowOr(I);
2360
0
  }
2361
2362
0
  void visitFCmpInst(FCmpInst &I) {
2363
0
    handleShadowOr(I);
2364
0
  }
2365
2366
8
  void handleShift(BinaryOperator &I) {
2367
8
    IRBuilder<> IRB(&I);
2368
8
    // If any of the S2 bits are poisoned, the whole thing is poisoned.
2369
8
    // Otherwise perform the same shift on S1.
2370
8
    Value *S1 = getShadow(&I, 0);
2371
8
    Value *S2 = getShadow(&I, 1);
2372
8
    Value *S2Conv = IRB.CreateSExt(IRB.CreateICmpNE(S2, getCleanShadow(S2)),
2373
8
                                   S2->getType());
2374
8
    Value *V2 = I.getOperand(1);
2375
8
    Value *Shift = IRB.CreateBinOp(I.getOpcode(), S1, V2);
2376
8
    setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2377
8
    setOriginForNaryOp(I);
2378
8
  }
2379
2380
8
  void visitShl(BinaryOperator &I) { handleShift(I); }
2381
0
  void visitAShr(BinaryOperator &I) { handleShift(I); }
2382
0
  void visitLShr(BinaryOperator &I) { handleShift(I); }
2383
2384
  /// Instrument llvm.memmove
2385
  ///
2386
  /// At this point we don't know if llvm.memmove will be inlined or not.
2387
  /// If we don't instrument it and it gets inlined,
2388
  /// our interceptor will not kick in and we will lose the memmove.
2389
  /// If we instrument the call here, but it does not get inlined,
2390
  /// we will memove the shadow twice: which is bad in case
2391
  /// of overlapping regions. So, we simply lower the intrinsic to a call.
2392
  ///
2393
  /// Similar situation exists for memcpy and memset.
2394
4
  void visitMemMoveInst(MemMoveInst &I) {
2395
4
    IRBuilder<> IRB(&I);
2396
4
    IRB.CreateCall(
2397
4
        MS.MemmoveFn,
2398
4
        {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2399
4
         IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2400
4
         IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2401
4
    I.eraseFromParent();
2402
4
  }
2403
2404
  // Similar to memmove: avoid copying shadow twice.
2405
  // This is somewhat unfortunate as it may slowdown small constant memcpys.
2406
  // FIXME: consider doing manual inline for small constant sizes and proper
2407
  // alignment.
2408
12
  void visitMemCpyInst(MemCpyInst &I) {
2409
12
    IRBuilder<> IRB(&I);
2410
12
    IRB.CreateCall(
2411
12
        MS.MemcpyFn,
2412
12
        {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2413
12
         IRB.CreatePointerCast(I.getArgOperand(1), IRB.getInt8PtrTy()),
2414
12
         IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2415
12
    I.eraseFromParent();
2416
12
  }
2417
2418
  // Same as memcpy.
2419
4
  void visitMemSetInst(MemSetInst &I) {
2420
4
    IRBuilder<> IRB(&I);
2421
4
    IRB.CreateCall(
2422
4
        MS.MemsetFn,
2423
4
        {IRB.CreatePointerCast(I.getArgOperand(0), IRB.getInt8PtrTy()),
2424
4
         IRB.CreateIntCast(I.getArgOperand(1), IRB.getInt32Ty(), false),
2425
4
         IRB.CreateIntCast(I.getArgOperand(2), MS.IntptrTy, false)});
2426
4
    I.eraseFromParent();
2427
4
  }
2428
2429
25
  void visitVAStartInst(VAStartInst &I) {
2430
25
    VAHelper->visitVAStartInst(I);
2431
25
  }
2432
2433
4
  void visitVACopyInst(VACopyInst &I) {
2434
4
    VAHelper->visitVACopyInst(I);
2435
4
  }
2436
2437
  /// Handle vector store-like intrinsics.
2438
  ///
2439
  /// Instrument intrinsics that look like a simple SIMD store: writes memory,
2440
  /// has 1 pointer argument and 1 vector argument, returns void.
2441
0
  bool handleVectorStoreIntrinsic(IntrinsicInst &I) {
2442
0
    IRBuilder<> IRB(&I);
2443
0
    Value* Addr = I.getArgOperand(0);
2444
0
    Value *Shadow = getShadow(&I, 1);
2445
0
    Value *ShadowPtr, *OriginPtr;
2446
0
2447
0
    // We don't know the pointer alignment (could be unaligned SSE store!).
2448
0
    // Have to assume to worst case.
2449
0
    std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2450
0
        Addr, IRB, Shadow->getType(), /*Alignment*/ 1, /*isStore*/ true);
2451
0
    IRB.CreateAlignedStore(Shadow, ShadowPtr, 1);
2452
0
2453
0
    if (ClCheckAccessAddress)
2454
0
      insertShadowCheck(Addr, &I);
2455
0
2456
0
    // FIXME: factor out common code from materializeStores
2457
0
    if (MS.TrackOrigins) IRB.CreateStore(getOrigin(&I, 1), OriginPtr);
2458
0
    return true;
2459
0
  }
2460
2461
  /// Handle vector load-like intrinsics.
2462
  ///
2463
  /// Instrument intrinsics that look like a simple SIMD load: reads memory,
2464
  /// has 1 pointer argument, returns a vector.
2465
4
  bool handleVectorLoadIntrinsic(IntrinsicInst &I) {
2466
4
    IRBuilder<> IRB(&I);
2467
4
    Value *Addr = I.getArgOperand(0);
2468
4
2469
4
    Type *ShadowTy = getShadowTy(&I);
2470
4
    Value *ShadowPtr, *OriginPtr;
2471
4
    if (PropagateShadow) {
2472
4
      // We don't know the pointer alignment (could be unaligned SSE load!).
2473
4
      // Have to assume to worst case.
2474
4
      unsigned Alignment = 1;
2475
4
      std::tie(ShadowPtr, OriginPtr) =
2476
4
          getShadowOriginPtr(Addr, IRB, ShadowTy, Alignment, /*isStore*/ false);
2477
4
      setShadow(&I,
2478
4
                IRB.CreateAlignedLoad(ShadowTy, ShadowPtr, Alignment, "_msld"));
2479
4
    } else {
2480
0
      setShadow(&I, getCleanShadow(&I));
2481
0
    }
2482
4
2483
4
    if (ClCheckAccessAddress)
2484
0
      insertShadowCheck(Addr, &I);
2485
4
2486
4
    if (MS.TrackOrigins) {
2487
2
      if (PropagateShadow)
2488
2
        setOrigin(&I, IRB.CreateLoad(MS.OriginTy, OriginPtr));
2489
0
      else
2490
0
        setOrigin(&I, getCleanOrigin());
2491
2
    }
2492
4
    return true;
2493
4
  }
2494
2495
  /// Handle (SIMD arithmetic)-like intrinsics.
2496
  ///
2497
  /// Instrument intrinsics with any number of arguments of the same type,
2498
  /// equal to the return type. The type should be simple (no aggregates or
2499
  /// pointers; vectors are fine).
2500
  /// Caller guarantees that this intrinsic does not access memory.
2501
16
  bool maybeHandleSimpleNomemIntrinsic(IntrinsicInst &I) {
2502
16
    Type *RetTy = I.getType();
2503
16
    if (!(RetTy->isIntOrIntVectorTy() ||
2504
16
          
RetTy->isFPOrFPVectorTy()12
||
2505
16
          
RetTy->isX86_MMXTy()12
))
2506
12
      return false;
2507
4
2508
4
    unsigned NumArgOperands = I.getNumArgOperands();
2509
4
2510
12
    for (unsigned i = 0; i < NumArgOperands; 
++i8
) {
2511
8
      Type *Ty = I.getArgOperand(i)->getType();
2512
8
      if (Ty != RetTy)
2513
0
        return false;
2514
8
    }
2515
4
2516
4
    IRBuilder<> IRB(&I);
2517
4
    ShadowAndOriginCombiner SC(this, IRB);
2518
12
    for (unsigned i = 0; i < NumArgOperands; 
++i8
)
2519
8
      SC.Add(I.getArgOperand(i));
2520
4
    SC.Done(&I);
2521
4
2522
4
    return true;
2523
4
  }
2524
2525
  /// Heuristically instrument unknown intrinsics.
2526
  ///
2527
  /// The main purpose of this code is to do something reasonable with all
2528
  /// random intrinsics we might encounter, most importantly - SIMD intrinsics.
2529
  /// We recognize several classes of intrinsics by their argument types and
2530
  /// ModRefBehaviour and apply special intrumentation when we are reasonably
2531
  /// sure that we know what the intrinsic does.
2532
  ///
2533
  /// We special-case intrinsics where this approach fails. See llvm.bswap
2534
  /// handling as an example of that.
2535
140
  bool handleUnknownIntrinsic(IntrinsicInst &I) {
2536
140
    unsigned NumArgOperands = I.getNumArgOperands();
2537
140
    if (NumArgOperands == 0)
2538
0
      return false;
2539
140
2540
140
    if (NumArgOperands == 2 &&
2541
140
        
I.getArgOperand(0)->getType()->isPointerTy()89
&&
2542
140
        
I.getArgOperand(1)->getType()->isVectorTy()0
&&
2543
140
        
I.getType()->isVoidTy()0
&&
2544
140
        
!I.onlyReadsMemory()0
) {
2545
0
      // This looks like a vector store.
2546
0
      return handleVectorStoreIntrinsic(I);
2547
0
    }
2548
140
2549
140
    if (NumArgOperands == 1 &&
2550
140
        
I.getArgOperand(0)->getType()->isPointerTy()25
&&
2551
140
        
I.getType()->isVectorTy()23
&&
2552
140
        
I.onlyReadsMemory()4
) {
2553
4
      // This looks like a vector load.
2554
4
      return handleVectorLoadIntrinsic(I);
2555
4
    }
2556
136
2557
136
    if (I.doesNotAccessMemory())
2558
16
      if (maybeHandleSimpleNomemIntrinsic(I))
2559
4
        return true;
2560
132
2561
132
    // FIXME: detect and handle SSE maskstore/maskload
2562
132
    return false;
2563
132
  }
2564
2565
85
  void handleLifetimeStart(IntrinsicInst &I) {
2566
85
    if (!PoisonStack)
2567
17
      return;
2568
68
    DenseMap<Value *, AllocaInst *> AllocaForValue;
2569
68
    AllocaInst *AI =
2570
68
        llvm::findAllocaForValue(I.getArgOperand(1), AllocaForValue);
2571
68
    if (!AI)
2572
11
      InstrumentLifetimeStart = false;
2573
68
    LifetimeStartList.push_back(std::make_pair(&I, AI));
2574
68
  }
2575
2576
4
  void handleBswap(IntrinsicInst &I) {
2577
4
    IRBuilder<> IRB(&I);
2578
4
    Value *Op = I.getArgOperand(0);
2579
4
    Type *OpType = Op->getType();
2580
4
    Function *BswapFunc = Intrinsic::getDeclaration(
2581
4
      F.getParent(), Intrinsic::bswap, makeArrayRef(&OpType, 1));
2582
4
    setShadow(&I, IRB.CreateCall(BswapFunc, getShadow(Op)));
2583
4
    setOrigin(&I, getOrigin(Op));
2584
4
  }
2585
2586
  // Instrument vector convert instrinsic.
2587
  //
2588
  // This function instruments intrinsics like cvtsi2ss:
2589
  // %Out = int_xxx_cvtyyy(%ConvertOp)
2590
  // or
2591
  // %Out = int_xxx_cvtyyy(%CopyOp, %ConvertOp)
2592
  // Intrinsic converts \p NumUsedElements elements of \p ConvertOp to the same
2593
  // number \p Out elements, and (if has 2 arguments) copies the rest of the
2594
  // elements from \p CopyOp.
2595
  // In most cases conversion involves floating-point value which may trigger a
2596
  // hardware exception when not fully initialized. For this reason we require
2597
  // \p ConvertOp[0:NumUsedElements] to be fully initialized and trap otherwise.
2598
  // We copy the shadow of \p CopyOp[NumUsedElements:] to \p
2599
  // Out[NumUsedElements:]. This means that intrinsics without \p CopyOp always
2600
  // return a fully initialized value.
2601
4
  void handleVectorConvertIntrinsic(IntrinsicInst &I, int NumUsedElements) {
2602
4
    IRBuilder<> IRB(&I);
2603
4
    Value *CopyOp, *ConvertOp;
2604
4
2605
4
    switch (I.getNumArgOperands()) {
2606
4
    case 3:
2607
0
      assert(isa<ConstantInt>(I.getArgOperand(2)) && "Invalid rounding mode");
2608
0
      LLVM_FALLTHROUGH;
2609
0
    case 2:
2610
0
      CopyOp = I.getArgOperand(0);
2611
0
      ConvertOp = I.getArgOperand(1);
2612
0
      break;
2613
4
    case 1:
2614
4
      ConvertOp = I.getArgOperand(0);
2615
4
      CopyOp = nullptr;
2616
4
      break;
2617
0
    default:
2618
0
      llvm_unreachable("Cvt intrinsic with unsupported number of arguments.");
2619
4
    }
2620
4
2621
4
    // The first *NumUsedElements* elements of ConvertOp are converted to the
2622
4
    // same number of output elements. The rest of the output is copied from
2623
4
    // CopyOp, or (if not available) filled with zeroes.
2624
4
    // Combine shadow for elements of ConvertOp that are used in this operation,
2625
4
    // and insert a check.
2626
4
    // FIXME: consider propagating shadow of ConvertOp, at least in the case of
2627
4
    // int->any conversion.
2628
4
    Value *ConvertShadow = getShadow(ConvertOp);
2629
4
    Value *AggShadow = nullptr;
2630
4
    if (ConvertOp->getType()->isVectorTy()) {
2631
4
      AggShadow = IRB.CreateExtractElement(
2632
4
          ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
2633
6
      for (int i = 1; i < NumUsedElements; 
++i2
) {
2634
2
        Value *MoreShadow = IRB.CreateExtractElement(
2635
2
            ConvertShadow, ConstantInt::get(IRB.getInt32Ty(), i));
2636
2
        AggShadow = IRB.CreateOr(AggShadow, MoreShadow);
2637
2
      }
2638
4
    } else {
2639
0
      AggShadow = ConvertShadow;
2640
0
    }
2641
4
    assert(AggShadow->getType()->isIntegerTy());
2642
4
    insertShadowCheck(AggShadow, getOrigin(ConvertOp), &I);
2643
4
2644
4
    // Build result shadow by zero-filling parts of CopyOp shadow that come from
2645
4
    // ConvertOp.
2646
4
    if (CopyOp) {
2647
0
      assert(CopyOp->getType() == I.getType());
2648
0
      assert(CopyOp->getType()->isVectorTy());
2649
0
      Value *ResultShadow = getShadow(CopyOp);
2650
0
      Type *EltTy = ResultShadow->getType()->getVectorElementType();
2651
0
      for (int i = 0; i < NumUsedElements; ++i) {
2652
0
        ResultShadow = IRB.CreateInsertElement(
2653
0
            ResultShadow, ConstantInt::getNullValue(EltTy),
2654
0
            ConstantInt::get(IRB.getInt32Ty(), i));
2655
0
      }
2656
0
      setShadow(&I, ResultShadow);
2657
0
      setOrigin(&I, getOrigin(CopyOp));
2658
4
    } else {
2659
4
      setShadow(&I, getCleanShadow(&I));
2660
4
      setOrigin(&I, getCleanOrigin());
2661
4
    }
2662
4
  }
2663
2664
  // Given a scalar or vector, extract lower 64 bits (or less), and return all
2665
  // zeroes if it is zero, and all ones otherwise.
2666
10
  Value *Lower64ShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2667
10
    if (S->getType()->isVectorTy())
2668
4
      S = CreateShadowCast(IRB, S, IRB.getInt64Ty(), /* Signed */ true);
2669
10
    assert(S->getType()->getPrimitiveSizeInBits() <= 64);
2670
10
    Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2671
10
    return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2672
10
  }
2673
2674
  // Given a vector, extract its first element, and return all
2675
  // zeroes if it is zero, and all ones otherwise.
2676
8
  Value *LowerElementShadowExtend(IRBuilder<> &IRB, Value *S, Type *T) {
2677
8
    Value *S1 = IRB.CreateExtractElement(S, (uint64_t)0);
2678
8
    Value *S2 = IRB.CreateICmpNE(S1, getCleanShadow(S1));
2679
8
    return CreateShadowCast(IRB, S2, T, /* Signed */ true);
2680
8
  }
2681
2682
6
  Value *VariableShadowExtend(IRBuilder<> &IRB, Value *S) {
2683
6
    Type *T = S->getType();
2684
6
    assert(T->isVectorTy());
2685
6
    Value *S2 = IRB.CreateICmpNE(S, getCleanShadow(S));
2686
6
    return IRB.CreateSExt(S2, T);
2687
6
  }
2688
2689
  // Instrument vector shift instrinsic.
2690
  //
2691
  // This function instruments intrinsics like int_x86_avx2_psll_w.
2692
  // Intrinsic shifts %In by %ShiftSize bits.
2693
  // %ShiftSize may be a vector. In that case the lower 64 bits determine shift
2694
  // size, and the rest is ignored. Behavior is defined even if shift size is
2695
  // greater than register (or field) width.
2696
16
  void handleVectorShiftIntrinsic(IntrinsicInst &I, bool Variable) {
2697
16
    assert(I.getNumArgOperands() == 2);
2698
16
    IRBuilder<> IRB(&I);
2699
16
    // If any of the S2 bits are poisoned, the whole thing is poisoned.
2700
16
    // Otherwise perform the same shift on S1.
2701
16
    Value *S1 = getShadow(&I, 0);
2702
16
    Value *S2 = getShadow(&I, 1);
2703
16
    Value *S2Conv = Variable ? 
VariableShadowExtend(IRB, S2)6
2704
16
                             : 
Lower64ShadowExtend(IRB, S2, getShadowTy(&I))10
;
2705
16
    Value *V1 = I.getOperand(0);
2706
16
    Value *V2 = I.getOperand(1);
2707
16
    Value *Shift = IRB.CreateCall(I.getFunctionType(), I.getCalledValue(),
2708
16
                                  {IRB.CreateBitCast(S1, V1->getType()), V2});
2709
16
    Shift = IRB.CreateBitCast(Shift, getShadowTy(&I));
2710
16
    setShadow(&I, IRB.CreateOr(Shift, S2Conv));
2711
16
    setOriginForNaryOp(I);
2712
16
  }
2713
2714
  // Get an X86_MMX-sized vector type.
2715
4
  Type *getMMXVectorTy(unsigned EltSizeInBits) {
2716
4
    const unsigned X86_MMXSizeInBits = 64;
2717
4
    assert(EltSizeInBits != 0 && (X86_MMXSizeInBits % EltSizeInBits) == 0 &&
2718
4
           "Illegal MMX vector element size");
2719
4
    return VectorType::get(IntegerType::get(*MS.C, EltSizeInBits),
2720
4
                           X86_MMXSizeInBits / EltSizeInBits);
2721
4
  }
2722
2723
  // Returns a signed counterpart for an (un)signed-saturate-and-pack
2724
  // intrinsic.
2725
6
  Intrinsic::ID getSignedPackIntrinsic(Intrinsic::ID id) {
2726
6
    switch (id) {
2727
6
      case Intrinsic::x86_sse2_packsswb_128:
2728
0
      case Intrinsic::x86_sse2_packuswb_128:
2729
0
        return Intrinsic::x86_sse2_packsswb_128;
2730
0
2731
2
      case Intrinsic::x86_sse2_packssdw_128:
2732
2
      case Intrinsic::x86_sse41_packusdw:
2733
2
        return Intrinsic::x86_sse2_packssdw_128;
2734
2
2735
2
      case Intrinsic::x86_avx2_packsswb:
2736
2
      case Intrinsic::x86_avx2_packuswb:
2737
2
        return Intrinsic::x86_avx2_packsswb;
2738
2
2739
2
      case Intrinsic::x86_avx2_packssdw:
2740
0
      case Intrinsic::x86_avx2_packusdw:
2741
0
        return Intrinsic::x86_avx2_packssdw;
2742
0
2743
2
      case Intrinsic::x86_mmx_packsswb:
2744
2
      case Intrinsic::x86_mmx_packuswb:
2745
2
        return Intrinsic::x86_mmx_packsswb;
2746
2
2747
2
      case Intrinsic::x86_mmx_packssdw:
2748
0
        return Intrinsic::x86_mmx_packssdw;
2749
2
      default:
2750
0
        llvm_unreachable("unexpected intrinsic id");
2751
6
    }
2752
6
  }
2753
2754
  // Instrument vector pack instrinsic.
2755
  //
2756
  // This function instruments intrinsics like x86_mmx_packsswb, that
2757
  // packs elements of 2 input vectors into half as many bits with saturation.
2758
  // Shadow is propagated with the signed variant of the same intrinsic applied
2759
  // to sext(Sa != zeroinitializer), sext(Sb != zeroinitializer).
2760
  // EltSizeInBits is used only for x86mmx arguments.
2761
6
  void handleVectorPackIntrinsic(IntrinsicInst &I, unsigned EltSizeInBits = 0) {
2762
6
    assert(I.getNumArgOperands() == 2);
2763
6
    bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2764
6
    IRBuilder<> IRB(&I);
2765
6
    Value *S1 = getShadow(&I, 0);
2766
6
    Value *S2 = getShadow(&I, 1);
2767
6
    assert(isX86_MMX || S1->getType()->isVectorTy());
2768
6
2769
6
    // SExt and ICmpNE below must apply to individual elements of input vectors.
2770
6
    // In case of x86mmx arguments, cast them to appropriate vector types and
2771
6
    // back.
2772
6
    Type *T = isX86_MMX ? 
getMMXVectorTy(EltSizeInBits)2
:
S1->getType()4
;
2773
6
    if (isX86_MMX) {
2774
2
      S1 = IRB.CreateBitCast(S1, T);
2775
2
      S2 = IRB.CreateBitCast(S2, T);
2776
2
    }
2777
6
    Value *S1_ext = IRB.CreateSExt(
2778
6
        IRB.CreateICmpNE(S1, Constant::getNullValue(T)), T);
2779
6
    Value *S2_ext = IRB.CreateSExt(
2780
6
        IRB.CreateICmpNE(S2, Constant::getNullValue(T)), T);
2781
6
    if (isX86_MMX) {
2782
2
      Type *X86_MMXTy = Type::getX86_MMXTy(*MS.C);
2783
2
      S1_ext = IRB.CreateBitCast(S1_ext, X86_MMXTy);
2784
2
      S2_ext = IRB.CreateBitCast(S2_ext, X86_MMXTy);
2785
2
    }
2786
6
2787
6
    Function *ShadowFn = Intrinsic::getDeclaration(
2788
6
        F.getParent(), getSignedPackIntrinsic(I.getIntrinsicID()));
2789
6
2790
6
    Value *S =
2791
6
        IRB.CreateCall(ShadowFn, {S1_ext, S2_ext}, "_msprop_vector_pack");
2792
6
    if (isX86_MMX) 
S = IRB.CreateBitCast(S, getShadowTy(&I))2
;
2793
6
    setShadow(&I, S);
2794
6
    setOriginForNaryOp(I);
2795
6
  }
2796
2797
  // Instrument sum-of-absolute-differencies intrinsic.
2798
4
  void handleVectorSadIntrinsic(IntrinsicInst &I) {
2799
4
    const unsigned SignificantBitsPerResultElement = 16;
2800
4
    bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2801
4
    Type *ResTy = isX86_MMX ? 
IntegerType::get(*MS.C, 64)2
:
I.getType()2
;
2802
4
    unsigned ZeroBitsPerResultElement =
2803
4
        ResTy->getScalarSizeInBits() - SignificantBitsPerResultElement;
2804
4
2805
4
    IRBuilder<> IRB(&I);
2806
4
    Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2807
4
    S = IRB.CreateBitCast(S, ResTy);
2808
4
    S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2809
4
                       ResTy);
2810
4
    S = IRB.CreateLShr(S, ZeroBitsPerResultElement);
2811
4
    S = IRB.CreateBitCast(S, getShadowTy(&I));
2812
4
    setShadow(&I, S);
2813
4
    setOriginForNaryOp(I);
2814
4
  }
2815
2816
  // Instrument multiply-add intrinsic.
2817
  void handleVectorPmaddIntrinsic(IntrinsicInst &I,
2818
4
                                  unsigned EltSizeInBits = 0) {
2819
4
    bool isX86_MMX = I.getOperand(0)->getType()->isX86_MMXTy();
2820
4
    Type *ResTy = isX86_MMX ? 
getMMXVectorTy(EltSizeInBits * 2)2
:
I.getType()2
;
2821
4
    IRBuilder<> IRB(&I);
2822
4
    Value *S = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2823
4
    S = IRB.CreateBitCast(S, ResTy);
2824
4
    S = IRB.CreateSExt(IRB.CreateICmpNE(S, Constant::getNullValue(ResTy)),
2825
4
                       ResTy);
2826
4
    S = IRB.CreateBitCast(S, getShadowTy(&I));
2827
4
    setShadow(&I, S);
2828
4
    setOriginForNaryOp(I);
2829
4
  }
2830
2831
  // Instrument compare-packed intrinsic.
2832
  // Basically, an or followed by sext(icmp ne 0) to end up with all-zeros or
2833
  // all-ones shadow.
2834
2
  void handleVectorComparePackedIntrinsic(IntrinsicInst &I) {
2835
2
    IRBuilder<> IRB(&I);
2836
2
    Type *ResTy = getShadowTy(&I);
2837
2
    Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2838
2
    Value *S = IRB.CreateSExt(
2839
2
        IRB.CreateICmpNE(S0, Constant::getNullValue(ResTy)), ResTy);
2840
2
    setShadow(&I, S);
2841
2
    setOriginForNaryOp(I);
2842
2
  }
2843
2844
  // Instrument compare-scalar intrinsic.
2845
  // This handles both cmp* intrinsics which return the result in the first
2846
  // element of a vector, and comi* which return the result as i32.
2847
8
  void handleVectorCompareScalarIntrinsic(IntrinsicInst &I) {
2848
8
    IRBuilder<> IRB(&I);
2849
8
    Value *S0 = IRB.CreateOr(getShadow(&I, 0), getShadow(&I, 1));
2850
8
    Value *S = LowerElementShadowExtend(IRB, S0, getShadowTy(&I));
2851
8
    setShadow(&I, S);
2852
8
    setOriginForNaryOp(I);
2853
8
  }
2854
2855
4
  void handleStmxcsr(IntrinsicInst &I) {
2856
4
    IRBuilder<> IRB(&I);
2857
4
    Value* Addr = I.getArgOperand(0);
2858
4
    Type *Ty = IRB.getInt32Ty();
2859
4
    Value *ShadowPtr =
2860
4
        getShadowOriginPtr(Addr, IRB, Ty, /*Alignment*/ 1, /*isStore*/ true)
2861
4
            .first;
2862
4
2863
4
    IRB.CreateStore(getCleanShadow(Ty),
2864
4
                    IRB.CreatePointerCast(ShadowPtr, Ty->getPointerTo()));
2865
4
2866
4
    if (ClCheckAccessAddress)
2867
2
      insertShadowCheck(Addr, &I);
2868
4
  }
2869
2870
4
  void handleLdmxcsr(IntrinsicInst &I) {
2871
4
    if (!InsertChecks) 
return0
;
2872
4
2873
4
    IRBuilder<> IRB(&I);
2874
4
    Value *Addr = I.getArgOperand(0);
2875
4
    Type *Ty = IRB.getInt32Ty();
2876
4
    unsigned Alignment = 1;
2877
4
    Value *ShadowPtr, *OriginPtr;
2878
4
    std::tie(ShadowPtr, OriginPtr) =
2879
4
        getShadowOriginPtr(Addr, IRB, Ty, Alignment, /*isStore*/ false);
2880
4
2881
4
    if (ClCheckAccessAddress)
2882
2
      insertShadowCheck(Addr, &I);
2883
4
2884
4
    Value *Shadow = IRB.CreateAlignedLoad(Ty, ShadowPtr, Alignment, "_ldmxcsr");
2885
4
    Value *Origin = MS.TrackOrigins ? 
IRB.CreateLoad(MS.OriginTy, OriginPtr)0
2886
4
                                    : getCleanOrigin();
2887
4
    insertShadowCheck(Shadow, Origin, &I);
2888
4
  }
2889
2890
12
  void handleMaskedStore(IntrinsicInst &I) {
2891
12
    IRBuilder<> IRB(&I);
2892
12
    Value *V = I.getArgOperand(0);
2893
12
    Value *Addr = I.getArgOperand(1);
2894
12
    unsigned Align = cast<ConstantInt>(I.getArgOperand(2))->getZExtValue();
2895
12
    Value *Mask = I.getArgOperand(3);
2896
12
    Value *Shadow = getShadow(V);
2897
12
2898
12
    Value *ShadowPtr;
2899
12
    Value *OriginPtr;
2900
12
    std::tie(ShadowPtr, OriginPtr) = getShadowOriginPtr(
2901
12
        Addr, IRB, Shadow->getType(), Align, /*isStore*/ true);
2902
12
2903
12
    if (ClCheckAccessAddress) {
2904
4
      insertShadowCheck(Addr, &I);
2905
4
      // Uninitialized mask is kind of like uninitialized address, but not as
2906
4
      // scary.
2907
4
      insertShadowCheck(Mask, &I);
2908
4
    }
2909
12
2910
12
    IRB.CreateMaskedStore(Shadow, ShadowPtr, Align, Mask);
2911
12
2912
12
    if (MS.TrackOrigins) {
2913
4
      auto &DL = F.getParent()->getDataLayout();
2914
4
      paintOrigin(IRB, getOrigin(V), OriginPtr,
2915
4
                  DL.getTypeStoreSize(Shadow->getType()),
2916
4
                  std::max(Align, kMinOriginAlignment));
2917
4
    }
2918
12
  }
2919
2920
12
  bool handleMaskedLoad(IntrinsicInst &I) {
2921
12
    IRBuilder<> IRB(&I);
2922
12
    Value *Addr = I.getArgOperand(0);
2923
12
    unsigned Align = cast<ConstantInt>(I.getArgOperand(1))->getZExtValue();
2924
12
    Value *Mask = I.getArgOperand(2);
2925
12
    Value *PassThru = I.getArgOperand(3);
2926
12
2927
12
    Type *ShadowTy = getShadowTy(&I);
2928
12
    Value *ShadowPtr, *OriginPtr;
2929
12
    if (PropagateShadow) {
2930
6
      std::tie(ShadowPtr, OriginPtr) =
2931
6
          getShadowOriginPtr(Addr, IRB, ShadowTy, Align, /*isStore*/ false);
2932
6
      setShadow(&I, IRB.CreateMaskedLoad(ShadowPtr, Align, Mask,
2933
6
                                         getShadow(PassThru), "_msmaskedld"));
2934
6
    } else {
2935
6
      setShadow(&I, getCleanShadow(&I));
2936
6
    }
2937
12
2938
12
    if (ClCheckAccessAddress) {
2939
4
      insertShadowCheck(Addr, &I);
2940
4
      insertShadowCheck(Mask, &I);
2941
4
    }
2942
12
2943
12
    if (MS.TrackOrigins) {
2944
4
      if (PropagateShadow) {
2945
2
        // Choose between PassThru's and the loaded value's origins.
2946
2
        Value *MaskedPassThruShadow = IRB.CreateAnd(
2947
2
            getShadow(PassThru), IRB.CreateSExt(IRB.CreateNeg(Mask), ShadowTy));
2948
2
2949
2
        Value *Acc = IRB.CreateExtractElement(
2950
2
            MaskedPassThruShadow, ConstantInt::get(IRB.getInt32Ty(), 0));
2951
8
        for (int i = 1, N = PassThru->getType()->getVectorNumElements(); i < N;
2952
6
             ++i) {
2953
6
          Value *More = IRB.CreateExtractElement(
2954
6
              MaskedPassThruShadow, ConstantInt::get(IRB.getInt32Ty(), i));
2955
6
          Acc = IRB.CreateOr(Acc, More);
2956
6
        }
2957
2
2958
2
        Value *Origin = IRB.CreateSelect(
2959
2
            IRB.CreateICmpNE(Acc, Constant::getNullValue(Acc->getType())),
2960
2
            getOrigin(PassThru), IRB.CreateLoad(MS.OriginTy, OriginPtr));
2961
2
2962
2
        setOrigin(&I, Origin);
2963
2
      } else {
2964
2
        setOrigin(&I, getCleanOrigin());
2965
2
      }
2966
4
    }
2967
12
    return true;
2968
12
  }
2969
2970
  // Instrument BMI / BMI2 intrinsics.
2971
  // All of these intrinsics are Z = I(X, Y)
2972
  // where the types of all operands and the result match, and are either i32 or i64.
2973
  // The following instrumentation happens to work for all of them:
2974
  //   Sz = I(Sx, Y) | (sext (Sy != 0))
2975
16
  void handleBmiIntrinsic(IntrinsicInst &I) {
2976
16
    IRBuilder<> IRB(&I);
2977
16
    Type *ShadowTy = getShadowTy(&I);
2978
16
2979
16
    // If any bit of the mask operand is poisoned, then the whole thing is.
2980
16
    Value *SMask = getShadow(&I, 1);
2981
16
    SMask = IRB.CreateSExt(IRB.CreateICmpNE(SMask, getCleanShadow(ShadowTy)),
2982
16
                           ShadowTy);
2983
16
    // Apply the same intrinsic to the shadow of the first operand.
2984
16
    Value *S = IRB.CreateCall(I.getCalledFunction(),
2985
16
                              {getShadow(&I, 0), I.getOperand(1)});
2986
16
    S = IRB.CreateOr(SMask, S);
2987
16
    setShadow(&I, S);
2988
16
    setOriginForNaryOp(I);
2989
16
  }
2990
2991
325
  void visitIntrinsicInst(IntrinsicInst &I) {
2992
325
    switch (I.getIntrinsicID()) {
2993
325
    case Intrinsic::lifetime_start:
2994
85
      handleLifetimeStart(I);
2995
85
      break;
2996
325
    case Intrinsic::bswap:
2997
4
      handleBswap(I);
2998
4
      break;
2999
325
    case Intrinsic::masked_store:
3000
12
      handleMaskedStore(I);
3001
12
      break;
3002
325
    case Intrinsic::masked_load:
3003
12
      handleMaskedLoad(I);
3004
12
      break;
3005
325
    case Intrinsic::x86_sse_stmxcsr:
3006
4
      handleStmxcsr(I);
3007
4
      break;
3008
325
    case Intrinsic::x86_sse_ldmxcsr:
3009
4
      handleLdmxcsr(I);
3010
4
      break;
3011
325
    case Intrinsic::x86_avx512_vcvtsd2usi64:
3012
2
    case Intrinsic::x86_avx512_vcvtsd2usi32:
3013
2
    case Intrinsic::x86_avx512_vcvtss2usi64:
3014
2
    case Intrinsic::x86_avx512_vcvtss2usi32:
3015
2
    case Intrinsic::x86_avx512_cvttss2usi64:
3016
2
    case Intrinsic::x86_avx512_cvttss2usi:
3017
2
    case Intrinsic::x86_avx512_cvttsd2usi64:
3018
2
    case Intrinsic::x86_avx512_cvttsd2usi:
3019
2
    case Intrinsic::x86_avx512_cvtusi2ss:
3020
2
    case Intrinsic::x86_avx512_cvtusi642sd:
3021
2
    case Intrinsic::x86_avx512_cvtusi642ss:
3022
2
    case Intrinsic::x86_sse2_cvtsd2si64:
3023
2
    case Intrinsic::x86_sse2_cvtsd2si:
3024
2
    case Intrinsic::x86_sse2_cvtsd2ss:
3025
2
    case Intrinsic::x86_sse2_cvttsd2si64:
3026
2
    case Intrinsic::x86_sse2_cvttsd2si:
3027
2
    case Intrinsic::x86_sse_cvtss2si64:
3028
2
    case Intrinsic::x86_sse_cvtss2si:
3029
2
    case Intrinsic::x86_sse_cvttss2si64:
3030
2
    case Intrinsic::x86_sse_cvttss2si:
3031
2
      handleVectorConvertIntrinsic(I, 1);
3032
2
      break;
3033
2
    case Intrinsic::x86_sse_cvtps2pi:
3034
2
    case Intrinsic::x86_sse_cvttps2pi:
3035
2
      handleVectorConvertIntrinsic(I, 2);
3036
2
      break;
3037
2
3038
10
    case Intrinsic::x86_avx512_psll_w_512:
3039
10
    case Intrinsic::x86_avx512_psll_d_512:
3040
10
    case Intrinsic::x86_avx512_psll_q_512:
3041
10
    case Intrinsic::x86_avx512_pslli_w_512:
3042
10
    case Intrinsic::x86_avx512_pslli_d_512:
3043
10
    case Intrinsic::x86_avx512_pslli_q_512:
3044
10
    case Intrinsic::x86_avx512_psrl_w_512:
3045
10
    case Intrinsic::x86_avx512_psrl_d_512:
3046
10
    case Intrinsic::x86_avx512_psrl_q_512:
3047
10
    case Intrinsic::x86_avx512_psra_w_512:
3048
10
    case Intrinsic::x86_avx512_psra_d_512:
3049
10
    case Intrinsic::x86_avx512_psra_q_512:
3050
10
    case Intrinsic::x86_avx512_psrli_w_512:
3051
10
    case Intrinsic::x86_avx512_psrli_d_512:
3052
10
    case Intrinsic::x86_avx512_psrli_q_512:
3053
10
    case Intrinsic::x86_avx512_psrai_w_512:
3054
10
    case Intrinsic::x86_avx512_psrai_d_512:
3055
10
    case Intrinsic::x86_avx512_psrai_q_512:
3056
10
    case Intrinsic::x86_avx512_psra_q_256:
3057
10
    case Intrinsic::x86_avx512_psra_q_128:
3058
10
    case Intrinsic::x86_avx512_psrai_q_256:
3059
10
    case Intrinsic::x86_avx512_psrai_q_128:
3060
10
    case Intrinsic::x86_avx2_psll_w:
3061
10
    case Intrinsic::x86_avx2_psll_d:
3062
10
    case Intrinsic::x86_avx2_psll_q:
3063
10
    case Intrinsic::x86_avx2_pslli_w:
3064
10
    case Intrinsic::x86_avx2_pslli_d:
3065
10
    case Intrinsic::x86_avx2_pslli_q:
3066
10
    case Intrinsic::x86_avx2_psrl_w:
3067
10
    case Intrinsic::x86_avx2_psrl_d:
3068
10
    case Intrinsic::x86_avx2_psrl_q:
3069
10
    case Intrinsic::x86_avx2_psra_w:
3070
10
    case Intrinsic::x86_avx2_psra_d:
3071
10
    case Intrinsic::x86_avx2_psrli_w:
3072
10
    case Intrinsic::x86_avx2_psrli_d:
3073
10
    case Intrinsic::x86_avx2_psrli_q:
3074
10
    case Intrinsic::x86_avx2_psrai_w:
3075
10
    case Intrinsic::x86_avx2_psrai_d:
3076
10
    case Intrinsic::x86_sse2_psll_w:
3077
10
    case Intrinsic::x86_sse2_psll_d:
3078
10
    case Intrinsic::x86_sse2_psll_q:
3079
10
    case Intrinsic::x86_sse2_pslli_w:
3080
10
    case Intrinsic::x86_sse2_pslli_d:
3081
10
    case Intrinsic::x86_sse2_pslli_q:
3082
10
    case Intrinsic::x86_sse2_psrl_w:
3083
10
    case Intrinsic::x86_sse2_psrl_d:
3084
10
    case Intrinsic::x86_sse2_psrl_q:
3085
10
    case Intrinsic::x86_sse2_psra_w:
3086
10
    case Intrinsic::x86_sse2_psra_d:
3087
10
    case Intrinsic::x86_sse2_psrli_w:
3088
10
    case Intrinsic::x86_sse2_psrli_d:
3089
10
    case Intrinsic::x86_sse2_psrli_q:
3090
10
    case Intrinsic::x86_sse2_psrai_w:
3091
10
    case Intrinsic::x86_sse2_psrai_d:
3092
10
    case Intrinsic::x86_mmx_psll_w:
3093
10
    case Intrinsic::x86_mmx_psll_d:
3094
10
    case Intrinsic::x86_mmx_psll_q:
3095
10
    case Intrinsic::x86_mmx_pslli_w:
3096
10
    case Intrinsic::x86_mmx_pslli_d:
3097
10
    case Intrinsic::x86_mmx_pslli_q:
3098
10
    case Intrinsic::x86_mmx_psrl_w:
3099
10
    case Intrinsic::x86_mmx_psrl_d:
3100
10
    case Intrinsic::x86_mmx_psrl_q:
3101
10
    case Intrinsic::x86_mmx_psra_w:
3102
10
    case Intrinsic::x86_mmx_psra_d:
3103
10
    case Intrinsic::x86_mmx_psrli_w:
3104
10
    case Intrinsic::x86_mmx_psrli_d:
3105
10
    case Intrinsic::x86_mmx_psrli_q:
3106
10
    case Intrinsic::x86_mmx_psrai_w:
3107
10
    case Intrinsic::x86_mmx_psrai_d:
3108
10
      handleVectorShiftIntrinsic(I, /* Variable */ false);
3109
10
      break;
3110
10
    case Intrinsic::x86_avx2_psllv_d:
3111
6
    case Intrinsic::x86_avx2_psllv_d_256:
3112
6
    case Intrinsic::x86_avx512_psllv_d_512:
3113
6
    case Intrinsic::x86_avx2_psllv_q:
3114
6
    case Intrinsic::x86_avx2_psllv_q_256:
3115
6
    case Intrinsic::x86_avx512_psllv_q_512:
3116
6
    case Intrinsic::x86_avx2_psrlv_d:
3117
6
    case Intrinsic::x86_avx2_psrlv_d_256:
3118
6
    case Intrinsic::x86_avx512_psrlv_d_512:
3119
6
    case Intrinsic::x86_avx2_psrlv_q:
3120
6
    case Intrinsic::x86_avx2_psrlv_q_256:
3121
6
    case Intrinsic::x86_avx512_psrlv_q_512:
3122
6
    case Intrinsic::x86_avx2_psrav_d:
3123
6
    case Intrinsic::x86_avx2_psrav_d_256:
3124
6
    case Intrinsic::x86_avx512_psrav_d_512:
3125
6
    case Intrinsic::x86_avx512_psrav_q_128:
3126
6
    case Intrinsic::x86_avx512_psrav_q_256:
3127
6
    case Intrinsic::x86_avx512_psrav_q_512:
3128
6
      handleVectorShiftIntrinsic(I, /* Variable */ true);
3129
6
      break;
3130
6
3131
6
    case Intrinsic::x86_sse2_packsswb_128:
3132
4
    case Intrinsic::x86_sse2_packssdw_128:
3133
4
    case Intrinsic::x86_sse2_packuswb_128:
3134
4
    case Intrinsic::x86_sse41_packusdw:
3135
4
    case Intrinsic::x86_avx2_packsswb:
3136
4
    case Intrinsic::x86_avx2_packssdw:
3137
4
    case Intrinsic::x86_avx2_packuswb:
3138
4
    case Intrinsic::x86_avx2_packusdw:
3139
4
      handleVectorPackIntrinsic(I);
3140
4
      break;
3141
4
3142
4
    case Intrinsic::x86_mmx_packsswb:
3143
2
    case Intrinsic::x86_mmx_packuswb:
3144
2
      handleVectorPackIntrinsic(I, 16);
3145
2
      break;
3146
2
3147
2
    case Intrinsic::x86_mmx_packssdw:
3148
0
      handleVectorPackIntrinsic(I, 32);
3149
0
      break;
3150
2
3151
4
    case Intrinsic::x86_mmx_psad_bw:
3152
4
    case Intrinsic::x86_sse2_psad_bw:
3153
4
    case Intrinsic::x86_avx2_psad_bw:
3154
4
      handleVectorSadIntrinsic(I);
3155
4
      break;
3156
4
3157
4
    case Intrinsic::x86_sse2_pmadd_wd:
3158
2
    case Intrinsic::x86_avx2_pmadd_wd:
3159
2
    case Intrinsic::x86_ssse3_pmadd_ub_sw_128:
3160
2
    case Intrinsic::x86_avx2_pmadd_ub_sw:
3161
2
      handleVectorPmaddIntrinsic(I);
3162
2
      break;
3163
2
3164
2
    case Intrinsic::x86_ssse3_pmadd_ub_sw:
3165
2
      handleVectorPmaddIntrinsic(I, 8);
3166
2
      break;
3167
2
3168
2
    case Intrinsic::x86_mmx_pmadd_wd:
3169
0
      handleVectorPmaddIntrinsic(I, 16);
3170
0
      break;
3171
2
3172
8
    case Intrinsic::x86_sse_cmp_ss:
3173
8
    case Intrinsic::x86_sse2_cmp_sd:
3174
8
    case Intrinsic::x86_sse_comieq_ss:
3175
8
    case Intrinsic::x86_sse_comilt_ss:
3176
8
    case Intrinsic::x86_sse_comile_ss:
3177
8
    case Intrinsic::x86_sse_comigt_ss:
3178
8
    case Intrinsic::x86_sse_comige_ss:
3179
8
    case Intrinsic::x86_sse_comineq_ss:
3180
8
    case Intrinsic::x86_sse_ucomieq_ss:
3181
8
    case Intrinsic::x86_sse_ucomilt_ss:
3182
8
    case Intrinsic::x86_sse_ucomile_ss:
3183
8
    case Intrinsic::x86_sse_ucomigt_ss:
3184
8
    case Intrinsic::x86_sse_ucomige_ss:
3185
8
    case Intrinsic::x86_sse_ucomineq_ss:
3186
8
    case Intrinsic::x86_sse2_comieq_sd:
3187
8
    case Intrinsic::x86_sse2_comilt_sd:
3188
8
    case Intrinsic::x86_sse2_comile_sd:
3189
8
    case Intrinsic::x86_sse2_comigt_sd:
3190
8
    case Intrinsic::x86_sse2_comige_sd:
3191
8
    case Intrinsic::x86_sse2_comineq_sd:
3192
8
    case Intrinsic::x86_sse2_ucomieq_sd:
3193
8
    case Intrinsic::x86_sse2_ucomilt_sd:
3194
8
    case Intrinsic::x86_sse2_ucomile_sd:
3195
8
    case Intrinsic::x86_sse2_ucomigt_sd:
3196
8
    case Intrinsic::x86_sse2_ucomige_sd:
3197
8
    case Intrinsic::x86_sse2_ucomineq_sd:
3198
8
      handleVectorCompareScalarIntrinsic(I);
3199
8
      break;
3200
8
3201
8
    case Intrinsic::x86_sse_cmp_ps:
3202
2
    case Intrinsic::x86_sse2_cmp_pd:
3203
2
      // FIXME: For x86_avx_cmp_pd_256 and x86_avx_cmp_ps_256 this function
3204
2
      // generates reasonably looking IR that fails in the backend with "Do not
3205
2
      // know how to split the result of this operator!".
3206
2
      handleVectorComparePackedIntrinsic(I);
3207
2
      break;
3208
2
3209
16
    case Intrinsic::x86_bmi_bextr_32:
3210
16
    case Intrinsic::x86_bmi_bextr_64:
3211
16
    case Intrinsic::x86_bmi_bzhi_32:
3212
16
    case Intrinsic::x86_bmi_bzhi_64:
3213
16
    case Intrinsic::x86_bmi_pdep_32:
3214
16
    case Intrinsic::x86_bmi_pdep_64:
3215
16
    case Intrinsic::x86_bmi_pext_32:
3216
16
    case Intrinsic::x86_bmi_pext_64:
3217
16
      handleBmiIntrinsic(I);
3218
16
      break;
3219
16
3220
16
    case Intrinsic::is_constant:
3221
4
      // The result of llvm.is.constant() is always defined.
3222
4
      setShadow(&I, getCleanShadow(&I));
3223
4
      setOrigin(&I, getCleanOrigin());
3224
4
      break;
3225
16
3226
140
    default:
3227
140
      if (!handleUnknownIntrinsic(I))
3228
132
        visitInstruction(I);
3229
140
      break;
3230
325
    }
3231
325
  }
3232
3233
261
  void visitCallSite(CallSite CS) {
3234
261
    Instruction &I = *CS.getInstruction();
3235
261
    assert(!I.getMetadata("nosanitize"));
3236
261
    assert((CS.isCall() || CS.isInvoke() || CS.isCallBr()) &&
3237
261
           "Unknown type of CallSite");
3238
261
    if (CS.isCallBr() || 
(257
CS.isCall()257
&&
cast<CallInst>(&I)->isInlineAsm()257
)) {
3239
48
      // For inline asm (either a call to asm function, or callbr instruction),
3240
48
      // do the usual thing: check argument shadow and mark all outputs as
3241
48
      // clean. Note that any side effects of the inline asm that are not
3242
48
      // immediately visible in its constraints are not handled.
3243
48
      if (ClHandleAsmConservative && 
MS.CompileKernel24
)
3244
24
        visitAsmInstruction(I);
3245
24
      else
3246
24
        visitInstruction(I);
3247
48
      return;
3248
48
    }
3249
213
    if (CS.isCall()) {
3250
213
      CallInst *Call = cast<CallInst>(&I);
3251
213
      assert(!isa<IntrinsicInst>(&I) && "intrinsics are handled elsewhere");
3252
213
3253
213
      // We are going to insert code that relies on the fact that the callee
3254
213
      // will become a non-readonly function after it is instrumented by us. To
3255
213
      // prevent this code from being optimized out, mark that function
3256
213
      // non-readonly in advance.
3257
213
      if (Function *Func = Call->getCalledFunction()) {
3258
213
        // Clear out readonly/readnone attributes.
3259
213
        AttrBuilder B;
3260
213
        B.addAttribute(Attribute::ReadOnly)
3261
213
          .addAttribute(Attribute::ReadNone);
3262
213
        Func->removeAttributes(AttributeList::FunctionIndex, B);
3263
213
      }
3264
213
3265
213
      maybeMarkSanitizerLibraryCallNoBuiltin(Call, TLI);
3266
213
    }
3267
213
    IRBuilder<> IRB(&I);
3268
213
3269
213
    unsigned ArgOffset = 0;
3270
213
    LLVM_DEBUG(dbgs() << "  CallSite: " << I << "\n");
3271
213
    for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
3272
1.79k
         ArgIt != End; 
++ArgIt1.58k
) {
3273
1.59k
      Value *A = *ArgIt;
3274
1.59k
      unsigned i = ArgIt - CS.arg_begin();
3275
1.59k
      if (!A->getType()->isSized()) {
3276
0
        LLVM_DEBUG(dbgs() << "Arg " << i << " is not sized: " << I << "\n");
3277
0
        continue;
3278
0
      }
3279
1.59k
      unsigned Size = 0;
3280
1.59k
      Value *Store = nullptr;
3281
1.59k
      // Compute the Shadow for arg even if it is ByVal, because
3282
1.59k
      // in that case getShadow() will copy the actual arg shadow to
3283
1.59k
      // __msan_param_tls.
3284
1.59k
      Value *ArgShadow = getShadow(A);
3285
1.59k
      Value *ArgShadowBase = getShadowPtrForArgument(A, IRB, ArgOffset);
3286
1.59k
      LLVM_DEBUG(dbgs() << "  Arg#" << i << ": " << *A
3287
1.59k
                        << " Shadow: " << *ArgShadow << "\n");
3288
1.59k
      bool ArgIsInitialized = false;
3289
1.59k
      const DataLayout &DL = F.getParent()->getDataLayout();
3290
1.59k
      if (CS.paramHasAttr(i, Attribute::ByVal)) {
3291
18
        assert(A->getType()->isPointerTy() &&
3292
18
               "ByVal argument is not a pointer!");
3293
18
        Size = DL.getTypeAllocSize(A->getType()->getPointerElementType());
3294
18
        if (ArgOffset + Size > kParamTLSSize) 
break0
;
3295
18
        unsigned ParamAlignment = CS.getParamAlignment(i);
3296
18
        unsigned Alignment = std::min(ParamAlignment, kShadowTLSAlignment);
3297
18
        Value *AShadowPtr =
3298
18
            getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), Alignment,
3299
18
                               /*isStore*/ false)
3300
18
                .first;
3301
18
3302
18
        Store = IRB.CreateMemCpy(ArgShadowBase, Alignment, AShadowPtr,
3303
18
                                 Alignment, Size);
3304
18
        // TODO(glider): need to copy origins.
3305
1.57k
      } else {
3306
1.57k
        Size = DL.getTypeAllocSize(A->getType());
3307
1.57k
        if (ArgOffset + Size > kParamTLSSize) 
break12
;
3308
1.56k
        Store = IRB.CreateAlignedStore(ArgShadow, ArgShadowBase,
3309
1.56k
                                       kShadowTLSAlignment);
3310
1.56k
        Constant *Cst = dyn_cast<Constant>(ArgShadow);
3311
1.56k
        if (Cst && 
Cst->isNullValue()1.43k
)
ArgIsInitialized = true1.43k
;
3312
1.56k
      }
3313
1.59k
      
if (1.58k
MS.TrackOrigins1.58k
&&
!ArgIsInitialized84
)
3314
32
        IRB.CreateStore(getOrigin(A),
3315
32
                        getOriginPtrForArgument(A, IRB, ArgOffset));
3316
1.58k
      (void)Store;
3317
1.58k
      assert(Size != 0 && Store != nullptr);
3318
1.58k
      LLVM_DEBUG(dbgs() << "  Param:" << *Store << "\n");
3319
1.58k
      ArgOffset += alignTo(Size, 8);
3320
1.58k
    }
3321
213
    LLVM_DEBUG(dbgs() << "  done with call args\n");
3322
213
3323
213
    FunctionType *FT = CS.getFunctionType();
3324
213
    if (FT->isVarArg()) {
3325
82
      VAHelper->visitCallSite(CS, IRB);
3326
82
    }
3327
213
3328
213
    // Now, get the shadow for the RetVal.
3329
213
    if (!I.getType()->isSized()) 
return93
;
3330
120
    // Don't emit the epilogue for musttail call returns.
3331
120
    if (CS.isCall() && cast<CallInst>(&I)->isMustTailCall()) 
return8
;
3332
112
    IRBuilder<> IRBBefore(&I);
3333
112
    // Until we have full dynamic coverage, make sure the retval shadow is 0.
3334
112
    Value *Base = getShadowPtrForRetval(&I, IRBBefore);
3335
112
    IRBBefore.CreateAlignedStore(getCleanShadow(&I), Base, kShadowTLSAlignment);
3336
112
    BasicBlock::iterator NextInsn;
3337
112
    if (CS.isCall()) {
3338
112
      NextInsn = ++I.getIterator();
3339
112
      assert(NextInsn != I.getParent()->end());
3340
112
    } else {
3341
0
      BasicBlock *NormalDest = cast<InvokeInst>(&I)->getNormalDest();
3342
0
      if (!NormalDest->getSinglePredecessor()) {
3343
0
        // FIXME: this case is tricky, so we are just conservative here.
3344
0
        // Perhaps we need to split the edge between this BB and NormalDest,
3345
0
        // but a naive attempt to use SplitEdge leads to a crash.
3346
0
        setShadow(&I, getCleanShadow(&I));
3347
0
        setOrigin(&I, getCleanOrigin());
3348
0
        return;
3349
0
      }
3350
0
      // FIXME: NextInsn is likely in a basic block that has not been visited yet.
3351
0
      // Anything inserted there will be instrumented by MSan later!
3352
0
      NextInsn = NormalDest->getFirstInsertionPt();
3353
0
      assert(NextInsn != NormalDest->end() &&
3354
0
             "Could not find insertion point for retval shadow load");
3355
0
    }
3356
112
    IRBuilder<> IRBAfter(&*NextInsn);
3357
112
    Value *RetvalShadow = IRBAfter.CreateAlignedLoad(
3358
112
        getShadowTy(&I), getShadowPtrForRetval(&I, IRBAfter),
3359
112
        kShadowTLSAlignment, "_msret");
3360
112
    setShadow(&I, RetvalShadow);
3361
112
    if (MS.TrackOrigins)
3362
24
      setOrigin(&I, IRBAfter.CreateLoad(MS.OriginTy,
3363
24
                                        getOriginPtrForRetval(IRBAfter)));
3364
112
  }
3365
3366
494
  bool isAMustTailRetVal(Value *RetVal) {
3367
494
    if (auto *I = dyn_cast<BitCastInst>(RetVal)) {
3368
4
      RetVal = I->getOperand(0);
3369
4
    }
3370
494
    if (auto *I = dyn_cast<CallInst>(RetVal)) {
3371
168
      return I->isMustTailCall();
3372
168
    }
3373
326
    return false;
3374
326
  }
3375
3376
912
  void visitReturnInst(ReturnInst &I) {
3377
912
    IRBuilder<> IRB(&I);
3378
912
    Value *RetVal = I.getReturnValue();
3379
912
    if (!RetVal) 
return418
;
3380
494
    // Don't emit the epilogue for musttail call returns.
3381
494
    if (isAMustTailRetVal(RetVal)) 
return8
;
3382
486
    Value *ShadowPtr = getShadowPtrForRetval(RetVal, IRB);
3383
486
    if (CheckReturnValue) {
3384
13
      insertShadowCheck(RetVal, &I);
3385
13
      Value *Shadow = getCleanShadow(RetVal);
3386
13
      IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
3387
473
    } else {
3388
473
      Value *Shadow = getShadow(RetVal);
3389
473
      IRB.CreateAlignedStore(Shadow, ShadowPtr, kShadowTLSAlignment);
3390
473
      if (MS.TrackOrigins)
3391
168
        IRB.CreateStore(getOrigin(RetVal), getOriginPtrForRetval(IRB));
3392
473
    }
3393
486
  }
3394
3395
54
  void visitPHINode(PHINode &I) {
3396
54
    IRBuilder<> IRB(&I);
3397
54
    if (!PropagateShadow) {
3398
46
      setShadow(&I, getCleanShadow(&I));
3399
46
      setOrigin(&I, getCleanOrigin());
3400
46
      return;
3401
46
    }
3402
8
3403
8
    ShadowPHINodes.push_back(&I);
3404
8
    setShadow(&I, IRB.CreatePHI(getShadowTy(&I), I.getNumIncomingValues(),
3405
8
                                "_msphi_s"));
3406
8
    if (MS.TrackOrigins)
3407
6
      setOrigin(&I, IRB.CreatePHI(MS.OriginTy, I.getNumIncomingValues(),
3408
6
                                  "_msphi_o"));
3409
8
  }
3410
3411
110
  Value *getLocalVarDescription(AllocaInst &I) {
3412
110
    SmallString<2048> StackDescriptionStorage;
3413
110
    raw_svector_ostream StackDescription(StackDescriptionStorage);
3414
110
    // We create a string with a description of the stack allocation and
3415
110
    // pass it into __msan_set_alloca_origin.
3416
110
    // It will be printed by the run-time if stack-originated UMR is found.
3417
110
    // The first 4 bytes of the string are set to '----' and will be replaced
3418
110
    // by __msan_va_arg_overflow_size_tls at the first call.
3419
110
    StackDescription << "----" << I.getName() << "@" << F.getName();
3420
110
    return createPrivateNonConstGlobalForString(*F.getParent(),
3421
110
                                                StackDescription.str());
3422
110
  }
3423
3424
181
  void poisonAllocaUserspace(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
3425
181
    if (PoisonStack && 
ClPoisonStackWithCall148
) {
3426
20
      IRB.CreateCall(MS.MsanPoisonStackFn,
3427
20
                     {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
3428
161
    } else {
3429
161
      Value *ShadowBase, *OriginBase;
3430
161
      std::tie(ShadowBase, OriginBase) =
3431
161
          getShadowOriginPtr(&I, IRB, IRB.getInt8Ty(), 1, /*isStore*/ true);
3432
161
3433
161
      Value *PoisonValue = IRB.getInt8(PoisonStack ? 
ClPoisonStackPattern128
:
033
);
3434
161
      IRB.CreateMemSet(ShadowBase, PoisonValue, Len, I.getAlignment());
3435
161
    }
3436
181
3437
181
    if (PoisonStack && 
MS.TrackOrigins148
) {
3438
52
      Value *Descr = getLocalVarDescription(I);
3439
52
      IRB.CreateCall(MS.MsanSetAllocaOrigin4Fn,
3440
52
                     {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
3441
52
                      IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy()),
3442
52
                      IRB.CreatePointerCast(&F, MS.IntptrTy)});
3443
52
    }
3444
181
  }
3445
3446
58
  void poisonAllocaKmsan(AllocaInst &I, IRBuilder<> &IRB, Value *Len) {
3447
58
    Value *Descr = getLocalVarDescription(I);
3448
58
    if (PoisonStack) {
3449
55
      IRB.CreateCall(MS.MsanPoisonAllocaFn,
3450
55
                     {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len,
3451
55
                      IRB.CreatePointerCast(Descr, IRB.getInt8PtrTy())});
3452
55
    } else {
3453
3
      IRB.CreateCall(MS.MsanUnpoisonAllocaFn,
3454
3
                     {IRB.CreatePointerCast(&I, IRB.getInt8PtrTy()), Len});
3455
3
    }
3456
58
  }
3457
3458
239
  void instrumentAlloca(AllocaInst &I, Instruction *InsPoint = nullptr) {
3459
239
    if (!InsPoint)
3460
204
      InsPoint = &I;
3461
239
    IRBuilder<> IRB(InsPoint->getNextNode());
3462
239
    const DataLayout &DL = F.getParent()->getDataLayout();
3463
239
    uint64_t TypeSize = DL.getTypeAllocSize(I.getAllocatedType());
3464
239
    Value *Len = ConstantInt::get(MS.IntptrTy, TypeSize);
3465
239
    if (I.isArrayAllocation())
3466
44
      Len = IRB.CreateMul(Len, I.getArraySize());
3467
239
3468
239
    if (MS.CompileKernel)
3469
58
      poisonAllocaKmsan(I, IRB, Len);
3470
181
    else
3471
181
      poisonAllocaUserspace(I, IRB, Len);
3472
239
  }
3473
3474
228
  void visitAllocaInst(AllocaInst &I) {
3475
228
    setShadow(&I, getCleanShadow(&I));
3476
228
    setOrigin(&I, getCleanOrigin());
3477
228
    // We'll get to this alloca later unless it's poisoned at the corresponding
3478
228
    // llvm.lifetime.start.
3479
228
    AllocaSet.insert(&I);
3480
228
  }
3481
3482
44
  void visitSelectInst(SelectInst& I) {
3483
44
    IRBuilder<> IRB(&I);
3484
44
    // a = select b, c, d
3485
44
    Value *B = I.getCondition();
3486
44
    Value *C = I.getTrueValue();
3487
44
    Value *D = I.getFalseValue();
3488
44
    Value *Sb = getShadow(B);
3489
44
    Value *Sc = getShadow(C);
3490
44
    Value *Sd = getShadow(D);
3491
44
3492
44
    // Result shadow if condition shadow is 0.
3493
44
    Value *Sa0 = IRB.CreateSelect(B, Sc, Sd);
3494
44
    Value *Sa1;
3495
44
    if (I.getType()->isAggregateType()) {
3496
8
      // To avoid "sign extending" i1 to an arbitrary aggregate type, we just do
3497
8
      // an extra "select". This results in much more compact IR.
3498
8
      // Sa = select Sb, poisoned, (select b, Sc, Sd)
3499
8
      Sa1 = getPoisonedShadow(getShadowTy(I.getType()));
3500
36
    } else {
3501
36
      // Sa = select Sb, [ (c^d) | Sc | Sd ], [ b ? Sc : Sd ]
3502
36
      // If Sb (condition is poisoned), look for bits in c and d that are equal
3503
36
      // and both unpoisoned.
3504
36
      // If !Sb (condition is unpoisoned), simply pick one of Sc and Sd.
3505
36
3506
36
      // Cast arguments to shadow-compatible type.
3507
36
      C = CreateAppToShadowCast(IRB, C);
3508
36
      D = CreateAppToShadowCast(IRB, D);
3509
36
3510
36
      // Result shadow if condition shadow is 1.
3511
36
      Sa1 = IRB.CreateOr({IRB.CreateXor(C, D), Sc, Sd});
3512
36
    }
3513
44
    Value *Sa = IRB.CreateSelect(Sb, Sa1, Sa0, "_msprop_select");
3514
44
    setShadow(&I, Sa);
3515
44
    if (MS.TrackOrigins) {
3516
26
      // Origins are always i32, so any vector conditions must be flattened.
3517
26
      // FIXME: consider tracking vector origins for app vectors?
3518
26
      if (B->getType()->isVectorTy()) {
3519
2
        Type *FlatTy = getShadowTyNoVec(B->getType());
3520
2
        B = IRB.CreateICmpNE(IRB.CreateBitCast(B, FlatTy),
3521
2
                                ConstantInt::getNullValue(FlatTy));
3522
2
        Sb = IRB.CreateICmpNE(IRB.CreateBitCast(Sb, FlatTy),
3523
2
                                      ConstantInt::getNullValue(FlatTy));
3524
2
      }
3525
26
      // a = select b, c, d
3526
26
      // Oa = Sb ? Ob : (b ? Oc : Od)
3527
26
      setOrigin(
3528
26
          &I, IRB.CreateSelect(Sb, getOrigin(I.getCondition()),
3529
26
                               IRB.CreateSelect(B, getOrigin(I.getTrueValue()),
3530
26
                                                getOrigin(I.getFalseValue()))));
3531
26
    }
3532
44
  }
3533
3534
0
  void visitLandingPadInst(LandingPadInst &I) {
3535
0
    // Do nothing.
3536
0
    // See https://github.com/google/sanitizers/issues/504
3537
0
    setShadow(&I, getCleanShadow(&I));
3538
0
    setOrigin(&I, getCleanOrigin());
3539
0
  }
3540
3541
0
  void visitCatchSwitchInst(CatchSwitchInst &I) {
3542
0
    setShadow(&I, getCleanShadow(&I));
3543
0
    setOrigin(&I, getCleanOrigin());
3544
0
  }
3545
3546
0
  void visitFuncletPadInst(FuncletPadInst &I) {
3547
0
    setShadow(&I, getCleanShadow(&I));
3548
0
    setOrigin(&I, getCleanOrigin());
3549
0
  }
3550
3551
117
  void visitGetElementPtrInst(GetElementPtrInst &I) {
3552
117
    handleShadowOr(I);
3553
117
  }
3554
3555
80
  void visitExtractValueInst(ExtractValueInst &I) {
3556
80
    IRBuilder<> IRB(&I);
3557
80
    Value *Agg = I.getAggregateOperand();
3558
80
    LLVM_DEBUG(dbgs() << "ExtractValue:  " << I << "\n");
3559
80
    Value *AggShadow = getShadow(Agg);
3560
80
    LLVM_DEBUG(dbgs() << "   AggShadow:  " << *AggShadow << "\n");
3561
80
    Value *ResShadow = IRB.CreateExtractValue(AggShadow, I.getIndices());
3562
80
    LLVM_DEBUG(dbgs() << "   ResShadow:  " << *ResShadow << "\n");
3563
80
    setShadow(&I, ResShadow);
3564
80
    setOriginForNaryOp(I);
3565
80
  }
3566
3567
24
  void visitInsertValueInst(InsertValueInst &I) {
3568
24
    IRBuilder<> IRB(&I);
3569
24
    LLVM_DEBUG(dbgs() << "InsertValue:  " << I << "\n");
3570
24
    Value *AggShadow = getShadow(I.getAggregateOperand());
3571
24
    Value *InsShadow = getShadow(I.getInsertedValueOperand());
3572
24
    LLVM_DEBUG(dbgs() << "   AggShadow:  " << *AggShadow << "\n");
3573
24
    LLVM_DEBUG(dbgs() << "   InsShadow:  " << *InsShadow << "\n");
3574
24
    Value *Res = IRB.CreateInsertValue(AggShadow, InsShadow, I.getIndices());
3575
24
    LLVM_DEBUG(dbgs() << "   Res:        " << *Res << "\n");
3576
24
    setShadow(&I, Res);
3577
24
    setOriginForNaryOp(I);
3578
24
  }
3579
3580
0
  void dumpInst(Instruction &I) {
3581
0
    if (CallInst *CI = dyn_cast<CallInst>(&I)) {
3582
0
      errs() << "ZZZ call " << CI->getCalledFunction()->getName() << "\n";
3583
0
    } else {
3584
0
      errs() << "ZZZ " << I.getOpcodeName() << "\n";
3585
0
    }
3586
0
    errs() << "QQQ " << I << "\n";
3587
0
  }
3588
3589
0
  void visitResumeInst(ResumeInst &I) {
3590
0
    LLVM_DEBUG(dbgs() << "Resume: " << I << "\n");
3591
0
    // Nothing to do here.
3592
0
  }
3593
3594
0
  void visitCleanupReturnInst(CleanupReturnInst &CRI) {
3595
0
    LLVM_DEBUG(dbgs() << "CleanupReturn: " << CRI << "\n");
3596
0
    // Nothing to do here.
3597
0
  }
3598
3599
0
  void visitCatchReturnInst(CatchReturnInst &CRI) {
3600
0
    LLVM_DEBUG(dbgs() << "CatchReturn: " << CRI << "\n");
3601
0
    // Nothing to do here.
3602
0
  }
3603
3604
  void instrumentAsmArgument(Value *Operand, Instruction &I, IRBuilder<> &IRB,
3605
68
                             const DataLayout &DL, bool isOutput) {
3606
68
    // For each assembly argument, we check its value for being initialized.
3607
68
    // If the argument is a pointer, we assume it points to a single element
3608
68
    // of the corresponding type (or to a 8-byte word, if the type is unsized).
3609
68
    // Each such pointer is instrumented with a call to the runtime library.
3610
68
    Type *OpType = Operand->getType();
3611
68
    // Check the operand value itself.
3612
68
    insertShadowCheck(Operand, &I);
3613
68
    if (!OpType->isPointerTy() || 
!isOutput36
) {
3614
48
      assert(!isOutput);
3615
48
      return;
3616
48
    }
3617
20
    Type *ElType = OpType->getPointerElementType();
3618
20
    if (!ElType->isSized())
3619
0
      return;
3620
20
    int Size = DL.getTypeStoreSize(ElType);
3621
20
    Value *Ptr = IRB.CreatePointerCast(Operand, IRB.getInt8PtrTy());
3622
20
    Value *SizeVal = ConstantInt::get(MS.IntptrTy, Size);
3623
20
    IRB.CreateCall(MS.MsanInstrumentAsmStoreFn, {Ptr, SizeVal});
3624
20
  }
3625
3626
  /// Get the number of output arguments returned by pointers.
3627
24
  int getNumOutputArgs(InlineAsm *IA, CallBase *CB) {
3628
24
    int NumRetOutputs = 0;
3629
24
    int NumOutputs = 0;
3630
24
    Type *RetTy = dyn_cast<Value>(CB)->getType();
3631
24
    if (!RetTy->isVoidTy()) {
3632
16
      // Register outputs are returned via the CallInst return value.
3633
16
      StructType *ST = dyn_cast_or_null<StructType>(RetTy);
3634
16
      if (ST)
3635
12
        NumRetOutputs = ST->getNumElements();
3636
4
      else
3637
4
        NumRetOutputs = 1;
3638
16
    }
3639
24
    InlineAsm::ConstraintInfoVector Constraints = IA->ParseConstraints();
3640
190
    for (size_t i = 0, n = Constraints.size(); i < n; 
i++166
) {
3641
166
      InlineAsm::ConstraintInfo Info = Constraints[i];
3642
166
      switch (Info.Type) {
3643
166
      case InlineAsm::isOutput:
3644
50
        NumOutputs++;
3645
50
        break;
3646
166
      default:
3647
116
        break;
3648
166
      }
3649
166
    }
3650
24
    return NumOutputs - NumRetOutputs;
3651
24
  }
3652
3653
24
  void visitAsmInstruction(Instruction &I) {
3654
24
    // Conservative inline assembly handling: check for poisoned shadow of
3655
24
    // asm() arguments, then unpoison the result and all the memory locations
3656
24
    // pointed to by those arguments.
3657
24
    // An inline asm() statement in C++ contains lists of input and output
3658
24
    // arguments used by the assembly code. These are mapped to operands of the
3659
24
    // CallInst as follows:
3660
24
    //  - nR register outputs ("=r) are returned by value in a single structure
3661
24
    //  (SSA value of the CallInst);
3662
24
    //  - nO other outputs ("=m" and others) are returned by pointer as first
3663
24
    // nO operands of the CallInst;
3664
24
    //  - nI inputs ("r", "m" and others) are passed to CallInst as the
3665
24
    // remaining nI operands.
3666
24
    // The total number of asm() arguments in the source is nR+nO+nI, and the
3667
24
    // corresponding CallInst has nO+nI+1 operands (the last operand is the
3668
24
    // function to be called).
3669
24
    const DataLayout &DL = F.getParent()->getDataLayout();
3670
24
    CallBase *CB = dyn_cast<CallBase>(&I);
3671
24
    IRBuilder<> IRB(&I);
3672
24
    InlineAsm *IA = cast<InlineAsm>(CB->getCalledValue());
3673
24
    int OutputArgs = getNumOutputArgs(IA, CB);
3674
24
    // The last operand of a CallInst is the function itself.
3675
24
    int NumOperands = CB->getNumOperands() - 1;
3676
24
3677
24
    // Check input arguments. Doing so before unpoisoning output arguments, so
3678
24
    // that we won't overwrite uninit values before checking them.
3679
72
    for (int i = OutputArgs; i < NumOperands; 
i++48
) {
3680
48
      Value *Operand = CB->getOperand(i);
3681
48
      instrumentAsmArgument(Operand, I, IRB, DL, /*isOutput*/ false);
3682
48
    }
3683
24
    // Unpoison output arguments. This must happen before the actual InlineAsm
3684
24
    // call, so that the shadow for memory published in the asm() statement
3685
24
    // remains valid.
3686
44
    for (int i = 0; i < OutputArgs; 
i++20
) {
3687
20
      Value *Operand = CB->getOperand(i);
3688
20
      instrumentAsmArgument(Operand, I, IRB, DL, /*isOutput*/ true);
3689
20
    }
3690
24
3691
24
    setShadow(&I, getCleanShadow(&I));
3692
24
    setOrigin(&I, getCleanOrigin());
3693
24
  }
3694
3695
293
  void visitInstruction(Instruction &I) {
3696
293
    // Everything else: stop propagating and check for poisoned shadow.
3697
293
    if (ClDumpStrictInstructions)
3698
0
      dumpInst(I);
3699
293
    LLVM_DEBUG(dbgs() << "DEFAULT: " << I << "\n");
3700
1.03k
    for (size_t i = 0, n = I.getNumOperands(); i < n; 
i++738
) {
3701
738
      Value *Operand = I.getOperand(i);
3702
738
      if (Operand->getType()->isSized())
3703
512
        insertShadowCheck(Operand, &I);
3704
738
    }
3705
293
    setShadow(&I, getCleanShadow(&I));
3706
293
    setOrigin(&I, getCleanOrigin());
3707
293
  }
3708
};
3709
3710
/// AMD64-specific implementation of VarArgHelper.
3711
struct VarArgAMD64Helper : public VarArgHelper {
3712
  // An unfortunate workaround for asymmetric lowering of va_arg stuff.
3713
  // See a comment in visitCallSite for more details.
3714
  static const unsigned AMD64GpEndOffset = 48;  // AMD64 ABI Draft 0.99.6 p3.5.7
3715
  static const unsigned AMD64FpEndOffsetSSE = 176;
3716
  // If SSE is disabled, fp_offset in va_list is zero.
3717
  static const unsigned AMD64FpEndOffsetNoSSE = AMD64GpEndOffset;
3718
3719
  unsigned AMD64FpEndOffset;
3720
  Function &F;
3721
  MemorySanitizer &MS;
3722
  MemorySanitizerVisitor &MSV;
3723
  Value *VAArgTLSCopy = nullptr;
3724
  Value *VAArgTLSOriginCopy = nullptr;
3725
  Value *VAArgOverflowSize = nullptr;
3726
3727
  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3728
3729
  enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
3730
3731
  VarArgAMD64Helper(Function &F, MemorySanitizer &MS,
3732
                    MemorySanitizerVisitor &MSV)
3733
844
      : F(F), MS(MS), MSV(MSV) {
3734
844
    AMD64FpEndOffset = AMD64FpEndOffsetSSE;
3735
2.65k
    for (const auto &Attr : F.getAttributes().getFnAttributes()) {
3736
2.65k
      if (Attr.isStringAttribute() &&
3737
2.65k
          
(Attr.getKindAsString() == "target-features")1.15k
) {
3738
97
        if (Attr.getValueAsString().contains("-sse"))
3739
6
          AMD64FpEndOffset = AMD64FpEndOffsetNoSSE;
3740
97
        break;
3741
97
      }
3742
2.65k
    }
3743
844
  }
3744
3745
362
  ArgKind classifyArgument(Value* arg) {
3746
362
    // A very rough approximation of X86_64 argument classification rules.
3747
362
    Type *T = arg->getType();
3748
362
    if (T->isFPOrFPVectorTy() || T->isX86_MMXTy())
3749
0
      return AK_FloatingPoint;
3750
362
    if (T->isIntegerTy() && 
T->getPrimitiveSizeInBits() <= 64360
)
3751
339
      return AK_GeneralPurpose;
3752
23
    if (T->isPointerTy())
3753
2
      return AK_GeneralPurpose;
3754
21
    return AK_Memory;
3755
21
  }
3756
3757
  // For VarArg functions, store the argument shadow in an ABI-specific format
3758
  // that corresponds to va_list layout.
3759
  // We do this because Clang lowers va_arg in the frontend, and this pass
3760
  // only sees the low level code that deals with va_list internals.
3761
  // A much easier alternative (provided that Clang emits va_arg instructions)
3762
  // would have been to associate each live instance of va_list with a copy of
3763
  // MSanParamTLS, and extract shadow on va_arg() call in the argument list
3764
  // order.
3765
36
  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3766
36
    unsigned GpOffset = 0;
3767
36
    unsigned FpOffset = AMD64GpEndOffset;
3768
36
    unsigned OverflowOffset = AMD64FpEndOffset;
3769
36
    const DataLayout &DL = F.getParent()->getDataLayout();
3770
36
    for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
3771
406
         ArgIt != End; 
++ArgIt370
) {
3772
370
      Value *A = *ArgIt;
3773
370
      unsigned ArgNo = CS.getArgumentNo(ArgIt);
3774
370
      bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
3775
370
      bool IsByVal = CS.paramHasAttr(ArgNo, Attribute::ByVal);
3776
370
      if (IsByVal) {
3777
8
        // ByVal arguments always go to the overflow area.
3778
8
        // Fixed arguments passed through the overflow area will be stepped
3779
8
        // over by va_start, so don't count them towards the offset.
3780
8
        if (IsFixed)
3781
0
          continue;
3782
8
        assert(A->getType()->isPointerTy());
3783
8
        Type *RealTy = A->getType()->getPointerElementType();
3784
8
        uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
3785
8
        Value *ShadowBase = getShadowPtrForVAArgument(
3786
8
            RealTy, IRB, OverflowOffset, alignTo(ArgSize, 8));
3787
8
        Value *OriginBase = nullptr;
3788
8
        if (MS.TrackOrigins)
3789
4
          OriginBase = getOriginPtrForVAArgument(RealTy, IRB, OverflowOffset);
3790
8
        OverflowOffset += alignTo(ArgSize, 8);
3791
8
        if (!ShadowBase)
3792
0
          continue;
3793
8
        Value *ShadowPtr, *OriginPtr;
3794
8
        std::tie(ShadowPtr, OriginPtr) =
3795
8
            MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(), kShadowTLSAlignment,
3796
8
                                   /*isStore*/ false);
3797
8
3798
8
        IRB.CreateMemCpy(ShadowBase, kShadowTLSAlignment, ShadowPtr,
3799
8
                         kShadowTLSAlignment, ArgSize);
3800
8
        if (MS.TrackOrigins)
3801
4
          IRB.CreateMemCpy(OriginBase, kShadowTLSAlignment, OriginPtr,
3802
4
                           kShadowTLSAlignment, ArgSize);
3803
362
      } else {
3804
362
        ArgKind AK = classifyArgument(A);
3805
362
        if (AK == AK_GeneralPurpose && 
GpOffset >= AMD64GpEndOffset341
)
3806
250
          AK = AK_Memory;
3807
362
        if (AK == AK_FloatingPoint && 
FpOffset >= AMD64FpEndOffset0
)
3808
0
          AK = AK_Memory;
3809
362
        Value *ShadowBase, *OriginBase = nullptr;
3810
362
        switch (AK) {
3811
362
          case AK_GeneralPurpose:
3812
91
            ShadowBase =
3813
91
                getShadowPtrForVAArgument(A->getType(), IRB, GpOffset, 8);
3814
91
            if (MS.TrackOrigins)
3815
49
              OriginBase =
3816
49
                  getOriginPtrForVAArgument(A->getType(), IRB, GpOffset);
3817
91
            GpOffset += 8;
3818
91
            break;
3819
362
          case AK_FloatingPoint:
3820
0
            ShadowBase =
3821
0
                getShadowPtrForVAArgument(A->getType(), IRB, FpOffset, 16);
3822
0
            if (MS.TrackOrigins)
3823
0
              OriginBase =
3824
0
                  getOriginPtrForVAArgument(A->getType(), IRB, FpOffset);
3825
0
            FpOffset += 16;
3826
0
            break;
3827
362
          case AK_Memory:
3828
271
            if (IsFixed)
3829
0
              continue;
3830
271
            uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
3831
271
            ShadowBase =
3832
271
                getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset, 8);
3833
271
            if (MS.TrackOrigins)
3834
15
              OriginBase =
3835
15
                  getOriginPtrForVAArgument(A->getType(), IRB, OverflowOffset);
3836
271
            OverflowOffset += alignTo(ArgSize, 8);
3837
362
        }
3838
362
        // Take fixed arguments into account for GpOffset and FpOffset,
3839
362
        // but don't actually store shadows for them.
3840
362
        // TODO(glider): don't call get*PtrForVAArgument() for them.
3841
362
        if (IsFixed)
3842
26
          continue;
3843
336
        if (!ShadowBase)
3844
94
          continue;
3845
242
        Value *Shadow = MSV.getShadow(A);
3846
242
        IRB.CreateAlignedStore(Shadow, ShadowBase, kShadowTLSAlignment);
3847
242
        if (MS.TrackOrigins) {
3848
48
          Value *Origin = MSV.getOrigin(A);
3849
48
          unsigned StoreSize = DL.getTypeStoreSize(Shadow->getType());
3850
48
          MSV.paintOrigin(IRB, Origin, OriginBase, StoreSize,
3851
48
                          std::max(kShadowTLSAlignment, kMinOriginAlignment));
3852
48
        }
3853
242
      }
3854
370
    }
3855
36
    Constant *OverflowSize =
3856
36
      ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AMD64FpEndOffset);
3857
36
    IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
3858
36
  }
3859
3860
  /// Compute the shadow address for a given va_arg.
3861
  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
3862
370
                                   unsigned ArgOffset, unsigned ArgSize) {
3863
370
    // Make sure we don't overflow __msan_va_arg_tls.
3864
370
    if (ArgOffset + ArgSize > kParamTLSSize)
3865
94
      return nullptr;
3866
276
    Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
3867
276
    Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3868
276
    return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
3869
276
                              "_msarg_va_s");
3870
276
  }
3871
3872
  /// Compute the origin address for a given va_arg.
3873
68
  Value *getOriginPtrForVAArgument(Type *Ty, IRBuilder<> &IRB, int ArgOffset) {
3874
68
    Value *Base = IRB.CreatePointerCast(MS.VAArgOriginTLS, MS.IntptrTy);
3875
68
    // getOriginPtrForVAArgument() is always called after
3876
68
    // getShadowPtrForVAArgument(), so __msan_va_arg_origin_tls can never
3877
68
    // overflow.
3878
68
    Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
3879
68
    return IRB.CreateIntToPtr(Base, PointerType::get(MS.OriginTy, 0),
3880
68
                              "_msarg_va_o");
3881
68
  }
3882
3883
17
  void unpoisonVAListTagForInst(IntrinsicInst &I) {
3884
17
    IRBuilder<> IRB(&I);
3885
17
    Value *VAListTag = I.getArgOperand(0);
3886
17
    Value *ShadowPtr, *OriginPtr;
3887
17
    unsigned Alignment = 8;
3888
17
    std::tie(ShadowPtr, OriginPtr) =
3889
17
        MSV.getShadowOriginPtr(VAListTag, IRB, IRB.getInt8Ty(), Alignment,
3890
17
                               /*isStore*/ true);
3891
17
3892
17
    // Unpoison the whole __va_list_tag.
3893
17
    // FIXME: magic ABI constants.
3894
17
    IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
3895
17
                     /* size */ 24, Alignment, false);
3896
17
    // We shouldn't need to zero out the origins, as they're only checked for
3897
17
    // nonzero shadow.
3898
17
  }
3899
3900
13
  void visitVAStartInst(VAStartInst &I) override {
3901
13
    if (F.getCallingConv() == CallingConv::Win64)
3902
0
      return;
3903
13
    VAStartInstrumentationList.push_back(&I);
3904
13
    unpoisonVAListTagForInst(I);
3905
13
  }
3906
3907
4
  void visitVACopyInst(VACopyInst &I) override {
3908
4
    if (F.getCallingConv() == CallingConv::Win64) 
return0
;
3909
4
    unpoisonVAListTagForInst(I);
3910
4
  }
3911
3912
844
  void finalizeInstrumentation() override {
3913
844
    assert(!VAArgOverflowSize && !VAArgTLSCopy &&
3914
844
           "finalizeInstrumentation called twice");
3915
844
    if (!VAStartInstrumentationList.empty()) {
3916
13
      // If there is a va_start in this function, make a backup copy of
3917
13
      // va_arg_tls somewhere in the function entry block.
3918
13
      IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
3919
13
      VAArgOverflowSize =
3920
13
          IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
3921
13
      Value *CopySize =
3922
13
        IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AMD64FpEndOffset),
3923
13
                      VAArgOverflowSize);
3924
13
      VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3925
13
      IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
3926
13
      if (MS.TrackOrigins) {
3927
9
        VAArgTLSOriginCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
3928
9
        IRB.CreateMemCpy(VAArgTLSOriginCopy, 8, MS.VAArgOriginTLS, 8, CopySize);
3929
9
      }
3930
13
    }
3931
844
3932
844
    // Instrument va_start.
3933
844
    // Copy va_list shadow from the backup copy of the TLS contents.
3934
857
    for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; 
i++13
) {
3935
13
      CallInst *OrigInst = VAStartInstrumentationList[i];
3936
13
      IRBuilder<> IRB(OrigInst->getNextNode());
3937
13
      Value *VAListTag = OrigInst->getArgOperand(0);
3938
13
3939
13
      Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
3940
13
      Value *RegSaveAreaPtrPtr = IRB.CreateIntToPtr(
3941
13
          IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3942
13
                        ConstantInt::get(MS.IntptrTy, 16)),
3943
13
          PointerType::get(RegSaveAreaPtrTy, 0));
3944
13
      Value *RegSaveAreaPtr =
3945
13
          IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
3946
13
      Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
3947
13
      unsigned Alignment = 16;
3948
13
      std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
3949
13
          MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
3950
13
                                 Alignment, /*isStore*/ true);
3951
13
      IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
3952
13
                       AMD64FpEndOffset);
3953
13
      if (MS.TrackOrigins)
3954
9
        IRB.CreateMemCpy(RegSaveAreaOriginPtr, Alignment, VAArgTLSOriginCopy,
3955
9
                         Alignment, AMD64FpEndOffset);
3956
13
      Type *OverflowArgAreaPtrTy = Type::getInt64PtrTy(*MS.C);
3957
13
      Value *OverflowArgAreaPtrPtr = IRB.CreateIntToPtr(
3958
13
          IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
3959
13
                        ConstantInt::get(MS.IntptrTy, 8)),
3960
13
          PointerType::get(OverflowArgAreaPtrTy, 0));
3961
13
      Value *OverflowArgAreaPtr =
3962
13
          IRB.CreateLoad(OverflowArgAreaPtrTy, OverflowArgAreaPtrPtr);
3963
13
      Value *OverflowArgAreaShadowPtr, *OverflowArgAreaOriginPtr;
3964
13
      std::tie(OverflowArgAreaShadowPtr, OverflowArgAreaOriginPtr) =
3965
13
          MSV.getShadowOriginPtr(OverflowArgAreaPtr, IRB, IRB.getInt8Ty(),
3966
13
                                 Alignment, /*isStore*/ true);
3967
13
      Value *SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSCopy,
3968
13
                                             AMD64FpEndOffset);
3969
13
      IRB.CreateMemCpy(OverflowArgAreaShadowPtr, Alignment, SrcPtr, Alignment,
3970
13
                       VAArgOverflowSize);
3971
13
      if (MS.TrackOrigins) {
3972
9
        SrcPtr = IRB.CreateConstGEP1_32(IRB.getInt8Ty(), VAArgTLSOriginCopy,
3973
9
                                        AMD64FpEndOffset);
3974
9
        IRB.CreateMemCpy(OverflowArgAreaOriginPtr, Alignment, SrcPtr, Alignment,
3975
9
                         VAArgOverflowSize);
3976
9
      }
3977
13
    }
3978
844
  }
3979
};
3980
3981
/// MIPS64-specific implementation of VarArgHelper.
3982
struct VarArgMIPS64Helper : public VarArgHelper {
3983
  Function &F;
3984
  MemorySanitizer &MS;
3985
  MemorySanitizerVisitor &MSV;
3986
  Value *VAArgTLSCopy = nullptr;
3987
  Value *VAArgSize = nullptr;
3988
3989
  SmallVector<CallInst*, 16> VAStartInstrumentationList;
3990
3991
  VarArgMIPS64Helper(Function &F, MemorySanitizer &MS,
3992
18
                    MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
3993
3994
12
  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
3995
12
    unsigned VAArgOffset = 0;
3996
12
    const DataLayout &DL = F.getParent()->getDataLayout();
3997
12
    for (CallSite::arg_iterator ArgIt = CS.arg_begin() +
3998
12
         CS.getFunctionType()->getNumParams(), End = CS.arg_end();
3999
512
         ArgIt != End; 
++ArgIt500
) {
4000
500
      Triple TargetTriple(F.getParent()->getTargetTriple());
4001
500
      Value *A = *ArgIt;
4002
500
      Value *Base;
4003
500
      uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4004
500
      if (TargetTriple.getArch() == Triple::mips64) {
4005
250
        // Adjusting the shadow for argument with size < 8 to match the placement
4006
250
        // of bits in big endian system
4007
250
        if (ArgSize < 8)
4008
2
          VAArgOffset += (8 - ArgSize);
4009
250
      }
4010
500
      Base = getShadowPtrForVAArgument(A->getType(), IRB, VAArgOffset, ArgSize);
4011
500
      VAArgOffset += ArgSize;
4012
500
      VAArgOffset = alignTo(VAArgOffset, 8);
4013
500
      if (!Base)
4014
80
        continue;
4015
420
      IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4016
420
    }
4017
12
4018
12
    Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(), VAArgOffset);
4019
12
    // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
4020
12
    // a new class member i.e. it is the total size of all VarArgs.
4021
12
    IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
4022
12
  }
4023
4024
  /// Compute the shadow address for a given va_arg.
4025
  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4026
500
                                   unsigned ArgOffset, unsigned ArgSize) {
4027
500
    // Make sure we don't overflow __msan_va_arg_tls.
4028
500
    if (ArgOffset + ArgSize > kParamTLSSize)
4029
80
      return nullptr;
4030
420
    Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4031
420
    Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4032
420
    return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4033
420
                              "_msarg");
4034
420
  }
4035
4036
4
  void visitVAStartInst(VAStartInst &I) override {
4037
4
    IRBuilder<> IRB(&I);
4038
4
    VAStartInstrumentationList.push_back(&I);
4039
4
    Value *VAListTag = I.getArgOperand(0);
4040
4
    Value *ShadowPtr, *OriginPtr;
4041
4
    unsigned Alignment = 8;
4042
4
    std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4043
4
        VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4044
4
    IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4045
4
                     /* size */ 8, Alignment, false);
4046
4
  }
4047
4048
0
  void visitVACopyInst(VACopyInst &I) override {
4049
0
    IRBuilder<> IRB(&I);
4050
0
    VAStartInstrumentationList.push_back(&I);
4051
0
    Value *VAListTag = I.getArgOperand(0);
4052
0
    Value *ShadowPtr, *OriginPtr;
4053
0
    unsigned Alignment = 8;
4054
0
    std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4055
0
        VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4056
0
    IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4057
0
                     /* size */ 8, Alignment, false);
4058
0
  }
4059
4060
18
  void finalizeInstrumentation() override {
4061
18
    assert(!VAArgSize && !VAArgTLSCopy &&
4062
18
           "finalizeInstrumentation called twice");
4063
18
    IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
4064
18
    VAArgSize = IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4065
18
    Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
4066
18
                                    VAArgSize);
4067
18
4068
18
    if (!VAStartInstrumentationList.empty()) {
4069
4
      // If there is a va_start in this function, make a backup copy of
4070
4
      // va_arg_tls somewhere in the function entry block.
4071
4
      VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4072
4
      IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
4073
4
    }
4074
18
4075
18
    // Instrument va_start.
4076
18
    // Copy va_list shadow from the backup copy of the TLS contents.
4077
22
    for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; 
i++4
) {
4078
4
      CallInst *OrigInst = VAStartInstrumentationList[i];
4079
4
      IRBuilder<> IRB(OrigInst->getNextNode());
4080
4
      Value *VAListTag = OrigInst->getArgOperand(0);
4081
4
      Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4082
4
      Value *RegSaveAreaPtrPtr =
4083
4
          IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4084
4
                             PointerType::get(RegSaveAreaPtrTy, 0));
4085
4
      Value *RegSaveAreaPtr =
4086
4
          IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
4087
4
      Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4088
4
      unsigned Alignment = 8;
4089
4
      std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4090
4
          MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4091
4
                                 Alignment, /*isStore*/ true);
4092
4
      IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4093
4
                       CopySize);
4094
4
    }
4095
18
  }
4096
};
4097
4098
/// AArch64-specific implementation of VarArgHelper.
4099
struct VarArgAArch64Helper : public VarArgHelper {
4100
  static const unsigned kAArch64GrArgSize = 64;
4101
  static const unsigned kAArch64VrArgSize = 128;
4102
4103
  static const unsigned AArch64GrBegOffset = 0;
4104
  static const unsigned AArch64GrEndOffset = kAArch64GrArgSize;
4105
  // Make VR space aligned to 16 bytes.
4106
  static const unsigned AArch64VrBegOffset = AArch64GrEndOffset;
4107
  static const unsigned AArch64VrEndOffset = AArch64VrBegOffset
4108
                                             + kAArch64VrArgSize;
4109
  static const unsigned AArch64VAEndOffset = AArch64VrEndOffset;
4110
4111
  Function &F;
4112
  MemorySanitizer &MS;
4113
  MemorySanitizerVisitor &MSV;
4114
  Value *VAArgTLSCopy = nullptr;
4115
  Value *VAArgOverflowSize = nullptr;
4116
4117
  SmallVector<CallInst*, 16> VAStartInstrumentationList;
4118
4119
  enum ArgKind { AK_GeneralPurpose, AK_FloatingPoint, AK_Memory };
4120
4121
  VarArgAArch64Helper(Function &F, MemorySanitizer &MS,
4122
8
                    MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
4123
4124
266
  ArgKind classifyArgument(Value* arg) {
4125
266
    Type *T = arg->getType();
4126
266
    if (T->isFPOrFPVectorTy())
4127
6
      return AK_FloatingPoint;
4128
260
    if ((T->isIntegerTy() && T->getPrimitiveSizeInBits() <= 64)
4129
260
        || 
(T->isPointerTy())0
)
4130
260
      return AK_GeneralPurpose;
4131
0
    return AK_Memory;
4132
0
  }
4133
4134
  // The instrumentation stores the argument shadow in a non ABI-specific
4135
  // format because it does not know which argument is named (since Clang,
4136
  // like x86_64 case, lowers the va_args in the frontend and this pass only
4137
  // sees the low level code that deals with va_list internals).
4138
  // The first seven GR registers are saved in the first 56 bytes of the
4139
  // va_arg tls arra, followers by the first 8 FP/SIMD registers, and then
4140
  // the remaining arguments.
4141
  // Using constant offset within the va_arg TLS array allows fast copy
4142
  // in the finalize instrumentation.
4143
4
  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
4144
4
    unsigned GrOffset = AArch64GrBegOffset;
4145
4
    unsigned VrOffset = AArch64VrBegOffset;
4146
4
    unsigned OverflowOffset = AArch64VAEndOffset;
4147
4
4148
4
    const DataLayout &DL = F.getParent()->getDataLayout();
4149
4
    for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
4150
270
         ArgIt != End; 
++ArgIt266
) {
4151
266
      Value *A = *ArgIt;
4152
266
      unsigned ArgNo = CS.getArgumentNo(ArgIt);
4153
266
      bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
4154
266
      ArgKind AK = classifyArgument(A);
4155
266
      if (AK == AK_GeneralPurpose && 
GrOffset >= AArch64GrEndOffset260
)
4156
228
        AK = AK_Memory;
4157
266
      if (AK == AK_FloatingPoint && 
VrOffset >= AArch64VrEndOffset6
)
4158
0
        AK = AK_Memory;
4159
266
      Value *Base;
4160
266
      switch (AK) {
4161
266
        case AK_GeneralPurpose:
4162
32
          Base = getShadowPtrForVAArgument(A->getType(), IRB, GrOffset, 8);
4163
32
          GrOffset += 8;
4164
32
          break;
4165
266
        case AK_FloatingPoint:
4166
6
          Base = getShadowPtrForVAArgument(A->getType(), IRB, VrOffset, 8);
4167
6
          VrOffset += 16;
4168
6
          break;
4169
266
        case AK_Memory:
4170
228
          // Don't count fixed arguments in the overflow area - va_start will
4171
228
          // skip right over them.
4172
228
          if (IsFixed)
4173
0
            continue;
4174
228
          uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4175
228
          Base = getShadowPtrForVAArgument(A->getType(), IRB, OverflowOffset,
4176
228
                                           alignTo(ArgSize, 8));
4177
228
          OverflowOffset += alignTo(ArgSize, 8);
4178
228
          break;
4179
266
      }
4180
266
      // Count Gp/Vr fixed arguments to their respective offsets, but don't
4181
266
      // bother to actually store a shadow.
4182
266
      if (IsFixed)
4183
4
        continue;
4184
262
      if (!Base)
4185
74
        continue;
4186
188
      IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4187
188
    }
4188
4
    Constant *OverflowSize =
4189
4
      ConstantInt::get(IRB.getInt64Ty(), OverflowOffset - AArch64VAEndOffset);
4190
4
    IRB.CreateStore(OverflowSize, MS.VAArgOverflowSizeTLS);
4191
4
  }
4192
4193
  /// Compute the shadow address for a given va_arg.
4194
  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4195
266
                                   unsigned ArgOffset, unsigned ArgSize) {
4196
266
    // Make sure we don't overflow __msan_va_arg_tls.
4197
266
    if (ArgOffset + ArgSize > kParamTLSSize)
4198
74
      return nullptr;
4199
192
    Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4200
192
    Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4201
192
    return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4202
192
                              "_msarg");
4203
192
  }
4204
4205
2
  void visitVAStartInst(VAStartInst &I) override {
4206
2
    IRBuilder<> IRB(&I);
4207
2
    VAStartInstrumentationList.push_back(&I);
4208
2
    Value *VAListTag = I.getArgOperand(0);
4209
2
    Value *ShadowPtr, *OriginPtr;
4210
2
    unsigned Alignment = 8;
4211
2
    std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4212
2
        VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4213
2
    IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4214
2
                     /* size */ 32, Alignment, false);
4215
2
  }
4216
4217
0
  void visitVACopyInst(VACopyInst &I) override {
4218
0
    IRBuilder<> IRB(&I);
4219
0
    VAStartInstrumentationList.push_back(&I);
4220
0
    Value *VAListTag = I.getArgOperand(0);
4221
0
    Value *ShadowPtr, *OriginPtr;
4222
0
    unsigned Alignment = 8;
4223
0
    std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4224
0
        VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4225
0
    IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4226
0
                     /* size */ 32, Alignment, false);
4227
0
  }
4228
4229
  // Retrieve a va_list field of 'void*' size.
4230
6
  Value* getVAField64(IRBuilder<> &IRB, Value *VAListTag, int offset) {
4231
6
    Value *SaveAreaPtrPtr =
4232
6
      IRB.CreateIntToPtr(
4233
6
        IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4234
6
                      ConstantInt::get(MS.IntptrTy, offset)),
4235
6
        Type::getInt64PtrTy(*MS.C));
4236
6
    return IRB.CreateLoad(Type::getInt64Ty(*MS.C), SaveAreaPtrPtr);
4237
6
  }
4238
4239
  // Retrieve a va_list field of 'int' size.
4240
4
  Value* getVAField32(IRBuilder<> &IRB, Value *VAListTag, int offset) {
4241
4
    Value *SaveAreaPtr =
4242
4
      IRB.CreateIntToPtr(
4243
4
        IRB.CreateAdd(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4244
4
                      ConstantInt::get(MS.IntptrTy, offset)),
4245
4
        Type::getInt32PtrTy(*MS.C));
4246
4
    Value *SaveArea32 = IRB.CreateLoad(IRB.getInt32Ty(), SaveAreaPtr);
4247
4
    return IRB.CreateSExt(SaveArea32, MS.IntptrTy);
4248
4
  }
4249
4250
8
  void finalizeInstrumentation() override {
4251
8
    assert(!VAArgOverflowSize && !VAArgTLSCopy &&
4252
8
           "finalizeInstrumentation called twice");
4253
8
    if (!VAStartInstrumentationList.empty()) {
4254
2
      // If there is a va_start in this function, make a backup copy of
4255
2
      // va_arg_tls somewhere in the function entry block.
4256
2
      IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
4257
2
      VAArgOverflowSize =
4258
2
          IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4259
2
      Value *CopySize =
4260
2
        IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, AArch64VAEndOffset),
4261
2
                      VAArgOverflowSize);
4262
2
      VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4263
2
      IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
4264
2
    }
4265
8
4266
8
    Value *GrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64GrArgSize);
4267
8
    Value *VrArgSize = ConstantInt::get(MS.IntptrTy, kAArch64VrArgSize);
4268
8
4269
8
    // Instrument va_start, copy va_list shadow from the backup copy of
4270
8
    // the TLS contents.
4271
10
    for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; 
i++2
) {
4272
2
      CallInst *OrigInst = VAStartInstrumentationList[i];
4273
2
      IRBuilder<> IRB(OrigInst->getNextNode());
4274
2
4275
2
      Value *VAListTag = OrigInst->getArgOperand(0);
4276
2
4277
2
      // The variadic ABI for AArch64 creates two areas to save the incoming
4278
2
      // argument registers (one for 64-bit general register xn-x7 and another
4279
2
      // for 128-bit FP/SIMD vn-v7).
4280
2
      // We need then to propagate the shadow arguments on both regions
4281
2
      // 'va::__gr_top + va::__gr_offs' and 'va::__vr_top + va::__vr_offs'.
4282
2
      // The remaning arguments are saved on shadow for 'va::stack'.
4283
2
      // One caveat is it requires only to propagate the non-named arguments,
4284
2
      // however on the call site instrumentation 'all' the arguments are
4285
2
      // saved. So to copy the shadow values from the va_arg TLS array
4286
2
      // we need to adjust the offset for both GR and VR fields based on
4287
2
      // the __{gr,vr}_offs value (since they are stores based on incoming
4288
2
      // named arguments).
4289
2
4290
2
      // Read the stack pointer from the va_list.
4291
2
      Value *StackSaveAreaPtr = getVAField64(IRB, VAListTag, 0);
4292
2
4293
2
      // Read both the __gr_top and __gr_off and add them up.
4294
2
      Value *GrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 8);
4295
2
      Value *GrOffSaveArea = getVAField32(IRB, VAListTag, 24);
4296
2
4297
2
      Value *GrRegSaveAreaPtr = IRB.CreateAdd(GrTopSaveAreaPtr, GrOffSaveArea);
4298
2
4299
2
      // Read both the __vr_top and __vr_off and add them up.
4300
2
      Value *VrTopSaveAreaPtr = getVAField64(IRB, VAListTag, 16);
4301
2
      Value *VrOffSaveArea = getVAField32(IRB, VAListTag, 28);
4302
2
4303
2
      Value *VrRegSaveAreaPtr = IRB.CreateAdd(VrTopSaveAreaPtr, VrOffSaveArea);
4304
2
4305
2
      // It does not know how many named arguments is being used and, on the
4306
2
      // callsite all the arguments were saved.  Since __gr_off is defined as
4307
2
      // '0 - ((8 - named_gr) * 8)', the idea is to just propagate the variadic
4308
2
      // argument by ignoring the bytes of shadow from named arguments.
4309
2
      Value *GrRegSaveAreaShadowPtrOff =
4310
2
        IRB.CreateAdd(GrArgSize, GrOffSaveArea);
4311
2
4312
2
      Value *GrRegSaveAreaShadowPtr =
4313
2
          MSV.getShadowOriginPtr(GrRegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4314
2
                                 /*Alignment*/ 8, /*isStore*/ true)
4315
2
              .first;
4316
2
4317
2
      Value *GrSrcPtr = IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4318
2
                                              GrRegSaveAreaShadowPtrOff);
4319
2
      Value *GrCopySize = IRB.CreateSub(GrArgSize, GrRegSaveAreaShadowPtrOff);
4320
2
4321
2
      IRB.CreateMemCpy(GrRegSaveAreaShadowPtr, 8, GrSrcPtr, 8, GrCopySize);
4322
2
4323
2
      // Again, but for FP/SIMD values.
4324
2
      Value *VrRegSaveAreaShadowPtrOff =
4325
2
          IRB.CreateAdd(VrArgSize, VrOffSaveArea);
4326
2
4327
2
      Value *VrRegSaveAreaShadowPtr =
4328
2
          MSV.getShadowOriginPtr(VrRegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4329
2
                                 /*Alignment*/ 8, /*isStore*/ true)
4330
2
              .first;
4331
2
4332
2
      Value *VrSrcPtr = IRB.CreateInBoundsGEP(
4333
2
        IRB.getInt8Ty(),
4334
2
        IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4335
2
                              IRB.getInt32(AArch64VrBegOffset)),
4336
2
        VrRegSaveAreaShadowPtrOff);
4337
2
      Value *VrCopySize = IRB.CreateSub(VrArgSize, VrRegSaveAreaShadowPtrOff);
4338
2
4339
2
      IRB.CreateMemCpy(VrRegSaveAreaShadowPtr, 8, VrSrcPtr, 8, VrCopySize);
4340
2
4341
2
      // And finally for remaining arguments.
4342
2
      Value *StackSaveAreaShadowPtr =
4343
2
          MSV.getShadowOriginPtr(StackSaveAreaPtr, IRB, IRB.getInt8Ty(),
4344
2
                                 /*Alignment*/ 16, /*isStore*/ true)
4345
2
              .first;
4346
2
4347
2
      Value *StackSrcPtr =
4348
2
        IRB.CreateInBoundsGEP(IRB.getInt8Ty(), VAArgTLSCopy,
4349
2
                              IRB.getInt32(AArch64VAEndOffset));
4350
2
4351
2
      IRB.CreateMemCpy(StackSaveAreaShadowPtr, 16, StackSrcPtr, 16,
4352
2
                       VAArgOverflowSize);
4353
2
    }
4354
8
  }
4355
};
4356
4357
/// PowerPC64-specific implementation of VarArgHelper.
4358
struct VarArgPowerPC64Helper : public VarArgHelper {
4359
  Function &F;
4360
  MemorySanitizer &MS;
4361
  MemorySanitizerVisitor &MSV;
4362
  Value *VAArgTLSCopy = nullptr;
4363
  Value *VAArgSize = nullptr;
4364
4365
  SmallVector<CallInst*, 16> VAStartInstrumentationList;
4366
4367
  VarArgPowerPC64Helper(Function &F, MemorySanitizer &MS,
4368
36
                    MemorySanitizerVisitor &MSV) : F(F), MS(MS), MSV(MSV) {}
4369
4370
30
  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {
4371
30
    // For PowerPC, we need to deal with alignment of stack arguments -
4372
30
    // they are mostly aligned to 8 bytes, but vectors and i128 arrays
4373
30
    // are aligned to 16 bytes, byvals can be aligned to 8 or 16 bytes,
4374
30
    // and QPX vectors are aligned to 32 bytes.  For that reason, we
4375
30
    // compute current offset from stack pointer (which is always properly
4376
30
    // aligned), and offset for the first vararg, then subtract them.
4377
30
    unsigned VAArgBase;
4378
30
    Triple TargetTriple(F.getParent()->getTargetTriple());
4379
30
    // Parameter save area starts at 48 bytes from frame pointer for ABIv1,
4380
30
    // and 32 bytes for ABIv2.  This is usually determined by target
4381
30
    // endianness, but in theory could be overriden by function attribute.
4382
30
    // For simplicity, we ignore it here (it'd only matter for QPX vectors).
4383
30
    if (TargetTriple.getArch() == Triple::ppc64)
4384
16
      VAArgBase = 48;
4385
14
    else
4386
14
      VAArgBase = 32;
4387
30
    unsigned VAArgOffset = VAArgBase;
4388
30
    const DataLayout &DL = F.getParent()->getDataLayout();
4389
30
    for (CallSite::arg_iterator ArgIt = CS.arg_begin(), End = CS.arg_end();
4390
578
         ArgIt != End; 
++ArgIt548
) {
4391
548
      Value *A = *ArgIt;
4392
548
      unsigned ArgNo = CS.getArgumentNo(ArgIt);
4393
548
      bool IsFixed = ArgNo < CS.getFunctionType()->getNumParams();
4394
548
      bool IsByVal = CS.paramHasAttr(ArgNo, Attribute::ByVal);
4395
548
      if (IsByVal) {
4396
8
        assert(A->getType()->isPointerTy());
4397
8
        Type *RealTy = A->getType()->getPointerElementType();
4398
8
        uint64_t ArgSize = DL.getTypeAllocSize(RealTy);
4399
8
        uint64_t ArgAlign = CS.getParamAlignment(ArgNo);
4400
8
        if (ArgAlign < 8)
4401
0
          ArgAlign = 8;
4402
8
        VAArgOffset = alignTo(VAArgOffset, ArgAlign);
4403
8
        if (!IsFixed) {
4404
8
          Value *Base = getShadowPtrForVAArgument(
4405
8
              RealTy, IRB, VAArgOffset - VAArgBase, ArgSize);
4406
8
          if (Base) {
4407
8
            Value *AShadowPtr, *AOriginPtr;
4408
8
            std::tie(AShadowPtr, AOriginPtr) =
4409
8
                MSV.getShadowOriginPtr(A, IRB, IRB.getInt8Ty(),
4410
8
                                       kShadowTLSAlignment, /*isStore*/ false);
4411
8
4412
8
            IRB.CreateMemCpy(Base, kShadowTLSAlignment, AShadowPtr,
4413
8
                             kShadowTLSAlignment, ArgSize);
4414
8
          }
4415
8
        }
4416
8
        VAArgOffset += alignTo(ArgSize, 8);
4417
540
      } else {
4418
540
        Value *Base;
4419
540
        uint64_t ArgSize = DL.getTypeAllocSize(A->getType());
4420
540
        uint64_t ArgAlign = 8;
4421
540
        if (A->getType()->isArrayTy()) {
4422
8
          // Arrays are aligned to element size, except for long double
4423
8
          // arrays, which are aligned to 8 bytes.
4424
8
          Type *ElementTy = A->getType()->getArrayElementType();
4425
8
          if (!ElementTy->isPPC_FP128Ty())
4426
8
            ArgAlign = DL.getTypeAllocSize(ElementTy);
4427
532
        } else if (A->getType()->isVectorTy()) {
4428
6
          // Vectors are naturally aligned.
4429
6
          ArgAlign = DL.getTypeAllocSize(A->getType());
4430
6
        }
4431
540
        if (ArgAlign < 8)
4432
0
          ArgAlign = 8;
4433
540
        VAArgOffset = alignTo(VAArgOffset, ArgAlign);
4434
540
        if (DL.isBigEndian()) {
4435
274
          // Adjusting the shadow for argument with size < 8 to match the placement
4436
274
          // of bits in big endian system
4437
274
          if (ArgSize < 8)
4438
20
            VAArgOffset += (8 - ArgSize);
4439
274
        }
4440
540
        if (!IsFixed) {
4441
510
          Base = getShadowPtrForVAArgument(A->getType(), IRB,
4442
510
                                           VAArgOffset - VAArgBase, ArgSize);
4443
510
          if (Base)
4444
430
            IRB.CreateAlignedStore(MSV.getShadow(A), Base, kShadowTLSAlignment);
4445
510
        }
4446
540
        VAArgOffset += ArgSize;
4447
540
        VAArgOffset = alignTo(VAArgOffset, 8);
4448
540
      }
4449
548
      if (IsFixed)
4450
30
        VAArgBase = VAArgOffset;
4451
548
    }
4452
30
4453
30
    Constant *TotalVAArgSize = ConstantInt::get(IRB.getInt64Ty(),
4454
30
                                                VAArgOffset - VAArgBase);
4455
30
    // Here using VAArgOverflowSizeTLS as VAArgSizeTLS to avoid creation of
4456
30
    // a new class member i.e. it is the total size of all VarArgs.
4457
30
    IRB.CreateStore(TotalVAArgSize, MS.VAArgOverflowSizeTLS);
4458
30
  }
4459
4460
  /// Compute the shadow address for a given va_arg.
4461
  Value *getShadowPtrForVAArgument(Type *Ty, IRBuilder<> &IRB,
4462
518
                                   unsigned ArgOffset, unsigned ArgSize) {
4463
518
    // Make sure we don't overflow __msan_va_arg_tls.
4464
518
    if (ArgOffset + ArgSize > kParamTLSSize)
4465
80
      return nullptr;
4466
438
    Value *Base = IRB.CreatePointerCast(MS.VAArgTLS, MS.IntptrTy);
4467
438
    Base = IRB.CreateAdd(Base, ConstantInt::get(MS.IntptrTy, ArgOffset));
4468
438
    return IRB.CreateIntToPtr(Base, PointerType::get(MSV.getShadowTy(Ty), 0),
4469
438
                              "_msarg");
4470
438
  }
4471
4472
4
  void visitVAStartInst(VAStartInst &I) override {
4473
4
    IRBuilder<> IRB(&I);
4474
4
    VAStartInstrumentationList.push_back(&I);
4475
4
    Value *VAListTag = I.getArgOperand(0);
4476
4
    Value *ShadowPtr, *OriginPtr;
4477
4
    unsigned Alignment = 8;
4478
4
    std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4479
4
        VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4480
4
    IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4481
4
                     /* size */ 8, Alignment, false);
4482
4
  }
4483
4484
0
  void visitVACopyInst(VACopyInst &I) override {
4485
0
    IRBuilder<> IRB(&I);
4486
0
    Value *VAListTag = I.getArgOperand(0);
4487
0
    Value *ShadowPtr, *OriginPtr;
4488
0
    unsigned Alignment = 8;
4489
0
    std::tie(ShadowPtr, OriginPtr) = MSV.getShadowOriginPtr(
4490
0
        VAListTag, IRB, IRB.getInt8Ty(), Alignment, /*isStore*/ true);
4491
0
    // Unpoison the whole __va_list_tag.
4492
0
    // FIXME: magic ABI constants.
4493
0
    IRB.CreateMemSet(ShadowPtr, Constant::getNullValue(IRB.getInt8Ty()),
4494
0
                     /* size */ 8, Alignment, false);
4495
0
  }
4496
4497
36
  void finalizeInstrumentation() override {
4498
36
    assert(!VAArgSize && !VAArgTLSCopy &&
4499
36
           "finalizeInstrumentation called twice");
4500
36
    IRBuilder<> IRB(MSV.ActualFnStart->getFirstNonPHI());
4501
36
    VAArgSize = IRB.CreateLoad(IRB.getInt64Ty(), MS.VAArgOverflowSizeTLS);
4502
36
    Value *CopySize = IRB.CreateAdd(ConstantInt::get(MS.IntptrTy, 0),
4503
36
                                    VAArgSize);
4504
36
4505
36
    if (!VAStartInstrumentationList.empty()) {
4506
4
      // If there is a va_start in this function, make a backup copy of
4507
4
      // va_arg_tls somewhere in the function entry block.
4508
4
      VAArgTLSCopy = IRB.CreateAlloca(Type::getInt8Ty(*MS.C), CopySize);
4509
4
      IRB.CreateMemCpy(VAArgTLSCopy, 8, MS.VAArgTLS, 8, CopySize);
4510
4
    }
4511
36
4512
36
    // Instrument va_start.
4513
36
    // Copy va_list shadow from the backup copy of the TLS contents.
4514
40
    for (size_t i = 0, n = VAStartInstrumentationList.size(); i < n; 
i++4
) {
4515
4
      CallInst *OrigInst = VAStartInstrumentationList[i];
4516
4
      IRBuilder<> IRB(OrigInst->getNextNode());
4517
4
      Value *VAListTag = OrigInst->getArgOperand(0);
4518
4
      Type *RegSaveAreaPtrTy = Type::getInt64PtrTy(*MS.C);
4519
4
      Value *RegSaveAreaPtrPtr =
4520
4
          IRB.CreateIntToPtr(IRB.CreatePtrToInt(VAListTag, MS.IntptrTy),
4521
4
                             PointerType::get(RegSaveAreaPtrTy, 0));
4522
4
      Value *RegSaveAreaPtr =
4523
4
          IRB.CreateLoad(RegSaveAreaPtrTy, RegSaveAreaPtrPtr);
4524
4
      Value *RegSaveAreaShadowPtr, *RegSaveAreaOriginPtr;
4525
4
      unsigned Alignment = 8;
4526
4
      std::tie(RegSaveAreaShadowPtr, RegSaveAreaOriginPtr) =
4527
4
          MSV.getShadowOriginPtr(RegSaveAreaPtr, IRB, IRB.getInt8Ty(),
4528
4
                                 Alignment, /*isStore*/ true);
4529
4
      IRB.CreateMemCpy(RegSaveAreaShadowPtr, Alignment, VAArgTLSCopy, Alignment,
4530
4
                       CopySize);
4531
4
    }
4532
36
  }
4533
};
4534
4535
/// A no-op implementation of VarArgHelper.
4536
struct VarArgNoOpHelper : public VarArgHelper {
4537
  VarArgNoOpHelper(Function &F, MemorySanitizer &MS,
4538
2
                   MemorySanitizerVisitor &MSV) {}
4539
4540
0
  void visitCallSite(CallSite &CS, IRBuilder<> &IRB) override {}
4541
4542
2
  void visitVAStartInst(VAStartInst &I) override {}
4543
4544
0
  void visitVACopyInst(VACopyInst &I) override {}
4545
4546
2
  void finalizeInstrumentation() override {}
4547
};
4548
4549
} // end anonymous namespace
4550
4551
static VarArgHelper *CreateVarArgHelper(Function &Func, MemorySanitizer &Msan,
4552
908
                                        MemorySanitizerVisitor &Visitor) {
4553
908
  // VarArg handling is only implemented on AMD64. False positives are possible
4554
908
  // on other platforms.
4555
908
  Triple TargetTriple(Func.getParent()->getTargetTriple());
4556
908
  if (TargetTriple.getArch() == Triple::x86_64)
4557
844
    return new VarArgAMD64Helper(Func, Msan, Visitor);
4558
64
  else if (TargetTriple.isMIPS64())
4559
18
    return new VarArgMIPS64Helper(Func, Msan, Visitor);
4560
46
  else if (TargetTriple.getArch() == Triple::aarch64)
4561
8
    return new VarArgAArch64Helper(Func, Msan, Visitor);
4562
38
  else if (TargetTriple.getArch() == Triple::ppc64 ||
4563
38
           
TargetTriple.getArch() == Triple::ppc64le19
)
4564
36
    return new VarArgPowerPC64Helper(Func, Msan, Visitor);
4565
2
  else
4566
2
    return new VarArgNoOpHelper(Func, Msan, Visitor);
4567
908
}
4568
4569
1.05k
bool MemorySanitizer::sanitizeFunction(Function &F, TargetLibraryInfo &TLI) {
4570
1.05k
  if (!CompileKernel && 
(&F == MsanCtorFunction)946
)
4571
145
    return false;
4572
908
  MemorySanitizerVisitor Visitor(F, *this, TLI);
4573
908
4574
908
  // Clear out readonly/readnone attributes.
4575
908
  AttrBuilder B;
4576
908
  B.addAttribute(Attribute::ReadOnly)
4577
908
    .addAttribute(Attribute::ReadNone);
4578
908
  F.removeAttributes(AttributeList::FunctionIndex, B);
4579
908
4580
908
  return Visitor.runOnFunction();
4581
908
}