hardware testing

Sandi Metz's RailsConf 2013 talk offers practical guidelines for deciding what to test and how, so tests stay cheap and stable. She argues for testing the public interface rather than implementation and using mocks appropriately.

Will Wilson explains autonomous testing: using deterministic simulation and fuzzing, building on FoundationDB's approach, to test software continuously. It sets out the ideas behind his later company Antithesis.

John Hughes, co-creator of QuickCheck, talks about property-based testing of hard systems. He shows production examples where it found very low-level bugs, including issues in the CAN bus standard.

A 36C3 talk by Bunnie Huang arguing that open source is necessary but not sufficient for trustworthy hardware, because verifying hardware is destructive and so creates a time-of-check/time-of-use gap. It surveys supply chain security and classes of hardware attacks, and presents the Betrusted approach to narrowing that gap.

Hillel Wayne's PyCon 2018 talk on testing beyond pytest and mypy. It introduces property-based testing with the Hypothesis library and design-by-contract with dpcontracts, and shows how to combine them with an existing test suite.

Timothy Roscoe of ETH Zurich delivers the joint USENIX ATC/OSDI 2021 keynote arguing that operating systems must reckon with modern hardware, which is a heterogeneous collection of processors, firmware and coprocessors rather than a simple machine. It shows how much runs below and beside Linux.

Alen Ladavac of Croteam talks about continuous integration and automated testing pipelines in games, with case studies from The Talos Principle and Serious Sam.

Ted Young of LightStep proposes trace testing, using distributed tracing data to unify testing and observability. The KubeCon-era talk argues for testing against trace data rather than treating tests and monitoring separately.

Cliff Click of Azul Systems gives a Google Tech Talk on running Java business applications on a machine with about a thousand cores. He covers hardware and software codesign, including garbage collection hardware and the JIT.

Joe Grand recounts being hired to hack a Trezor One hardware wallet and recover $2 million of cryptocurrency. He describes three months of experimentation, voltage glitching, failures and high-stakes attempts with only one chance to get it right.

Christopher Domas's Black Hat talk demonstrates a hardware backdoor in a third-party x86 processor, a hidden core that grants root from userland. He uses it to discuss how feasible broader hardware backdoors might be.

Will Wilson of FoundationDB explains how the team tested their distributed database using deterministic simulation. By running the whole system in a simulator with injected faults and reproducible seeds, they could find rare concurrency and network bugs.

Gary Bernhardt's RubyConf 2012 talk on testing in isolation without mocking everything. It goes through behavior versus data, mutation versus immutability, and how a functional core with an imperative shell yields natural isolation and parallelism.

Microsoft's Tony Chen explains how the Xbox One was designed to resist physical attacks, including secure boot, encrypted memory, a security processor and a hypervisor. The goal was to make any hardware attack cost more than the price of many games.

Andrew "bunnie" Huang examines how realistic hardware supply chain attacks are, from rogue insiders to nation states, in the wake of the Bloomberg Businessweek report. He explains how undocumented components could be hidden on boards and how hard they are to spot.

Deviant Ollam, head of a physical penetration testing team, presents tactics for getting into buildings, from bypassing door levers with a wire to defeating bank doors. He argues that physical security is often surprisingly weak.

Christopher Domas presents research at DEF CON 26 showing a hidden alternate instruction set and undocumented hardware backdoor in some x86 processors, found on a third-party CPU. He uses it to ask how feasible wider hardware backdoors are.

Bill Dally of NVIDIA gives a Hot Chips 2023 keynote on the challenges of building hardware for deep learning. He traces how GPUs have been adapted for machine learning, especially training.

Raymond Hettinger's PyBay 2018 keynote on preventing, finding and fixing bugs under a time budget. He covers testing practice, type annotation tradeoffs and the limits of coverage metrics.

Michael Steil's 33c3 talk explains essentially everything about the Game Boy: CPU, video hardware, peripherals, and the tricks used by games and demos. It revives the era of counting clock cycles around hardware limits.

J. B. Rainsberger argues that large integrated tests are a scam: they are slow, fail opaquely, and give a false sense of coverage. He advocates focused collaboration and contract tests at boundaries instead.

Tony Chen of Microsoft explains at BlueHat Seattle 2019 how the Xbox One was designed to resist physical attacks. He covers the security design and hypervisor-based OS architecture that kept the console uncracked for years.

Misko Hevery's Clean Code Talk shows how to use object-oriented techniques such as polymorphism to remove repeated if and switch statements. The result is code that is easier to test, understand and maintain.

A 35C3 talk presenting a proof-of-concept SPI bus hardware implant with capabilities like those alleged in the Bloomberg Supermicro story, plus countermeasures for detecting such modchips.

Cliff Click walks through a small performance example on a modern out-of-order CPU. He covers pipelining, ILP, CISC vs RISC, multicore, and why cache misses and the memory subsystem dominate performance.

An AWS re:Invent 2018 session diving into the Nitro system, a set of hardware offload and security building blocks behind next-generation EC2 instances. It covers the design and architecture and how Nitro enables new instance types.

Tom Murphy (suckerpinch) shows 'reverse emulation', using a Raspberry Pi in an NES cartridge to give an unmodified NES new abilities such as playing SNES games. The video starts in a strange way and gets stranger.

Ian Cooper's DevTernity talk revisits Kent Beck's original TDD ideas and shows where misunderstandings crept in. It argues tests should target behaviors through public interfaces rather than implementation details, so suites support refactoring instead of resisting it.

Casey Muratori's long talk arguing that modern software has ballooned to tens of millions of lines because of hardware and OS diversity. He proposes a stable ISA for entire system-on-a-chip packages and recalls when games shipped with their own boot disk.

Jason Turner's CppCon 2016 talk builds a simple Commodore 64 game using C++17, showing how constexpr and const let a modern compiler produce tight 6502 code. He walks through assembly output to show abstractions being optimized away.

Gil Tene of Azul Systems explains common pitfalls in measuring latency and response time, including why averages mislead and the problem of coordinated omission. He introduces open source tools like HdrHistogram for better measurement and reporting.

SQLite creator Richard Hipp gives a history of SQLite and explains where it shines as a database at the edge of the network with no DBA. He covers long-term support, testing and how it is used in products.

A DEF CON 21 talk by Adam Laurie and Zac Franken on decapping chips, covering how to remove chip packaging and image the die to read out contents such as ROM.

Chris Lattner's ASPLOS 2021 keynote on the golden age of compiler design in an era of hardware and software co-design, covering compiler infrastructure for accelerators and unifying hardware IRs.

Deviant Ollam shows that most physical attacks on doors bypass the lock, with photo and video examples of how doors, frames and hardware get defeated. It is a physical security talk from Shakacon.

Simon Knowles, co-founder and CTO of Graphcore, talks about "intelligence processors", chips designed specifically for machine learning. The talk was given at the RAAIS 2017 summit and covers Graphcore's architecture.

A Ropecon 2019 talk by a lighting designer and a software engineer on how the Odysseus larp spaceship was built, using over ten IT systems, hundreds of lights and dozens of speakers. It covers the lessons learned.

James Hamilton, AWS VP and distinguished engineer, gives an insider view of the innovations behind the AWS cloud at re:Invent 2014. He covers custom hardware, networking, regions and availability zones, and infrastructure at scale.

A Thoughtworks conversation among Martin Fowler, Kent Beck and David Heinemeier Hansson on whether TDD is dead. They discuss the tradeoffs of test-driven development, mocking and where TDD helps or hurts design.

Michael Abrash of Valve explains what affordable VR hardware will be capable of within a couple of years, and describes a prototype showing what that level of hardware can do.