• Contents
• Introduction
• The “Birthday Problem”
• Attack #8: The “Birthday Problem” attack
• Defense #8: Widen the nonces
• Other defenses
  • The problem with “oracles”
• Updated threat model
• Conclusion

Introduction

When we last left the unlocking process, things seemed pretty secure: the car generates a totally random nonce (“number used only once”) which the attacker couldn’t possibly have guessed ahead of time. The fob then has to provide the proper MAC value of this nonce, computed using the unlock_key, which is unknown to an attacker. What could go wrong?

Ah, famous last words. In this article, we’ll discover yet another type of replay attack based on something called the “birthday problem” and the fact that even PRNGs repeat values WAY more often than you’d think they would.

The “Birthday Problem”

Let’s take a closer look at the phrase “totally random nonce.” In reality, this isn’t quite true. Sure, an attacker can’t guess what nonce values are coming next. But the nonce itself has a predetermined size, so there are only so many nonces that could be generated before our PRNG has to emit a duplicate. If our car repeats a nonce and an attacker happened to save the fob’s MAC response for that first nonce, then we’re back in replay territory!

All an attacker has to do is collect enough (nonce, response) pairs for the car to repeat a nonce and then BOOM: they send back the recorded response and they can get the car to unlock.

Two examples of an attacker who could collect these (nonce, response) pairs are

1. a valet who is holding a person’s key fob while they’re eating dinner or
2. an attacker who places a listening device outside a person’s house to capture unlocks every time that person gets into their car over the course of a year.

In both cases, the attacker records enough valid unlock responses that when they return to the car and try to unlock it themselves, they have a good chance of knowing the correct MAC response without needing to know the unlock key.

“But c’mon! Nonces are currently 32 bits wide, giving over 4.3 billion possible values. How likely is it, really, for an attacker to find a repeated nonce?”

Let’s find out! First, let’s assume that an attacker can trigger an unlock and store the (nonce, response) pair every 3.4 ms¹. This means that if an attacker had control of a fob for a mere 30 minutes (like a valet might) they could potentially trigger 530,973 unlocks. We want to find out how long it would take that attacker to later break into that car by requesting unlocks until the car challenged them with one of the 530,973 nonces they recorded earlier.

The general statement for this problem is “Given a set of T values from a distribution of size N, what’s the probability that in M random draws from that distribution at least one of them matches a value from T?” It’s known as the “birthday problem” since another example of this exact same question is “Assume you have a group of T people with different birthdays and you ask M random strangers for their birthdays. What are the chances that at least one of those M people shares a birthday with someone in the group of T people?”² (in this case, “dates in a year” forms the pool of “N” values, N being 365).

The formula for this probability is \(P(M) = 1 - ( 1 - {T \over N})^M\)³, which can be approximated as \(P(M) \approx 1 - e^{-TM/N}\). Weirdly, M doesn’t need to be that big to have a really good chance of finding a match! If you took a group of 20 people with different birthdays, it would only take about 12 random strangers to have a 50/50 chance of finding someone who shared a birthday with a person from the original group!

In a group of 20 people, it would only take about 12 random strangers to have a 50% chance of finding someone who shared a birthday with one of the people from the group. Image from https://picryl.com/media/crowd-human-silhouettes-6333fd.

❗ Test it out for yourself!

Generate a set of 20 random values from 1 to 365. How long does it feel like it will take to get a repeated value (regardless of what the math says)?

Now generate random values until you get one that matches one of the original 20 values. How many random numbers did you need to generate to find that match? What was the probability of finding a match when you did?

In the case of our key fob, \(T = 530,973\) and \(N = 2^{32} = 4,294,967,296\). Let’s say the attacker had another 30 minutes to try to get into the car using that table, during which time they could request another 942,408 unlocks (our value for M), since each failed unlock attempt would only take 1.91 ms⁴.

❓ What’s the probability?

So, what’s the probability that our attacker finds a repeated nonce within that window? Calculate it before looking at my answer below!

⮮ ANSWER BELOW ⮯

Let’s see, \(P(M) = 1 - ( 1 - {T \over N})^M = 1 - (1 - {530973 \over 4294967296})^{942408} = 1\).

Woah, it’s basically guaranteed?! I’m afraid so. In fact, it gets even worse. We can rearrange the “birthday problem” formula to calculate approximately how many unlocks (M) we’d need to have a 50% chance of a repeated value, and that formula is \(M = {\log (1-P) \over \log (1-T/N)}\) . For our 4.3 billion nonces and 530,973 table size, an attacker would only need to trigger about 5,606 unlocks before they had a 50% chance of seeing a repeated value, which they could do with our system in as little as 11 seconds.

❓ What about the second scenario?

In the second scenario above, an attacker placed a listening device outside a person’s house to capture each of their unlocks over the course of a year.

What’s a reasonable assumption for how many unlocks that attacker might capture in that year (i.e. the size of T)?

How long would it take an attacker who had that table to unlock a car using this attack? (Answer⁵)

Attack #8: The “Birthday Problem” attack

This is exactly the threat model described by Car #2 in the competition, in which an attacker (such as a valet or anyone who “borrows” your keys) only has temporary access to your fob.

The security test I wrote performs exactly the attack just described, using getBoardMsgLog to simulate eavesdropping on each unlock transaction. It collects a set number of (nonce, response) pairs (configurable on the command-line with --oracle-table-size; default: 65535) then triggers unlocks until it finds a nonce that matches one from the table (or until --oracle-max-iter is reached; default: 0, meaning run indefinitely).

In simulation, this test only takes an astonishing 40 seconds to fail.

Defense #8: Widen the nonces

So, apparently 4.3 billion possible nonce values isn’t enough. (Perhaps you were thinking exactly that way back in Part 7 when I made nonces only 32 bits long.) The easy fix is to make nonces longer, up to the 16 bytes that are generated by our PRNG each time we request a new random number.

// messages.h
#define NONCE_SIZE 16  // nonce is 16 bytes

If an attacker now attempts to unlock our car using their table of 530,973 stored (nonce, response) pairs, it will take them over 4.16 x 10²⁰ centuries (over 3 trillion times the age of our universe) to even have a 50% chance of success! (Note that the time per attempt went up from 1.91 ms to 2.95 ms, since nonce messages are now 18 bytes instead of 6.)

Our current test would try to corroborate this by running for at least 4.16 x 10¹⁷ centuries before finding a nonce collision. This is, clearly, terribly unreasonable. Although we can, and should, let this test run for long periods of time to substantiate that the attack is infeasible (I’ve run the actual “birthday problem” attack for longer than 8 hours with no repeated nonce values), it would be nice if we had a faster test to still check that we’ve implemented a reasonable defense against the attack. To that end, I’ve added another security test that just checks the length of the nonce values and calculates the probability of the attack using the equations above. This does not simulate the “birthday problem” attack, but it’s better than nothing for a quick security test. This test passes.

Lowering the nonce length back to 4 bytes and re-running the test also confirms that our earlier design would have failed this test.

Other defenses

Widening the nonces may have been the obvious solution, but there are others that we could consider. For instance, we could apply to this situation the same reasoning that was described in Part 8 about protecting the pairing pin from brute force attacks by:

• Making each attempt categorically more difficult: Requiring an attacker to provide a fingerprint scan alongside the unlock request or using a non-repeating PRNG algorithm and locking/erasing the device after 2³² total unlock attempts over the lifetime of the car.
• Making each individual attempt take longer: Enforcing a rate-limit of one unlock per second or having a 1 minute timeout after 3 unsuccessful unlock attempts.
• Decreasing the odds of any single guess being successful by widening the nonce (which we did above)

As with the pairing pin, though, the competition and its specific threat model eliminate many of these as viable options.

• We can’t change the competition rules (or the hardware!) to require a fingerprint scan or other additional requirement to make unlocking more difficult.
• Attackers can still reflash the hardware at will, making it meaningless to save counters or timeout values or to even have any sort of saved system state whatsoever (attackers in the competition could simply reflash the target with fresh firmware whenever they wanted, resetting any variables that were intended to be stored between power cycles).

Adding a roughly 1 second delay for each unlock request (the maximum time an unlock could take, per the competition rules) would definitely help, but it would be insufficient on its own to prevent the “birthday problem” attack.

❓ What about that “non-repeating PRNG algorithm” thing?

The problem with pseudo-random number generation is that every possible answer should have the same probability of showing up on any given step, giving rise to the “birthday problem.” An alternative to this form of random number generation might be to encrypt a monotonically increasing counter using something like AES, a technique that was brought up during our initial discussion about PRNGs in Part 7:

static uint32_t counter = 0;
uint32_t rand(void)
{
 // return upper 4 bytes of AES_CMAC(prng_key, counter++);
}

The counter would increment on each unlock, ensuring that the output from AES (the random number) would be different on each unlock and, critically, non-repeating⁶. (At least until the counter rolled over from 2³²-1 to 0, at which point the car could just permanently disable itself, preventing any further unlocks. This number could be made high enough to never be reached during normal operation.)

This would seem to solve our “birthday problem” attack, since there’s no birthday problem to contend with! Unfortunately, it wouldn’t survive the attackers in this competition. Can you explain why? (Answer⁷)

The problem with “oracles”

Part of the reason the “birthday problem” attack works at all is that both the fob and the car act as oracles: devices that an attacker can query at will to learn some information about the system.

1. Attackers can send unlock messages to a car whenever they want to see which nonce message is sent back.
2. They can also trigger an unlock on the fob and then send the fob a made-up nonce message to see what response the fob sends back.

Right now, the “birthday problem” attack uses primarily the second fact above to generate its table of (nonce, response) pairs (though an attacker could also get that information by just observing valid unlock transactions, as was depicted originally) and then the first fact above during the actual attack to start an unlock sequence without the fob being present.

Ideally, neither the fob nor the car ever talks to a device without authenticating it as a valid car or fob, though. If the car had a way to verify that an unlock request came from an authentic fob, then it would cease to be an oracle, eliminating situation #1. And if the fob had a way to verify that the nonce messages it sees after sending an unlock request came from an authentic car, then it would cease to be an oracle in the sense of situation #2. Eliminating both the fob and car as oracles would categorically eliminate the “birthday problem” attack, not simply make it less likely to succeed.

I spent some time trying to think of ways to do that, however, and couldn’t find a solution that still offered any meaningful improvements after accounting for the fact that attackers in the competition could reflash a car or fob whenever they wanted.

For example, say we try to eliminate the car as an oracle by expanding the unlock message (which is back to being [ 0x56 | 0x6 | ‘unlock’ ] right now) to include a rolling counter and MAC value.

        Plaintext input                    MAC
  /—————————————————————————————\   /————————————————\
[ 0x56 | 0xB | 0x0C | 0xED | 0x0C | 0xA15B32390B17FF4C ]
   ^      ^    \——/   \—————————/
   |   Length  Fob ID   Counter
 UNLOCK_MAGIC

The car would refuse to send a nonce for any unlock message that didn’t have the right counter and MAC values.

❓ Can you spot the problem?

There’s a problem with the above design! What is it? (Answer⁸)

Fundamentally, systems that try to authenticate devices with a single message (static passwords, rolling codes + MAC, etc) end up being vulnerable to attacks that simply reflash the device and reset whatever state was supposed to be saved between unlock attempts. The challenge-response we developed in Part 7 fixed this, but at the cost of the car acting like an oracle, for which there isn’t a workaround: the car can’t withhold a challenge before a device has authenticated itself if it’s using that challenge to authenticate the other device!

Or perhaps we try to eliminate the fob as an oracle by requiring that nonce messages also be accompanied by a MAC value (using a new, unique key, separate from the unlock key, of course!), verifying that they came from a real car.

The fob would refuse to reply with its own MAC response (computed using the unlock key) unless it could validate the nonce MAC using its own “nonce key.”

This one is interesting, since it does prevent an attacker from querying a fob by itself to generate the table of (nonce, response) pairs. However, in both of the realistic attack scenarios that were mentioned above (a valet or an attacker who has installed a listening device outside a person’s home), this would have no effect on the attacker, since the fob is already interacting with an authentic car and the attacker is merely listening to that conversation.

All of which goes to show the importance of things like hardware monotonic counters (which can’t be reset to 0) and of designing your system so that an attacker can’t reset or reflash your device.

Updated threat model

Our updated threat model with this new attack (scoped just to show the relevant features) is below.

The primary pitfall (a plausibly wrong solution) in trying to defend against the “birthday problem” attack is thinking we could add a rolling code+MAC to the unlock message to close off the car as an oracle. This doesn’t work because it’s vulnerable to exactly the attacks we developed in Part 7: RollJam, Forced rollback, and Forced rollover.

Two additional pitfalls are the ideas that:

• A valid PRNG algorithm like the one we’re using won’t repeat any values (it will; that’s kind of the whole point of this article) and
• That it’s possible to prevent a nonce from being repeated by simply tracking in software which ones have already been sent out. This pitfall is vulnerable to devices being reflashed and having that history reset.

The only two alternatives to widening the nonce (defenses that would or could have worked as well as it) would have been to have:

• Added something to the unlock process like a fingerprint scan or requiring the fob to be within a certain distance of the car (such as in a passive keyless entry system, which uses RF signals to detect if a fob is near the car) or
• Replaced our PRNG with a non-repeating algorithm whose counter was stored in an anti-rollback counter.

Adding some sort of timeout after N unlock attempts within a certain small window might also have worked, had our threat model not included reflashing (i.e. had our design/the competition prevented that from happening).

In addition to widening the nonce, there are a few things that could be add-ons to the design for additional security or better features.

• Adding a 0.75 or 1 sec delay to each unlock (the maximum length of time the rules say unlocking is allowed to take) would make the “birthday problem” attack take a few hundred times longer than it would otherwise.
• Adding a MAC to the nonce message would prevent an attacker from querying a fob directly for response values, forcing them to obtain that data via a live unlock session between a valid fob and car.
• Had the hardware/rules allowed it, logging, locking, or erasing the device upon a rapid sequence of unlock requests or failed attempts could have made this attack much harder or even categorically infeasible.

Conclusion

In this article we discovered that 4.3 billion isn’t actually that big of a number and there’s a real chance that a random number generated from that pool will repeat often enough to facilitate another replay attack! The simple solution was to generate nonces from a much larger pool (up to 2¹²⁸, instead of 2³²), making repeated values much less likely to occur. Along the way, we discussed the idea of an “oracle”, including what it is and why it can’t always be avoided.

If you’ve made it this far, thanks for reading and happy hacking!

• Contents
• Introduction
• “Enable feature”: A practical security exercise
  • Design review of the “feature” feature
  • Security analysis of “features”
  • A possible attack and its defense (Attack/Defense #6)
• The “blast radius” problem again
• In the driver’s seat
• Updated threat model
• Conclusion
• Attacking and defending “start” messages

Introduction

In the last article we shifted our attention away from the unlocking process (which would seem to have required some rather sophisticated attacks after Part 7) to the pairing process, which was vulnerable to some very simple brute force and timing attacks. The last thing left to secure for the competition is the process of packaging and enabling a feature.

As with pairing, this code is woefully insecure at the moment. But the good news is that we already have all the skills we need to make it secure! In this article, I’ll ask you to apply what you’ve learned so far in this series to do exactly that, walking you through my own solution. Before we close, we’ll also get to talk about per-device keys and asymmetric cryptography.

“Enable feature”: A practical security exercise

Let’s put those security skills you’ve been developing in this series to the test! Can you protect the “enable feature” aspect of our system from exploitation?

We’ll tackle this challenge in three steps:

1. Reviewing how features are packaged, formatted, and enabled
2. Identifying the security requirements our system needs
3. Demonstrating a successful attack on the system (by exploiting a security requirement which is lacking enforcement) and then implementing a defense against that attack

Each step is composed of several questions which I hope you take the time to answer (even if it’s just briefly and in your own head!) before I provide you with my answer.

Design review of the “feature” feature

❓ How are features packaged?

Before a feature can be enabled, it must be “packaged” into a feature file. What is the shell command that packages a feature for our system? Who is responsible for executing that command? How were these features distributed to the teams in the competition?

Hint: You can find this information in either the repo README, HHGH (Part 2): On-boarding, and/or the competition rules.

⮮ ANSWER BELOW ⮯

Features are packaged by executing the following command in the terminal.

# Packages Feature 1 for Car 1234
./tools/package.py --id 1234 --num 1

In the competition, the package.py tool also had read/write access to any “host secrets” (e.g. secrets.json).

Features were packaged at the “factory” (i.e. by the MITRE competition organizers) and were distributed to attacking teams alongside the encrypted firmware binaries for each of the car scenarios (except for Car #5; more on this in just a bit).

❓ What is the format of a feature file?

What does a feature file actually look like? What is actually being saved to disk or sent over the UART port when a feature is being packaged or enabled?

Hint: You can find this information in either the repo README, HHGH (Part 1), or by reviewing the code.

⮮ ANSWER BELOW ⮯

Feature files are simply binary files composed of the car ID and a feature number.

 FEATURE PACKET:
   ┌──────────────────┬───────────────────┐
   │Car ID (11 bytes) │Feature # (1 byte) │
   └──────────────────┴───────────────────┘

❓ How are features enabled on a fob?

What initiated the process of enabling a feature? What steps were taken by the fob before that feature was possibly enabled?

Hint: You can find this information in either the repo README or HHGH (Part 1).

⮮ ANSWER BELOW ⮯

Features were enabled on a paired fob by sending it the ASCII message enable <feature>\n to the HOST UART port, where <feature> is the Base16-encoded contents of the binary feature file. (Base16 encoding converts binary or hex values like 0b1001 1111 (0x9F) into ASCII “9F” (0x39 0x46), which makes it easier to read and debug later on when looking at a transcript of what was sent to a fob.)

An unpaired fob would reject the message straightaway. A paired fob would check that the feature file was valid (length was greater than or equal to the expected value, car ID in the feature file matched its own car ID, feature number within range) and that it could be enabled (feature list wasn’t full, feature wasn’t already enabled) before enabling it, sending back OK or ERROR (with an error message) as appropriate.

Security analysis of “features”

Now that we have a thorough understanding of the “enable feature” process, we can begin to analyze what security requirements must be present to meet the competition rules/our design goals.

❓ What flags are at stake?

What flags could an attacking team capture by abusing the “enable feature” part of our system? What things should our system prevent a malicious person from doing?

⮮ ANSWER BELOW ⮯

In the MITRE eCTF competition, teams could capture a flag if they could cause Car #5 to emit its flag for “Feature 2” (which wasn’t given to the attacking teams; see the graphic above of the attack scenarios) and which was only supposed to be possible if the paired fob for Car #5 had Feature 2 legitimately enabled.

More broadly, features should only ever be made for owners who’ve paid for them, car owners should not be able to enable a feature without a valid feature file from the manufacturer, and they should also not be able to enable a feature on their car using a valid feature file that was created by the manufacturer for a different car (these are, essentially, security requirements #5 and #6 from the competition rules).

In a real production system, you may also want to ensure that feature files are unique to certain drivers or that feature files eventually “expire” or are “one-time use”.

❓ What security requirements do we need?

In Part 5 we listed 5 important security principles: confidentiality, integrity, availability, authentication, and non-repudiation. Which, specifically, do we need and might possibly be missing from our current implementation? (Hint: there may be more than one!)

⮮ ANSWER BELOW ⮯

If a fob needs to enforce that a feature file which it has received has come from none other than the manufacturer themselves, then it needs to authenticate that feature file. Additionally, if it’s important that a valid feature file not be modified by an attacker to enable a feature on a different car or to enable a different feature on the same car, then the fob needs a way to verify the integrity of each feature file.

A possible attack and its defense (Attack/Defense #6)

Knowing what security requirements we need, let’s discuss how an attacker can exploit the fact that we’re not doing anything right now to enforce those security requirements and then show how to enforce those requirements and close off the attacks.

❓ What is an attack that could exploit our current system?

What is the easiest possible attack you can think of to capture the “Feature 2” flag from Car #5 by exploiting our current system?

⮮ ANSWER BELOW ⮯

The easiest attack that could be conducted against a feature file is forgery. Forging a feature file is quite simple, since the feature files are simply binary files composed of the car ID and a feature number.

 FEATURE PACKET:
   ┌──────────────────┬───────────────────┐
   │Car ID (11 bytes) │Feature # (1 byte) │
   └──────────────────┴───────────────────┘

We can easily modify either field to be whatever we want, allowing us to enable any feature on any car.

We’ll call this “Attack #6: Forging a feature file”.

❓ What would a security test for this attack look like?

How would you write the test to go in test_security.py that implements this attack? Write it out in Python (or in pseudocode) and then run it to make sure that it fails, as expected!

⮮ ANSWER BELOW ⮯

The failing test that I wrote forges a feature file by performing the following actions.

1. Pull the data off a fob (to ensure feature 2 isn’t enabled yet),
2. Package feature 1,
3. Create a forged feature 2 file by modifying the “feature number” field, and
4. Attempt to enable the forged feature file.

As expected, this test currently fails.

❓ How can we defend against that attack?

What’s a way we can prevent an attacker from enabling a forged feature file?

Hint: We’ve already talked about how a car can authenticate an unlock message from a fob (Parts 5-7); how could you adapt or modify that solution to the problem of authenticating a feature file?

⮮ ANSWER BELOW ⮯

The current problem is that an attacker is allowed to modify a feature file without a fob being able to detect it; they can easily forge any feature file they want. Any solution we come up with has to be able to detect this forgery and reject those forged messages.

Actually, we’ve already seen the best defense for this attack: computing a MAC value over the feature file and appending that to each feature packet.

The most important property that a MAC algorithm actually has is “resistance to forgery”, which provides both authenticity and integrity: an attacker could neither create their own valid feature message nor could they modify a valid feature message and have it remain valid. In either case, they wouldn’t be able to compute the correct MAC value without knowing the secret key.

Previously we used MAC values to authenticate the unlock messages from a paired fob. In that case, the MAC value gave us integrity in addition to authenticity, though integrity wasn’t necessarily a requirement of the design. The only thing an attacker could do by modifying an unlock message was have it rejected by the car.

Our new feature file will have a MAC value appended to the end, computed over the car ID and feature number using a new “feature key” (which every fob will get at build-time). The new format looks like this:

 FEATURE PKT:
   ┌──────────────────┬───────────────────┬────────────────────┐
   │Car ID (11 bytes) │Feature # (1 byte) │MAC value (8 bytes) │
   └──────────────────┴───────────────────┴────────────────────┘

❓ Why not use the unlock key?

You might be asking, at this point, “There’s already a secret key on the fob; why can’t we use that?”

The short answer is that we could, but it’s considered bad practice. Much like toothbrushes, everybody should get their own; meaning, it’s best to only use each secret key for a single purpose. Although outside the scope of this series, it’s possible for an attacker to gain information about a key if it’s reused for multiple purposes. And if they were able to compute the key value, then they would have access to both forging a new feature file and impersonating a fob during the unlock process. If, instead, we use separate keys then the damage isn’t as bad if they happen to correctly compute one of them. This is known as key separation.

An alternative to storing unique keys on the fob for each unique purpose is to store one key and derive any other keys from it (key derivation). For instance, a fob could get a single fob_key and then compute the unlock and feature keys on-the-fly using the HKDF algorithm. HKDF is composed of two operations: Extract and Expand, the first of which can be skipped if the seed material is uniform (as would be the case if it were the output from a CSPRNG).

https://www.researchgate.net/figure/Extract-then-expand-model-for-KDFs_fig2_287478235

If keys are being derived from a fob_key, then fob_key should never be used in any other context (to ensure proper key separation).

The fob will test that this MAC value matches the one it computes over the first part of the message using its feature key before going on to check any of the other fields, failing early if it doesn’t match.

Since an attacker doesn’t have access to the feature key, they can’t recompute a new MAC for a modified message nor can they construct new messages. Even a one-bit change to either the car ID or the feature number causes the MACs to mismatch, which the fob will then reject.

We’ll call this “Defense #6: Authenticating feature files with MACs”.

After implementing this defense, our test now passes!

There’s a subtlety to this defense, though; a vulnerability we’ve seen before and need to make sure we keep closed in our new defense.

❓ What key property does the MAC comparison need to have?

In other words, what attack is left open by the naive code below?

uint8_t computed_mac[16] = {0};
AES_CMAC_digest(&feature_cmac_ctx,
                (uint8_t*)enable_message,
                offsetof(ENABLE_PACKET, mac),
                computed_mac);
if( memcmp(&computed_mac[8], enable_message->mac, 8) != 0 )
{
    sendError("bad MAC");
    return;
}

Hint: The pairing pin suffered this attack until we fixed it!

⮮ ANSWER BELOW ⮯

When comparing MACs, don’t forget to use a constant-time memcmp (i.e. memcmp_ct), like we did in Part 8 when comparing pairing pins. Otherwise you’ll leave yourself open to a timing attack!

❗ Attack it!

Write a test to conduct a timing attack on the naive code above and run it to prove that it’s possible. Then switch out memcmp for memcmp_ct and prove that the code is no longer susceptible to a timing attack on the feature file’s MAC value. (One possible answer)

The “blast radius” problem again

Although our code is secure at this point, there are two modifications we could make that would help limit the negative effects of an attacker learning the value of “feature key” in the future. This is the “blast radius” problem that was mentioned in Part 8. If an attacker is able to extract the “feature key” that’s stored on every fob then they can have full control over the entire feature file verification system: they can forge whatever feature file they want for as long as that key remains in use. This would be a catastrophic failure of security, so instead we might use one of the techniques below (in addition to locking the debug port and doing our best to ensure that an attacker can’t actually extract that “feature key”).

First, we could use a per-device feature key, as opposed to a company-wide one like we’re using now. In this system, each paired fob would get a feature key that’s unique to a single car ID, similar to how each paired fob gets an unlock key that’s unique to each car ID. If an attacker ever extracts that feature key, the most damage they can do is forge feature files for that specific car, but not for every car we’ve ever produced.

The downside is that our design gets a little more complex. Our company database (i.e. secrets.json in our codebase) would need to keep track of the feature key that’s been associated with each car ID. Additionally, this key would need to be transmitted to an unpaired fob that was being paired as part of the PAIR_PACKET it gets sent.

If we wanted to reduce the blast radius even further, we could replace our MAC algorithm with a digital signature, which uses asymmetric cryptography, a.k.a. public-key cryptography. Asymmetric cryptography uses two keys, a “public” and a “private” key, as opposed to symmetric cryptography, which uses just one key. (We’ve been using symmetric cryptographic algorithms this whole time, as evidenced by the fact that our unlock and feature keys needed to be shared by all devices computing the same MAC values: one key on all devices.)

In asymmetric cryptography the private key is kept secret from everybody except the person or device holding the key while the public key can be distributed to anyone. The weird and neat thing about asymmetric cryptography is that you can’t “double dip”: if the public key is used to encrypt a message then only the private key can be used to decrypt it; trying to decrypt it with the public key just yields garbledy-gook. Similarly, if a message is signed with a private key then only the matching public key can verify it.

That’s the essence of a digital signature: an entity signs a message with its private key, which anyone else can then verify using that same entity’s public key. If the verify operation returns true, then it had to have been that actual entity that did the signing in the first place (since no one else could produce a valid signature to pass the verify operation without knowing the value of the private key). What’s more, there’s no shared secret that could get leaked if an attacker extracts a device’s keys; the only key that’s actually on the device is already public knowledge.

Using this system, we would generate a public/private key pair during build time and put the public key inside every fob. The private key would be used to sign any feature files, which the fobs could then verify. This system has the smallest blast radius (it’s essentially none), since there’s no key on the fob that could be extracted to allow an attacker the ability to forge a feature file.

In the driver’s seat

Now I’m going to step back even further, by letting you run through that same process to identify where else the feature data is vulnerable, to describe and mount the attack, and then to design and implement an appropriate defense.

To clarify, yes, there is one other place in the current code where an attacker could force a car to divulge its feature message despite not having that feature enabled on the fob. But where? (Take a second to consider your answer before moving on!)

Sending a feature file to a fob is only the first of two places where the feature data can be attacked. The second is when it’s being transmitted as part of a start message after a car has been successfully unlocked!

Each start message contains the fob’s list of enabled features, and these are sent to the car in plaintext, with no additional security features whatsoever.

The process of attacking and securing this exchange is the same as what we’ve just walked through, even if the exact attack and defense steps are slightly different. I’ll leave it to you to complete that process for this part of the design, all the way from “What’s a start message and how is it used?” to “What’s the easiest way to attack this and force Car #5 to spit out the Feature #2 flag?” and “How to protect against that attack?” If you get stuck, you can follow along with my explanation at the end of this article. Good luck!

Updated threat model

SPOILER ALERT: This section and the conclusion assume you’ve worked through the process of securing the start messages, the vulnerability identified in the last section.

The updated threat model (scoped just to show the latest attacks) is depicted below.

A few things worth noting that weren’t previously discussed:

• One pitfall to watch out for is using an algorithm that provides integrity but not authentication. An example would be a hash, like SHA256 or MD5.

  These algorithms could help a fob or car detect accidental errors in a feature file or start message, but they wouldn’t stop an attacker from changing a value and then computing a new hash (these are public algorithms, after all!).

• An alternative to adding a MAC value to each feature file or start message is to encrypt them using an authenticated encryption algorithm like AES-GCM. This would be only moderately more complicated than computing a MAC and would add “confidentiality” to our feature enabling process, but that’s not necessarily something we care about.
• The Python secrets module could be used to generate the keys instead of randbytes (previously mentioned when discussing how unlock keys were generated in Part 6).
• We could remove the explicit chain of error messages in enableFeature. If an attacker is able to somehow generate a valid MAC, then the rest of this function is set up to inform them exactly which part of their feature file (if any) is invalid for this specific fob. This is another example of a “side-channel leakage” (introduced in Part 8) and although it may not be exploitable today, it’s a thing to keep your eye on.
• Compiler optimizations could still remove the “constant-time” feature of our constant-time memcmp, memcmp_ct.
• An added delay during each enabling attempt would further help to defend against timing attacks or any other attack that relied on repeatedly and quickly attempting to enable a new feature.

Conclusion

In this article, we applied the security principles we’ve been developing in this series to the process of enabling new features on a fob.

Upon analyzing the firmware, it became clear that attackers could easily forge their own feature files and start messages, representing a lack of integrity and authentication in our current system. As the primary feature of a MAC (message authentication code) is “resistance to forgery”, appending one both to each feature file and to each start message cleanly solved the problem of forged messages. But don’t forget to make sure that you compare the MAC values using a constant-time memcmp! Otherwise you’ll leave yourself open to a timing attack. At this point, an attacker must find a way to defeat our MAC algorithm (an exceedingly hard thing to do) to be able to forge a feature file or a start message.

If you’ve made it this far, thanks for reading and happy hacking!

Attacking and defending “start” messages

We’re going to start this process of security analysis over again, walking gradually toward yet another important defense. As before, I want to emphasize that we’ve already discussed all of the tools you’d need to do this on your own and I want to encourage you to answer each question to yourself (even just in your head) before reading my answers.

Let’s start at the top.

❓ What is the format of a start message?

What does a start message actually look like? What is actually being sent over the UART port when a fob is telling a car which features have been enabled?

Hint: You can find this information in either the repo README, HHGH (Part 1), or by reviewing the code.

A start message consists of 17 bytes in TLV (tag-length-value) format:

                 Tag Len
       (START_MAGIC)  │
                  │   │
                  ▼   ▼
               ┌────┬────┬────────────────────────────┐
START MSG:     │0x57│0x0F│  Feature info (15 bytes)   │
               └────┴────┴─────────────┬──────────────┘
                                       │
                                       ▼
┌───────────────────┬───────────────────────────┬───────────────────────────┐
│ Car ID (11 bytes) │# active features (1 byte) │List of features (3 bytes) │
└───────────────────┴───────────────────────────┴───────────────────────────┘

❓ How are active features sent from a fob to a car?

When are features sent to the car? What steps are taken by the car before it prints out the associated feature flags?

Hint: You can find this information in either the repo README or HHGH (Part 1).

The fob sends active features to the car after it receives the ACK SUCCESS message from the car during the unlocking process.

The car checks only that the ID in the start message matches its own and that each active feature number is within a valid range (1-3 for us) before sending out the associated flag over its HOST UART port.

The same flag is at stake as before: the Feature 2 flag from Car #5 (which, again, doesn’t have a valid Feature 2 file for it to be enabled legitimately).

❓ What security requirements do we need?

Which of the five (confidentiality, integrity, availability, authentication, and non-repudiation) do we need and might possibly be missing from our current implementation? (Hint: there may be more than one!)

A car needs to know that the feature info it’s getting from a fob has come from a valid paired fob and was untampered-with before it got to the car. This means we need authentication and integrity.

❓ What is an attack that could exploit our current system?

What, now, is the easiest possible attack you can think of to capture the “Feature 2” flag from Car #5 by exploiting our current system?

Hint: It specifically targets the start message.

The easiest attack that could be conducted against a start message is still forgery, though forging a start message is a little more complicated than forging a feature file. The challenge is in the timing: we need Car #5’s paired fob to begin the unlock process, getting as far as issuing a correct response to the car’s nonce challenge, and then we need to send our own start message, instead of the one the fob wants to send.

This is a type of attack called “man-in-the-middle” (MITM), since the attacker literally sits in between the two devices trying to communicate, with the ability to modify the messages that are being passed back and forth.

Modifying (i.e. forging) a start message in this scenario is as easy as it was to forge a feature file (before we added the MAC, that is): all we have to do is change two binary values in the start message (the number of active features and the feature or features which is/are active). This means that this attack would allow us to, once again, enable any feature on any car.

We’ll call this “Attack #7: Forging a start message”.

❓ What would a security test for this attack look like?

How would you write the test to go in test_security.py that implements this attack? Write it out in Python (or in pseudocode) and then run it to make sure that it fails, as expected!

The biggest challenge facing us in implementing this attack is that our current hardware setup has no way to “insert” an attacker in between a fob and a car to modify any start messages; they’re directly connected to each other over UART. We don’t have a way to conduct a real MITM-type attack.

So it’s simulation to the rescue! To avoid needing to include a real “MITM” in our hardware setup, I’ve opted to create a new test command called setStartMsg, which will store a binary start message in the fob to be sent out in place of its normal start message the next time the fob tries to unlock a car.

1. Test triggers unlock and reads car message log to get the start message that was sent by the fob.
2. Test modifies the start message to enable feature #2 and sets this as the stored start message on the fob.
3. Test triggers unlock.

The failing test that I wrote first triggers an unlock and then extracts the last start message from the board message log (simulating a MITM attacker who can see a valid start message being transmitted by a fob). It then modifies the start message so that feature #2 is enabled and uses setStartMsg to save that modified message to the fob. Then it triggers an unlock, during which the fob uses the modified message.

To verify that the test currently fails, I’ve also added a car test command called getFeatures which lists the features that were sent over by a paired fob during the last valid unlock attempt.

❓ How can we defend against that attack?

What’s a way we can prevent an attacker from enabling a forged start message?

Hint: We’ve already talked about how a car can authenticate an unlock message from a fob (Parts 5-7); how could you adapt or modify that solution to the problem of authenticating a start message?

Given the similarity of each of these attacks on the feature data, it will hopefully come as no surprise to learn that the easiest defense against this attack is to add a MAC to the end of each start message, which the car will also verify before it sends out any flags. If an attacker modifies even a single bit of the start message sent by the paired fob (or tries to forge their own), they won’t be able to correctly compute the MAC value without the “start message” key.

We’ll call this “Defense #7: Authenticating start messages with MACs”.

Implementing this requires creating a new “start message” key to go into each fob/car (which will also need to be added to the pair packet when pairing a new fob) and adding one more conditional check during the car’s unlock process. Don’t forget to use memcmp_ct!

With challenge-response and a properly seeded PRNG in place, the unlock process from Part 7 would take something like breaking AES-CMAC to defeat. So Part 8 does what a real attacker would do: it goes looking for the weakest part of the system that gives the same payoff. That’s the pairing process. A paired fob hands over the car ID, the unlock key, and the pairing pin to anyone who sends it pair <pin> with the right 6-hex-digit pin. There are only about 16.7 million possible pins, and at roughly 2.5 ms per guess a script can brute force its way through all of them in under 12 hours (about 6 on average). The payoff is the unlock key itself.

Brute force defenses come in three flavors: make each attempt harder (a fingerprint scan, or erasing the device after too many failures), make each attempt slower, or make any single guess less likely to succeed (a longer pin). The competition rules fix the pin format, and an attacker who can reset or re-flash the fob can wipe out any counter of failed attempts, so the defense here is a fixed delay added to every pairing attempt, no counter needed. A user never notices it, but it pushes the worst-case brute force time from 11.7 hours to 194 days.

That defense assumes a wrong guess tells the attacker nothing except that it’s wrong. It doesn’t. memcmp bails out at the first mismatched byte, so the error message comes back a few nanoseconds later for each leading digit the guess got right. Consistent differences that small are still measurable, and they turn the problem into a game of Wordle: guess all 16 values of the first digit, keep the slow one, repeat for the next digit. That’s at most 96 guesses instead of 16.7 million, or an hour and a half even at a full minute per guess. This is a timing attack, one kind of side-channel attack, in which a device leaks information through how long it takes, how much power it draws, or even what its error messages say.

The fix is a constant-time comparison that XORs every byte pair together and checks the accumulated result only at the end, so the run time no longer depends on where the pins differ. Because C has no way to express “constant-time,” you have to check the generated assembly to make sure the compiler kept it that way. Comparing MACs of the pins would also work, since the avalanche effect makes a byte-by-byte timing leak useless to an attacker. The article also covers two tempting fixes that fall short. A random delay only adds noise, which averaging over many guesses (or watching the power trace) removes. Encrypting the pairing packet requires a key shared by every fob, and in this competition that key sits in a binary every attacker already has.

Read the full article →

This one covers the same territory as my Make Your Own MCU Boards talk, but live and on camera: how little you actually need to bring up a microcontroller. The chip in question is an STM32C071KBT3 — a 32-pin QFP part — paired with a breakout board from Adafruit. Solder the chip down, add header pins, and you’re most of the way to a working circuit.

What’s left is the same list of five things every MCU needs, no matter the vendor:

1. Power — digital supply pins tied to the rails, with decoupling capacitors as a nice-to-have rather than a requirement. Separate analog power pins can share the digital rail too, at the cost of a noisier ADC.
2. Reset — the STM32C0 has an internal pull-up on NRST, so the pin needs nothing extra. A two-pin breadboard-mount button gives you a manual reset if you want one.
3. Clocking — the internal RC oscillator is enough for most projects. An external clock only earns its keep when you need precision timing, like USB or a real-time clock, and a packaged oscillator (rather than a hand-tuned crystal circuit) is the more reliable way to add one.
4. Programming — an ST-Link V2 debug adapter supplies SWDIO, SWCLK, and reset directly; no extra circuitry needed.
5. Specialty pins — BOOT0 controls where the STM32C0 starts executing after reset, and it’s disabled by default, so it’s another pin you can leave alone.

Add power, wire up SWD, and drop in a quick CubeMX Blinky project, and that’s a working STM32 for only a few dollars in new parts — cheaper if you use a less expensive STM32C0 variant and make your own breakout boards.

Related

• Getting started with STM32C0 MCU hardware development (AN5673) — ST application note describing the minimal circuit for an STM32C0 in great detail.
• Make Your Own MCU Boards (Teardown 2023) — the talk this video’s premise is drawn from, with more on sourcing, layout, and assembly
• STM32F103C8T6 breakout board (GitHub) — my own minimal MCU board design, as a reference

Every device that someone can physically get their hands on is a hardware security problem, whether or not it has a radio. The question worth asking isn’t “if” someone wants to get their hands on your device for malicious purposes — it’s “who are they, and what could they do once they had it?” The first question is specific to your product; the second is answerable once you know how attackers actually approach hardware, which is what this talk works through.

Six bins, roughly by cost and invasiveness

1. Research — datasheets, third-party teardowns, extracted firmware, even social engineering against support staff. The goal isn’t just “how does this work” but “where did the designers make an assumption they never actually enforced” — every attack downstream starts by invalidating one of those.
2. Passive observation and logical attacks — probing exposed ports and internal signals, checking whether the debug port is locked, brute-forcing a short password, replaying a captured transaction, or just fuzzing the UART with garbage. The 2010 PlayStation 3 hack started here: watching signals to the console’s memory revealed a free() operation that could be glitched, resulting in a full compromise.
3. Physical modification — cutting a trace, swapping a strap resistor to change what hardware revision the firmware thinks it’s running on, injecting a false signal onto an analog sensor line, or pushing the device outside its rated environmental range to see what breaks.
4. Side-channel attacks — extracting secrets from what a device leaks just by operating, without any modification at all. Timing is the easiest to grasp: a naive strcmp-style password check returns faster on the first mismatched byte, which turns a billion-guess brute force into 36 guesses per digit (a real, shipped “secure” external hard drive was broken exactly this way through its error LED). Power analysis goes further — simple power analysis can read key bits directly off a single trace for algorithms like textbook RSA, and differential power analysis uses difference-of-means statistics across thousands of traces to recover an AES key one byte at a time, without ever attacking AES itself.
5. Fault injection — deliberately pushing a device outside its operating envelope to induce a skipped instruction, a corrupted fetch, or a flipped memory value: a crowbar glitch on the supply rail, clock double-pulsing, EM pulses, or a laser on a decapped die. This is exactly how the Xbox 360 was broken in 2011 — the second-stage bootloader’s kernel hash check was a single conditional, and glitching past that one branch let attackers load an unsigned kernel.
6. IC decapping and FIB editing — for a sufficiently determined, resourced attacker, exposing the silicon opens up optical side-channels, laser fault injection, and literal circuit edits with a focused ion beam. Most devices don’t need to defend against this tier — and part of good security engineering is deciding, deliberately, that they don’t.

The takeaway

None of these attacks require a state-sponsored budget, and none of them are hypothetical — every technique in this talk has a real production device it’s been used against. The point isn’t to induce despair; it’s to work backward from “what am I actually protecting, from whom, and via which of these six avenues” and spend defensive effort accordingly. Sometimes that’s as simple as locking a debug port. Sometimes it’s a lot more.

Additional Resources

• Nothing is Safe (JawnCon 0x2) — the same attack spectrum talk, with a full recording and a live DPA/fault injection demo on a ChipWhisperer Nano
• All Your Bytes are Belong to Us — the written companion on DigiKey/Maker.io, covering the same six levels in more depth
• How Hardware Gets Hacked → — a multi-part series walking through a complete attack and defense, from start to finish

Session page →

Code and exercises for this workshop are on GitHub: nathancharlesjones/authenticated_exchanges.

Authentication is one of five foundational security principles — alongside confidentiality, integrity, availability, and non-repudiation — and it’s the one this workshop dug into: how do you prove a message or piece of data actually came from who it claims to have come from? The running example is a physical access control system: badges and a reader attached to a door. Each exercise breaks a specific assumption in the previous defense, and the fix that follows repairs it.

Five defenses, each closing off the last attack

• Baseline: the UID allowlist — A badge presents its ID number; the reader checks it against an allowlist and opens the door if it matches. Fine when an attacker can’t observe traffic or guess a UID — broken the moment one leaks from a log, a firmware image, or a public repo.
• Challenge-response — The reader sends a nonce and the badge computes a message authentication code (MAC) over it with a shared key, proving knowledge of a secret without ever transmitting it. This raises new questions of its own, like where a good nonce comes from (PRNG vs. TRNG).
• Per-device keys — A single MAC key shared fleet-wide means one stolen badge compromises everyone. Deriving a unique device key per badge — HMAC(fleet_key, UID) — caps the blast radius to one badge and makes key management (generation, storage, revocation) a first-class concern.
• Mutual authentication — Nothing so far stops a rogue reader from “authenticating” a badge and reading its private data. The reader now has to prove itself too, using asymmetric cryptography and digital signatures so no shared secret sits on the reader side waiting to be extracted.
• Session keys — Even after mutual authentication, an eavesdropper can still read the traffic that follows. A session key, derived the same way as the device key but tagged with an info string, encrypts post-handshake traffic without touching the long-lived device key or forward secrecy.

What’s still open

The workshop closes on the questions it deliberately left out: authorization (what a badge is allowed to do once authenticated), what happens when a reader’s public key isn’t known in advance, resource constraints on real hardware, and how to handle failures — logging, lockouts, and lengthening timeouts — without those becoming a denial-of-service vector of their own. Every defense here closes off one avenue while relying on a fresh assumption of its own; the interesting part of security engineering is knowing what those assumptions are and deciding, deliberately, which ones you’re willing to make.

Additional Resources

• How Hardware Gets Hacked (Part 6): Authenticated Exchanges — the eCTF key fob replaces its broken PRNG-based unlock with a rolling counter and AES-CMAC, the same MAC-based authentication idea applied to a real embedded target
• GitHub repo → — all five exercises, tests, and the worksheet

Session page →

Part 6 closed the door on naive replay with a rolling counter and an AES-CMAC. Part 7 shows the door was only half-shut: three more sophisticated attacks still let an attacker replay a stale message. RollJam intercepts every unlock attempt, forwards the second-most-recent message to the car while banking the newest one for later use. Forced rollback re-flashes the car’s firmware to reset its stored counter to zero, resurrecting an old captured message. Forced rollover just triggers enough unlocks (up to 65,536 for a 16-bit counter) to wrap the counter back around to a value an old message still satisfies. All three exploit the same gap: the scheme proves a message came from an authenticated fob at some point in the past, never that it was generated right now.

That distinction — between authenticity and freshness — matters differently depending on what’s being verified. A signed firmware update is fine no matter how old it is; the signature alone proves authenticity. A live “unlock” command isn’t, because an attacker watching the wire can capture and hold a genuinely valid message and nothing about it says when it happened. The fix is challenge-response: the car generates a random value (a “nonce,” a number used once) and asks the fob to compute a MAC over it. Since the fob can’t know the nonce until the car sends it, a correct response proves the fob is answering this conversation, not replaying pieces of a previous one.

That scheme is only as strong as the nonce generator behind it. A naive LCG-style rand() is out — Part 5 already showed one is invertible from three intercepted outputs. Instead, the article reuses AES-CMAC itself as a one-way PRNG (the same construction NIST recommends for CTR-DRBG), driven by a key kept separate from the unlock key through key separation. That still leaves one gap: if the PRNG always starts from the same internal state, an attacker who can reset the device sees the identical “random” sequence every time. Closing it means seeding the PRNG from actual hardware noise — RAM startup values, clock jitter, ADC readings on Vcc, temperature, and a floating pin — accumulated into an entropy pool and measured for real using NIST’s SP800-90B toolkit rather than assumed.

The article closes with a look at what else could get you there: TOTP (if fob and car can keep synchronized time), rate-limiting unlock attempts, a hardware anti-rollback counter, or an MCU with a built-in TRNG — each a different way of buying the same property the whole article is really about: proving a message is fresh, not just authentic.

Read the full article →

An interactive serial REPL is one of the fastest ways to prototype on an embedded system, inspect state, or tweak configuration without a debugger. The article puts four options for reaching one head-to-head on an STM32 Nucleo-F411: MicroPython, the Zephyr shell utility, Lua, and Forth — scored on how fast each reaches a working prompt (on supported hardware and on a brand-new device), how large the compiled binary is, how much slower than native C/C++ it runs, and how hard it is to call into (or be called from) existing C/C++ code.

MicroPython wins on capability — the full Python language plus its library ecosystem — but it’s also the heaviest: 277 kB compiled, 50–250x slower than native code, and porting it to an unsupported MCU can run from a few days to two months. The Zephyr shell sits at the opposite extreme: 44 kB, the fastest of the four since application code is just C, and trivial to interface with existing C/C++ — but it isn’t an interpreted language, so there’s no REPL in the “declare a variable and write a loop” sense. Lua (144 kB, 10–50x slower than native) lands in the middle: mature, genuinely interpreted, easy to embed, and consistently cheaper to port to a new MCU than either MicroPython or Zephyr. Forth is the outlier: only 25 kB and the smallest interpreted option by far, but its stack-based RPN syntax gets unwieldy fast once you’re juggling more than a couple of variables.

The article’s recommendation is Lua for most projects: capable enough for real prototyping, small enough for constrained devices, and — critically for anyone targeting more than one MCU — the one option where porting effort stays predictable instead of ballooning with each new device tier.

Read the full article →

Part 5 ended with the fob’s PRNG-based defense broken by simple algebra — a linear congruential generator turned out to be invertible from just three intercepted messages. Part 6 replaces it with real cryptography: a secret key generated at build time and embedded in both car and fob firmware, combined with a rolling counter that increments on every unlock attempt. Before transmitting, the fob runs its ID, the counter, and the unlock command through AES-CMAC with the shared key and appends the resulting MAC to the message.

The car doesn’t need to predict anything; it recomputes the same MAC over the received fields and checks that it matches, and that the counter falls within an acceptable window of the last value it saw. A replayed message fails the counter check. A forged message fails the MAC check, because producing a valid one requires the secret key, which never travels over the wire in the clear. That’s the difference between authenticity and mere obfuscation.

The article closes by turning the “what’s still broken?” question back on the reader: counter rollover, flash wear from writing the counter on every unlock, revoking a lost or stolen fob, and capping how many fobs can pair to one car all need real answers before this ships to production.

Read the full article →

Your brain doesn’t stop working on a problem just because you’ve stopped looking at the screen. The default mode network — the same circuitry that’s active when you’re washing dishes, walking, or showering — keeps consolidating and recombining what you already know, often surfacing the fix that hours of focused staring couldn’t find. That’s not procrastination; it’s how insight actually works.

Lin-Manuel Miranda put it well: “A good idea doesn’t come when you’re doing a million things. The good idea comes in the moment of rest.” Next time a bug won’t budge, step away from the keyboard for a few minutes — the fix might be waiting when you get back.

Read the full article →