This is what the measurement should have been, and the finding that came out when I did it properly — which is more useful than the one I thought I had.

The thing I was about to publish

Four CloudFormation templates with a Rules: block. Assertions on the parameters: no $Latest for a version pin, no empty certificate ARN, no SSH from 0.0.0.0/0.

I fed them ten deliberately invalid parameter sets. All ten came back clean — create-stack returned success, exit 0, every time. Conclusion: Rules: is a Service Catalog feature that does nothing in plain CloudFormation, and anyone relying on it has no guardrails.

That conclusion was wrong.

What the measurement should have looked like

Six cases, each one there to kill a specific alternative explanation. The fixture is a single AWS::CloudFormation::WaitConditionHandle — no VPC, no compute, nothing billable.

And from describe-stack-events on A:

Parameter validation failed: assertion error: For a production environment,
the instance type must be t3.large. Rollback requested by user.

That is my own AssertDescription, read back to me. The guardrail fired. It fired in every case that violated it and in none that didn’t.

Each control is there for a reason. C proves the rollback came from the rule and not from something else in the template. E kills “your RuleCondition evaluated false so the assertion never ran.” F kills “your YAML shorthand tags didn’t parse.” And D is the one that matters most — it’s the instrument check, an AllowedValues constraint that is known to be enforced. If D had also been accepted, my harness couldn’t observe a refusal at all and none of the other rows would mean anything.

My original ten-probe run had no D. That’s the whole story.

The actual finding

CloudFormation has two parameter-validation mechanisms with completely different observable signatures, and nothing in the docs puts them side by side:

So this line is a lie for half of CloudFormation’s own validation features:

aws cloudformation create-stack ... && echo "guardrails passed"

create-stack returning success means CloudFormation accepted your request. For a Rules: assertion the evaluation happens after that, during stack creation, and the failure surfaces as a rollback. If your pipeline checks $? and moves on, a rejected stack looks exactly like an accepted one — and if you’re checking whether a guardrail works, an accepted stack looks exactly like a guardrail that never fired.

That’s the trap, and it’s worse than a guardrail that doesn’t exist. A missing guardrail fails open and you find out. This one fails closed and reports open.

Two things I’d do differently

Every probe needs a case that must fail. Ten inputs that all came back the same told me nothing, and I read it as a finding because the finding was interesting. One AllowedValues control would have shown me my harness could see refusals — which would have made the ten silent results a question rather than an answer.

“The command succeeded” and “the thing I wanted happened” are different observations. I know this. I’d written it down. I checked $? anyway, because checking $? is what your fingers do.

Worth adding: when I first doubted the result, I went looking for documentation and found an AWS sample repo that contradicted me. I nearly settled it there. That would have been right by luck — a doc search is not a measurement, and the experiment took four minutes and cost nothing. If the question is empirical and the fixture is cheap, build the fixture.

The bits that held up

Measured the same way, with controls that discriminated:

• AllowedValues / AllowedPattern / MinLength are enforced synchronously. AllowedPattern also works on CommaDelimitedList — an empty member is refused with “Each value of parameter … failed to satisfy constraint.”
• cfn-lint returned exit 0 on a template that aws cloudformation validate-template rejects outright — Fn::Join isn’t legal inside Rules. The linter is not the oracle. The service is.
• Legal intrinsics in Rules: And, Contains, EachMemberEquals, EachMemberIn, Equals, If, Not, Or, RefAll, ValueOf, ValueOfAll, Ref. Not Fn::Join, Fn::Sub, Fn::Select.

If you want synchronous rejection, use parameter constraints — failing at the API beats creating a stack and rolling it back. But do it because the failure mode is better, not because Rules: doesn’t work. It works. I just couldn’t see it.