Skip to content

Add mTLS app-to-app routing support (RFC draft)#4910

Draft
rkoster wants to merge 21 commits intomainfrom
feature/app-to-app-mtls-routing
Draft

Add mTLS app-to-app routing support (RFC draft)#4910
rkoster wants to merge 21 commits intomainfrom
feature/app-to-app-mtls-routing

Conversation

@rkoster
Copy link
Copy Markdown
Contributor

@rkoster rkoster commented Mar 5, 2026

Summary

Implements Part 2 (CF Identity & Authorization) of the RFC for Identity-Aware Routing for GoRouter. This adds a new Access Rules API that allows developers to control which Cloud Foundry apps can access routes on identity-aware domains.

Note: This PR is a draft because the RFC has not been approved yet.

What This Enables

Developers can now secure routes on identity-aware domains with platform-enforced access control:

# Create a route on an identity-aware domain
cf map-route my-backend apps.identity --hostname backend

# Allow specific apps to access it
cf add-access-rule apps.identity --source-app frontend-app --hostname backend
cf add-access-rule apps.identity --source-app monitoring-app --hostname backend

# List access rules
cf access-rules apps.identity --hostname backend

GoRouter enforces these rules automatically - no code changes needed in the backend app.

Implementation Summary

1. Access Rules API (RFC-Compliant)

New /v3/access_rules endpoints for managing route-level access control:

Method Path Description
GET /v3/access_rules List access rules with filtering
GET /v3/access_rules/:guid Get a single access rule
POST /v3/access_rules Create an access rule
PATCH /v3/access_rules/:guid Update metadata (labels/annotations only)
DELETE /v3/access_rules/:guid Delete an access rule

Query Parameters:

  • guids, route_guids, space_guids - Filter by resource GUIDs
  • selectors - Filter by exact selector strings
  • selector_resource_guids - Text-match against GUIDs in selectors (for stale rule detection)
  • include - Sideload related resources: route, app, space, organization, selector_resource

Example Request:

POST /v3/access_rules
{
  "selector": "cf:app:d76446a1-f429-4444-8797-be2f78b75b08",
  "metadata": {
    "labels": { "team": "payments" },
    "annotations": { "description": "Allow frontend to call payments API" }
  },
  "relationships": {
    "route": { "data": { "guid": "route-guid" } }
  }
}

Response includes:

  • Selector string (e.g., cf:app:<guid>, cf:space:<guid>, cf:org:<guid>, cf:any)
  • Read-only relationships extracted from selector (app, space, organization)
  • Metadata (labels and annotations)
  • Created/updated timestamps

2. Domain-Level Enforcement Configuration

Domains can now be configured with enforcement settings at creation time (immutable):

cf create-shared-domain apps.identity --enforce-access-rules --scope org

New Domain Fields:

  • enforce_access_rules (boolean) - When true, GoRouter enforces access rules for routes on this domain
  • access_rules_scope (string: any, org, space) - Operator-level boundary for allowed callers

These fields are set via POST /v3/domains and are not present on PATCH /v3/domains (immutable by design).

3. Validation Rules

Access rule creation enforces all RFC-specified validations:

  • ✅ Route's domain must have enforce_access_rules: true
  • ✅ Route's domain cannot be internal (internal routes bypass GoRouter)
  • cf:any cannot be combined with other selectors on the same route
  • ✅ Duplicate selectors per route are rejected
  • ✅ Selector GUIDs are not validated at creation time (intentional - allows cross-org sharing)

4. Metadata Support

Access rules support labels and annotations for auditability:

{
  "metadata": {
    "labels": { "team": "frontend", "env": "prod" },
    "annotations": { 
      "contact": "frontend-team@example.com",
      "slack": "#frontend-support"
    }
  }
}

Implemented via:

  • RouteAccessRuleLabelModel and RouteAccessRuleAnnotationModel
  • Database tables: route_access_rule_labels, route_access_rule_annotations
  • Migrations applied

5. Read-Only Relationships

Access rules expose read-only relationships extracted from the selector:

Selector Populated Relationship Others
cf:app:<guid> app: { data: { guid } } space, organizationnull
cf:space:<guid> space: { data: { guid } } app, organizationnull
cf:org:<guid> organization: { data: { guid } } app, spacenull
cf:any All null

When the referenced resource no longer exists, the relationship data is null but the selector string is preserved (for metadata like contact info).

6. Diego Sync Integration

Access rules are automatically translated into GoRouter-compatible route options during Diego sync:

# Cloud Controller converts access rules to:
route.options = {
  "access_scope": "org",  # From domain.access_rules_scope
  "access_rules": "cf:app:guid1,cf:space:guid2"  # Comma-joined selectors
}

Process timestamps are updated on access rule create/delete to trigger ProcessesSync.

Database Migrations

Three new migrations:

  1. 20260415000001_remove_name_from_route_access_rules.rb - Remove name field per RFC update
  2. 20260415000002_create_route_access_rule_labels.rb - Labels table for metadata
  3. 20260415000003_create_route_access_rule_annotations.rb - Annotations table for metadata

Previous migrations (already applied):

  • 20260407100000_add_enforce_access_rules_to_domains.rb - Domain enforcement fields
  • 20260407100001_create_route_access_rules.rb - Access rules table

Files Changed

Controllers:

  • app/controllers/v3/access_rules_controller.rb - Access Rules API implementation

Models:

  • app/models/runtime/route_access_rule.rb - Core model with Diego sync callbacks
  • app/models/runtime/route_access_rule_label_model.rb - Labels metadata
  • app/models/runtime/route_access_rule_annotation_model.rb - Annotations metadata
  • app/models/runtime/domain.rb - Add enforcement fields
  • app/models/runtime/route.rb - Add access_rules relationship
  • app/models.rb - Explicit requires for new models (Rails autoloading disabled)

Messages:

  • app/messages/access_rule_create_message.rb - Create validation
  • app/messages/access_rule_update_message.rb - Update validation (metadata only)
  • app/messages/access_rules_list_message.rb - List filtering
  • app/messages/domain_create_message.rb - Add enforcement fields

Presenters:

  • app/presenters/v3/access_rule_presenter.rb - API response format
  • app/presenters/v3/domain_presenter.rb - Add enforcement fields

Decorators:

  • app/decorators/include_access_rule_selector_resource_decorator.rb - Sideload selector resources
  • app/decorators/include_access_rule_route_decorator.rb - Sideload routes

Actions:

  • app/actions/domain_create.rb - Set enforcement fields (immutable after creation)

Diego Integration:

  • lib/cloud_controller/diego/protocol/routing_info.rb - Generate route options

Tests:

  • spec/request/access_rules_spec.rb - Full API integration tests
  • spec/unit/models/runtime/route_access_rule_spec.rb - Model unit tests
  • spec/unit/messages/access_rule*_spec.rb - Message validation tests

Related PRs

Testing

All RFC requirements have test coverage:

  • ✅ Access Rules CRUD operations
  • ✅ Query parameter filtering (guids, route_guids, space_guids, selectors, selector_resource_guids)
  • ✅ Include parameter (route, app, space, organization, selector_resource)
  • ✅ Metadata (labels/annotations)
  • ✅ Read-only relationships
  • ✅ Validation rules (enforce_access_rules, internal domains, cf:any exclusivity, duplicates)
  • ✅ Permissions (Space Developer required)
  • ✅ Diego sync on create/delete

Run tests:

bundle exec rspec spec/request/access_rules_spec.rb
bundle exec rspec spec/unit/models/runtime/route_access_rule_spec.rb

Deployment Notes

  1. Run migrations: rake db:migrate
  2. Ensure routing-release is deployed with mTLS support
  3. No feature flags required - gated by domain configuration (enforce_access_rules)

Breaking Changes

None - this is additive functionality. Existing routes are unaffected.

rkoster added 5 commits March 5, 2026 08:33
- Add app_to_app_mtls_routing feature flag (default: false)
- Add allowed_sources to RouteOptionsMessage with validation
- Validate allowed_sources structure (apps/spaces/orgs arrays, any boolean)
- Validate that app/space/org GUIDs exist in database
- Enforce mutual exclusivity of 'any' with apps/spaces/orgs lists
Tests cover:
- Feature flag disabled: allowed_sources rejected as unknown field
- Structure validation: object type, valid keys, array types, boolean any
- any exclusivity: cannot combine any:true with apps/spaces/orgs lists
- GUID existence validation: apps, spaces, orgs must exist in database
- Combined options: allowed_sources works with loadbalancing
Rails parses JSON with symbol keys, but validation was comparing
against string keys. Add normalized_allowed_sources helper to
transform keys to strings for consistent comparison.
Rename the route options field from allowed_sources to
mtls_allowed_sources for better clarity about its purpose
in mTLS app-to-app routing.

Updates RouteOptionsMessage to use the new field name in:
- Allowed keys registration
- Feature flag gating
- Validation methods
- All related tests
Change from nested mtls_allowed_sources object to flat options:
- mtls_allowed_apps: comma-separated app GUIDs (string)
- mtls_allowed_spaces: comma-separated space GUIDs (string)
- mtls_allowed_orgs: comma-separated org GUIDs (string)
- mtls_allow_any: boolean (true/false)

This complies with RFC-0027 which requires route options to only use
numbers, strings, and boolean values (no nested objects or arrays).
rkoster added 12 commits April 9, 2026 07:52
Replace POC route-options-based mTLS implementation with RFC-compliant architecture:

Domain model changes:
- Add enforce_access_rules (boolean) and access_rules_scope (any/org/space) to domains
- Fields are immutable after domain creation
- Update DomainCreateMessage, DomainPresenter, and DomainCreate action

Access Rules resource:
- New /v3/access_rules API with full CRUD operations
- RouteAccessRule model with guid, name, selector, route_id
- Selector format: cf:app:<uuid>, cf:space:<uuid>, cf:org:<uuid>, or cf:any
- Enforce cf:any exclusivity and per-route name/selector uniqueness
- Space Developer can manage rules for routes in their space

Diego sync path:
- Inject access_scope and access_rules into route options for GoRouter
- Filter internal mTLS keys (access_scope, access_rules) from public /v3/routes API
- Add access_rules to eager load to avoid N+1 queries

Tests:
- Unit tests for AccessRuleCreateMessage (selector validation, cf:any rules)
- Request specs for /v3/access_rules CRUD (create, show, list, delete, metadata update)
- Updated domain_create_message_spec for enforce_access_rules validation
- Updated routing_info_spec to verify mTLS options injection
- Updated route_presenter_spec to verify internal keys are filtered

Remove POC artifacts:
- Remove app_to_app_mtls_routing feature flag
- Remove mtls_allowed_* keys from route_options_message
- Replace non-existent readable_space_scoped_space_guids_query with proper subquery
- Use readable_space_scoped_spaces_query for non-global readers
- Handle global readers separately with all routes
- Add after_create and after_destroy callbacks to touch associated processes
- Updates process.updated_at to trigger Diego ProcessesSync immediately
- Eliminates 30-second wait for access rule changes to propagate to GoRouter
- Add comprehensive unit tests for callbacks and validations
- Ensure RouteAccessRule model is loaded in app/models.rb

This enables automatic synchronization of access rules to Diego/GoRouter
within seconds instead of requiring manual app restarts or waiting for
the next sync cycle.
- Add include parameter support to AccessRulesListMessage
- Refactor IncludeAccessRuleSelectorResourceDecorator to match RFC format:
  - Return separate arrays for apps, spaces, organizations instead of selector_resources
  - Include full resource details using appropriate presenters
  - Batch resource fetching by type with eager loading
  - Auto-deduplicate resources
  - Gracefully handle stale/missing resources
- Wire up decorator to AccessRulesController
- Add comprehensive request specs for include=selector_resource

Fixes: uninitialized constant error by adding proper require statement
Implement space-based filtering for access rules endpoint to enable
querying all access rules within a given space using ?space_guids=<guid>
query parameter.

Changes:
- Add space_guids to AccessRulesListMessage with array validation
- Implement space filtering in AccessRulesController#build_dataset
- Add comprehensive unit tests for AccessRulesListMessage
- Add request specs for single/multiple space filtering and combinations
- Follow existing CAPI patterns for space_guids filtering

The filter joins through the routes table to filter access rules by
the space_id of their associated routes.
Add support for including route resources when listing access rules
via the ?include=route query parameter.

Changes:
- Create IncludeAccessRuleRouteDecorator to handle route inclusion
- Wire up decorator in AccessRulesController
- Add comprehensive request specs for include=route
- Test single/multiple routes, uniqueness, and combining with selector_resource
- Follow existing CAPI decorator patterns for resource inclusion

The decorator fetches and presents Route resources referenced by the
access rules, adding them to the 'included' section of the response.
…RFC updates

Update /v3/access_rules API to align with latest RFC changes:
- Remove 'name' field from RouteAccessRule model and API
- Add database migration to drop name column and unique index
- Use labels/annotations for metadata instead of name field
- Add read-only relationships (app, space, organization) to responses
  extracted from selector (cf:app:X, cf:space:X, cf:org:X, cf:any)
- Replace 'names' filter with 'guids' filter
- Add 'selector_resource_guids' filter for text-match against selectors
- Update include support: add individual app, space, organization
  (in addition to existing selector_resource and route)
- Remove name-based uniqueness validation (keep selector uniqueness)
- Update all tests to remove name references

Breaking changes:
- POST /v3/access_rules no longer accepts 'name' field
- GET /v3/access_rules responses no longer include 'name' field
- Filter parameter 'names' removed, use 'guids' instead
- Access rule responses now include app/space/organization relationships
- Create RouteAccessRuleLabelModel and RouteAccessRuleAnnotationModel
- Add one_to_many relationships for labels and annotations to RouteAccessRule
- Add database migrations for route_access_rule_labels and route_access_rule_annotations tables
- Fixes: undefined method 'labels' error in AccessRulePresenter

This enables metadata (labels/annotations) support for access rules,
required by the RFC changes that removed the 'name' field in favor
of using labels/annotations for metadata storage.
Add require statements for RouteAccessRuleLabelModel and
RouteAccessRuleAnnotationModel to app/models.rb.

Rails autoloading is disabled for app/** so all models must be
explicitly required. This fixes the error:
  uninitialized constant VCAP::CloudController::RouteAccessRuleLabelModel
Per RFC requirement (line 246-247):
Access rules cannot be created for routes on internal domains
(domains created with --internal). Internal routes use container-to-container
networking and bypass GoRouter entirely, so GoRouter cannot enforce
access rules.

Changes:
- Add validation in AccessRulesController#create to reject access rules
  on internal domains with 422 status
- Add test coverage for internal domain validation
- Error message explains why: internal domains bypass GoRouter
…up tests

- Collapse 4 migrations into 1 consolidated migration (20260407100001) that
  creates route_access_rules, route_access_rule_labels, and
  route_access_rule_annotations tables
- Remove name field from access rules per RFC updates
- Fix all RuboCop offenses: Style/CollectionQuerying (.count > 0 -> .any?),
  Migration/AddConstraintName (primary_key :id, name: :id),
  Metrics/BlockLength, Metrics/CyclomaticComplexity, and others
- Add stale resource detection in presenter (null data for deleted resources)
- Extract controller methods to reduce complexity
- Use relative class names within VCAP::CloudController module
- Fix test shadowing of rack-test app method (let(:app) -> let(:frontend_app))
- Fix Sequel validation assertion style (.include(:presence) not strings)
…ain API surface in access rules

- Wrap create action in transaction with FOR UPDATE lock to prevent
  concurrent inserts from violating cf:any exclusivity constraints
- Rescue Sequel::UniqueConstraintViolation to return 422 instead of 500
- Join routes table at most once when both route_guids and space_guids
  filters are requested, preventing ambiguous column references
- Escape LIKE metacharacters (% and _) in selector_resource_guids filter
- Replace deprecated routes__column syntax with Sequel[:routes][:column]
- Remove per-row DB existence checks in AccessRulePresenter to eliminate
  N+1 queries; relationship GUIDs are now included directly from selector
- Only include enforce_access_rules and access_rules_scope in domain
  responses when enforce_access_rules is true
rkoster added 4 commits April 15, 2026 14:03
…tization)

Escape backslash characters before % and _ in selector_resource_guids
LIKE filtering to prevent backslash-based injection.
Expand selector_resource_guids filtering tests to cover all three
LIKE metacharacters: %, _, and backslash.
Annotations table used key VARCHAR(1000) with a unique index on
(resource_guid, key), totaling 5020 bytes in utf8mb4 — exceeding
MySQL's 3072-byte max key length.

Align with codebase convention established in migration
20240102150000: use key_name VARCHAR(63) with a three-column unique
index on (resource_guid, key_prefix, key_name). Also add NOT NULL
default '' to key_prefix on both labels and annotations tables.
…Rule

RouteAccessRule does not have a name column. The test was passing
name: to create(), triggering Sequel::MassAssignmentRestriction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants