DataCollector.finalise() is not idempotent — middleware retry loop triggers TypeError: SQLQuery() got unexpected keyword arguments: 'model'

[newpronik]

Summary

silk.collector.DataCollector.finalise() mutates each query dict by
adding a model reference after its first pass. Because
SilkyMiddleware.process_response has a max_attempts = 2 retry loop
around _process_response, any AttributeError or DatabaseError on
the first attempt causes finalise() to be called a second time — and
on that second pass each query dict still carries a stale model key
from the previous run. The next models.SQLQuery(**query) line then
explodes with:

TypeError: SQLQuery() got unexpected keyword arguments: 'model'

The result is that silk turns retryable failures into a hard 500 whose
traceback points inside silk's middleware rather than at the real issue.

Environment

• django-silk 5.5.0 (latest on PyPI as of this writing)
• Django 5.2.13
• Python 3.13
• Running in DEBUG=True dev server

Observed traceback

Internal Server Error: /api/regions/states/
Traceback (most recent call last):                                                                                                                                                         
  File ".../django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)                                                                                                                                                       
  File ".../silk/middleware.py", line 97, in __call__
    response = self.process_response(request, response)                                                                                                                                    
  File ".../silk/middleware.py", line 180, in process_response                                                                                                                             
    self._process_response(request, response)
  File ".../silk/middleware.py", line 158, in _process_response                                                                                                                            
    collector.finalise()
  File ".../silk/collector.py", line 164, in finalise                                                                                                                                      
    sql_query = models.SQLQuery(**query)                                                                                                                                                   
  File ".../django/db/models/base.py", line 569, in __init__
    raise TypeError(                                                                                                                                                                       
        "%s() got unexpected keyword arguments: %s"                                                                                                                                        
        % (cls.__name__, ", ".join(kwargs))
    )                                                                                                                                                                                      
TypeError: SQLQuery() got unexpected keyword arguments: 'model'

Root cause

In silk/collector.py::DataCollector.finalise():

# silk/collector.py, finalise()
sql_queries = []                                                                                                                                                                           
for identifier, query in self.queries.items():
    query['identifier'] = identifier                                                                                                                                                       
    sql_query = models.SQLQuery(**query)   # <-- line 164: fails on 2nd call                                                                                                               
    sql_queries += [sql_query]                                                                                                                                                             
                                                                                                                                                                                           
models.SQLQuery.objects.bulk_create(sql_queries)                                                                                                                                           
sql_queries = models.SQLQuery.objects.filter(request=self.request)
for sql_query in sql_queries.all():                                                                                                                                                        
    query = self.queries.get(sql_query.identifier)                                                                                                                                         
    if query:                                                                                                                                                                              
        query['model'] = sql_query         # <-- line 172: mutates dict                                                                                                                    

Line 172 attaches a model reference onto each query dict so the
profile-processing block below can correlate profiles with their
SQLQuery model instances. The dicts stored in self.queries are held
by reference (see register_query → register_objects), so this
mutation persists for the lifetime of the collector (and DataCollector
is a Singleton).

Meanwhile silk/middleware.py::SilkyMiddleware.process_response:

# silk/middleware.py, process_response()
def process_response(self, request, response):
    max_attempts = 2                                                                                                                                                                       
    attempts = 1
    if getattr(request, 'silk_is_intercepted', False):                                                                                                                                     
        while attempts <= max_attempts:                                                                                                                                                    
            if attempts > 1:
                Logger.debug('Retrying _process_response; attempt %s' % attempts)                                                                                                          
            try:
                self._process_response(request, response)                                                                                                                                  
                break
            except (AttributeError, DatabaseError):                                                                                                                                        
                if attempts >= max_attempts:
                    Logger.warning('Exhausted _process_response attempts; not processing request')                                                                                         
                    break                                                                                                                                                                  
            attempts += 1                                                                                                                                                                  
    return response                                                                                                                                                                        

If the first _process_response call raises AttributeError or
DatabaseError after finalise() has already set query['model']
for any subset of queries (for example, silk_request.save() at line
169 fails, or something in profile processing fails), the loop retries
and runs finalise() again. The second iteration's
models.SQLQuery(**query) is now passed a model=<SQLQuery> kwarg that
the model class has no field for, so Django 5.x raises TypeError,
which is not caught by the except (AttributeError, DatabaseError)
clause — it propagates out of middleware and becomes a 500.

Minimal reproduction (verified)

Drop this into a Django shell (manage.py shell) of any project that
has silk in INSTALLED_APPS and has run migrate:

from django.utils import timezone                                                                                                                                                          
from silk.collector import DataCollector
from silk import models as silk_models                                                                                                                                                     
 
request = silk_models.Request.objects.create(                                                                                                                                              
    path="/test/repro",
    method="GET",
    start_time=timezone.now(),                                                                                                                                                             
)
                                                                                                                                                                                           
collector = DataCollector()
collector.configure(request=request, should_profile=False)
collector.register_query({
    "query": "SELECT 1",
    "start_time": timezone.now(),                                                                                                                                                          
    "end_time": timezone.now(),
    "traceback": "",                                                                                                                                                                       
    "request": request,
    "analysis": None,
})

print("keys before:", list(collector.queries[1].keys()))                                                                                                                                   
 
collector.finalise()   # ok                                                                                                                                                                
print("keys after 1st:", list(collector.queries[1].keys()))
                                                                                                                                                                                           
collector.finalise()   # TypeError

Actual output on django-silk 5.5.0 + Django 5.2.13:

keys before: ['query', 'start_time', 'end_time', 'traceback', 'request', 'analysis']
keys after 1st: ['query', 'start_time', 'end_time', 'traceback', 'request', 'analysis', 'identifier', 'model']
Traceback (most recent call last):                                                                                                                                                         
  ...
  File ".../silk/collector.py", line 164, in finalise                                                                                                                                      
    sql_query = models.SQLQuery(**query)
  File ".../django/db/models/base.py", line 569, in __init__                                                                                                                               
    raise TypeError(...)                                                                                                                                                                   
TypeError: SQLQuery() got unexpected keyword arguments: 'model'

So after the first successful finalise(), the dict has picked up both
identifier and model keys. identifier is harmless (it's a real
field on SQLQuery), but model is not — which is exactly what the
second call unpacks into the model constructor.

Expected behavior

finalise() should be idempotent — calling it multiple times on the
same collector state should either succeed or raise the same underlying
error the first call raised, not a confusing TypeError about an
unexpected kwarg.