Hmm... For now, I ran the public Xylaria endpoint through all 541 English IFEval prompts.

Endpoint:
https://xylaria2s.vercel.app/api/v1/chat/completions

Submitted model label:
xylaria-2-senoa-max

Backend model identity:
Unverified

IFEval results:

• Prompt-level strict: 67.8%
• Prompt-level loose: 71.3%
• Instruction-level strict: 70.6%
• Instruction-level loose: 73.5%

Reference instruction-level strict scores, recomputed with the same prompt file and scorer:

• GPT-4o 2024-08-06: 88.7%
• Claude 3.5 Sonnet: 87.9%
• GPT-4o mini: 85.6%
• Qwen 2.5 7B Int4: 80.8%
• Llama 3.1 8B Int4: 80.1%
• Xylaria endpoint: 70.6%
• Mistral 7B Instruct v0.3: 58.9%

The main issue was endpoint reliability:

• 541 requests
• 425 successful non-empty responses
• 114 HTTP 200 responses with empty content
• 2 HTTP 504 timeouts
• 78.56% non-empty response rate
• 8.68-second median latency for successful responses
• 28.85-second p95 latency

As a diagnostic only, among the 425 requests that returned non-empty content, Xylaria achieved 90.1% instruction-level strict accuracy. This conditional figure is not directly comparable to the full reference-model scores because it excludes the 116 failed responses.

Overall: when the endpoint returns text, instruction following appears strong. The current bottleneck is response reliability rather than output quality.

All requests were sequential, unauthenticated, temperature 0, with no automatic retries. Raw responses, per-question scores, configuration, scorer source, and SHA-256 checksums are included in the proof package.