Element A & Element B: Problem Justification and Past Solution Review

[code259]

Element A & B

Links

• https://sites.google.com/view/longbearscapstone/

Ideas

1. Host & Serve an LLM

• While it seems pretty simple, serving and managing proper inference workloads is much harder. The field of inference engineering exists for a reason. We also have a unique hardware scenario, so we will have to work with that.
• model → kernels/runtime → inference engine → node configuration → distributed serving → scheduler → networking → autoscaling → observability → reliability → cost optimization
• even for the earlier parts of the stack like kernel & runtime that have good tooling like flash attention and vLLM there is still optimizations you can do based on your workload and usecase
• potential model to run: Qwen3.8 27B (equivalent to GPt 5.6 luna on max; https://artificialanalysis.ai/models/qwen3-8-27b)
• in our case serving that model is nontrivial because of our hardware constraints, so it would be a good learning experience. also, the model is actually good, so people in OCS can use it for programming tasks (would likely be like a subagent with a larger orchestrator like opus, fable, gpt 5.6 sol, or something would delegate to), so would need some interface for people to access the model we serve in their harnesses

Metrics to Track

• TTFT (Time to first token)
• TPS (tokens per second)

Capabilities that this enables

Other Ideas

• Make RL envs or use the OCS github/codebase to finetune/improve the model for our usecase.