Inconsistent Results Between HF Inference API and StarCoder Playground for Code Completion Task
Hello,
I am currently facing an issue with the Hugging Face (HF) Inference API when trying to perform code completion tasks using the StarCoder model. Although I follow the provided examples and match my API call parameters to those in the StarCoder Playground, my requests often return an empty string, or, if I adjust the temperature parameter closer to zero, the output becomes nonsensical and impractical. This happens with prompts that return valid and meaningful completions on the StarCoder Playground and web inference API next to the model card. I also observe inconsistencies between the StarCoder Playground and the web inference API next to the model card.
Technical Details:
Here's how I structure my API calls:
def query(self, payload):
response = requests.post(API_URL, headers=headers, json=payload)
return response.json()
def _draw_sample(self, prompt: str, temperature=0.2, max_new_tokens=256, top_p=0.9, repetition_penalty=1.2) -> str:
# Define the payload with the modified prompt and additional parameters
payload = {
"inputs": prompt,
"parameters": {
"temperature": temperature,
"max_new_tokens": max_new_tokens,
"top_p": top_p,
"repetition_penalty": repetition_penalty,
"return_full_text": False,
"do_sample": True,
},
"options": {"use_cache": True}
}
response = self.query(payload)
logging.info(f"response from LLM is {response}")
# Extract the generated code from the response
return response
Prompt Details:
The prompt includes a docstring that describes the script's functionality and a brief explanation of the code completion task (e.g., 'Improve priority_v1 over priority_v0 from the previous iteration'), context functions, the function from the previous iteration, and finally, the function header and docstring for which StarCoder is expected to generate the body.
Any help understanding the reason for the discrepancy between the API and Playground results or alternative approaches to structuring my prompt and API requests would be greatly appreciated.
Thank you!
Attached below are screenshots showcasing the outputs on the Playground and the web inference tool next to the model card for comparison.
