Introduction to the Open Leaderboard for Japanese LLMs
•
35
LLM-jp
university
AI & ML interests
None defined yet.
Recent Activity
Articles
llm-jp's activity
speed
updated
a
collection
2 days ago
t0-0
updated
3
datasets
5 days ago
Taishi-N324
updated
a
collection
6 days ago
Taishi-N324
authored
3
papers
19 days ago
Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs
Paper
•
2411.08719
•
Published
Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs
Paper
•
2412.14471
•
Published
Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search
Paper
•
2503.04412
•
Published
•
1
Taishi-N324
authored
a
paper
27 days ago
nokazaki
authored
a
paper
about 1 month ago
shintaro-ozaki
authored
a
paper
3 months ago
AkimfromParis
posted
an
update
3 months ago
Post
1732
💵 Polymarket is leveraging “Chatbot Arena LLM Leaderboard” on HuggingFace for online gambling on the “Top AI model on January 31?”. 🤗
As of January 3rd, 2025:
-1./ Gemini (83%) -2./ ChatGPT (13%) -3./ Other (2%) -4./ Claude (2%) -5./ Grok (1%) -6./ Llama (<1%)
🇺🇸 The market opinion is following historical data. It's clearly bias towards US historical AI giants, yet Polymarket is forbidden in the USA and for US citizens.
🇨🇳 In the “Other”, you might have Chinese AI labs that are probably the future AI leaders (Qwen, DeepSeek, Yi).
⚖️ In the market resolution, if two models are tied in the evaluation, they will take the alphabetical order. (e.g. if both were tied, “Google” would resolve to “Yes”, and “xAI” would resolve to “No”). 🙃
That might be illegal usage of the Chatbot Arena policy? And maybe HuggingFace? @clem
Or maybe authors and contributors should get a cut each month as “market markers”. @weichiang @angelopoulos
As of January 3rd, 2025:
-1./ Gemini (83%) -2./ ChatGPT (13%) -3./ Other (2%) -4./ Claude (2%) -5./ Grok (1%) -6./ Llama (<1%)
🇺🇸 The market opinion is following historical data. It's clearly bias towards US historical AI giants, yet Polymarket is forbidden in the USA and for US citizens.
🇨🇳 In the “Other”, you might have Chinese AI labs that are probably the future AI leaders (Qwen, DeepSeek, Yi).
⚖️ In the market resolution, if two models are tied in the evaluation, they will take the alphabetical order. (e.g. if both were tied, “Google” would resolve to “Yes”, and “xAI” would resolve to “No”). 🙃
That might be illegal usage of the Chatbot Arena policy? And maybe HuggingFace? @clem
Or maybe authors and contributors should get a cut each month as “market markers”. @weichiang @angelopoulos
- 1 reply
AkimfromParis
posted
an
update
3 months ago
Post
1726
🇺🇸 🇨🇦 🇬🇧 Nobel Prize winners against USSR & Japanese AI pioneers ☭🇯🇵
🇩🇪 Prof. Jürgen Schmidhuber: “The #NobelPrize in Physics 2024 for Hopfield & Hinton turns out to be a Nobel Prize for plagiarism. They republished methodologies developed in #Ukraine and #Japan by Ivakhnenko and Amari in the 1960s & 1970s, as well as other techniques, without citing the original inventors.”
1965 - First Deep Learning - USSR ☭ (Ukraine 🇺🇦 now)
Ivakhnenko and Lapa introduced the first deep learning in deep MLPs that learn internal representations of input data.
1967/68 - Deep Learning by Stochastic Gradient Descent - Japan 🇯🇵
Shun-Ichi Amari trained MLPs with many layers in non-incremental end-to-end fashion from scratch by stochastic gradient descent (SGD).
1969 - Rectified linear unit - Japan 🇯🇵
In 1969, Kunihiko Fukushima introduced ReLU in the context of visual feature extraction in hierarchical neural networks.
1970 - Backpropagation - Finland 🇫🇮 😃
In 1970, Seppo Linnainmaa was the first the reverse mode of automatic differentiation, now known as backpropagation.
1972 - Recurrent Neural Network - Japan 🇯🇵
In 1972, Shun-Ichi Amari published a learning recurrent neural network based on Lenz-Ising model (Amari's net was later called the "Hopfield network". Hopfield republished in 1982, without citing Amari papers.)
1979 - First Convolutional neural network - Japan 🇯🇵
CNN architecture was introduced in 1979 by Kunihiko Fukushima, also known as Neocognitron.
https://people.idsia.ch/~juergen/deep-learning-history.html#AMH2
🇩🇪 Prof. Jürgen Schmidhuber: “The #NobelPrize in Physics 2024 for Hopfield & Hinton turns out to be a Nobel Prize for plagiarism. They republished methodologies developed in #Ukraine and #Japan by Ivakhnenko and Amari in the 1960s & 1970s, as well as other techniques, without citing the original inventors.”
1965 - First Deep Learning - USSR ☭ (Ukraine 🇺🇦 now)
Ivakhnenko and Lapa introduced the first deep learning in deep MLPs that learn internal representations of input data.
1967/68 - Deep Learning by Stochastic Gradient Descent - Japan 🇯🇵
Shun-Ichi Amari trained MLPs with many layers in non-incremental end-to-end fashion from scratch by stochastic gradient descent (SGD).
1969 - Rectified linear unit - Japan 🇯🇵
In 1969, Kunihiko Fukushima introduced ReLU in the context of visual feature extraction in hierarchical neural networks.
1970 - Backpropagation - Finland 🇫🇮 😃
In 1970, Seppo Linnainmaa was the first the reverse mode of automatic differentiation, now known as backpropagation.
1972 - Recurrent Neural Network - Japan 🇯🇵
In 1972, Shun-Ichi Amari published a learning recurrent neural network based on Lenz-Ising model (Amari's net was later called the "Hopfield network". Hopfield republished in 1982, without citing Amari papers.)
1979 - First Convolutional neural network - Japan 🇯🇵
CNN architecture was introduced in 1979 by Kunihiko Fukushima, also known as Neocognitron.
https://people.idsia.ch/~juergen/deep-learning-history.html#AMH2
-
- 11 replies
shintaro-ozaki
authored
4
papers
3 months ago
Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?
Paper
•
2410.03240
•
Published
BQA: Body Language Question Answering Dataset for Video Large Language Models
Paper
•
2410.13206
•
Published
Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models
Paper
•
2409.01584
•
Published
An Implementation of Werewolf Agent That does not Truly Trust LLMs
Paper
•
2409.01575
•
Published
AkimfromParis
posted
an
update
4 months ago
Post
1493
🇯🇵 The Open Japanese LLM Leaderboard created by LLM-jp 🌸 in partnership with HuggingFace 🤗 was released today!
Blog: https://huggingface.co/blog/leaderboard-japanese
Space: llm-jp/open-japanese-llm-leaderboard
🌍 The leaderboard is available in both Japanese and English
📚 Based on the evaluation tool, llm-jp-eval with more than 20 datasets for Japanese LLMs
📊 The leaderboard showcases all the metrics for NLP experts, plus averages for NLP beginners
💻 For the comfort of users, we chose a horizontal UI, and implemented it in a light and dark theme on Gradio
🔬 The radar chart provides a very interesting visualization of metrics!
🌱 We are using the Japanese research platform, MDX, so please be patient!
⚡ LLMs bigger than +70B will be evaluated soon…
How do you say “GPUs Go Brrr” in Japanese - > GPUがブンブン～! (To pronounce "GPU ga bunbun!") 🔥
Blog: https://huggingface.co/blog/leaderboard-japanese
Space: llm-jp/open-japanese-llm-leaderboard
🌍 The leaderboard is available in both Japanese and English
📚 Based on the evaluation tool, llm-jp-eval with more than 20 datasets for Japanese LLMs
📊 The leaderboard showcases all the metrics for NLP experts, plus averages for NLP beginners
💻 For the comfort of users, we chose a horizontal UI, and implemented it in a light and dark theme on Gradio
🔬 The radar chart provides a very interesting visualization of metrics!
🌱 We are using the Japanese research platform, MDX, so please be patient!
⚡ LLMs bigger than +70B will be evaluated soon…
How do you say “GPUs Go Brrr” in Japanese - > GPUがブンブン～! (To pronounce "GPU ga bunbun!") 🔥
-
- 4 replies
Taishi-N324
authored
a
paper
5 months ago