French & English Audio LLM Leaderboard

Overview

1st 2nd Second to last Last

#	Model	Size	Aggregation			ASR	FR	EN	AR	DE	ES	IT	PT	NL	AST	ES-FR · Multilingual_TEDx	ES-IT · Multilingual_TEDx	FR-EN · Multilingual_TEDx	FR-ES · Multilingual_TEDx	Question Answering	FR	EN	Others	Emotion Recognition	Gender Recognition	Age Recognition	Dialogue Summarization	Diarization	Spoken Language Identification	Temporal	Music	Music Question Answering	Music Captioning	Sound	Audio Captioning	Audio Question Answering
#	Model	Size	Min-Max	Z-Score	Avg Rank	ASR	FR	EN	AR	DE	ES	IT	PT	NL	AST	ES-FR · Multilingual_TEDx	ES-IT · Multilingual_TEDx	FR-EN · Multilingual_TEDx	FR-ES · Multilingual_TEDx	Question Answering	FR	EN	Others	Emotion Recognition	Gender Recognition	Age Recognition	Dialogue Summarization	Diarization	Spoken Language Identification	Temporal	Music	Music Question Answering	Music Captioning	Sound	Audio Captioning	Audio Question Answering
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				9.35	10.50	6.84	79.43	15.44	6.28	6.82	8.58	10.55	60.54	57.16	57.27	65.31	62.41	62.87	61.19	63.59	43.88	19.89	58.13	38.28	43.28	45.12	80.40	22.03	40.29	39.22	41.36	47.07	41.92	52.21
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				7.94	9.45	5.56	54.24	12.97	5.15	5.27	7.31	8.14	65.09	62.15	62.31	68.82	67.09	69.27	66.02	70.66	51.63	29.51	67.60	40.24	40.96	44.32	94.80	44.02	44.58	47.24	41.92	50.54	45.64	55.44
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				23.37	30.84	10.64	91.90	29.60	15.21	20.10	17.84	38.14	47.31	45.58	45.00	50.91	47.73	62.89	50.78	68.08	48.71	31.51	88.40	29.86	51.12	40.96	88.40	10.73	61.26	62.92	59.60	57.21	53.08	61.33
	Qwen/Qwen2.5-Omni-3B	5.9B				28.25	32.73	21.58	64.70	30.61	24.37	27.53	26.24	29.83	44.66	37.33	42.21	51.56	47.52	64.60	49.64	71.01	29.35	28.17	16.40	5.16	54.00	38.00	53.20	10.50	39.22	42.68	35.76	43.47	34.56	52.37
	Qwen/Qwen2.5-Omni-7B	11B				22.80	26.26	22.34	61.45	26.46	16.62	14.64	20.54	29.34	46.84	43.80	40.63	55.49	47.45	63.79	50.64	69.43	35.01	30.44	19.47	1.45	53.36	38.32	89.60	12.41	38.04	42.80	33.28	51.89	47.40	56.37
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				11.43	13.17	6.22	43.56	17.38	10.28	11.41	8.38	13.14	61.47	58.60	57.19	65.75	64.34	71.94	60.45	76.87	52.22	44.99	89.73	9.47	58.88	47.04	96.80	18.64	62.40	65.92	58.88	59.30	54.36	64.24
	google/gemma-4-E2B-it	5.1B				12.83	15.68	10.76	44.33	17.40	7.62	8.98	12.23	12.15	56.69	53.81	53.09	59.77	60.11	60.20	49.30	64.87	31.22	23.50	24.13	1.43	54.88	34.80	70.00	9.83	31.30	30.68	31.92	27.39	20.92	33.87
	microsoft/Phi-4-multimodal-instruct	5.6B				24.14	25.69	7.50	99.22	31.23	16.64	20.31	22.28	76.99	42.04	29.93	40.36	58.43	39.43	65.44	52.87	70.83	31.35	26.83	46.93	2.83	56.24	38.32	44.00	4.27	45.44	46.64	44.24	52.13	45.60	58.67
	mistralai/Voxtral-Mini-3B-2507	4.68B				8.88	11.11	6.68	76.42	13.45	5.20	6.30	8.03	7.12	56.93	51.79	51.83	64.28	59.83	69.31	56.26	74.90	34.93	31.50	30.67	7.94	59.92	34.24	70.40	9.84	40.65	36.50	44.80	47.47	40.12	54.83
	mistralai/Voxtral-Small-24B-2507	24.3B				7.70	10.23	5.45	68.71	11.75	4.38	5.13	6.69	5.00	60.90	58.48	57.21	68.42	59.48	73.16	60.55	78.57	36.60	36.85	32.93	5.10	61.68	40.48	67.60	11.59	53.00	53.28	52.72	45.44	34.64	56.24
	nvidia/audio-flamingo-3-hf	8.2B				55.74	60.96	11.72	91.39	75.20	54.43	76.81	63.89	61.74	24.27	17.58	12.69	46.03	20.79	58.78	45.09	64.64	39.71	37.92	80.27	22.61	54.48	-1.28	76.40	7.54	59.99	59.10	60.88	59.36	58.88	59.84

Overview (FR/EN — ASR, AST, QA)

1st 2nd Second to last Last

#	Model	Size	Aggregation			ASR	FR	EN	AST	ES-FR · Multilingual_TEDx	FR-EN · Multilingual_TEDx	FR-ES · Multilingual_TEDx	Question Answering	FR	EN
#	Model	Size	Min-Max	Z-Score	Avg Rank	ASR	FR	EN	AST	ES-FR · Multilingual_TEDx	FR-EN · Multilingual_TEDx	FR-ES · Multilingual_TEDx	Question Answering	FR	EN
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				9.28	10.50	6.84	61.63	57.16	65.31	62.41	62.87	61.19	63.59
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				8.16	9.45	5.56	66.02	62.15	68.82	67.09	69.27	66.02	70.66
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				24.11	30.84	10.64	48.08	45.58	50.91	47.73	62.89	50.78	68.08
	Qwen/Qwen2.5-Omni-3B	5.9B				29.01	32.73	21.58	45.47	37.33	51.56	47.52	64.60	49.64	71.01
	Qwen/Qwen2.5-Omni-7B	11B				24.96	26.26	22.34	48.92	43.80	55.49	47.45	63.79	50.64	69.43
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				10.85	13.17	6.22	62.90	58.60	65.75	64.34	71.94	60.45	76.87
	google/gemma-4-E2B-it	5.1B				14.04	15.68	10.76	57.90	53.81	59.77	60.11	60.20	49.30	64.87
	microsoft/Phi-4-multimodal-instruct	5.6B				19.62	25.69	7.50	42.60	29.93	58.43	39.43	65.44	52.87	70.83
	mistralai/Voxtral-Mini-3B-2507	4.68B				9.63	11.11	6.68	58.63	51.79	64.28	59.83	69.31	56.26	74.90
	mistralai/Voxtral-Small-24B-2507	24.3B				8.64	10.23	5.45	62.13	58.48	68.42	59.48	73.16	60.55	78.57
	nvidia/audio-flamingo-3-hf	8.2B				44.54	60.96	11.72	28.13	17.58	46.03	20.79	58.78	45.09	64.64

Tasks · ASR

1st 2nd Second to last Last

WER (%)

#	Model	Size	Aggregation			Average	FR	CommonVoice	Fleurs	Multilingual_TEDx	SUMM-RE	VoxPopuli	YouTubeFr	EN	CommonVoice	Fleurs	VoxPopuli	AR	Fleurs	Multilingual_TEDx	DE	Fleurs	Multilingual_TEDx	ES	Fleurs	Multilingual_TEDx	IT	Fleurs	Multilingual_TEDx	PT	Fleurs	Multilingual_TEDx	NL - Fleurs
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR	CommonVoice	Fleurs	Multilingual_TEDx	SUMM-RE	VoxPopuli	YouTubeFr	EN	CommonVoice	Fleurs	VoxPopuli	AR	Fleurs	Multilingual_TEDx	DE	Fleurs	Multilingual_TEDx	ES	Fleurs	Multilingual_TEDx	IT	Fleurs	Multilingual_TEDx	PT	Fleurs	Multilingual_TEDx	NL - Fleurs
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				9.29	10.50	7.30	5.94	7.97	13.28	7.83	20.67	6.84	8.84	6.05	5.63	79.43	68.66	90.20	15.44	7.95	22.92	6.28	4.84	7.72	6.82	5.40	8.25	8.58	7.20	9.96	10.55
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				7.69	9.45	6.09	5.32	7.32	12.66	6.87	18.46	5.56	6.69	4.80	5.19	54.24	30.17	78.30	12.97	5.47	20.46	5.15	3.90	6.39	5.27	3.66	6.87	7.31	6.26	8.36	8.14
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				23.20	30.84	18.59	26.57	23.69	37.79	35.44	42.96	10.64	9.97	7.11	14.83	91.90	87.74	96.06	29.60	20.63	38.57	15.21	13.38	17.05	20.10	20.33	19.88	17.84	13.32	22.36	38.14
	Qwen/Qwen2.5-Omni-3B	5.9B				27.56	32.73	18.61	14.32	28.37	52.41	24.76	57.90	21.58	26.80	14.06	23.89	64.70	43.61	85.79	30.61	18.54	42.67	24.37	16.32	32.42	27.53	20.37	34.69	26.24	15.90	36.58	29.83
	Qwen/Qwen2.5-Omni-7B	11B				22.32	26.26	14.21	11.04	15.70	37.22	24.49	54.92	22.34	28.75	18.46	19.82	61.45	38.04	84.86	26.46	15.96	36.97	16.62	13.16	20.09	14.64	12.53	16.74	20.54	13.69	27.39	29.34
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				11.43	13.17	6.63	6.56	10.94	19.72	11.08	24.11	6.22	5.31	4.28	9.07	43.56	15.83	71.30	17.38	9.92	24.83	10.28	8.59	11.96	11.41	9.24	13.58	8.38	5.51	11.26	13.14
	google/gemma-4-E2B-it	5.1B				12.12	15.68	13.65	9.94	13.14	18.32	11.24	27.77	10.76	15.61	8.23	8.45	44.33	15.87	72.79	17.40	8.58	26.22	7.62	5.18	10.06	8.98	5.80	12.16	12.23	7.70	16.77	12.15
	microsoft/Phi-4-multimodal-instruct	5.6B				28.66	25.69	18.17	18.78	24.64	30.67	20.94	40.91	7.50	7.18	4.83	10.49	99.22	99.08	99.36	31.23	21.38	41.09	16.64	12.62	20.65	20.31	15.92	24.70	22.28	18.67	25.89	76.99
	mistralai/Voxtral-Mini-3B-2507	4.68B				8.27	11.11	7.30	5.51	8.99	18.45	9.18	17.21	6.68	8.58	4.74	6.73	76.42	62.76	90.08	13.45	5.52	21.39	5.20	3.50	6.89	6.30	3.91	8.69	8.03	5.50	10.56	7.12
	mistralai/Voxtral-Small-24B-2507	24.3B				6.95	10.23	5.65	4.58	7.78	18.14	8.89	16.31	5.45	5.83	3.98	6.55	68.71	58.51	78.91	11.75	3.73	19.78	4.38	2.95	5.81	5.13	3.27	6.99	6.69	4.02	9.36	5.00
	nvidia/audio-flamingo-3-hf	8.2B				57.82	60.96	32.58	46.09	64.82	63.43	83.32	75.50	11.72	11.63	11.62	11.91	91.39	85.47	97.31	75.20	65.45	84.95	54.43	52.67	56.19	76.81	75.93	77.68	63.89	60.45	67.33	61.74

Tasks · AST

1st 2nd Second to last Last

BLEU

#	Model	Size	Aggregation			Average	FR-EN · Multilingual_TEDx (FR→EN)	FR-ES · Multilingual_TEDx (FR→ES)	ES-FR · Multilingual_TEDx (ES→FR)	ES-IT · Multilingual_TEDx (ES→IT)
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR-EN · Multilingual_TEDx (FR→EN)	FR-ES · Multilingual_TEDx (FR→ES)	ES-FR · Multilingual_TEDx (ES→FR)	ES-IT · Multilingual_TEDx (ES→IT)
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				40.15	41.25	43.00	39.67	36.70
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				44.79	45.85	45.63	45.85	41.83
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				26.97	29.20	24.33	29.00	25.33
	Qwen/Qwen2.5-Omni-3B	5.9B				20.73	23.63	22.64	17.68	18.96
	Qwen/Qwen2.5-Omni-7B	11B				23.70	27.17	22.90	26.51	18.23
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				37.89	36.11	44.33	40.97	30.15
	google/gemma-4-E2B-it	5.1B				42.50	42.59	46.10	42.54	38.79
	microsoft/Phi-4-multimodal-instruct	5.6B				26.59	39.12	21.24	22.67	23.35
	mistralai/Voxtral-Mini-3B-2507	4.68B				36.33	41.47	39.86	33.99	30.01
	mistralai/Voxtral-Small-24B-2507	24.3B				42.79	47.48	43.35	41.42	38.91
	nvidia/audio-flamingo-3-hf	8.2B				16.41	31.19	14.74	13.06	6.66

METEOR (%)

#	Model	Size	Aggregation			Average	FR-EN · Multilingual_TEDx (FR→EN)	FR-ES · Multilingual_TEDx (FR→ES)	ES-FR · Multilingual_TEDx (ES→FR)	ES-IT · Multilingual_TEDx (ES→IT)
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR-EN · Multilingual_TEDx (FR→EN)	FR-ES · Multilingual_TEDx (FR→ES)	ES-FR · Multilingual_TEDx (ES→FR)	ES-IT · Multilingual_TEDx (ES→IT)
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				60.54	65.31	62.41	57.16	57.27
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				65.09	68.82	67.09	62.15	62.31
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				47.31	50.91	47.73	45.58	45.00
	Qwen/Qwen2.5-Omni-3B	5.9B				44.66	51.56	47.52	37.33	42.21
	Qwen/Qwen2.5-Omni-7B	11B				46.84	55.49	47.45	43.80	40.63
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				61.47	65.75	64.34	58.60	57.19
	google/gemma-4-E2B-it	5.1B				56.69	59.77	60.11	53.81	53.09
	microsoft/Phi-4-multimodal-instruct	5.6B				42.04	58.43	39.43	29.93	40.36
	mistralai/Voxtral-Mini-3B-2507	4.68B				56.93	64.28	59.83	51.79	51.83
	mistralai/Voxtral-Small-24B-2507	24.3B				60.90	68.42	59.48	58.48	57.21
	nvidia/audio-flamingo-3-hf	8.2B				24.27	46.03	20.79	17.58	12.69

Tasks · QA

1st 2nd Second to last Last

Math Question Answering

ACC (%)

#	Model	Size	Aggregation			Average	EN · spoken-mqa_short_digit
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · spoken-mqa_short_digit
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				13.00	13.00
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				58.00	58.00
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				66.00	66.00
	Qwen/Qwen2.5-Omni-3B	5.9B				85.00	85.00
	Qwen/Qwen2.5-Omni-7B	11B				89.00	89.00
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				88.00	88.00
	google/gemma-4-E2B-it	5.1B				79.00	79.00
	microsoft/Phi-4-multimodal-instruct	5.6B				24.00	24.00
	mistralai/Voxtral-Mini-3B-2507	4.68B				71.00	71.00
	mistralai/Voxtral-Small-24B-2507	24.3B				88.00	88.00
	nvidia/audio-flamingo-3-hf	8.2B				63.00	63.00

Question Answering

FLOW_JUDGE

#	Model	Size	Aggregation			Average	FR · CohereLabs-Aya_collection	FR · Vigogne--Alpaca	FR · VoxPopuli-QA	EN · NationalSpeechCorpus_SQA	EN · OpenHermes_audio	EN · SLUE-P2-SQA5	EN · SpokenWOZ_AIR-Bench	EN · alpaca_audio	EN · fisher_AIR-Bench	EN · public-sg-speech
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR · CohereLabs-Aya_collection	FR · Vigogne--Alpaca	FR · VoxPopuli-QA	EN · NationalSpeechCorpus_SQA	EN · OpenHermes_audio	EN · SLUE-P2-SQA5	EN · SpokenWOZ_AIR-Bench	EN · alpaca_audio	EN · fisher_AIR-Bench	EN · public-sg-speech
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				62.39	47.74	55.12	80.72	55.36	59.20	80.80	61.76	65.80	67.30	54.88
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				68.34	56.77	58.08	83.20	58.56	70.60	85.20	68.91	78.20	68.70	64.48
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				59.43	32.90	42.72	76.72	56.96	61.40	87.36	68.60	65.20	72.50	64.56
	Qwen/Qwen2.5-Omni-3B	5.9B				60.33	36.13	47.04	65.76	65.36	63.20	88.08	68.39	70.20	69.70	72.16
	Qwen/Qwen2.5-Omni-7B	11B				60.04	41.29	45.28	65.36	61.92	64.40	87.44	67.98	66.20	70.70	67.36
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				68.66	47.74	54.96	78.64	67.68	78.40	86.32	73.26	76.60	77.90	77.92
	google/gemma-4-E2B-it	5.1B				57.08	36.13	40.80	70.96	64.48	47.40	77.60	72.12	45.40	76.70	70.40
	microsoft/Phi-4-multimodal-instruct	5.6B				61.85	34.84	43.84	79.92	63.76	58.00	92.56	73.99	56.40	75.90	75.20
	mistralai/Voxtral-Mini-3B-2507	4.68B				65.58	46.45	49.52	72.80	67.20	72.80	86.00	74.20	75.20	75.70	73.20
	mistralai/Voxtral-Small-24B-2507	24.3B				69.56	47.74	50.56	83.36	66.40	80.20	90.72	77.62	79.00	79.30	76.72
	nvidia/audio-flamingo-3-hf	8.2B				54.87	37.42	39.36	58.48	64.48	47.60	76.48	69.95	52.40	70.40	71.20

Tasks · Others

1st 2nd Second to last Last

Age Recognition

LABEL_MATCH

#	Model	Size	Aggregation			Average	FR · CommonVoice_Age	EN · CommonVoice_Age
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR · CommonVoice_Age	EN · CommonVoice_Age
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				38.28	27.35	49.20
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				40.24	32.48	48.00
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				29.86	20.51	39.20
	Qwen/Qwen2.5-Omni-3B	5.9B				5.16	5.13	5.20
	Qwen/Qwen2.5-Omni-7B	11B				1.45	1.71	1.20
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				9.47	8.55	10.40
	google/gemma-4-E2B-it	5.1B				1.43	0.85	2.00
	microsoft/Phi-4-multimodal-instruct	5.6B				2.83	0.85	4.80
	mistralai/Voxtral-Mini-3B-2507	4.68B				7.94	4.27	11.60
	mistralai/Voxtral-Small-24B-2507	24.3B				5.10	2.99	7.20
	nvidia/audio-flamingo-3-hf	8.2B				22.61	6.41	38.80

Dialogue Summarization

FLOW_JUDGE

#	Model	Size	Aggregation			Average	EN · NationalSpeechCorpus_SDS
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · NationalSpeechCorpus_SDS
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				43.28	43.28
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				40.96	40.96
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				51.12	51.12
	Qwen/Qwen2.5-Omni-3B	5.9B				54.00	54.00
	Qwen/Qwen2.5-Omni-7B	11B				53.36	53.36
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				58.88	58.88
	google/gemma-4-E2B-it	5.1B				54.88	54.88
	microsoft/Phi-4-multimodal-instruct	5.6B				56.24	56.24
	mistralai/Voxtral-Mini-3B-2507	4.68B				59.92	59.92
	mistralai/Voxtral-Small-24B-2507	24.3B				61.68	61.68
	nvidia/audio-flamingo-3-hf	8.2B				54.48	54.48

Diarization

FLOW_JUDGE

#	Model	Size	Aggregation			Average	EN · AMI
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · AMI
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				45.12	45.12
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				44.32	44.32
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				40.96	40.96
	Qwen/Qwen2.5-Omni-3B	5.9B				38.00	38.00
	Qwen/Qwen2.5-Omni-7B	11B				38.32	38.32
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				47.04	47.04
	google/gemma-4-E2B-it	5.1B				34.80	34.80
	microsoft/Phi-4-multimodal-instruct	5.6B				38.32	38.32
	mistralai/Voxtral-Mini-3B-2507	4.68B				34.24	34.24
	mistralai/Voxtral-Small-24B-2507	24.3B				40.48	40.48
	nvidia/audio-flamingo-3-hf	8.2B				-1.28	-1.28

Emotion Recognition

LABEL_MATCH

#	Model	Size	Aggregation			Average	FR · MELD_Emotion	EN · IEMOCAP-Emotion	EN · MELD_Emotion
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR · MELD_Emotion	EN · IEMOCAP-Emotion	EN · MELD_Emotion
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				20.02	20.40	20.88	18.40
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				29.33	28.80	29.72	30.00
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				30.14	26.00	35.34	33.20
	Qwen/Qwen2.5-Omni-3B	5.9B				27.13	24.00	28.11	32.40
	Qwen/Qwen2.5-Omni-7B	11B				30.13	29.20	28.11	34.00
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				44.84	44.40	41.37	49.20
	google/gemma-4-E2B-it	5.1B				22.72	20.40	24.10	26.00
	microsoft/Phi-4-multimodal-instruct	5.6B				26.02	23.60	24.90	32.00
	mistralai/Voxtral-Mini-3B-2507	4.68B				31.32	30.80	22.49	41.20
	mistralai/Voxtral-Small-24B-2507	24.3B				37.83	40.80	34.54	35.20
	nvidia/audio-flamingo-3-hf	8.2B				35.44	28.00	42.57	43.20

Gender Recognition

LABEL_MATCH

#	Model	Size	Aggregation			Average	FR · CommonVoice_Gender	EN · CommonVoice_Gender	EN · IEMOCAP-gender
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR · CommonVoice_Gender	EN · CommonVoice_Gender	EN · IEMOCAP-gender
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				57.40	55.20	55.60	63.60
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				66.10	61.60	68.40	72.80
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				84.50	72.80	94.40	98.00
	Qwen/Qwen2.5-Omni-3B	5.9B				13.10	3.20	10.40	35.60
	Qwen/Qwen2.5-Omni-7B	11B				18.40	15.20	9.20	34.00
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				87.70	81.60	90.40	97.20
	google/gemma-4-E2B-it	5.1B				20.50	9.60	23.60	39.20
	microsoft/Phi-4-multimodal-instruct	5.6B				45.20	40.00	46.80	54.00
	mistralai/Voxtral-Mini-3B-2507	4.68B				33.00	40.00	31.20	20.80
	mistralai/Voxtral-Small-24B-2507	24.3B				33.50	35.20	43.20	20.40
	nvidia/audio-flamingo-3-hf	8.2B				73.70	54.00	93.60	93.20

Spoken Language Identification

LABEL_MATCH

#	Model	Size	Aggregation			Average	EN · CoVost2_AIR-Bench
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · CoVost2_AIR-Bench
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				80.40	80.40
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				94.80	94.80
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				88.40	88.40
	Qwen/Qwen2.5-Omni-3B	5.9B				53.20	53.20
	Qwen/Qwen2.5-Omni-7B	11B				89.60	89.60
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				96.80	96.80
	google/gemma-4-E2B-it	5.1B				70.00	70.00
	microsoft/Phi-4-multimodal-instruct	5.6B				44.00	44.00
	mistralai/Voxtral-Mini-3B-2507	4.68B				70.40	70.40
	mistralai/Voxtral-Small-24B-2507	24.3B				67.60	67.60
	nvidia/audio-flamingo-3-hf	8.2B				76.40	76.40

Temporal

TEMPORAL_REGEX

#	Model	Size	Aggregation			Average	EN · SLUE-SQA5_time2word	EN · SLUE-SQA5_word2time
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · SLUE-SQA5_time2word	EN · SLUE-SQA5_word2time
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				22.03	28.80	15.26
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				44.02	44.80	43.23
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				10.73	9.20	12.25
	Qwen/Qwen2.5-Omni-3B	5.9B				10.50	7.60	13.40
	Qwen/Qwen2.5-Omni-7B	11B				12.41	10.00	14.82
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				18.64	9.60	27.68
	google/gemma-4-E2B-it	5.1B				9.83	11.20	8.47
	microsoft/Phi-4-multimodal-instruct	5.6B				4.27	3.60	4.94
	mistralai/Voxtral-Mini-3B-2507	4.68B				9.84	6.80	12.89
	mistralai/Voxtral-Small-24B-2507	24.3B				11.59	8.80	14.38
	nvidia/audio-flamingo-3-hf	8.2B				7.54	4.40	10.68

Tasks · Music

1st 2nd Second to last Last

Music Captioning

FLOW_JUDGE

#	Model	Size	Aggregation			Average	EN · MusicCaps
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · MusicCaps
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				41.36	41.36
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				41.92	41.92
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				59.60	59.60
	Qwen/Qwen2.5-Omni-3B	5.9B				35.76	35.76
	Qwen/Qwen2.5-Omni-7B	11B				33.28	33.28
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				58.88	58.88
	google/gemma-4-E2B-it	5.1B				31.92	31.92
	microsoft/Phi-4-multimodal-instruct	5.6B				44.24	44.24
	mistralai/Voxtral-Mini-3B-2507	4.68B				44.80	44.80
	mistralai/Voxtral-Small-24B-2507	24.3B				52.72	52.72
	nvidia/audio-flamingo-3-hf	8.2B				60.88	60.88

Music Question Answering

FLOW_JUDGE

#	Model	Size	Aggregation			Average	FR · MusicCaps-QA	EN · MTJ-Jamendo_AIR-Bench	EN · MuChoMusic	EN · MusicCaps-QA
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	FR · MusicCaps-QA	EN · MTJ-Jamendo_AIR-Bench	EN · MuChoMusic	EN · MusicCaps-QA
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				42.31	48.48	23.20	38.40	46.80
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				49.01	52.56	30.40	56.00	50.00
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				60.32	55.12	62.80	71.20	62.56
	Qwen/Qwen2.5-Omni-3B	5.9B				40.56	36.32	24.80	69.20	40.40
	Qwen/Qwen2.5-Omni-7B	11B				40.67	36.40	30.80	69.60	34.40
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				64.43	61.44	63.60	75.60	63.04
	google/gemma-4-E2B-it	5.1B				30.83	31.12	2.80	60.40	28.40
	microsoft/Phi-4-multimodal-instruct	5.6B				48.53	52.32	31.20	52.40	50.64
	mistralai/Voxtral-Mini-3B-2507	4.68B				41.61	51.84	10.40	30.40	53.36
	mistralai/Voxtral-Small-24B-2507	24.3B				53.33	53.44	45.20	63.20	51.28
	nvidia/audio-flamingo-3-hf	8.2B				55.91	49.52	65.60	66.80	54.48

Model	Size	EN · MTJ-Jamendo_AIR-Bench	EN · MusicCaps-QA	MCQ (Text Instruction + Audio Context) · MuChoMusic
OpenLLM-France/Luciole-1B-Audio-1.0	1.9B	40.63	48.48	23.20	46.80	38.40
OpenLLM-France/Luciole-8B-Audio-1.0	8.8B	49.59	52.56	30.40	50.00	56.00
Qwen/Qwen2-Audio-7B-Instruct	8.4B	63.00	55.12	62.80	62.56	71.20
Qwen/Qwen2.5-Omni-3B	5.9B	46.04	36.32	24.80	40.40	69.20
Qwen/Qwen2.5-Omni-7B	11B	46.20	36.40	30.80	34.40	69.60
Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B	66.79	61.44	63.60	63.04	75.60
google/gemma-4-E2B-it	5.1B	35.71	31.12	2.80	28.40	60.40
microsoft/Phi-4-multimodal-instruct	5.6B	48.55	52.32	31.20	50.64	52.40
mistralai/Voxtral-Mini-3B-2507	4.68B	38.04	51.84	10.40	53.36	30.40
mistralai/Voxtral-Small-24B-2507	24.3B	54.96	53.44	45.20	51.28	63.20
nvidia/audio-flamingo-3-hf	8.2B	58.79	49.52	65.60	54.48	66.80

Tasks · Sound

1st 2nd Second to last Last

Audio Captioning

FLOW_JUDGE

#	Model	Size	Aggregation			Average	EN · AudioCaps	EN · WavCaps
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · AudioCaps	EN · WavCaps
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				41.92	42.16	41.68
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				45.64	47.68	43.60
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				53.08	54.16	52.00
	Qwen/Qwen2.5-Omni-3B	5.9B				34.56	35.52	33.60
	Qwen/Qwen2.5-Omni-7B	11B				47.40	48.80	46.00
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				54.36	52.80	55.92
	google/gemma-4-E2B-it	5.1B				20.92	20.88	20.96
	microsoft/Phi-4-multimodal-instruct	5.6B				45.60	47.12	44.08
	mistralai/Voxtral-Mini-3B-2507	4.68B				40.12	38.64	41.60
	mistralai/Voxtral-Small-24B-2507	24.3B				34.64	33.36	35.92
	nvidia/audio-flamingo-3-hf	8.2B				58.88	63.92	53.84

Audio Question Answering

FLOW_JUDGE

#	Model	Size	Aggregation			Average	EN · AudioCaps-QA	EN · Clotho-AQA	EN · WavCaps-QA
#	Model	Size	Min-Max	Z-Score	Avg Rank	Average	EN · AudioCaps-QA	EN · Clotho-AQA	EN · WavCaps-QA
	OpenLLM-France/Luciole-1B-Audio-1.0	1.9B				52.21	44.00	63.36	49.28
	OpenLLM-France/Luciole-8B-Audio-1.0	8.8B				55.44	46.48	69.68	50.16
	Qwen/Qwen2-Audio-7B-Instruct	8.4B				61.33	64.64	56.64	62.72
	Qwen/Qwen2.5-Omni-3B	5.9B				52.37	51.68	57.76	47.68
	Qwen/Qwen2.5-Omni-7B	11B				56.37	55.60	61.84	51.68
	Qwen/Qwen3-Omni-30B-A3B-Instruct	35.3B				64.24	63.68	67.60	61.44
	google/gemma-4-E2B-it	5.1B				33.87	33.68	36.08	31.84
	microsoft/Phi-4-multimodal-instruct	5.6B				58.67	57.68	62.16	56.16
	mistralai/Voxtral-Mini-3B-2507	4.68B				54.83	56.32	55.12	53.04
	mistralai/Voxtral-Small-24B-2507	24.3B				56.24	52.48	61.60	54.64
	nvidia/audio-flamingo-3-hf	8.2B				59.84	62.56	54.08	62.88