Skip to content

[fix](ann-index) Fix ivf recall zero and oom.#64082

Open
kaka11chen wants to merge 3 commits into
apache:masterfrom
kaka11chen:ann-build-full-buffer-no-spill
Open

[fix](ann-index) Fix ivf recall zero and oom.#64082
kaka11chen wants to merge 3 commits into
apache:masterfrom
kaka11chen:ann-build-full-buffer-no-spill

Conversation

@kaka11chen
Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Copilot AI review requested due to automatic review settings June 3, 2026 12:01
@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen kaka11chen changed the title Ann build full buffer no spill [fix](ann-index) Fix ivf recall zero and oom. Jun 3, 2026
@kaka11chen
Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@kaka11chen
Copy link
Copy Markdown
Contributor Author

run buildall

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@kaka11chen kaka11chen force-pushed the ann-build-full-buffer-no-spill branch from a66b5d7 to 582071f Compare June 3, 2026 13:21
@kaka11chen
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 29269 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 582071f0eeb96aa5a7754df2d0ef4ec12745e462, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17827	4092	4069	4069
q2	q3	10781	1462	831	831
q4	4692	477	345	345
q5	7544	876	599	599
q6	181	170	137	137
q7	773	873	627	627
q8	9412	1593	1572	1572
q9	5712	4555	4464	4464
q10	6760	1803	1534	1534
q11	440	277	253	253
q12	634	440	289	289
q13	18142	3384	2813	2813
q14	267	264	240	240
q15	q16	833	782	713	713
q17	1020	947	950	947
q18	6921	5860	5590	5590
q19	1362	1186	1034	1034
q20	518	413	253	253
q21	6177	2870	2637	2637
q22	472	376	322	322
Total cold run time: 100468 ms
Total hot run time: 29269 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5113	4833	4845	4833
q2	q3	4887	5277	4748	4748
q4	2119	2219	1416	1416
q5	4790	4861	4656	4656
q6	228	176	127	127
q7	1850	1827	1561	1561
q8	2405	2136	2159	2136
q9	7877	7608	7402	7402
q10	4720	4691	4229	4229
q11	534	386	352	352
q12	727	756	522	522
q13	3013	3363	2818	2818
q14	281	278	263	263
q15	q16	675	713	612	612
q17	1287	1259	1259	1259
q18	7369	6856	6835	6835
q19	1129	1122	1135	1122
q20	2225	2206	1942	1942
q21	5278	4622	4494	4494
q22	516	476	419	419
Total cold run time: 57023 ms
Total hot run time: 51746 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 169468 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 582071f0eeb96aa5a7754df2d0ef4ec12745e462, data reload: false

query5	4322	636	479	479
query6	443	203	184	184
query7	4814	565	303	303
query8	378	220	208	208
query9	8783	4008	4018	4008
query10	475	331	259	259
query11	5927	2321	2229	2229
query12	158	103	100	100
query13	1264	625	455	455
query14	6423	5360	4995	4995
query14_1	4369	4344	4350	4344
query15	211	201	177	177
query16	1017	449	434	434
query17	921	679	576	576
query18	2452	468	342	342
query19	198	176	138	138
query20	110	111	104	104
query21	218	137	115	115
query22	13736	13525	13321	13321
query23	17626	16887	16554	16554
query23_1	16584	16352	16254	16254
query24	7673	1716	1296	1296
query24_1	1323	1310	1338	1310
query25	572	469	418	418
query26	1294	332	172	172
query27	2675	530	351	351
query28	4453	2033	2003	2003
query29	1095	641	508	508
query30	305	239	201	201
query31	1130	1092	957	957
query32	121	68	62	62
query33	546	322	274	274
query34	1188	1126	646	646
query35	764	789	686	686
query36	1379	1457	1231	1231
query37	157	107	93	93
query38	3192	3133	3056	3056
query39	920	921	890	890
query39_1	889	903	872	872
query40	223	158	101	101
query41	65	62	62	62
query42	98	96	93	93
query43	320	321	282	282
query44	
query45	196	187	178	178
query46	1094	1250	770	770
query47	2344	2422	2254	2254
query48	411	401	306	306
query49	628	468	359	359
query50	994	351	265	265
query51	4359	4337	4220	4220
query52	87	90	77	77
query53	237	271	199	199
query54	262	213	195	195
query55	80	77	69	69
query56	219	239	246	239
query57	1460	1395	1338	1338
query58	247	215	212	212
query59	1580	1620	1415	1415
query60	281	283	226	226
query61	160	160	155	155
query62	705	671	591	591
query63	224	181	186	181
query64	2559	767	628	628
query65	
query66	1821	469	348	348
query67	29740	29774	29591	29591
query68	
query69	430	295	263	263
query70	988	929	962	929
query71	305	225	251	225
query72	3008	2772	2419	2419
query73	843	740	465	465
query74	5129	4932	4768	4768
query75	2662	2589	2264	2264
query76	2329	1141	765	765
query77	365	371	294	294
query78	12462	12331	12014	12014
query79	1234	1056	767	767
query80	514	480	385	385
query81	445	282	241	241
query82	233	159	123	123
query83	275	278	248	248
query84	294	139	111	111
query85	833	531	434	434
query86	346	298	304	298
query87	3352	3345	3236	3236
query88	3608	2729	2721	2721
query89	414	384	328	328
query90	2190	183	179	179
query91	173	167	137	137
query92	63	61	54	54
query93	1481	1478	849	849
query94	549	356	322	322
query95	664	373	344	344
query96	1098	825	340	340
query97	2721	2699	2575	2575
query98	213	206	203	203
query99	1159	1175	1053	1053
Total cold run time: 251293 ms
Total hot run time: 169468 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 96.20% (76/79) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.91% (21032/39013)
Line Coverage 37.57% (199809/531817)
Region Coverage 33.67% (156804/465736)
Branch Coverage 34.63% (68582/198021)

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 96.20% (76/79) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.92% (21036/39013)
Line Coverage 37.61% (199993/531817)
Region Coverage 33.69% (156893/465736)
Branch Coverage 34.65% (68611/198021)

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 96.25% (77/80) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.88% (27478/38229)
Line Coverage 55.48% (294351/530545)
Region Coverage 52.30% (245926/470246)
Branch Coverage 53.41% (106186/198814)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants