KDD 2026 Paper Review

25분 분량

174: CCQ: An LLM-Curated Child Care Quality Dataset to Support AI Research for Children’s Health

Title*

Brief summary of your review.

The paper introduces CCQ, a de-identified dataset of 7,929 child care providers from Georgia with 245 variables, alongside an LLM-based curation pipeline. While the domain is socially important, the benchmark lacks baseline results and the End-to-End vs Coder comparison suffers from a model size confound.

본 논문은 조지아주의 7,929개 보육 시설 정보를 245개 변수로 비식별화한 CCQ 데이터셋과 LLM 기반 큐레이션 파이프라인을 소개한다. 해당 도메인은 사회적으로 중요하나, 벤치마크에 베이스라인 결과가 없고 End-to-End vs Coder 비교에 모델 크기 교란 변인이 존재한다.

Review*

Please provide an evaluation of the quality, clarity, originality and significance of this work, including a list of its pros and cons (max 200000 characters). Add formatting using Markdown and formulas using LaTeX. For more information see
TeX is supported

The dataset addresses a socially important but underexplored domain and demonstrates strong reproducibility efforts. However, two critical flaws undermine the core contributions: (1) the End-to-End vs Coder comparison confounds paradigm differences with a 3.75x model size gap, and (2) the proposed benchmark tasks lack any baseline results. Additionally, the methodology novelty is overclaimed given uncited prior work on LLM-based data curation.

Pros:

  1. Under-explored domain: Child care quality is underexplored in AI; making structured data available is a meaningful contribution.
  2. Dual-version release: CCQ-Anon-Raw and CCQ-Anon-Norm serve different research needs (exploration vs. ready-to-analyze).
  3. Reproducibility: Open-source models (Qwen3), public code/prompts, and detailed pipeline documentation (Appendix C-D).
  4. Honest failure reporting: End-to-End’s 35.2% on DeID is transparently reported.

Cons:

  1. Critical experimental confound: End-to-End (Qwen3-8B) vs Coder (Qwen3-30B-Coder) differs by 3.75x in model size. Performance gaps cannot be attributed to paradigm vs capacity without same-size comparisons.
  2. No benchmark baselines: Three benchmark tasks are defined but zero methods evaluated—not even trivial baselines. The benchmark’s utility is undemonstrated.
  3. Overclaimed novelty: “Novel LLM-based multi-agent pipeline” ignores directly relevant prior work: Jellyfish (2023), Bendinelli et al. (2025), CodeGenWrangler (NAACL 2025), Narayan et al. (DEEM@SIGMOD 2024), DCA-Bench (KDD 2025).

Questions for Rebuttal:

  1. Can you provide same-size model comparisons (e.g., Qwen3-8B vs Qwen3-8B-Coder) to isolate paradigm effects?
  2. Can you include at least 2-3 baseline methods per benchmark task?
  3. How does CCQ improve upon the raw DECAL public data beyond what pandas scripting could achieve?
  4. The paper identifies complementary strengths between E2E and Coder (Discussion, Section 4) and recommends task-specific paradigm selection. Have you evaluated a hybrid pipeline that routes tasks to E2E or Coder based on task type (e.g., E2E for NormMlc/NormComplex, Coder for DeID/Removal/Norm2Num)? Without this experiment, the practical recommendation remains unvalidated, and the optimal pipeline configuration is unclear.

사회적으로 중요하지만 미개척된 도메인을 다루며 재현성 측면에서 노력이 돋보인다. 그러나 두 가지 치명적 결함이 핵심 기여를 약화시킨다: (1) End-to-End vs Coder 비교가 패러다임 차이와 3.75배 모델 크기 차이를 혼동하고, (2) 제안된 벤치마크 태스크에 베이스라인 결과가 전무하다. 또한 LLM 기반 데이터 큐레이션의 선행 연구가 인용되지 않아 방법론 신규성이 과대 주장되었다.

장점:

  1. 미개척 도메인: 보육 품질은 AI 연구에서 탐구가 부족한 영역으로, 구조화된 데이터 공개는 의미 있는 기여이다.
  2. 이중 버전 공개: CCQ-Anon-Raw(탐색용)와 CCQ-Anon-Norm(즉시 분석용)이 서로 다른 연구 요구를 충족한다.
  3. 재현성: 오픈소스 모델(Qwen3), 공개 코드/프롬프트, 상세 파이프라인 문서화(Appendix C-D).
  4. 정직한 실패 보고: End-to-End의 DeID 정확도 35.2%를 투명하게 보고.

단점:

  1. [치명적] 실험 교란 변인: End-to-End(Qwen3-8B) vs Coder(Qwen3-30B-Coder)는 모델 크기가 3.75배 차이. 동일 크기 비교 없이는 성능 차이를 패러다임 vs 용량으로 귀인할 수 없다.
  2. [치명적] 벤치마크 베이스라인 부재: 세 가지 벤치마크 태스크가 정의되었으나 기초적 베이스라인조차 평가되지 않아 유용성이 입증되지 않았다.
  3. [주요] 신규성 과대 주장: “새로운 LLM 기반 멀티에이전트 파이프라인” 주장이 직접 관련 선행 연구 미인용: Jellyfish (2023), Bendinelli et al. (2025), CodeGenWrangler (NAACL 2025), Narayan et al. (DEEM@SIGMOD 2024), DCA-Bench (KDD 2025).

반박을 위한 질문:

  1. 패러다임 효과 분리를 위한 동일 크기 모델 비교(예: Qwen3-8B vs Qwen3-8B-Coder)를 제공할 수 있는가?
  2. 각 벤치마크 태스크에 최소 2-3개 베이스라인을 포함할 수 있는가?
  3. CCQ가 원시 DECAL 공공 데이터 대비 pandas 스크립팅 이상의 부가 가치는 무엇인가?
  4. 논문이 Discussion(Section 4)에서 E2E와 Coder의 상보적 강점을 확인하고 태스크별 패러다임 선택을 추천하고 있는데, 태스크 유형에 따라 E2E 또는 Coder로 라우팅하는 하이브리드 파이프라인을 평가한 적이 있는가? (예: NormMlc/NormComplex는 E2E, DeID/Removal/Norm2Num은 Coder) 이 실험 없이는 실용적 추천이 검증되지 않으며, 최적 파이프라인 구성이 불명확하다.
Rating*

10: Top 5% of accepted papers, seminal paper
9: Top 15% of accepted papers, strong accept
8: Top 50% of accepted papers, clear accept
7: Good paper, accept
6: Marginally above acceptance threshold
5: Marginally below acceptance threshold
4: Ok but not good enough - rejection
3: Clear rejection
2: Strong rejection
1: Trivial or wrong

4
Paper Summary*

Please briefly summarize the main points and contributions of this paper.

This paper presents CCQ, a dataset of 7,929 child care providers from Georgia’s DECAL (Department of Early Care and Learning) public records, with 245 variables capturing facility characteristics, licensing compliance, and quality ratings. The dataset is released in two versions (minimally processed and fully normalized) and is accompanied by an LLM-based curation pipeline using two paradigms: End-to-End (Qwen3-8B, sample-by-sample) and Coder (Qwen3-30B-Coder, code generation + batch execution). The pipeline handles anonymization, validation, and normalization, validated against human-curated ground truth across 29 subtasks. Three downstream benchmark tasks are proposed: missing variable imputation, quality rating score estimation, and causal discovery of child care quality drivers.

본 논문은 조지아주 DECAL(Department of Early Care and Learning, 영유아 보육·교육부) 공공 기록에서 수집한 7,929개 보육 시설의 데이터셋 CCQ를 제시한다. 시설 특성, 인가 준수 여부, 품질 등급을 포착하는 245개 변수를 포함한다. 최소 처리 버전과 완전 정규화 버전의 두 가지로 공개되며, End-to-End(Qwen3-8B, 샘플 단위)와 Coder(Qwen3-30B-Coder, 코드 생성 + 배치 실행) 두 패러다임을 사용하는 LLM 기반 큐레이션 파이프라인이 동반된다. 파이프라인은 익명화, 검증, 정규화를 처리하며, 29개 서브태스크에 걸쳐 인간 큐레이션 정답과 비교 검증된다. 결측 변수 대치, 품질 등급 점수 추정, 보육 품질 동인의 인과 발견 등 세 가지 다운스트림 벤치마크 태스크가 제안된다.

Paper Strengths*

Please provide a list of the strengths of this paper, including but not limited to: innovative and practical methodology, insightful empirical findings or in-depth theoretical analysis, well-structured review of relevant literature, and any other factors that may make the paper valuable to readers.

  • Socially important, underexplored domain with structured data made accessible.
  • Dual-version release (Raw for exploration, Norm for analysis) serves different research needs.
  • Strong reproducibility: open-source models (Qwen3), public code/prompts, detailed pipeline documentation (Appendix C-D).
  • Honest failure reporting (End-to-End’s 35.2% on DeID) demonstrates scientific integrity.
  • 사회적으로 중요하고 AI 연구에서 미개척된 도메인의 구조화된 데이터 공개.
  • 이중 버전 공개(탐색용 Raw, 분석용 Norm)가 서로 다른 연구 요구를 충족.
  • 강력한 재현성: 오픈소스 모델(Qwen3), 공개 코드/프롬프트, 상세 파이프라인 문서화(Appendix C-D).
  • 실패 사례의 정직한 보고(End-to-End DeID 정확도 35.2%)가 과학적 진정성을 보여준다.
Paper Weaknesses*

Please provide a list of the weaknesses of this paper, including but not limited to: inadequate implementation details for reproducing the study, limited evaluation and ablation studies for the proposed method, correctness of the theoretical analysis or experimental results, lack of comparisons or discussions with widely-known baselines in the field, lack of clarity in exposition, or any other factors that may impede the reader’s understanding or benefit from the paper. Please kindly refrain from providing a general assessment of the paper’s novelty without providing detailed explanations.

  • [Critical] Model size confound: End-to-End (Qwen3-8B) vs Coder (Qwen3-30B-Coder) differs by 3.75x in parameters. Performance gaps cannot be attributed to paradigm vs capacity without same-size comparisons.

  • [Critical] No benchmark baselines: Three benchmark tasks defined but zero methods evaluated—not even trivial baselines. The benchmark’s utility is undemonstrated.

  • [Major] Novelty overclaimed: “Novel LLM-based multi-agent pipeline” ignores directly relevant prior work: Jellyfish (2023), Bendinelli et al. (2025), CodeGenWrangler (NAACL 2025), Narayan et al. (DEEM@SIGMOD 2024), DCA-Bench (KDD 2025).

    Prior work summaries:

    1. Jellyfish (Zhang et al., 2023): Instruction-tuned local LLMs (7-13B) as universal data preprocessing solvers for error detection, data imputation, schema matching, and entity matching. Achieves GPT-3.5/4 competitive performance on a single GPU without API dependency. Demonstrates that smaller, specialized LLMs can handle diverse DP tasks with strong generalizability.
    2. Bendinelli et al. (2025): Explores LLM agents paired with Python for cleaning tabular ML datasets through iterative tool calls and performance feedback. Benchmarks LLM agents on intentionally corrupted datasets, finding they correct simple row-level anomalies but struggle with distributional shifts. Published at ICLR 2025 Workshop on Foundation Models in the Wild.
    3. CodeGenWrangler (Akella et al., NAACL 2025): Automates data wrangling (imputation, error detection, correction) by generating executable code via LLMs combined with histogram-based feature selection. Uses RAG-inspired external knowledge base lookup to handle semantic context in tabular data. Outperforms traditional statistical and deep learning approaches at lower computational cost.
    4. Narayan et al. (VLDB 2022): Casts five classical data cleaning/integration tasks as foundation model prompting tasks, demonstrating SOTA performance without task-specific training. Establishes that large pretrained models generalize to data wrangling tasks like entity matching, error detection, and data imputation. Foundational work showing FM applicability to data management beyond NLP.
    5. DCA-Bench (Huang et al., KDD 2025): Benchmark for dataset curation agents with 221 real cases from 8 major data platforms, organized into 4 categories with 18 tags. Provides 4 levels of hints and LLM-based automatic evaluation replacing human annotators. Reveals even advanced models detect only ~30% of dataset issues without assistance, underscoring need for stronger curation tools.

  • [치명적] 모델 크기 교란 변인: End-to-End(Qwen3-8B) vs Coder(Qwen3-30B-Coder)는 파라미터 수가 3.75배 차이. 동일 크기 비교 없이는 성능 차이를 패러다임 vs 용량으로 귀인할 수 없다.

  • [치명적] 벤치마크 베이스라인 부재: 세 가지 벤치마크 태스크가 정의되었으나 기초적 베이스라인조차 평가되지 않아 유용성이 입증되지 않았다.

  • [주요] 신규성 과대 주장: “새로운 LLM 기반 멀티에이전트 파이프라인” 주장이 직접 관련 선행 연구 미인용: Jellyfish (2023), Bendinelli et al. (2025), CodeGenWrangler (NAACL 2025), Narayan et al. (DEEM@SIGMOD 2024), DCA-Bench (KDD 2025).

    선행 연구 요약:

    1. Jellyfish (Zhang et al., 2023): 명령어 튜닝된 로컬 LLM(7-13B)을 오류 감지, 데이터 대치, 스키마 매칭, 엔티티 매칭을 위한 범용 데이터 전처리 솔버로 활용. API 의존 없이 단일 GPU에서 GPT-3.5/4 경쟁 성능 달성. 소규모 특화 LLM이 다양한 데이터 전처리 태스크를 강한 일반화 능력으로 처리할 수 있음을 입증.
    2. Bendinelli et al. (2025): 반복적 도구 호출과 성능 피드백을 통해 표 형식 ML 데이터셋 정제를 위한 LLM 에이전트와 Python 활용 탐구. 의도적으로 오염된 데이터셋에서 LLM 에이전트를 벤치마킹하여, 단순 행 수준 이상치는 수정하나 분포 변화에는 어려움을 확인. ICLR 2025 Workshop on Foundation Models in the Wild에서 발표.
    3. CodeGenWrangler (Akella et al., NAACL 2025): LLM을 통한 실행 가능 코드 생성과 히스토그램 기반 피처 선택을 결합하여 데이터 정제(대치, 오류 감지, 수정)를 자동화. RAG 영감의 외부 지식 베이스 검색으로 표 형식 데이터의 의미적 맥락 처리. 기존 통계적 및 딥러닝 접근법 대비 낮은 계산 비용으로 우수한 성능 달성.
    4. Narayan et al. (VLDB 2022): 5가지 고전적 데이터 정제/통합 태스크를 파운데이션 모델 프롬프팅 태스크로 변환하여, 태스크 특화 학습 없이 SOTA 성능 입증. 대규모 사전학습 모델이 엔티티 매칭, 오류 감지, 데이터 대치 등 데이터 정제 태스크에 일반화됨을 확립. NLP를 넘어 데이터 관리 영역에 FM 적용 가능성을 보여준 기초적 연구.
    5. DCA-Bench (Huang et al., KDD 2025): 8개 주요 데이터 플랫폼의 221개 실제 사례로 구성된 데이터셋 큐레이션 에이전트 벤치마크. 4개 카테고리와 18개 태그로 조직되며, 4단계 힌트와 인간 평가자를 대체하는 LLM 기반 자동 평가 제공. 고급 모델도 보조 없이 데이터셋 문제의 ~30%만 감지함을 밝혀 강력한 큐레이션 도구의 필요성 강조.
Benchmark*

4: Strong Benchmark - A difficult-to-create benchmark (dataset, framework, or both) addressing an important problem, enabling broad algorithm evaluation, and having high potential for widespread use.
3: Good Benchmark - useful benchmark with some challenges and generalizability, likely to be adopted but with limitations in scope, coverage, or long-term impact.
2: Limited Benchmark Value - Includes a benchmarking component but is narrow in scope, lacks generality, or has limited potential for widespread adoption.
1: Not a Benchmark - Focuses on specialized algorithms rather than a reusable benchmark, with little value for broader evaluation.

Benchmark tasks are defined but lack any baseline evaluations, making the benchmark aspirational rather than realized.

2

벤치마크 태스크가 정의되었으나 베이스라인 평가가 전무하여, 벤치마크가 실현되지 않은 포부에 머물러 있다.

Relevance*

4: High - The dataset or the benchmark address an important problem in machine learning and is of broad interest to the community
3: Moderate - The work is somewhat relevant to KDD and is of narrow interest to a sub-community
2: Low - The connection to KDD is weak
1: Poor - The work is irrelevant to KDD

Child care quality is a socially relevant domain for KDD, but the narrow geographic scope (Georgia only) limits broader community interest.

3

보육 품질은 KDD에 사회적으로 관련된 도메인이나, 좁은 지리적 범위(조지아주 한정)가 광범위한 커뮤니티 관심을 제한한다.

Novelty*

4: High - The dataset is novel compared to existing resources.The paper introduces new data types, tasks, or problem formulations. The level of innovation is high, leading to major advancements and potentially inspiring further research and development.
3: Moderate - The dataset is moderately novel and introduces new and interesting data types, tasks, or problem formulations. The contribution is original and represents an advancement of existing knowledge.
2: Low - The contributions are relatively minor and largely incremental. The work builds heavily on existing datasets and benchmarks.
1: Poor - The paper presents datasets and benchmarks that are well-known and have been extensively covered previously. There are no new contributions or unique perspectives.

The dataset itself covers an underexplored domain, but the LLM curation pipeline closely mirrors uncited prior work (Jellyfish, CodeGenWrangler, DCA-Bench).

2

데이터셋 자체는 미개척 도메인을 다루나, LLM 큐레이션 파이프라인이 인용되지 않은 선행 연구(Jellyfish, CodeGenWrangler, DCA-Bench)와 유사하다.

Technical Quality*

4: High - The dataset is well-curated, clean, and representative of real-world scenarios. The data collection and evaluation procedures are well-documented and reproducible. Any biases or limitations are appropriately discussed.
3: Moderate - The dataset demonstrates solid technical quality with a sound methodology and thorough analysis. The data collection and evaluation procedures are reliable and well-supported. There may be minor issues, but they do not significantly undermine the overall quality. The work is competently executed and meets acceptable standards.
2: Low - The dataset, data collection, and evaluation procedures have several technical weaknesses, such as methodological flaws, insufficient analysis, or unsupported conclusions. While the work shows some level of competence, it lacks thoroughness and precision. Improvements are necessary for it to be considered robust.
1: Poor - The dataset, data collection, and evaluation procedures have significant technical errors, methodological flaws, or incorrect conclusions. The work lacks rigor, and the results are unreliable. The overall quality is below acceptable standards, and the technical execution is weak.

The End-to-End vs Coder comparison is confounded by model size, and single-run evaluations without statistical tests weaken reliability.

2

End-to-End vs Coder 비교가 모델 크기로 교란되었으며, 통계 검정 없는 단일 실행 평가가 신뢰성을 약화시킨다.

Usability And Accessibility*

4: High - The dataset and benchmarking are easily accessible and well-documented. The code, scripts, and metadata are provided for easy use. The dataset complies with legal and privacy considerations.
3: Moderate - The dataset and benchmarks are accessible with some effort. The code, scripts, and metadata can be understood with some effort.
2: Low - The dataset and benchmarks have noticeable issues with accessibility, legality, or privacy.
1: Poor - The dataset and benchmarks are poorly designed or hard to understand.

Dual-version release with Zenodo hosting and CC BY-NC-SA 4.0 license; documentation is thorough but benchmark tasks need starter code.

3

Zenodo 호스팅과 CC BY-NC-SA 4.0 라이선스를 갖춘 이중 버전 공개; 문서화는 충실하나 벤치마크 태스크에 스타터 코드가 필요하다.

Reproducibility*

4: High - The data collection procedures are well-documented and reproducible. Any biases or limitations are appropriately discussed. The evaluation setup is robust and aligned with best practices. Supplementary materials, including datasets and code, are complete, well-documented, and easily accessible. Reproducing the results would be straightforward and require minimal additional effort.
3: Moderate - The paper provides a clear and detailed description of the methods, data, and procedures used. Supplementary materials, such as datasets and code, are available and sufficiently documented. Reproducing the results would be feasible with the provided information, though some effort may still be required.
2: Low - The paper includes some information about the methods, data, and procedures, but key details are missing. There may be supplementary materials, but they are incomplete or unclear. Reproducing the results would require significant effort and additional information.
1: Poor - The paper provides insufficient details about the methods, data, and procedures used. There are no available supplementary materials, and the description is so vague that reproducing the results would be extremely difficult or impossible.

Open-source models, public code, and detailed prompts enable reproduction of the curation pipeline, though benchmark reproducibility is untested.

3

오픈소스 모델, 공개 코드, 상세 프롬프트가 큐레이션 파이프라인의 재현을 가능하게 하나, 벤치마크 재현성은 미검증 상태이다.

Reviewer Confidence*

4: High - The reviewer is an expert in the subject area and has extensive knowledge of the research methods and context of the paper. They are highly confident in their ability to provide an accurate and thorough assessment. Their evaluation is based on deep expertise and a comprehensive understanding of the work.
3: Moderate - The reviewer has a good understanding of the subject area and is familiar with the research methods and context of the paper. They feel confident in their ability to accurately assess the quality and significance of the work. Their evaluation is based on a solid grasp of the content and context.
2: Low - The reviewer has some knowledge of the subject area and is somewhat familiar with the research methods and context of the paper. They understand the main points but may lack depth in certain areas. The reviewer is reasonably confident in their assessment but acknowledges some limitations in their expertise.
1: Poor - The reviewer has limited knowledge of the subject area and is not very familiar with the specific research methods or context of the paper. The reviewer is unsure about their ability to accurately assess the paper and may have had difficulty understanding key aspects of the work. Their evaluation should be considered with caution.

Familiar with LLM-based data curation and dataset/benchmark evaluation methodology; less domain expertise in child care policy.

3

LLM 기반 데이터 큐레이션 및 데이터셋/벤치마크 평가 방법론에 익숙하나, 보육 정책 도메인 전문성은 부족하다.

Ethics Review Flag*

Please select Yes if there are ethical issues with the data collection, bias, or potential misuse, and specify the issue in the text box below. For guidance, please refer to ACM Code of Ethics (https://www.acm.org/code-of-ethics), ACM Policy on Plagiarism, Misrepresentation, and Falsification (https://www.acm.org/publications/policies/plagiarism-overview), ACM Publications Policy on Research Involving Human Participants and Subjects (https://www.acm.org/publications/policies/research-involving-human-participants-and-subjects). If in doubt, please enquire with the Program Chairs.
No

Ethics Review Description*

If you select Yes to the ethics review flag above please describe the issue.

No major ethical concerns identified. The study has IRB approval (Emory, Protocol 2025P012045), uses facility-level public records only, contains no individual-level data, and applies appropriate de-identification. Minor concern: re-identification risk when combining provider type, licensed capacity, and fee variables in small counties could potentially narrow down specific facilities, though the county variable has been randomized. The CC BY-NC-SA 4.0 license appropriately restricts commercial use.

주요 윤리적 우려 사항은 확인되지 않았다. 본 연구는 IRB 승인(Emory, Protocol 2025P012045)을 받았으며, 시설 수준 공공 기록만 사용하고, 개인 수준 데이터를 포함하지 않으며, 적절한 비식별화를 적용한다. 소규모 카운티에서 시설 유형, 인가 수용 인원, 비용 변수를 결합할 경우 특정 시설을 좁혀낼 수 있는 재식별 위험이 경미하게 존재하나, 카운티 변수는 무작위화되었다. CC BY-NC-SA 4.0 라이선스가 상업적 사용을 적절히 제한한다.

Llm Usage Description*

Please describe in what ways you have used LLMs in this review. This is not disclosed to authors.

LLMs (Claude) were used to assist in structuring this review. Two specialized review agents (AI/ML technical reviewer and comprehensive paper reviewer) independently analyzed the paper. The reviewer synthesized their findings with independent reading and evaluation. All technical assessments, scoring decisions, and constructive feedback were verified and refined by the human reviewer. The LLM did not generate any scores or final judgments autonomously.

LLM(Claude)이 본 리뷰의 구조화를 보조하는 데 사용되었다. 두 개의 전문 리뷰 에이전트(AI/ML 기술 리뷰어 및 종합 논문 리뷰어)가 독립적으로 논문을 분석하였다. 리뷰어는 독립적인 독해 및 평가와 함께 이들의 결과를 종합하였다. 모든 기술적 평가, 점수 결정, 건설적 피드백은 인간 리뷰어에 의해 검증 및 수정되었다. LLM은 어떤 점수나 최종 판단도 자율적으로 생성하지 않았다.