Dataset landscape
Compare multi-modal sensing resources—how they differ in modalities, tasks, and complementarity—with deep dives into OctoNet and RF-Behavior.
ACM UbiComp / ISWC 2026 · Half-day Tutorial
Towards Open-Sourced Datasets, Benchmarks, and Platforms for Multi-Modal Sensing
A structured and hands-on introduction to public datasets, benchmark design, multi-modal learning, and practical resources for multi-modal intelligence beyond vision.
Format
3-hour tutorial
Time
Oct 11, 2026 · 2:00–5:30 PM
Venue
Room 5I, Floor 5F
Time zone
UTC+8
Overview
Why this tutorial?
Ubiquitous computing is moving from bespoke, single-sensor systems toward robust multi-modal perception. Open datasets, shared platforms, and model designs—white-box, physics-informed, and signal-processing–learning co-design—are making it possible to compare modalities, reproduce results, and build reusable sensing pipelines.
This tutorial combines concise lectures with deep dives into OctoNet, RF-Behavior, and OctoSense, then three sensing model paradigms (RF-CRATE, TAP3D, RF-LEGO), plus a guided notebook session on streaming download, modality loading, and visualization.
What you will learn
From open datasets to reusable sensing models
OctoSense ecosystem
See how a unified data abstraction, signal operators, and benchmark engine can move wireless sensing toward ImageNet-style reproducibility.
Three model paradigms
White-box models (RF-CRATE), physics-informed thermal-to-3D learning (TAP3D), and signal-processing–learning co-design (RF-LEGO).
Hands-on practice
Stream, load, and visualize OctoNet and RF-Behavior modalities with shared notebook workflows you can reuse after the conference.
Representative program
Open datasets, sensing models, and guided practice
Scheduled for 2:00–5:30 PM (UTC+8) on Oct 11, 2026, in Room 5I (Floor 5F), with online participation via Zoom.
Opening
Motivation, scope, learning goals, and tutorial format.
Speaker: Prof. Chenshu Wu
Module I · Datasets and benchmarks
- Overview — Sensing modalities, multi-modal dataset trends, and how representative resources differ and complement one another
- OctoNet — Practical hosting and access for heterogeneous multi-modal sensing data Speaker: Dr. Xie Zhang
- RF-Behavior — Multimodal RF dataset for gestures, activities, and affective behavior Speaker: Dr. Si Zuo
- OctoSense — Unified ecosystem for open-source wireless sensing and reproducible benchmarks Speaker: Mr. Weiying Hou
Module II · Models and learning paradigms
- Overview — A brief review of multi-modal sensing models, how they differ from vision–language models, and why sensor data needs dedicated designs
- RF-CRATE — White-box deep wireless sensing Speaker: Dr. Xie Zhang
- TAP3D — Physics-informed learning for thermal-assisted 3D human point cloud generation Speaker: Dr. Xie Zhang
- RF-LEGO — Signal-processing and learning co-design via deep unrolling Speaker: Mr. Luca Jiang-Tao Yu
Guided hands-on
Workflows for OctoNet and RF-Behavior: streaming download, modality loading, and visualization.
Closing
Open research questions and discussion.
Speaker: Prof. Chenshu Wu
Open tutorial materials
Datasets, platform, and models
Linked resources below are ready to explore. Tutorial slides will be released closer to the conference.
Datasets
NeurIPS 2025
OctoNet
Large-scale multi-modal human activity dataset with motion-capture 3D pose labels; streaming download and visualization via the demo notebook.
ACM IMWUT 2026
RF-Behavior
Multimodal RF dataset (radar, RFID, LoRa, IMU, mocap) for gestures, activities, and affective behavior, from Aalto University.
Platform
arXiv 2026
OctoSense
A unified open-source wireless sensing ecosystem: data abstraction, standardized signal operators, and a benchmark engine toward ImageNet-style reproducibility.
Models & methods
UbiComp 2026
RF-CRATE
White-box deep wireless sensing with fully mathematically derivable architectures across RF modalities.
MobiCom 2026
TAP3D
Physics-informed learning for thermal-assisted 3D human point cloud generation.
MobiCom 2026
RF-LEGO
Modular signal-processing and deep-learning co-design for RF sensing via deep unrolling.
Slides:
Tutorial slides are TBA. Datasets, code, and demos are linked above.
Who should attend?
Open to newcomers and active researchers
The tutorial is intended for graduate students, postdoctoral researchers, faculty members, and industry researchers or engineers working in ubiquitous computing, mobile and wearable sensing, HCI, signal processing, and applied machine learning.
Prerequisites
- Working knowledge of Python and a framework such as PyTorch
- Basic familiarity with signal processing and machine learning
- No prior experience with a specific sensing modality is required
Open materials
All tutorial resources are open-sourced and can be downloaded anytime from the Materials section—datasets, code, and demos included.
People
Organizers & speakers
Organizers
Featured speakers