Developers Meeting 2025-07-27 — Data handling, storage, and tracking strategies скачать в хорошем качестве

Developers Meeting 2025-07-27 — Data handling, storage, and tracking strategies

Meeting Summary: Multi-part technical discussion involving system architecture, camera control, metadata handling, event detection, storage, and future database considerations. 🔧 System Architecture and Components A. Camera Configuration & Control -The camera's temperature, gain, and exposure are controlled automatically. -Frames are output with accompanying metadata: Timestamp Temperature Gain Exposure B. Metadata Handling & Storage -Metadata is stored per image and saved in a NumPy array in shared memory. -Archiving strategy discussed: likely to use HDF5 format. Possible approaches: one file per target, per event, or hierarchical structure. Consideration of integrating *ProtoBuf-based recording with eCAL*. 🧠 Data Structuring & Archiving A. HDF5 vs. Other Formats -HDF5 considered suitable for: Writing per-event or per-target files. Storing structured data including metadata headers and frame data. Supporting live playback via eCAL player/recorder. -Use of FITS format (from astronomy) discussed as an alternative but less sophisticated than HDF5. 📦 Database Strategy A. Purpose of the Database -Two different needs emerged: 　1. Real-time/streaming structure (e.g., shared memory buffer, HDF5 for archiving). 　2. Long-term retrieval and pattern matching: 　 - Ability to search for similar past events, e.g., “Have we seen this object before?” 　 - Interest in object comparison using cropped 64x64 px sub-images . B. Database Types Debated -SQL : Not ideal for real-time or fast event retrieval, but useful for dashboard/dev purposes. -Graph databases : More suitable for: Real-time linking of related data. Fast traversal through time-sequenced or object-based events. Tracking relationships (e.g., same object across different times/locations). -Idea: Use both in parallel: Graph DB for lookup/comparison , HDF5 for storage/archival . 📌 Object Detection & Tracking A. Object detection discussed as the next priority. B. Classification via CNNs , SIFT features , or simple blob tracking . C. Strategies: Tracking objects (PTF) vs. snapshotting all. Prioritizing “hot” events (e.g., unknowns or user-defined interest). D. Image crops (64x64 px) extracted for efficient classification or pattern recognition. These can be stored alongside events in HDF5. 🚨 Outstanding Questions & Decisions 1. HDF5 Structuring: Do we store one event per file or *multiple in a hierarchical structure*? How to handle multi-sensor input into the same event structure? 2. Database Integration: Do we begin integrating a graph database now or after the first ALOHA version is released? What database system is most effective for real-time event retrieval ? 3. Tracking & Identification Strategy: When multiple objects appear simultaneously, how do we associate and prioritize? How will re-identification be implemented for fast-moving or briefly visible objects? 4. System Next Steps: Prioritize object detection improvements. Finalize HDF5 archival strategy. Revisit database integration after ALOHA release . ✅ Current Status and Next Steps eCAL setup confirmed working on Ubuntu 24.04 systems. Object detection development is next. Archiving logic is good enough for current ALOHA goals. Graph/database exploration to be postponed until after base system is functional.