Skip to content

Acoustic embeddings via deep models (wav2vec2, HuBERT) #33

Description

@Hugo-W

Follow-up from #23.

Description

The stimulus feature extraction module (pyeeg/features/) currently supports classical acoustic features (envelope, filterbank, gammatone). Issue #23 originally called for deep-learning-based acoustic embeddings via wav2vec2 and HuBERT.

Tasks

  • Add DeepAcousticFeatureExtractor class to pyeeg/features/acoustic.py
  • Support wav2vec2 (HuggingFace facebook/wav2vec2-base-960h or similar)
  • Support HuBERT (HuggingFace facebook/hubert-base-ls960 or similar)
  • Extract hidden-layer embeddings as per-frame feature vectors
  • Integrate into FeaturePipeline via extractor_type='acoustic_deep' (or extend existing "acoustic" type)
  • Add StimulusEncoder.add_deep_acoustic_features() convenience method
  • Handle model download from ModelScope mirror (HF Hub may be firewalled)
  • Test suite (unit tests with synthetic audio, integration test with real model if available)

Context

  • The AcousticFeatureExtractor (envelope/filterbank/gammatone) is already implemented and tested
  • The pipeline architecture (FeatureSpec, FeaturePipeline, StimulusEncoder) supports adding new extractor types
  • Models can be downloaded from ModelScope (https://modelscope.ai) if HuggingFace Hub is unavailable

Priority: Medium

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions