Group 3 discussion thread

[gleenz]

Task 1: renaming files.

[rgattermeyer]

I'm in, too!

[jypwong]

Sample of inconsistently arranged and named folders
Samples for inconsistent arranged and names folders.pptx

[rachamiaman]

Thank you!

[rachamiaman]

Has anyone had any success or good leads of places to start to begin understanding how to help rename files and folders in bulk? Bulk Rename Utility may be the quickest way to address this but of course that is beside the point. I do want to say that if anyone in the group already has made progress related to the other highlighted topics in the google doc, or if they are maybe a better place for anyone to start, I say go ahead and dig into those.

[rachamiaman]

This was a good place for me to start to work: https://www.tutorialspoint.com/How-to-rename-directory-using-Python.

I think though that we may need something with a bit more logic like: in this directory look for a folder named "master". If it does not exist create it else ignore. In this directory look for a folder named "access". If it does not exist create it else ignore. Then find all pdfs and put them in the access folder.

For the different Woody Acres School formats, maybe a bit that recognizes a folder that may need to be reviewed based on level or on the date range which at least in this example is always present. I'm not certain how to build that at the moment but I'm curious if when we meet, we could have a mentor/skilled individual jump in and help.

[jessedyer-uma]

Below is an example of what I'm want to achieve with a Python renaming/restructuring script. I'm going to have a go at it with the example with additional steps in the Python Study Groups 'How to'. Will let you all know how I go next week.
Also, another thing I thought might be good to discuss is how is everyone writing their code? Using a text editor or an IDE? I just installed Visual Studio Code, not sure if it was necessary and it requires installing a Python extension which I'm hoping won't interfere with my actual Python install?

Current state:
~/2003-0112
├── Access
│   ├── 2003.0112.00006-s1.mp3
│   ├── 2003.0112.00006-s2.mp3
│   ├── 2003.0112.00007-s1.mp3
│   ├── 2003.0112.00007-s2.mp3
│   ├── 2003.0112.00008-s1.mp3
│   └── 2003.0112.00008-s2.mp3
└── Preservation
   ├── 2003.0112.00006-s1.wav
   ├── 2003.0112.00006-s2.wav
   ├── 2003.0112.00007-s1.wav
   ├── 2003.0112.00007-s2.wav
   ├── 2003.0112.00008-s1.wav
   └── 2003.0112.00008-s2.wav

Desired state:
~/2003-0112
├── 2003-0112-00006
│   ├── mp3
│   │   ├── 2003-0112-00006-00001.mp3
│   │   └── 2003-0112-00006-00002.mp3
│   └── wav
│   ├── 2003-0112-00006-00001.wav
│   └── 2003-0112-00006-00002.wav
├── 2003-0112-00007
│   ├── mp3
│   │   ├── 2003-0112-00007-00001.mp3
│   │   └── 2003-0112-00007-00002.mp3
│   └── wav
│   ├── 2003-0112-00007-00001.wav
│   └── 2003-0112-00007-00002.wav
└── 2003-0112-00008
   ├── mp3
   │   ├── 2003-0112-00008-00001.mp3
   │   └── 2003-0112-00008-00002.mp3
   └── wav
   ├── 2003-0112-00008-00001.wav
   └── 2003-0112-00008-00002.wav

[rachamiaman]

I'm using Visual Code Studio and it seems to work well not interfering but I don't use Python extensively...

[gleenz]

We use Sublime Text which works well for us. I like the way it colour codes the script which makes it easier to see what I'm doing. I tend to use it for importing a script and then changing a bit of code to suit our situation.

[eli-hu]

I used SublimeText for years; in fact, I still use it on my home computer with a license that I personally paid for. However, even though SublimeText license allows using on multiple computers (but only one at a time), my institution does not allow personal licenses on work computers. So I use free VSCode on my work computer. I suspect that most of my Pythoning will be done on my home computer and transferred to the work computer to finalize.

[jessedyer-uma]

Here are the scripts that I just showed you all in the meeting, first the one Lars wrote, which I still have to add a few things to, I'll update it here when I do:

import pathlib

errorlist = []
homedir = pathlib.Path('RenameAudio')
for x in homedir.rglob('*.*'):
    file_path = pathlib.Path(x)
    if file_path.match(f"{'[0-9]' * 4 }.{'[0-9]' * 4 }.{'[0-9]' * 5 }-s[0-9].*"):
        new_file_path = pathlib.Path('newfolder/')
        new_file_path = new_file_path / file_path.stem[0:15].replace('.', '-') / file_path.suffix.replace('.', '') / f"{file_path.stem[0:15].replace('.', '-')}-{file_path.stem[17:].zfill(5)}{file_path.suffix}"
        print(file_path)
        print(new_file_path)
    else:
        errorlist.append(file_path)
        print('this didn\'t work')

And then my collaboration with ChatGPT:

from pathlib import Path

# Define the directory path
rename_dir = Path("~/RenameAudio").expanduser()

# First set of filename modifications
for file in rename_dir.rglob("*.*"):
    new_filename = file.stem.replace(".", "-")+file.suffix
    new_file_path = file.with_name(new_filename)
    file.rename(new_file_path)

# Directory creation and filepath modification
for file in rename_dir.rglob("*.*"):
    subdir = file.stem.replace("-s1", "").replace("-s2", "")
    subdir_path = rename_dir / subdir
    subdir_path.mkdir(parents=True, exist_ok=True)
    subdir_wav_path = subdir_path / file.suffix.replace('.', '')
    subdir_wav_path.mkdir(parents=True, exist_ok=True)
    file.rename(subdir_wav_path / file.name)

# Second set of filename modifications
for file in rename_dir.rglob("*.*"):
    new_filename = file.name.replace("-s1", "-00001").replace("-s2", "-00002")
    new_file_path = file.with_name(new_filename)
    file.rename(new_file_path)

[rgattermeyer]

Here is my ChatGPT-derived script for generating sequential directories with user input about the location, number of directories, and base name of the directories:

import os

def create_folders(directory, num_folders):
    folder_name = input("Enter the base name for the folders: ")
    for i in range(1, num_folders + 1):
        new_folder_path = os.path.join(directory, f"{folder_name}_{i:03}")
        try:
            os.makedirs(new_folder_path)
            print(f"Folder '{folder_name}_{i:03}' created successfully at '{new_folder_path}'.")
        except FileExistsError:
            print(f"Folder '{folder_name}_{i:03}' already exists at '{new_folder_path}'.")

def main():
    directory = input("Enter the directory path where you want to create the folders: ")
    num_folders = int(input("Enter the number of folders you want to create: "))
    
    if not os.path.isdir(directory):
        print("Error: The specified directory does not exist.")
        return
    
    create_folders(directory, num_folders)

if __name__ == "__main__":
    main()

[rachamiaman]

Hello,

I've been working since last time to build a script that will parse a complex directory of xml files for a specific tag and print the results to a csv. So far the csv gets created but is empty except for the column headers. Sorry if the code formatting doesn't appear correct. The remainder of this comment should be code:

`
import os
import csv
import xml.etree.ElementTree as ET

def find_key_value(root):
"""
Recursively search for the key tag with attribute id="objectIdentifierValue"
within the specified hierarchy and return its value.
"""
for elem in root.findall(".//mets:mets/mets:amdSec/mets:techMD/mets:mdWrap/mets:xmlData/dnx/section/record/key[@id='objectIdentifierValue']", namespaces={"mets": "http://www.loc.gov/METS/", "dnx": "http://www.exlibrisgroup.com/dps/dnx"}):
return elem.text
return None

def process_xml_file(xml_file, csv_writer):
"""
Parse the given XML file, extract the key value, and write it to the CSV.
"""
try:
tree = ET.parse(xml_file)
root = tree.getroot()
key_value = find_key_value(root)
if key_value:
csv_writer.writerow([xml_file, key_value])
except ET.ParseError:
print(f"Error parsing {xml_file}")

def main(directory):
# Create a CSV file in the topmost directory
csv_file = os.path.join(directory, "result.csv")
with open(csv_file, "w", newline="") as csvfile:
csv_writer = csv.writer(csvfile)
csv_writer.writerow(["File", "objectIdentifierValue"])

    # Recursively search for XML files
    for root_dir, _, files in os.walk(directory):
        for file in files:
            if file.lower().endswith(".xml"):
                xml_file = os.path.join(root_dir, file)
                process_xml_file(xml_file, csv_writer)

print(f"Results saved to {csv_file}")

Set the directory where XML files are located

directory_to_search = "path\to\my\directory"
main(directory_to_search)
`

[lars-unimelb]

Hi, I'm going to guess that the error is in your xpath statement: ".//mets:mets/mets:amdSec/mets:techMD/mets:mdWrap/mets:xmlData/dnx/section/record/key[@id='objectIdentifierValue']"

Have you tried running just that and print all elem? You have specified a dnx namespace but it is not used in the xpath.

[rachamiaman]

I have not tried it just with that, but I think you're right. Something about this is not going down to the key element. This is slightly modified. Previously it had said ".//mets:mets/Mets:amdSec/Mets:techMD/mets:mdWrap/Mets:xmlData/dnx:dnx/dnx:section/dnx:record/key[@id='objectIdentifierValue']" but I was still getting an empty csv.

I did the following:
`import os
import csv
import xml.etree.ElementTree as ET

def find_key_value(root):
"""
Recursively search for the key tag with attribute id="objectIdentifierValue"
within the specified hierarchy and return its value.
"""
for elem in root.findall(".//mets:mets/mets:amdSec/mets:techMD/mets:mdWrap/mets:xmlData/dnx/section/record/key[@id='objectIdentifierValue']", namespaces={"mets": "http://www.loc.gov/METS/", "dnx": "http://www.exlibrisgroup.com/dps/dnx"}):
return elem.text
print(elem)
return None`

but nothing was printed.
I'm probably not dialing all the way to the element I need correctly

[rachamiaman]

Ok. I don't think I realized that you didn't have the specify the entire parent child relationship/path to get to the element you are looking for. It appears you can use a wild card * to search like so:

key_element = root.find(".//*[@id='objectIdentifierValue']") if key_element is not None: return key_element.text else: return None

[rachamiaman]

This one worked:

import os
import csv
import xml.etree.ElementTree as ET

def parse_xml(xml_path):
    """Parse an XML file and extract the specified key element."""
    tree = ET.parse(xml_path)
    root = tree.getroot()
    
    # Define the namespaces
    namespaces = {
        'mets': 'http://www.loc.gov/METS',
        'dnx': 'http://www.exlibrisgroup.com/dps/dnx'
    }
    
    # Find the key element with the specified attribute
    key_element = root.find(".//*[@id='objectIdentifierValue']")
    
    if key_element is not None:
        return key_element.text
    else:
        return None

def main():
    # Prompt the user for the input directory
    input_directory = input("Enter the directory path: ")
    
    # Initialize a list to store results
    results = []
    
    # Recursively search for XML files
    for root_dir, _, files in os.walk(input_directory):
        for filename in files:
            if filename.lower().endswith('.xml'):
                xml_path = os.path.join(root_dir, filename)
                key_value = parse_xml(xml_path)
                if key_value:
                    results.append((xml_path, filename, key_value))
    
    # Write results to a CSV file
    output_csv = os.path.join(input_directory, 'xml_results.csv')
    with open(output_csv, 'w', newline='') as csvfile:
        writer = csv.writer(csvfile)
        writer.writerow(['File Path', 'Filename', 'Key Element'])
        writer.writerows(results)
    
    print(f"Results saved to {output_csv}")

if __name__ == "__main__":
    main()

[rachamiaman]

Here is my python example to parse a directory of xml files for the mets:FLocat element where there is a URL. It gathers the path of the xml (so I know which one to look for depending on if the URL exists or not) the URL itself, and the Object ID from the Scholar's Archive repository. This prints to a CSV with the relevant information and leaves cells blank where there is no information. In this example, every xml has an Object ID always but not all have a mets:FLocat element.

import os
import csv
import xml.etree.ElementTree as ET

def parse_xml(directory):
    ns = {'mets': 'http://www.loc.gov/METS/', 'dnx': 'http://www.deluxe.com/dnx'}
    urls = []
    object_ids = []

    for dirpath, dirnames, filenames in os.walk(directory):
        for filename in filenames:
            if filename.endswith('.xml'):
                file_path = os.path.join(dirpath, filename)
                tree = ET.parse(file_path)
                root = tree.getroot()

                url = root.find('.//mets:FLocat', ns)
                if url is not None:
                    urls.append((file_path, url.get('{http://www.w3.org/1999/xlink}href')))
                else:
                    object_id = root.find('.//*[@id="objectIdentifierValue"]', ns)
                    if object_id is not None:
                        object_ids.append((file_path, object_id.text))

    return urls, object_ids

def write_to_csv(directory, urls, object_ids):
    with open(os.path.join(directory, 'results.csv'), 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(['File Path', 'URL/Object ID'])
        for url in urls:
            writer.writerow(url)
        for object_id in object_ids:
            writer.writerow(object_id)

if __name__ == "__main__":
    import sys
    directory = sys.argv[1]
    urls, object_ids = parse_xml(directory)
    write_to_csv(directory, urls, object_ids)

[rachamiaman]

This 2nd example makes use of some logic to decide that if there isn't a mets:FLocat element, then go harvest the object ID and print it to the csv. So the resulting csv has the xml path and the 2nd column is either the URL -or- the Object ID. That way I can quickly filter for object with no URL or in other words, only an object ID and let the person in charge of the repository know that there are some issues, or to say to management, that we have harvested everything and those things that don't appear in the Digital Preservation System have issues and everything is accounted for either way.

import os
import csv
import xml.etree.ElementTree as ET

def parse_xml(directory):
    ns = {'mets': 'http://www.loc.gov/METS/', 'dnx': 'http://www.deluxe.com/dnx'}
    data = []  # List to store extracted data

    for dirpath, dirnames, filenames in os.walk(directory):
        for filename in filenames:
            if filename.endswith('.xml'):
                file_path = os.path.join(dirpath, filename)
                tree = ET.parse(file_path)
                root = tree.getroot()

                # Extract URL (if it exists)
                url_elem = root.find('.//mets:FLocat', ns)
                url = url_elem.get('{http://www.w3.org/1999/xlink}href') if url_elem is not None else None

                # Extract object ID
                object_id_elem = root.find('.//*[@id="objectIdentifierValue"]', ns)
                object_id = object_id_elem.text if object_id_elem is not None else None

                # Append data to the list
                data.append((file_path, url, object_id))

    return data

def write_to_csv(directory, data):
    with open(os.path.join(directory, 'results.csv'), 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(['File Path', 'URL', 'Object ID'])
        for row in data:
            writer.writerow(row)

if __name__ == "__main__":
    import sys
    directory = sys.argv[1]
    extracted_data = parse_xml(directory)
    write_to_csv(directory, extracted_data)

[rachamiaman]

I wanted to post this here so Lars could make a comment that makes better sense than the notes I took last time!
We talked about using xpath.

import os
import csv
import xml.etree.ElementTree as ET

def parse_xml(directory):
    ns = {'mets': 'http://www.loc.gov/METS/', 'dnx': 'http://www.deluxe.com/dnx'}
    data = []  # List to store extracted data

    for dirpath, dirnames, filenames in os.walk(directory):
        for filename in filenames:
            if filename.endswith('.xml'):
                file_path = os.path.join(dirpath, filename)
                tree = ET.parse(file_path)
                root = tree.getroot()

                # Extract URL (if it exists)
                url_elem = root.find('.//mets:FLocat', ns)
                url = url_elem.get('{http://www.w3.org/1999/xlink}href') if url_elem is not None else None

                # Extract object ID
                object_id_elem = root.find('.//*[@id="objectIdentifierValue"]', ns)
                object_id = object_id_elem.text if object_id_elem is not None else None

                # Append data to the list
                data.append((file_path, url, object_id))

    return data

def write_to_csv(directory, data):
    with open(os.path.join(directory, 'results.csv'), 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(['File Path', 'URL', 'Object ID'])
        for row in data:
            writer.writerow(row)

if __name__ == "__main__":
    import sys
    directory = sys.argv[1]
    extracted_data = parse_xml(directory)
    write_to_csv(directory, extracted_data)

[lars-unimelb]

Hi Rachel, I've tried to have a little stab at it this morning using xpath (I haven't tested this at all):

import pathlib
import csv
import xml.etree.ElementTree as ET

def parse_xml(directory):
    ns = {'mets': 'http://www.loc.gov/METS/', 'dnx': 'http://www.deluxe.com/dnx', 'xlink': 'http://www.w3.org/1999/xlink'}
    data = []  # List to store extracted data

    for xml_file in directory.rglob('*.xml'):
        file_path = xml_file.parent.as_posix()
        tree = ET.parse(xml_file)
        root = tree.getroot()

        object_id = None
        object_id_elem = root.xpath('//*[@id="objectIdentifierValue"]', namespace=ns) # Isn't this usually the OBJDID in the mets root node?
        if object_id_elem != []:
            object_id = object_id_elem[0].text
        url_elem_list = root.xpath('//mets:FLocat', namespace=ns)
        for url_elem in url_elem_list:
            url = url_elem.get('{http://www.w3.org/1999/xlink}href') if url_elem is not None else None
            data.append((file_path, url, object_id))
            print((file_path, url, object_id))
        if len(url_elem_list) == 0:
            data.append((file_path, url, object_id))
            print((file_path, url, object_id))
            
    return data

def write_to_csv(directory, data):
    csv_path = directory / 'results.csv'
    with open(csv_path, 'w', newline='') as file:
        writer = csv.writer(file)
        writer.writerow(['File Path', 'URL', 'Object ID'])
        for row in data:
            writer.writerow(row)

if __name__ == "__main__":
    import sys
    directory = pathlib.Path(sys.argv[1])
    extracted_data = parse_xml(directory)
    write_to_csv(directory, extracted_data)

[rachamiaman]

I do get an attribute error 'xml.etree.ElementTree.Element' object has no attribute 'xpath'.

[lars-unimelb]

ah, sorry I forgot to reply to you, I guess you're going to need to install lxml for xpath and swap out import xml.etree.ElementTree as ET with import lxml.etree as ET

[rachamiaman]

No worries! Thank you!!

[rachamiaman]

I'm curious who all is planning to attend today. I believe it is the last of this series having been 6 months. Last month was a little sparse but it has been full of conferences and things.

[eli-hu]

For the past 10 years I've been using the conventions set forth by Chris Beams in his blog post How to Write a Git Commit Message. That is, something like this:

Add Norwegian language codes

Added codes for Bokmål and Nynorsk, also Norwegian macrolanguage.

Today in a Python4Lib meeting, I was introduced to Conventional Commits, which formalizes and extends those recommendations:

feat: add Norwegian language codes

Added codes for Bokmål and Nynorsk, also Norwegian macrolanguage.

[jypwong]

Sorry, didn’t manage attend the last session this morning. Will any of you be attending iPres? If yes, we can meet and say hi. :-) Glad to know you all and hope to stay in touch. Warmest regardsJoyceOn 26 Jun 2024, at 7:37 AM, elihu ***@***.***> wrote: For the past 10 years I've been using the conventions set forth by Chris Beams in his blog post How to Write a Git Commit Message. That is, something like this: Add Norwegian language codes Added codes for Bokmål and Nynorsk, also Norwegian macrolanguage. Today in a Python4Lib meeting, I was introduced to Conventional Commits, which formalizes and extends those recommendations: feat: add Norwegian language codes Added codes for Bokmål and Nynorsk, also Norwegian macrolanguage. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[rachamiaman]

I won't be going to iPres but my colleague, Tyler Thorsted will be attending - probably presenting as well. Maybe you already know him, but if not, he has a lot of experience and knowledge in Digital Preservation, especially around Rosetta and file formats. Take care!