Map Locations to Entity List Attribute by dan-tang-ssd · Pull Request #47 · stats4sd/groundswell_platform

dan-tang-ssd · 2026-07-10T14:11:46Z

This PR is submitted to fix #46

It is not yet ready for review. It is submitted for progress update.

Expected workflow:

for each team, someone creates location levels (user fills in Cluster, Group)
someone creates location records (manually enter or import csv / excel file)
someone creates farm records (import csv / excel file), farm is matched to locations
someone added a new farm by submitting Farm Registration form
- farm is not matched to location yet
- when fetching farm data from Entity List, need to match the farm to existing locations

It contains below items:

match farm to a location when retrieving farm data from Entity List
make sure the farms can be showed in Global Indicators form
make sure the farm data submitted by Farm Registration form can be retrieved to application

dan-tang-ssd · 2026-07-10T14:16:28Z

I did below testings:

refresh local database with seeder files
create xlsform template for Farm Registration form and Global Indicators form
publish both forms (need to publish form to test Entity List. Entity List is not available for draft form testing)
create a new farm using Farm registration form in Enketo, submit form
checked Entity List, a new farm is added
checked that the newly added farm can be showed in Global Indicators form
checked that the newly added farm can be showed in Survey Dashboard > Survey Locations > Farms page

Screen shots:

Admin Panel > Teams > Xlsform tab, published both forms

Submitted Farm Registration form in Enketo, new farm data found in Entity List

Global Indicators form in Enketo, new farm data showed properly

Survey Dashboard > Survey Locations > Farms page, new farm data showed

…when fetching farm data from ODK central entity list

… sync to ODK central entity list

dan-tang-ssd · 2026-07-13T15:37:17Z

I did testing to add a new farm by submitting Farm Registration form:

refresh local database with seeder files
create xlsform template for Farm Registration form and Global Indicators form
publish both forms (need to publish form to test Entity List. Entity List is not available for draft form testing)
create a new farm using Farm registration form in Enketo, submit form
checked Entity List, a new farm is added
checked that the newly added farm can be showed in Global Indicators form
checked that the newly added farm can be showed in Survey Dashboard > Survey Locations > Farms page. Farm is matched to an existing group

I did another testing to import farm list, imported farm can be showed in Global Indicator form corresponding location:

Import farm list excel file
imported farm belong to the corresponding group
checked Entity List, imported farm has correct value for attributes loc1, loc1_type, loc1_name, loc2, loc2_type, loc2_name
launch Global Indicator form, the imported farm can be showed for the corresponding group

Screen shots:

First testing - Farm is matched with correct group after fetching from ODK central entity list

Second testing - Imported farm has been showed in Global Indicator form for corresponding group

… name implicitly

dan-tang-ssd · 2026-07-14T15:46:13Z

Progress update:

GPS data is stored in geometry attribute of farm entity, revised to show them in GPS section properly. Also revised to save GPS section data to geometry attribute when sync data to Entity List.
Import farm list feature, change to match GPS data with column header name implicitly (case-insensitive comparison)
added unit tests for GPS data

TODO:

I just realised that attribute loc1 and loc2 stores internal cluster ID and group ID. These internal ID are defined in ODK forms. I will revise program so that the created / edited / imported farm will be put under the current group and cluster

…utes loc1 (cluster id) and loc2 (group id)

dan-tang-ssd · 2026-07-14T16:22:59Z

Just applied a fix to find and fill in value for attributes loc1 (cluster ID) and loc2 (group ID).
I did testing to import and edit farm (change group, then save), farm can be showed in correct group in ODK form.

To show how it works, I will refresh my local database then do testings with screen shots tomorrow morning.

dan-tang-ssd · 2026-07-15T08:22:01Z

I did below testings:

refresh local database with seeder files
create Xlsform Template for Farm Registration form and Global Indicators form
for "P1 Test Team 1", publish Farm Registration form and Global Indicators form
create location levels "Cluster" and "Group"
import Groups by importing an excel file containing clusters and groups
submit Farm Registration form in Enketo
check that newly added farm is found in ODK central entity list
open Global Indicators form in Enketo, newly added farm is showed for the corresponding group
Farms page, newly added farm is synchronised from ODK central entity list. It has been matched to the corresponding group
Cluster page, "# of Farms" column shows correct value for corresponding cluster (Note that the cache refreshes data every 5 minutes)
Group page, "# of Farms" column shows correct value for corresponding group (Note that the cache refreshes data every 5 minutes)

Screen shots:

Submitted farm is added to ODK central entity list

In Global Indicators form, newly added farm showed in corresponding group

Farms page, newly added farm has been synchronised from ODK central entity list. It is put under the corresponding group by matching group name in case-insensitive way

Cluster page, "# of Farms" column shows correct value for corresponding cluster

Group page, "# of Farms" column shows correct value for corresponding group

dan-tang-ssd · 2026-07-15T08:30:46Z

Continue testing:

Farms page, edit farm, change group from "Aapghari Pipalbot Farmer Group" to "Laligurans Misrit Krishak Samuha". Attribute values for loc1 and loc2 before sync data to ODK central entity list
Entity list, farm data has been updated. loc1 and loc2 have updated cluster id and group id now. It belongs to "Laligurans Misrit Krishak Samuha" now
open Global Indicators form, farm can be found in "Laligurans Misrit Krishak Samuha" group instead of "Aapghari Pipalbot Farmer Group" group now

Screen shots:

Farms edit page, original group is "Aapghari Pipalbot Farmer Group"

Farms edit page, change group to "Laligurans Misrit Krishak Samuha", save

Entity list, farm attribute loc1 and loc2 are updated to new cluster id and group id

Global Indicators form, farm cannot be found in "Aapghari Pipalbot Farmer Group" group. It can be found in "Laligurans Misrit Krishak Samuha" group now

dan-tang-ssd · 2026-07-15T08:49:56Z

Continue testing:

Import Farm List, import two farms belong to "Makhamali Womens Group" group. GPS data will be matched with header column name implicitly
ODK central entity list, two new farms added with correct attribute value for "loc1" (cluster id) and "loc2" (group id)
open Global Indicators form in Enketo, two new farms can be found in "Makhamali Womens Group" group

Screen shots

dan-tang-ssd · 2026-07-15T08:54:30Z

Continue testing:

Import Locations and Farm List, import two farms belong to "Chandeshwori Mahila Samuh B" group. GPS data will be matched with header column name implicitly
ODK central entity list, two new farms added with correct attribute value for "loc1" (cluster id) and "loc2" (group id)
open Global Indicators form in Enketo, two new farms can be found in "Chandeshwori Mahila Samuh B" group

Screen shots:

dan-tang-ssd · 2026-07-15T08:56:44Z

Here is the excel files for my testing:

Groups excel file
Group.xlsx

Import Farm List
Farms.xlsx

Import Locations and Farm List
Location_Level_with_Farms.xlsx

Morning @dave-mills - I think it is now ready for review.

dave-mills

The main point here is that we cannot be hardcoding location values. It looks like this is leaning heavily on the idea that we have "Group" and "Cluster" as the only 2 levels, (and using an old team's data as the canonical dataset). A team can have any number of location levels, and will come with their own locations - we do not know them in advance.

Second point - not related specifically to locations
I also noticed an issue in the main Farm ODK Entity handling that only arose because I used a 2nd team for testing this time around without refreshing the database. Because the teams share a "farm_entities" dataset, which tracks the variables already added, I got an error when pushing the farms to my new team's ODK EntityList:

"HTTP request returned status code 400: {"message":"The entity is invalid. You specified the dataset property [label_farm_entities] which does not exist." ...

This is caused because in reconcileProperties(), existing variables are not explicitly pushed to ODK Central as new properties. But they are new properties for the new team. I can see a few possible fixes to this:

always push the variables to ODK Central (quite wasteful; after the first time for each team it'll just return 409 (already exists) and quietly move on).
catch the 400 error thrown above, and then try to push the variables.
make the "farm_entities" dataset in our database team-owned, so each team has their own set of farm dataset_variables.

I think option 3 is the closest to the main intent - and also makes sense as different teams may have different farm properties.

dave-mills · 2026-07-15T09:55:30Z

+            $pos = $current->locationLevel->pos;
+            $attributes["loc{$pos}"] = '1';
+            $attributes["loc{$pos}_name"] = (string) $current->name;
+            $attributes["loc{$pos}_type"] = "Loc{$pos} name";


loc{pos}_type should be the type of location, e.g. "Group", "Cluster", "Village" etc.

(I suspect it was in the "Loc{pos} name" format because the example csv files were created without full knowledge of how the location levels system works)

dave-mills · 2026-07-15T10:05:39Z

+    // Hardcoded internal group ID / internal cluster ID per group name, keyed by the exact
+    // group name (lookup is case-insensitive - see resolveGroupClusterIds()). These specific
+    // IDs are defined in the ODK form's own choice list (used for Enketo's cascading
+    // selects) - they cannot be derived from this app's own Location/LocationLevel records,
+    // so per Dan (2026-07-15) they're hardcoded here instead. A group not in this table
+    // (e.g. a different team's data) simply falls back to the generic '1' placeholder in
+    // buildLocationAttributes() - see docs/plans/map-loc-attributes-to-location-levels.md.
+    //
+    // @var array<string, array{groupId: int, clusterId: int}>
+    protected const GROUP_CLUSTER_LOOKUP = [


We shouldn't be hardcoding any location data!

dave-mills · 2026-07-15T10:29:20Z

+     * `loc{n}` itself (per Dan, 2026-07-15): the given Location (the farm's own attachment
+     * point, i.e. the "Group") and its immediate parent (the "Cluster") get their real
+     * internal group ID / internal cluster ID from GROUP_CLUSTER_LOOKUP, matched by the
+     * Group's name - these specific IDs matter for Central-side cascading selects to place
+     * the farm under the correct cluster/group, unlike every other level's `loc{n}`, which
+     * stays a fixed placeholder value `"1"` (only its presence signals "this level has
+     * data" - see the 2026-07-13 note in git history for why that was fine before this).
+     * A Group name not found in the lookup (e.g. a different team's data) falls back to the
+     * same `"1"` placeholder for both loc{n} positions.


I don't understand this. What happens when a team registers and has 3 location levels: "District", "Sub-district" and "village"?

dan-tang-ssd · 2026-07-15T12:59:40Z

The main point here is that we cannot be hardcoding location values. It looks like this is leaning heavily on the idea that we have "Group" and "Cluster" as the only 2 levels, (and using an old team's data as the canonical dataset). A team can have any number of location levels, and will come with their own locations - we do not know them in advance.

I agree. Different team can have different location levels and locations. e.g.

Team A - Location levels: Cluster, Group
Team B - Location levels: District, Sub-district, Village

The location levels and location names are defined in ODK form "choices" sheet "loc1" and "loc2".
They are imported into choice_lists and choice_list_entries table.
I assume they will remain unchanged until a new ODK form is uploaded.

We allow user to maintain location levels and locations in application. When a new location level or location is added in application, it will not be synchronised to the ODK form.

Can I make an assumption that user will keep the same location levels and locations between applications and ODK forms? In case a new location is added in ODK form, user will need to upload the latest ODK form and add the new location in application.

I am going to remove the hardcoded location levels and locations now.

Screen shots:

dave-mills · 2026-07-15T13:54:20Z

We allow user to maintain location levels and locations in application. When a new location level or location is added in application, it will not be synchronised to the ODK form.

The whole point of this work is so that teams can manage the locations and location groups in one central location - the platform.

It should be synchronised to the ODK form via the helper function we looked at together - the one that regenerates the entire "Locations" module based on the team's location levels and locations.

Note, for that to happen:

We need to make sure that the ODK form's "locations" module is called "locations", or fix the hard-coding in that function (that currently searches for an XlsformModuleVersion called "Global Locations").
We need to review that helper function to make sure it creates the correct final "select_one_from_file" question; I bet it doesn't, because it was written for HOLPA which didn't use ODK Entities.

dave-mills · 2026-07-15T13:59:40Z

@dan-tang-ssd - I'll take a quick look at the ODK "Locations" module generator now, and share a small PR into this branch.

Replaces the legacy combined LocationSectionBuilder with two focused builders, split to match the new separate-farm-registration workflow.

Breaks the approved design (location-and-farm-info-module-rewrite.md) into 4 TDD tasks: can_be_replaced support in Xlsform::syncWithTemplate(), the two new builder classes, and wiring Team::localiseXlsforms() to them.

Captures operational gaps and local-DB findings from planning that aren't part of any task's code but matter for rollout: the dormant has_updated_locations flag, the has_latest_template resync gotcha, the current local dev DB template/module snapshot, and the jr:choice-name source citations.

…t. Resolve it by making dataset team specific

dan-tang-ssd · 2026-07-16T15:47:25Z

Some changes made:

remove hardcoded locations in program source code
find location data from locations table for location name matching
support location level with any level, rather than fixing as Cluster and Group
resolve error 400 returned from ODK central due to globally shared dataset. Resolve it by making dataset team-specific

For testing, I refreshed my local database with migration and seeders.

I created different location levels for different teams:

Team 1: Cluster, Group
Team 2: District, Sub-district, Village
Non Program Team: Level 1, Level 2, Level 3, Level 4, Level 5, Level 6 (to test if we can support more than 4 levels)

I imported locations by uploading excel file.
Then I imported farms by uploading excel file. The imported farms are sync to ODK central with correct attribute values in "loc" related fields.

For attributes loc1, loc2, loc3, loc4, etc. I put value of column locations.code for testing now.
Just because it allows string, which can be identified easier during testing.
I believe we will need to change to put locations.id instead.
I will evaluate and make changes after reviewing PR #48.

I think the program changes is in correct direction now.

Screen shots:

Team 1 - Location Levels, Locations, Farms, Farm Entities in ODK Central (one farm created by submitted Farm Registration form, one farm imported in application)

Team 2 - Location Levels, Locations, Farms, Farm Entities in ODK Central

Team 3 - Location Levels, Locations, Farms, Farm Entities in ODK Central

…min panel

Location odk module builder

dan-tang-ssd · 2026-07-18T10:57:35Z

After PR 48 merging into this PR. I run below commands before resuming my test.

git checkout map-location-to-entity-list-attribute

git pull

git submodule update

For Team 1, I added a new cluster then tried to publish, but I got the same error due to global dataset. I suspect that this is an error due to existing data, even using latest code does not help. Therefore, I refreshed my local database and start everything again.

In Admin Panel, I created Xlsform Template for Farm Registration form and Global Indicators form.

In Survey Dashboard, I selected Team 1. I created location levels as Cluster and Group. I imported locations by uploading excel file.

In Admin Panel > Teams > Team 1 > Xlsform tab, error occurred when I clicked "Publish" button.

Screen shots:

dan-tang-ssd · 2026-07-18T11:00:08Z

Ah.... I recall that In need to mark "locations" and "farm info" module as "can be replaced"...

I did that and tried to publish again, but problem persists...

Screen shots:

dan-tang-ssd · 2026-07-18T11:03:46Z

I tried the draft feature in Survey Dashboard, I got the same error.

Screen shots:

dan-tang-ssd · 2026-07-18T12:55:05Z

I applied one fix for dataset and another fix for variable name in Admin Panel > Teams > Team > Publish button.
Now I can publish the form with a replaced location level data now.

I added a new cluster called "Cluster 8" and a new group "Group 1-5" under cluster "Binata Groups".
They can be showed in the published Farm Registration form.

Screen shots:

dan-tang-ssd · 2026-07-18T13:01:26Z

I did similar testing for Team 2, which has location levels as District > Sub-district > Village.
The published farm registration form can show these location levels successfully.

Screen shots:

dan-tang-ssd · 2026-07-18T13:09:55Z

I did similar testing for Non Program Team, which has location levels as Level 1 > Level 2 > Level 3 > Level 4 > Level 5 > Level 6.
The published farm registration form can show these location levels successfully.

Screen shots:

dan-tang-ssd · 2026-07-18T13:34:30Z

Going back to Team 2, performed testing to check if the imported farms can be showed in Global Indicators forms.
It works!

Screen shots:

Team 2, imported 2 farms under village "Village 1-2-1"

Crossed check with Team 2 Entity List, 2 imported farms are sync to ODK central

Preview Global Indicators form, two farms can be showed for selection

dan-tang-ssd · 2026-07-18T13:41:54Z

I did similar test for Non Program Team, the imported farm is sync to ODK central entity list.
It can be showed in Global Indicators form for selection.

Screen shots:

dan-tang-ssd · 2026-07-18T13:50:39Z

I did similar test for Team 1, the imported farms are sync to ODK central entity list.
They can be showed in Global Indicators form for selection.

Screen shots:

dan-tang-ssd · 2026-07-18T13:58:16Z

After testing, locations levels data can be retrieved from database to Xlsform. Farm info data can be retrieved from entity list.
We can now support any level of location levels without knowing it in advance.

I think it is now ready for review.

dan-tang-ssd · 2026-07-18T14:51:00Z

I did one more testing and I found an issue.
I created a new team called "Dan Team". I used Team 2's location levels: District, Sub-district and Village.
I imported Team 2's locations excel file.

I published Farm Registration form.
I opened the published form in Enketo, it does not show any Sub-districts after selecting a District.
I downloaded the published form excel file for checking. It looks like Dan Team's Sub-district matched to Team 2's District.

Screen shots:

It looks like Dan Team's Sub-district matched to Team 2's District.

- make required variables required in locations + farms import forms - scope location import queries to current team

dave-mills · 2026-07-19T14:04:08Z

@dan-tang-ssd - thanks. Good find on the cross-team scoping problem. This is now fixed with specific team scoping in the locations import class when finding the "parent" location.

…ions)

- LocationModuleBuilder now deletes choice list entries for deleted locations - saved and deleted hooks for FarmEntity now flag forms with draft_needs_update - deleted locations also flag forms with draft_needs_update - added extra tests for locations and farms module builder; - module builders now build 1 local module version per form (i.e. per XlsformModule entry), rather than a single one; to avoid problems of resyncing with Xlsform syncFromTemplate()

add Claude code plan file

5bb5c4d

dan-tang-ssd added 3 commits July 13, 2026 14:03

match farm to an existing location (lowest location level with farm) …

7243a82

…when fetching farm data from ODK central entity list

enhance location matching to case-insensitive

5b761b7

after importing farm, prepare attributes for location matching before…

82b157e

… sync to ODK central entity list

dan-tang-ssd added 6 commits July 13, 2026 16:41

remove ray() calls

fc6da4f

add new unit tests

6e94305

show GPS data in GPS section, store GPS data in geometry attribute

a3dc8d8

add new unit tests

450ef90

Import farm list feature, change to match GPS data with column header…

db1bd82

… name implicitly

Import farm list feature, change to match GPS data with column header…

4bab032

… name implicitly

fix to put new farm under correct group with correct value for attrib…

10c9b1d

…utes loc1 (cluster id) and loc2 (group id)

dan-tang-ssd requested a review from dave-mills July 15, 2026 08:56

dave-mills requested changes Jul 15, 2026

View reviewed changes

dan-tang-ssd and others added 5 commits July 15, 2026 15:18

WIP: change hardcoded locations to a generic approach

642e52d

Add design doc for Locations/Farm Info module rewrite

0996d33

Replaces the legacy combined LocationSectionBuilder with two focused builders, split to match the new separate-farm-registration workflow.

Add implementation plan for Locations/Farm Info module rewrite

5e830f9

Breaks the approved design (location-and-farm-info-module-rewrite.md) into 4 TDD tasks: can_be_replaced support in Xlsform::syncWithTemplate(), the two new builder classes, and wiring Team::localiseXlsforms() to them.

Add LocationsModuleBuilder for the split locations XLSForm module

73067e9

fix error 400 returned from ODK central due to globally shared datase…

9ad3a05

…t. Resolve it by making dataset team specific

dave-mills and others added 2 commits July 17, 2026 10:40

ensure forms are localised for locations before publishing via the ad…

b09c741

…min panel

Merge pull request #48 from stats4sd/location-odk-module-builder

bdd0329

Location odk module builder

dan-tang-ssd added 2 commits July 18, 2026 13:44

fix dataset issue

0f64837

fix incorrect variable name

b6b2cee

sync locations.id instead of locations.code to ODK central entity list

e51f6ce

dan-tang-ssd requested a review from dave-mills July 18, 2026 13:58

fixes:

0ed6b81

- make required variables required in locations + farms import forms - scope location import queries to current team

dave-mills added 7 commits July 19, 2026 15:14

fix tests

26902f4

enable has_updated_locations flag

378c1b3

match adopted-entity location by locations.id, only use name as fallback

4c09c14

align module builder expectations with lower-case naming (Local locat…

2e1db08

…ions)

add GPS rules to avoid sending co-erced 0s to ODK Central

a52745c

update code review findings - add addressed list

e8c8b9f

dave-mills merged commit 466b47c into dev Jul 19, 2026
1 of 2 checks passed

dave-mills deleted the map-location-to-entity-list-attribute branch July 19, 2026 16:05

Uh oh!

Conversation

dan-tang-ssd commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dan-tang-ssd commented Jul 10, 2026

Uh oh!

dan-tang-ssd commented Jul 13, 2026

Uh oh!

dan-tang-ssd commented Jul 14, 2026

Uh oh!

dan-tang-ssd commented Jul 14, 2026

Uh oh!

dan-tang-ssd commented Jul 15, 2026

Uh oh!

dan-tang-ssd commented Jul 15, 2026

Uh oh!

dan-tang-ssd commented Jul 15, 2026

Uh oh!

dan-tang-ssd commented Jul 15, 2026

Uh oh!

dan-tang-ssd commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dave-mills left a comment

Choose a reason for hiding this comment

Uh oh!

dave-mills Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

dave-mills Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

dave-mills Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

dave-mills Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

dan-tang-ssd commented Jul 15, 2026

Uh oh!

dave-mills commented Jul 15, 2026

Uh oh!

dave-mills commented Jul 15, 2026

Uh oh!

dan-tang-ssd commented Jul 16, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dan-tang-ssd commented Jul 18, 2026

Uh oh!

dave-mills commented Jul 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

dan-tang-ssd commented Jul 10, 2026 •

edited

Loading

dan-tang-ssd commented Jul 15, 2026 •

edited

Loading

dan-tang-ssd commented Jul 18, 2026 •

edited

Loading

dan-tang-ssd commented Jul 18, 2026 •

edited

Loading

dan-tang-ssd commented Jul 18, 2026 •

edited

Loading