[DEV-12162] Agency loader by alburde1 · Pull Request #8 · fedspendingtransparency/brus-backend-common

alburde1 · 2026-05-14T19:04:09Z

Description:

Creating a loader to load in all agency data (CGAC, FREC, Subtier).

Technical Details:

Created 3 new gold tables and a new bronze table to house agency data. Updated the existing table models from what was in Broker to simplify without the availability of the FKs. FREC and Subtier tables now use the related codes directly instead of using IDs of higher tier tables.

Requirements for PR Merge:

Unit & integration tests updated
Documentation updated
Data validation completed (examples listed below)
1. Is performance impacted in the changes (e.g., API, pipeline, downloads, etc.)?
2. Is the expected data returned with the expected format?
Appropriate Operations ticket(s) created
Jira Ticket(s)
1. DEV-12162
Developer Reviews
- Broker Developer Review
- USAS Developer Review
Merge [DEV-12155] Funding Opportunity Extractor and Loader #6

Explain N/A in above checklist:

+ Creating the agency references + Creating the initial script containing most of the work for cgac and frec loading

+ Also add subtier data

…-12162-agency-loader

+ black + flake + removing prints

zachflanders-frb · 2026-05-18T21:22:41Z

+        cgac_data["display_name"] = cgac_data.apply(
+            lambda row: (
+                f"{row["agency_name"]} ({row["agency_abbreviation"]})"
+                if row["agency_abbreviation"]
+                else f"{row["agency_name"]} (nan)"
+            ),
+            axis=1,
+        )


This could be done column-wise instead of row-wise to increase performance:

Suggested change

cgac_data["display_name"] = cgac_data.apply(

lambda row: (

f"{row["agency_name"]} ({row["agency_abbreviation"]})"

if row["agency_abbreviation"]

else f"{row["agency_name"]} (nan)"

),

axis=1,

)

cgac_data["display_name"] = (

cgac_data["agency_name"]

+ " ("

+ cgac_data["agency_abbreviation"].replace("", "nan").fillna("nan")

+ ")"

)

zachflanders-frb · 2026-05-18T21:23:34Z

+        frec_data["display_name"] = frec_data.apply(
+            lambda row: (
+                f"{row["agency_name"]} ({row["agency_abbreviation"]})"
+                if row["agency_abbreviation"]
+                else f"{row["agency_name"]} (nan)"
+            ),
+            axis=1,
+        )


same thing here re: column-wise vs row-wise operations

zachflanders-frb · 2026-05-18T21:28:35Z

+    frec_data = clean_data(
+        raw_data,
+        frec_mapping,
+        {"frec": {"keep_null": False}, "cgac_code": {"pad_to_length": 3}, "frec_code": {"pad_to_length": 4}},
+    )
+
+    # de-dupe
+    frec_data = frec_data[frec_data.frec_cgac == "TRUE"]
+    frec_data.drop(["frec_cgac"], axis=1, inplace=True)
+    frec_data.drop_duplicates(subset=["frec_code"], inplace=True)


instead of mutating the data in place, you could chain these:

Suggested change

frec_data = clean_data(

raw_data,

frec_mapping,

{"frec": {"keep_null": False}, "cgac_code": {"pad_to_length": 3}, "frec_code": {"pad_to_length": 4}},

)

# de-dupe

frec_data = frec_data[frec_data.frec_cgac == "TRUE"]

frec_data.drop(["frec_cgac"], axis=1, inplace=True)

frec_data.drop_duplicates(subset=["frec_code"], inplace=True)

frec_data = (

clean_data(

raw_data,

frec_mapping,

{"frec": {"keep_null": False}, "cgac_code": {"pad_to_length": 3}, "frec_code": {"pad_to_length": 4}},

)

.loc[lambda df: df.frec_cgac == "TRUE"]

.drop(columns=["frec_cgac"])

.drop_duplicates(subset=["frec_code"])

)

zachflanders-frb · 2026-05-18T21:33:18Z

+    raw_data.loc[condition, "PRIORITY"] = 1
+    raw_data.loc[~condition, "PRIORITY"] = 2
+    raw_data["PRIORITY"] = raw_data["PRIORITY"].astype(int)
+    raw_data.replace({"TRUE": True, "FALSE": False}, inplace=True)


Same approach here re: method chaining.

Suggested change

raw_data.loc[condition, "PRIORITY"] = 1

raw_data.loc[~condition, "PRIORITY"] = 2

raw_data["PRIORITY"] = raw_data["PRIORITY"].astype(int)

raw_data.replace({"TRUE": True, "FALSE": False}, inplace=True)

raw_data = (

raw_data

.assign(PRIORITY=lambda df: np.where(df["TOPTIER_FLAG"] == "TRUE", 1, 2).astype(int))

.replace({"TRUE": True, "FALSE": False})

)

dpb-bah · 2026-05-21T17:36:07Z

+    DESCRIPTION = "CGAC Data after processing"
+    CSV_NAME = "cgac.csv"
+    PK = "cgac_id"
+    UNIQUE_CONSTRAINTS = []


I know UNIQUE_CONSTRAINTS hasn't been setup as an extra validation yet. The intent is for any other columns (or group of columns) needing a unique check. I'm gonna make another PR to 1) fix DEFCGroup's PK but also 2) update the type mapping of UNIQUE_CONSTRAINTS to be List[(str,) | str]; so it can be ['col x', ('col y', 'col z')] to support single column uniqueness but also multiple columns as well.

So in this case (as well as FRECGold and SubtierAgencyGold), since the PKs are x_id, the other unique col would be ["cgac_code"] (or ["frec_code"], ["subtier_code"]).

Again, not functional yet so I'm fine with leaving it alone as it may change in the future when we do implement it. Just letting you know where I'm thinking for now.

dpb-bah · 2026-05-21T17:58:04Z

+        cgac_data["display_name"] = (
+            cgac_data["agency_name"] + " (" + cgac_data["agency_abbreviation"].replace("", "nan").fillna("nan") + ")"
+        )
+        logger.info("Overwriting new CGAC data to Broker")


Suggested change

logger.info("Overwriting new CGAC data to Broker")

logger.info("Overwriting new CGAC data to CGACGold")

Same for FREC and Subtier

dpb-bah · 2026-05-21T18:05:26Z

+    metrics_json = load_cgac(raw_data, start_time, force_reload, metrics_json)
+    metrics_json = load_frec(raw_data, start_time, force_reload, metrics_json)
+    metrics_json = load_subtier(raw_data, start_time, force_reload, metrics_json)


Instead of passing in start_time, could you pull it from metrics_json? I know it gets messy with the strings vs datetimes but when saving the metrics file json.dump(metrics, metrics_file, default=str) including that default=str stringifys the dates automatically which is nice.

dpb-bah · 2026-05-21T20:10:03Z

+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description="Process the bronze defc data into the gold defc table.")


I know this was likely based on defc_gold.py which doesnt include this (making another PR fix for that). Could you add a

from brus_backend_common.logging import configure_logging ... if __name__ == "__main__": configure_logging()

here? Running it locally I could see none of the logs.

I'm going to also start looking into a shared Loader class or shared methods that standardizes all our scripts so that we don't need to remember little details like this.

dpb-bah · 2026-05-21T20:10:20Z

+    """Loads the FREC data into the gold table
+
+    Args:
+        raw_data: the raw agency codes bronze table\


Suggested change

raw_data: the raw agency codes bronze table\

raw_data: the raw agency codes bronze table

alburde1 added 6 commits May 13, 2026 10:47

Intermediate commit

62c4386

+ Creating the agency references + Creating the initial script containing most of the work for cgac and frec loading

Merge FON branch updates into this one

e56a7fc

Update with fixes based on new code

0ec97e7

+ Also add subtier data

Merge branch 'mod/dev-12155-funding-opportunity-loaders' into mod/dev…

a6e27b6

…-12162-agency-loader

Cleaning up updated files

6e2e028

+ black + flake + removing prints

Creating tests for agency loader

3415a9c

github-actions Bot assigned alburde1 May 14, 2026

zachflanders-frb reviewed May 18, 2026

View reviewed changes

github-actions Bot assigned zachflanders-frb May 18, 2026

alburde1 added 2 commits May 21, 2026 10:49

Merge branch 'qat' into mod/dev-12162-agency-loader

320a3a4

PR Review Updates

e653422

dpb-bah requested changes May 21, 2026

View reviewed changes

github-actions Bot assigned dpb-bah May 21, 2026

Review comments

49cce50

dpb-bah requested changes May 21, 2026

View reviewed changes

PR comments

8d295cf

dpb-bah approved these changes May 21, 2026

View reviewed changes

zachflanders-frb approved these changes May 22, 2026

View reviewed changes

alburde1 merged commit 3ecf577 into qat May 22, 2026
10 checks passed

alburde1 deleted the mod/dev-12162-agency-loader branch May 22, 2026 15:18

	logger.info("Overwriting new CGAC data to Broker")
	logger.info("Overwriting new CGAC data to CGACGold")



		if __name__ == "__main__":
		parser = argparse.ArgumentParser(description="Process the bronze defc data into the gold defc table.")

	raw_data: the raw agency codes bronze table\
	raw_data: the raw agency codes bronze table

Uh oh!

Conversation

alburde1 commented May 14, 2026 • edited by dpb-bah Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Technical Details:

Requirements for PR Merge:

Explain N/A in above checklist:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alburde1 commented May 14, 2026 •

edited by dpb-bah

Loading