KiDS Website Update

[cheymans]

As much as we love ESO, I'm pretty sure the majority of people access KiDS data through our website. My understanding is that you can only edit the website from a Leiden machine, so @kuijken this will have to be your task, but @AngusWright and I will help hacking the html files.

Going through the KiDS-DR4 pages we have the following tasks

• About DR5: I will hack the DR4 html for DR5
• Release notes: easy - I will hack the DR4 html for DR5
• Observations Table: Hopefully @jtadejong has a script that creates this?
• New for DR5: we need to add a table of the SLR corrections (Listing DMAGs for AW single-band products #13) for any users of the single-band catalogues
• File types and formats: @AngusWright could I assign the update of this page to you?
• Data Access: @kuijken could I assign the update of this page to you? It will require getting the relevant links from AstroWISE and ESO
• Acknowledgements: easy - I will hack the DR4 html for DR5

[cheymans]

Timeline: Target date to complete website is late April

• The paper will be published at A&A in ~ a month
• If we manage a successful test field upload to ESO soon, then I'd estimate 2 months before all the data is present and validated at ESO.
• We post the paper on arXiv to co-ordinate with the ESO release (including a contents page for the arXiv version)
• The website needs to be ready before the ESO release, as the release notes need to include a link for people to download a table of the SLR DMAG corrections that aren't included in the single-band catalogues.

[jtadejong]

I found the attached awe script in my local DR4 website copy that makes the ascii table used in the "Observations Table" page, except that it also adds a completeness column.
I suppose for some reason I removed that last column afterwards, although I don't remember why.

Since you need an AW installation to run this, I can try to do that. Are the single-band source lists for DR5 identified by the same kind of comment as the DR4 ones were (KiDS-ESO-DR4_SourceList-Single) ?
compile_data_table.py.txt

[cheymans]

Awesome awe script @jtadejong!

At the moment we seems to be in a bit of a muddle with what was DR5 and what was DR4 (#12) - but for the purposes of this table it shouldn't matter as the table should list all 1347 tiles. So I think you would want to query both KiDS-ESO-DR4_SourceList-Single and KiDS-ESO-DR5_SourceList-Single to create the table. I could then write a little script to go through afterwards removing duplicates (the mainly u-band tiles that were re-processed for DR5)

[jtadejong]

I ran the script today for "KiDS-ESO-DR5_SourceList-Single and got the attached file:
data_table_dr5.csv

This query returns exactly 6735 single-band catalogs (5x1347), which is what is supposed to be in DR5, correct?

Let me know if you still want me to do the same for the DR4 string.

[cheymans]

Amazing - yes that is everything!

now all we have to do is work out why it is everything as most of those fields shouldn’t have changed since DR4

@AngusWright - any thoughts on this? Have I just misunderstood the DR5 Astrowise processing, and it was all redone for DR5?

[AngusWright]

If this is the base source list for the multiband then that's understood: source lists are based on THELI data and that was fully rereduced. If it's to do with single band cats, though, then I'm not sure..

[andrejdvornik]

I ran the script today for "KiDS-ESO-DR5_SourceList-Single and got the attached file: data_table_dr5.csv

This query returns exactly 6735 single-band catalogs (5x1347), which is what is supposed to be in DR5, correct?

Let me know if you still want me to do the same for the DR4 string.

@jtadejong can you send me that script so I can help with the pickle of what catalogues are new and old?

[cheymans]

It's attached to this comment above :)

I found the attached awe script in my local DR4 website copy that makes the ascii table used in the "Observations Table" page, except that it also adds a completeness column. I suppose for some reason I removed that last column afterwards, although I don't remember why.

Since you need an AW installation to run this, I can try to do that. Are the single-band source lists for DR5 identified by the same kind of comment as the DR4 ones were (KiDS-ESO-DR4_SourceList-Single) ? compile_data_table.py.txt

[andrejdvornik]

Ah!

[cheymans]

@kuijken - what is your diary looking like for the next couple of weeks? We need to work on the webpages so they're ready to coincide with the ESO release date.

[cheymans]

Text for acknowledgements:

Users of data from this release should cite "Wright et al. (2024, A&A, 686, 170)".

According to the Data Access Policy for ESO data held in the ESO Science Archive Facility, all users are required to acknowledge the source of the data with appropriate citation in their publications. Since processed data downloaded from the ESO Archive are assigned Digital Object Identifiers (DOIs), the following statement must be included in all publications making use of them:

Based on data obtained from the ESO Science Archive Facility with DOI: https://doi.org/10.18727/archive/37, and https://doi.eso.org/10.18727/archive/59 and on data products produced by the KiDS consortium. The KiDS production team acknowledges support from: Deutsche Forschungsgemeinschaft, ERC, NOVA and NWO-M grants; Target; the University of Padova, and the University Federico II (Naples).

Publications making use of data which have been assigned an archive request number (of the form XXXXXX) must also include the following statement in a footnote or in the acknowledgement:
Based on data obtained from the ESO Science Archive Facility under request number <request_number>.

Science data products from the ESO archive may be distributed by third parties, and disseminated via other services, according to the terms of the Creative Commons Attribution 4.0 International license. Credit to the ESO provenance of the data must be acknowledged, and the file headers preserved.

[kuijken]

Around, should be able to make some time for this. I believe we need Jelte though to make actual changes.

…

On 16 Jan 2025, at 09:56, Catherine Heymans ***@***.***> wrote: @kuijken - what is your diary looking like for the next couple of weeks? We need to work on the webpages so they're ready to coincide with the ESO release date. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[cheymans]

@kuijken - what are your thoughts on this? I'm feeling pretty swamped and I'm figuring you are as well. We could not do a DR5 page on the Leiden website and just leave it to ESO?

[kuijken]

I will check with Jelte what can be done.

[jtadejong]

@kuijken, @cheymans, I will try to make some time for doing updates on the website.
If you deliver me the pages/content as much as possible, I should be able to move them to the webserver and include them in the website.