2023

View page source

Release Notes for 2023

mm-dd

12-22

  1. Extend the csv.predict to jobs that fail to fit or compute sample statistics, jobs correspond to (node, sex) pair. This was does by using the closest ancestor job that did fit and have sample statistics; see the new columns fit_node_name and fit_sex in the fit_predict.csv file.

  2. Fix a syntax warning when running the max_fit_option and mulcov_freeze examples.

12-19

Some of the references to the dismod_at documentation were pointing to an old version that was frozen on 2022-11-05. This has been fixed and they no point to the current dismod_at documentation.

12-15

Add the a Predictions item to the wish list (this was completed on 12-22 ).

12-08

The option_fit.csv no_ode_ignore option was extend to include the all choice. In addition, the No Ode Fit wish list item was removed.

12-03

Add no_ode_ignore to the option_all table and option_fit.csv . This completed the old No Ode Fit wish list item but it added a new No Ode Fit item to the csv.fit wish list.

12-02

Using balance_sex equal to true works correctly for versions of dismod_at that are greater than equal 20231201 (the bug mentioned below was fixed). Note that this will probably cause different results than before when the default value, true, is used for balance_sex .

11-30

The balance_sex option was added to the option_fit.csv file. Note that there currently is a bug in dismod_at and the default choice for balance_sex , true, does not work correctly.

11-25

  1. The sample_method option was added to the option_all table.

  2. The sample_method option was added to the option_fit.csv file.

  3. A definition for child jobs was added to the glossary.

  4. The split_reference_table for the csv.simulate, csv.fit, csv.predict routines was added to the documentation.

  5. Fix the sort of the items in the option_fit.csv documentation.

11-24

API Change

  1. Rename all_option_table -> option_all_table so that it is similar to the csv option file names; e.g., option_fit.csv .

  2. Rename all_omega_table -> omega_all Table so that it is similar to option_all_table.

  3. Change all the csv.module page names that began with csv_ to begin with csv.; e.g., change csv_fit -> csv.fit . This is not really an API change, but links to the corresponding documentation pages will need to be changed; e.g., change csv_fit.html -> csv.fit.html

11-22

Add documentation for the sample_index column in the fit_predict.csv and sam_predict.csv files output by csv.predict.

11-18

Mention difference between in the meaning of the standard deviation for linear and log-scaled priors; see value_prior , dage_prior , dtime_prior . Also make mention of it for data densities; see meas_std .

11-10

Add definition of true value for the average integrand to the discussion of meas_mean .

11-08

  1. Add the Duration item to the wish list.

  2. Move the Meta Regression item to a higher priority on the wish list; i.e., closer to the top of the list.

  3. Add See Priors for Each Fit to the wish list.

11-03

Add the copy_other_tbl routine. This is useful when moving the dismod_at database corresponding to a fit from one location to another.

10-21

  1. Add more discussion to the csv fit child_rate.csv documentation.

  2. Correct and improve the csv simulate omega documentation. Importantly, remove the following sentence: ‘The values in this column do not matter if omega does not appear in no_effect_rate.csv’ .

10-07

Improve the No Ode Fit wish list item and move it outside of the csv.fit group (it is not just for csv.fit ). (This wish list item was removed on 12-03 .)

10-05

Add the number_sample option to the csv.fit routine.

10-02

  1. Add mention that one is one of the absolute_covariates .

  2. Add Measurement Value Covariate wish list item.

09-26

  1. There was a mistake in the 08-25 changes that caused the cascade to abort after the first type of fit failed, instead of trying the other type. (Here the types of fits are both and fixed.) This has been fixed.

  2. Add the bound_random option to csv.fit.

08-25

  1. Add specification for the connection between data_in.csv data_id and the dismod_at data table data_id column.

  2. Add discussion about the model for the measurement mean needing to be positive when using a binomial density for the data.

08-22

There was a bug in how csv.fit handled the upper and lower age and time limits in data_in.csv . To be specific, the lower limit (instead of the mid point) was used for both the upper and lower limits. This has been fixed. In addition, the compress_interval option was added to the csv.fit routine.

08-09

Some bugs in the production of tru_predict.csv were fixed. This is only relevant when sim_dir is present was present in a csv.predict call.

07-25

Add the max_fit wish list item.

07-23

  1. Add the Csv Example Simulating, Fitting and Predicting example.

  2. Move the csv.fit_xam example to csv.break_fit_pred . This was done because it has changed from just a fit example to also being an example of breaking up breaking up the computation.

07-21

  1. Add job_descendant to the at_cascade module .

  2. Add the start_job_id and max_job_depth arguments to the check_log routine.

  3. Fix bug in csv.predict. To be specific, only check logs for jobs that we are predicting for. Checking logs for other jobs might try to read a log that was changing (when running in parallel).

  4. Add the job_name to the shared memory name; see shared_unique . This fixes a bug when running cascades in parallel. It also required adding job_name to the clear_shared routine.

  5. Change csv.break_fit_pred so that it runs jobs in parallel when breakup_computation is True .

07-20

The eta and nu columns were not actually Optional Columns in data_in.csv. This has been fixed.

07-06

  1. Add sample_size to the csv.fit data_in.csv file. This is used when modeling the data with a binomial distribution.

  2. Make certain columns in data_in.csv Optional Columns .

06-30

The csv.simulate routine ignored pini entries in the no_effect_rate.csv file; i.e., the initial prevalence was always zero. This has been fixed.

06-28

The csv.fit routine now reports (and exits) when a grid point appears more than once in parent_rate.csv , or any of its other input files.

06-26

The csv.fit routine was changed to adapt to version 20230626 of dismod_at. To be more specific, the other_database is now relative to the directory where the primary database is located.

06-25

  1. There was a bug in csv.predict that would cause it to crash at unpredictable times with a message like:

    ... /at_cascade/csv/read_table.py", line 49, in read_table
file_ptr   = open(file_name)
           ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file ... /fit_predict.csv

    This has been fixed.

  2. Sometime it takes a while for the trace.out file corresponding to a fit to have any information in it. The dismod_at initialization trace was included in trace.out so that one has more feedback that the program is making progress.

06-23

There was a bug in csv.fit that caused a crash when data_in.csv had sex had sex equal to both and there was a covariate, other than sex, in the covariate.csv file. This has been fixed.

06-20

Remove the Disk Space entry from the wish list (it was completed by merging the other_database branch on 06-18).

06-19

Add pini to the std_random_effects_rate in option_sim.csv. Also add discussion about the connection between the standard deviation of the random effects, no_effect_rate.csv and random_effect.csv .

06-18

Merge in other_database branch:

  1. API Change : The root_database was added to the option_all table. In addition it was removed as an argument to cascade_root_node and create_all_node_db .

  2. Change the get_fit_integrand argument from the name of the fit node database to a fit_or_root_class object.

  3. Read the constant tables from the root_database .

  4. Use copy_root_db to remove the constant tables from the fit node databases . This avoids multiple copies of these tables and thereby saves disk space.

05-16

  1. Add the float_precision to control the accuracy of the prediction files.

  2. Change the csv.prevalence2iota example to check for extremely high accuracy in the corresponding tru_predict.csv file.

05-15

  1. Improve documentation of correspondence between fit_predict.csv , tru_predict.csv and sam_predict.csv.

  2. Add trace to the option_sim.csv file.

  3. Change the csv_sim_and_fit example to csv.prevalence2iota .

05-14

Add the optional tru_predict.csv output file to csv.predict.

05-13

Add the new_random_effects option to the csv simulate routine.

05-12

  1. The random effects were not being used properly in the csv.simulate routine. To be specific, the random effect for the node being fit, was used for all the levels above the node being fit. This has been fixed.

  2. The csv.predict routine was running parallel code when max_number_cpu was one. This has been changed so that the execution stream simpler when only one processor is used.

05-11

  1. Improve the get_parent_node documentation.

  2. Define sex_name2value in the at_cascade.csv name space.

05-10

There was a problem with the csv fit_xam when breakup_computation was true. To be specific, multiple copies of the header appeared in the fit_predict.csv and sam_predict.csv files. This has been fixed by using the csv DictReader and DictWriter routines instead of the shutil copyfileobj routine.

05-03

  1. Add the following columns to the csv.fit data_in.csv table: density_name , eta , nu . Warning This is an API change because these columns must appear in data.csv. To get the same effect as before this change, add ,density,eta,nu to the end of the first line in data.csv and ,cen_gaussian,, to the end of the other lines in data.csv.

  2. The Data Density wish list item was removed.

04-26

In the examples, move the assignment to new from just before a call to dismod_at.create_connection to inside the call. In addition, add readonly to the call. For example, in absolute_covariates.py the text

new          = False
connection   = dismod_at.create_connection(root_database, new)

was changed to

connection   = dismod_at.create_connection(
   root_database, new = False, readonly = False
)

04-12

  1. Add Install instructions.

  2. Add the Data Density wish list item.

  3. Remove separating the predictions form the fits because this wish list item has been completed.

04-09

Merge in the predict branch. Warning this is an API change because you must run both the csv fit and predict routines to get what the fit routine used to produce; see comments below.

  1. Split the csv fit routine into a fit and predict steps; see csv.fit , csv.predict.

  2. Add the option_predict.csv file.

  3. Remove db2csv and plot from the option_fit.csv file (they have been moved to option_predict.csv).

  4. Add the start_job_name and max_job_depth arguments to csv.predict.

  5. Improve the documentation for the csv.break_fit_pred . In addition, include continuing the cascade in this example; see breakup_computation .

  6. Add the max_node_depth argument to csv.fit.

  7. Change avgint_parent_grid to predict for all children, so continue_cascade works for any change of the fit_goal_set.

04-06

Improve the way the calculation of the fit_predict.csv and sam_predict.csv calculations are parallelized. (The old method did not seem to work well on some systems.)

04-05

Add some new items to the csv.fit wish list.

03-28

  1. Add some new wish_list items.

  2. Add the ode_method csv fit option.

  3. Change csv.shock_cov to use trapezoidal ode_method .

03-25

Add child_prior_std_factor to the option_fit table. This is called shift_prior_std_factor in the option_all table.

03-22

Fix discussion of relative covariates under the covariate heading, under multiplier.csv for the csv fit routine.

03-20

This documentation Version number was not updating since 2023..1. This has been fixed.

03-17

If a covariate multiplier was frozen at the root level, and for the split reference id corresponding to the no_ode_fit , the covariate was frozen after the no_ode fit. This has been fixed so that the covariate is frozen after the full root level fit.

03-14

The fit_predict.csv and sam_predict.csv files are created after all the fits. We are predicting for all the nodes (we could predict for just the next fit we want to inspect the results for). This computation was changed from sequential to in parallel because it was slow when there are large number fits.

03-09

Add the test/csv_one_cov.py test which fails unless the version of dismod_at is greater than or equal dismod_at-20230309.

03-08

Add the minimum_meas_cv to the option_fit table.

02-04

  1. Change the csv.shock_cov example to use the simulated data, instead of its mean, during fitting. This results in zero data values for prevalence at non-zero ages and hence is a good test of the censored gaussian density for the data (true prevalence is zero at age zero).

  2. If a job failed, csv.fit would crash at the end when gathering the fit_predict.csv results. This has been fixed and the failed results are skipped when building fit_predict.csv (and sim_predict.csv).

02-03

  1. Change the simulated data percent_cv data column to meas_std_cv (which is now a ratio and not a percent), and add the meas_std_min column.

  2. Change the density used for the data from Gaussian to censored Gaussian; see meas_std .

01-13

If refit_split was false, the covariates were not frozen after the root level fit. This has been fixed; see Multiplier Freeze.

01-05

  1. Create the csv.shock_cov example which simulates and fits an incidence shock using prevalence data. The shock is modeled as a covariate that affects the rate iota. This is a good test of the covariate multiplier bug fix mentioned directly below.

  2. Add the covariate.csv wish list item.

01-03

Fix a bug in covariate multipliers that affect rate values. For example, the model for a prevalence measurement needs to compute rate values at ages and time that are previous to the prevalence measurement time. See the dismod_at documentation for the rate_eff_cov table for more details.

01-04

  1. Add the csv fit absolute_covariates option.

  2. Remove covariate_effect_depend_age_time from option_sim.csv . Now the bug in the covariate multipliers has been fixed, all simulations should be done with full age time variation in the covariate values.

  3. Split the std_random_effects option_sim.csv option into std_random_effects_iota, std_random_effects_chi, std_random_effects_rho .

  4. Remove the csv_absolute_covariate example (it will be replaced by the csv_cov_shock example)