Competition

Prepare submission:

1. Create zip file:
   • algorithm_summary.[any one of txt, doc, docx, pdf]
   • Prediction_Adipose.npy
   • Prediction_Calcification.npy
   • Prediction_Fibroglandular.npy
   
   In the training and validation phases, the ground truths your submissions will be compared against are 10x512x512 in float32, containing ten images (first 10 from training and all 10 from validation). In the testing phase predictions.npy is 100x512x512. For the all phases the algorithm report has acceptable formats of pdf, docx, and doc. Different filenames than those in the examples are not acceptable. Start a forum discussion if you believe you need to use another.
   
   
   In the validation phase, it is fine to submit a placeholder algorithm_summary.txt document which only has a few words and no algorithm description.
   
   
   
2. Submit the zip file
   Under the "Participate" tab select "Submit/View Results".
   Click on the "Submit" button and upload the prepared zip file.
   It should take less than a minute to process the submission, and in the training and validation phases the scoring log can be viewed.
   After receiving a score, the score can be submitted to the leaderboard.

Under the "Participate" tab, select "Files" and click on the "Starting Kit" button. In there is a zip file called sample_training_submission_txt.zip. If you submit these images you should get an RMSE score of 0.0199964586645 and worst-case ROI RMSE of 0.0214029662311. The purpose of submitting in the training phase is to practice submission and to understand the scoring system. There are unlimited submissions in the training and validation phases for all individuals even if they are on the same team.

In the testing phase, each team will be allowed three submissions. (Failed submissions do not count toward the three submissions)

Quantitative evaluation:

Submitted test-phase reconstructed tissue-map images will be evaluated by computing

• Root-mean-square-error (RMSE) averaged over the 100 test images:

  \( {1\over100} \sum_{i=1}^{100} \sqrt{\|t_i-r_i\|_2^2 \over 3n} \) where r and t represent the triplet of reconstructed and test tissue-map images, respectively, and n is the number of pixels in a single image.

• Worst-case 25x25 pixel ROI-RMSE over all 100 test tissue-map images:

  \( \underset{i,c}{max} \sqrt{\|b_c (t_i-r_i)\|_2^2 \over 3m} \) where \(b_c\) is a masking indicator function for the 25x25 ROI centered on coordinates c in the image, m=625 is the number of pixels in the ROI.

Mean RMSE will be the primary metric that determines the algorithm ranking. In case of a numerical tie (to be determined based on the distribution of results with the mean RMSE), the worst-case ROI-RMSE will be used to rank the algorithms.

The above methods evaluate your submissions as follows:

metrics.py

    
# This program demonstrates the mean RMSE and worst-case ROI RMSE metrics
    # The 128-view fbp images serve as the test images
    # The phantom images are taken as the ground truth
    
    import numpy as np, sys, os, time
    import pdb
    
    # production
    INPUT = sys.argv[1] # INPUT which has both ./ref and ./res - user submission
    OUT = sys.argv[2] # OUTPUT
    
    REFERENCE = os.path.join(INPUT, "ref") # Phantom GT
    PREDICTION_OUTPUT = os.path.join(INPUT, "res") # not FBP, but hopefully phantom like
    
    # Will try to recreate these. They serve as the input data.
    if len(os.listdir(REFERENCE)) == 3:
        # phantom_gt_file_name = os.listdir(REFERENCE)[0] # old
        # images are the ground truth
        images = np.array([np.load(os.path.join(REFERENCE,"Phantom_Adipose.npy")),
                           np.load(os.path.join(REFERENCE,"Phantom_Fibroglandular.npy")),
                           np.load(os.path.join(REFERENCE,"Phantom_Calcification.npy"))]
                         )
    else:
        raise Exception('Organizer, we need 3 *.npy files')
    
    #QA for algorithm summary and images
    acceptable_file_types = [".pdf",".docx",".txt",".doc",".npy"]
    if len(os.listdir(PREDICTION_OUTPUT)) <= 3:
        raise Exception('Remember for this phase we need 3 .npy files and one of a {} with your algorithm summary'.format(acceptable_file_types))
    elif len(os.listdir(PREDICTION_OUTPUT)) == 4:
        adipose_file_name = ""
        fibroglandular_file_name = ""
        calcification_file_name = ""
        for file in os.listdir(PREDICTION_OUTPUT):
            if file.find("_Adipose.npy") != -1:
                adipose_file_name = file
            elif file.find("_Fibroglandular.npy") != -1:
                fibroglandular_file_name = file
            elif file.find("_Calcification.npy") != -1:
                calcification_file_name = file 
            elif file[-4:] in acceptable_file_types or file[-5:] in acceptable_file_types and file[-4:] != ".npy":
                algorithm_summary = file
            else:
                raise Exception('Remember for this phase we need one of a {} with your algorithm summary. This error suggests you have multiple files, but that your algorithm summary does not have the right extension or you are missing this.'.format(acceptable_file_types))
        testimages = np.array([np.load(os.path.join(PREDICTION_OUTPUT, adipose_file_name)),
                                np.load(os.path.join(PREDICTION_OUTPUT, fibroglandular_file_name)),
                                np.load(os.path.join(PREDICTION_OUTPUT, calcification_file_name))]
                                )
    else:
        raise Exception('Maybe ask a question in the forum as your submission has neither 1 file, 2 files or more than two files.')
    
    # pdb.set_trace()
    nmat, nim, nx, ny = images.shape     #get the number of materials, images and number of pixels in x and y
    
    # BB
    # >>> nmat, nim, nx, ny
    # (3, 10, 512, 512)
    
    # mean RMSE computation
    diffsquared=(images-testimages)**2
    meanrmse  = np.sqrt( (((diffsquared/float(nmat*nx*ny)).sum(axis=3)).sum(axis=2)).sum(axis=0) ).mean()
    print("The mean RSME over %3i images is %8.6f "%(nim,meanrmse))
    
    
    # worst-case ROI RMSE computation
    roisize = 25  #width and height of test ROI in pixels
    x0 = 0        #tracks x-coordinate for the worst-case ROI
    y0 = 0        #tracks x-coordinate for the worst-case ROI
    im0 = 0       #tracks image index for the worst-case ROI
    
    def compute_maxroierror(images,testimages):
       maxerr = -1.
       for i in range(nim):
          print("Searching image %3i"%(i))
          im =  images[:,i].copy()
          imtest = testimages[:,i].copy()
          for ix in range(nx-roisize):
             for iy in range(ny-roisize):
                roi =  im[:,ix:ix+roisize,iy:iy+roisize].copy()
                roitest =  imtest[:,ix:ix+roisize,iy:iy+roisize].copy()
                if roi.max()>0.01:     #Don't search ROIs in regions where the truth image is zero
                   roirmse = np.sqrt( (((roi-roitest)**2)/float(nmat*roisize**2)).sum() )
                   if roirmse>maxerr:
                      maxerr = roirmse
                      x0 = ix
                      y0 = iy
                      im0 = i
       return maxerr,x0,y0,im0
    
    t1 = time.time()
    maxerr,x0,y0,im0 = compute_maxroierror(images,testimages)
    print("Worst ROI search took: ,",time.time()-t1," seconds")
    
    print("Worst-case ROI RMSE is %8.6f"%(maxerr))
    print("Worst-case ROI location is (%3i,%3i) in image number %3i "%(x0,y0,im0+1))
    
    with open(os.path.join(OUT,"scores.txt"), "w") as results:
       results.write("score_1: {}\n".format(meanrmse))
       results.write("score_2: {}".format(maxerr))
       #results.write("score_1: {}\n".format('0.00003000'))
       #results.write("score_2: {}".format('0.09748530'))
    

Terms and Conditions

By participating in this challenge, each participant agrees to:

The DL-spectral CT challenge is organised in the spirit of cooperative scientific progress. The following rules apply to those who register a team and download the data:

• Anonymous participation is not allowed.
• Participants from the host institution and colleagues of the organizers may participate, but they will not be eligible for prizes or recognition as a ranking participant.
• Entry by commercial entities is permitted, but should be disclosed.
• Once participants submit their outputs to the DL-spectral CT challenge organizers, they will be considered fully vested in the challenge, so that their performance results will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers.
• The downloaded datasets or any data derived from these datasets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
• The full data, including reference data associated with the test set cases, is expected to be made publicly available after the publication of the DL-spectral CT Challenge. Until the official public release of the data, data downloaded from this site may only be used for the purpose of preparing an entry to be submitted for the DL-spectral CT challenge. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products.